For custom Ruby code in Logstash which is better?  Ruby code in the pipeline or Ruby scripts or full blown plugins?  There are pros and cons to all the choices and it depends on what the requirements are and a lot on personal preference.

If it is a Input/Output/Codec it has to be a plugin, there is no other method.  While it is technically possible to write Ruby code in the filter section that outputs it is not advisable.  For filters it is generally far easier to use the Ruby filter plugin and if you have a large amount of code then put it into a Ruby script file and use the Ruby filter plugin.

Why use the Ruby filter plugin instead of creating filter plugins?

Take an example where you have 20 different custom Ruby scripts.  If they were plugins, they are 20 plugins that need to be generated into a gem file, 20 plugins to be updated for every Logstash release, and 20 plugins that must be installed into Logstash.  A Ruby script is either code in the Logstash config itself or a file on disk that you reference by the filepath.  It is far easier to avoid the Logstash plugin system if your use case allows for it.  Seeing as each of the 20 Ruby scripts is a file it also allows you to manage them as files with the rest of the Logstash configuration which is far easier of a task then automating the installation of plugins through an automation solution.

In short, because it is easier.

 

Plugins (ones installed from a .gem file)
https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html
Pro:
–Inputs, Outputs, and Codecs can only be a plugin
–The ability to use Ruby gems not included with Logstash by default
–Easy to publish and share
–If you intend to have a public plugin, most users will expect this format
–If you have hopes of Logstash adopting the plugin, it needs to be in this format
–Tight control over versioning
–Easy portability
–Logging is generally better and more verbose which is especially useful for troubleshooting
Con:
–If you are creating custom plugins to use internally then installed plugins are a PITA to manage
–Plugins are usually generated supporting specific versions of Logstash, update Logstash = update plugin and reinstall
–Must be installed which makes it more difficult to (directly) manage with automation tools
–If the Logstash server does not have internet access it requires installing on a ‘test’ server then generating an offline plugin bundle to then install on the other servers
When you would use:
–Publishing the plugin publically
–Policy decision to do all plugins the same way

Plugins (using –path.plugins option)
https://www.elastic.co/guide/en/logstash/current/working-with-plugins.html
Pro:
–Inputs, Outputs, and Codecs can only be a plugin
–The ability to use Ruby gems not included with Logstash by default
–Easy to convert to a gem file to install it
–Logging is generally better and more verbose which is especially useful for troubleshooting
Con:
–Elastic has intention to remove this feature but has yet to do so (noooo!!!?!)
When you would use:
–Want to have a plugin but do not wish to generate gem files and install them

path.plugins: [ "/etc/logstash/plugins" ]
#/etc/logstash/plugins/logstash/filters/test.rb
# encoding: utf-8
require "logstash/filters/base"
require "logstash/namespace"

class LogStash::Filters::Test < LogStash::Filters::Base
        config_name "test"

        public
        def register
                # nothing ro register
        end # def register

        public
        def filter(event)
                filter_matched(event)
        end
end

 

Ruby Filter with code in Logstash config
https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html
Pro:
–None of the difficulty involved when dealing with plugins
–Everything is in the Logstash config, easy to see and follow
Con:
–Values from an event must be referenced directly in the Ruby code, you cannot pass variables the same way as plugins
–Tends to be confusing for new users
–Can drastically inflate the config file size
When you would use:
–Short pieces of code that are not worth creating a script file over
–To avoid creating a Filter plugin

ruby {
  code => "new_event_block.call(event.clone)"
}

 

Ruby Filter using script files
https://www.elastic.co/guide/en/logstash/current/plugins-filters-ruby.html
Pro:
–None of the difficulty involved when dealing with plugins
–Config file is much cleaner and shorter
–If the Ruby code is used many times it makes it much easier to update since it is stored in one place
–Can pass variables to the Ruby script in a similar way to how variables are passed to plugins
Con:
–New feature so it is less well documented with fewer examples
When you would use:
–To avoid creating a Filter plugin

ruby {
  # Cancel 90% of events
  path => "/etc/logstash/drop_percentage.rb"
  script_params => { "percentage" => 0.9 }
}
#/etc/logstash/drop_percentage.rb
def register(params)
  @drop_percentage = params["percentage"]
end

def filter(event)
  if rand >= @drop_percentage
    return [event]
  else
    return [] # return empty array to cancel event
  end
end

Leave a Reply

Close Menu