Ever want graph the event processing rate for Logstash at a high resolution such as 1 second accuracy? It is possible to query the logstash monitoring index but these metrics are collected only every 10 seconds. Alternatively you could use the Logstash exec input plugin to have Logstash query its own API to get the event processing rate at a higher resolution. It is a lot of extra work but it will work.

But…what happens if you want the MBps rate that Logstash is processing? No built in tools will do this and the information is valuable because log messages will vary wildly in size, from small syslog messages to absolutely huge Windows logs from Winlogbeat.

 

The below will create a new event every 1 second (@flush_frequency variable) and output the following fields.  @pipeline, @server, @datacenter, @flush_frequency are all configurable.

 

#####################
## Author: packetrevolt.com
#####################
## logstash internal metrics
ruby {
  init => "
    @pipeline = 'main'
    @server = 'server1'
    @datacenter = 'chicago'
    @flush_frequency = 1
    @last_flush = Time.now.to_f
    @event_count = 0
    @bytesize = 0
  "
  code => "
    @event_count = @event_count + 1
    @bytesize = @bytesize.to_i + event.to_s.bytesize.to_i
    if (@last_flush + @flush_frequency < Time.now.to_f) @last_flush = Time.now.to_f new_event_block.call(LogStash::Event.new('metric' => 'logstash', 'server' => @server, 'datacenter' => @datacenter, 'pipeline' => @pipeline, 'events' => @event_count, 'bytes' => @bytesize, 'seconds' => @flush_frequency))
      @event_count = 0
      @bytesize = 0
    end
  "
}
#####################

 

Leave a Reply

Close Menu