A simple Logstash ratelimiter

There is a need to rate limit the event rate to an output in Logstash, how do you do it?  Perhaps you are outputting to some other system that is licensed based on a certain event rate per second, or perhaps it is a mechanism to protect Elasticsearch by preventing large spikes of ingested logs from being dumped at once into Elasticsearch.  (more…)

Continue Reading

ELK vs Graylog: Backend index management

Both ELK and Graylog use Elasticsearch for log storage and indexing.  Most of what makes both Kibana and Graylog work so well for searching through logs is mostly provided by Elasticsearch with Graylog and Kibana acting as a GUI to interface with it.  Data is stored in indexes in Elasticsearch and those indexes need to be maintained and managed.  How each product does it though is different. (more…)

Continue Reading

1 billion+ logs per day with ELK stack, hardware and scalable architecture

I remember hitting my first 100 logs per second still very distinctly. After a few weeks of tweaking Elasticsearch to get an understanding of it, and fighting with Logstash and grok to make it actually do what I wanted….all without crashing Elasticsearch due to the very anemic resources I had given it and having Logstash actually keep up with the ingestion rate without falling behind and dropping events. (more…)

Continue Reading
  • 1
  • 2
Close Menu