What happens if you want to have multiple Logstash services running on the same server? There are various reasons you might need to do this. An effort to reduce a huge config by splitting it in half based on log type. Once type of log is more important then the other and thus needs treated with higher priority. Or perhaps even a ‘test’ service.

Below is a very quick and easy install to get a three services of Logstash 5.4.x up and running on CentOS 7. This is based on the install scripts/instructions used for production Elasticsearch environments.

The service names are “logstash-1”, “logstash-2”, “logstash-3”. A simple search/replace is all that is needed to use your own names.

The install documented here is based on a fresh install of CentOS 7 x86_64 Minimal
http://isoredirect.centos.org/centos/7/isos/x86_64/CentOS-7-x86_64-Minimal-1611.iso

When you build your CentOS server do not create a swap file, this is the easiest and best way to keep the JVM from swapping and it is unnecessary on a dedicated Elasticsearch server.

First, make sure your CentOS install is up to date

sudo yum update
sudo yum upgrade -y<

Install Java

sudo yum install java -y

Download and install Logstash and then install the x-pack plugin

sudo wget https://artifacts.elastic.co/downloads/logstash/logstash-5.4.3.rpm
sudo rpm -ivh logstash-5.4.3.rpm
sudo /usr/share/logstash/bin/logstash-plugin install x-pack

Add the firewall rules to allow access to the server to send logs. In the example are ports for Syslog (don’t use 514 for syslog, CentOS considers them privileged ports and they are more difficult to use).

firewall-cmd --permanent --add-port=1514/tcp
firewall-cmd --permanent --add-port=1514/udp
firewall-cmd --reload

Create a copy of the installed default service and files into three separate services and delete the original default.

sudo cp /etc/systemd/system/logstash.service /etc/systemd/system/logstash-1.service
sudo cp /etc/systemd/system/logstash.service /etc/systemd/system/logstash-2.service
sudo cp /etc/systemd/system/logstash.service /etc/systemd/system/logstash-3.service
sudo rm /etc/systemd/system/logstash.service
sudo mkdir /etc/logstash-1/
sudo mkdir /etc/logstash-2/
sudo mkdir /etc/logstash-3/
sudo cp -r /etc/logstash/* /etc/logstash-1/
sudo cp -r /etc/logstash/* /etc/logstash-2/ 
sudo cp -r /etc/logstash/* /etc/logstash-3/
sudo rm -rf /etc/logstash/
sudo cp /etc/default/logstash /etc/default/logstash-1
sudo cp /etc/default/logstash /etc/default/logstash-2
sudo cp /etc/default/logstash /etc/default/logstash-3
sudo rm /etc/default/logstash
sudo mkdir /var/log/logstash-1
sudo mkdir /var/log/logstash-2
sudo mkdir /var/log/logstash-3
sudo chown logstash.root /var/log/logstash-1
sudo chown logstash.root /var/log/logstash-2
sudo chown logstash.root /var/log/logstash-3
sudo rm -rf /var/log/logstash/
sudo mkdir /var/lib/logstash-1
sudo mkdir /var/lib/logstash-2
sudo mkdir /var/lib/logstash-3
sudo chown logstash.logstash /var/lib/logstash-1
sudo chown logstash.logstash /var/lib/logstash-2
sudo chown logstash.logstash /var/lib/logstash-3
sudo rm -rf /var/lib/logstash/

Under most circumstances extra RAM for Logstash is not necessary unless you heavily use plugins with built in cache’s. The following changes the JVM to 2GB for the first service which is enough for 10k or more per second of logs, and 1GB for the remaining 2 services.

Because you are running without a swap partition it is important to set the min/max values for the JVM to be the same that way RAM usage on your server is predictable.

sudo sed -i -e 's#Xms256m#Xms2g#' /etc/logstash-1/jvm.options
sudo sed -i -e 's#Xmx1g#Xmx2g#' /etc/logstash-1/jvm.options
sudo sed -i -e 's#Xms256m#Xms1g#' /etc/logstash-2/jvm.options
sudo sed -i -e 's#Xmx1g#Xmx1g#' /etc/logstash-2/jvm.options
sudo sed -i -e 's#Xms256m#Xms1g#' /etc/logstash-3/jvm.options
sudo sed -i -e 's#Xmx1g#Xmx1g#' /etc/logstash-3/jvm.options

Change the node name for Logstash. This will be visibile in Kibana for monitoring purposes. The example below changes the node name for the first service to “LS-node1-service1”, change to suite your needs.

sudo sed -i -e 's|# node.name: test|node.name: LS-node1-service1|' /etc/logstash-1/logstash.yml
sudo sed -i -e 's|# node.name: test|node.name: LS-node1-service2|' /etc/logstash-2/logstash.yml
sudo sed -i -e 's|# node.name: test|node.name: LS-node1-service3|' /etc/logstash-3/logstash.yml

Change the 3 new services configs to reflect new directory structure.

sudo sed -i -e 's|path.config: /etc/logstash/conf.d|path.config: /etc/logstash-1/conf.d|' /etc/logstash-1/logstash.yml
sudo sed -i -e 's|path.config: /etc/logstash/conf.d|path.config: /etc/logstash-2/conf.d|' /etc/logstash-2/logstash.yml
sudo sed -i -e 's|path.config: /etc/logstash/conf.d|path.config: /etc/logstash-3/conf.d|' /etc/logstash-3/logstash.yml
sudo sed -i -e 's|# http.port: 9600-9700|http.port: 9600|' /etc/logstash-1/logstash.yml
sudo sed -i -e 's|# http.port: 9600-9700|http.port: 9601|' /etc/logstash-2/logstash.yml
sudo sed -i -e 's|# http.port: 9600-9700|http.port: 9602|' /etc/logstash-3/logstash.yml
sudo sed -i -e 's|path.logs: /var/log/logstash|path.logs: /var/log/logstash-1|' /etc/logstash-1/logstash.yml
sudo sed -i -e 's|path.logs: /var/log/logstash|path.logs: /var/log/logstash-2|' /etc/logstash-2/logstash.yml
sudo sed -i -e 's|path.logs: /var/log/logstash|path.logs: /var/log/logstash-3|' /etc/logstash-3/logstash.yml
sudo sed -i -e 's|/var/lib/logstash|/var/lib/logstash-1|' /etc/logstash-1/logstash.yml
sudo sed -i -e 's|/var/lib/logstash|/var/lib/logstash-2|' /etc/logstash-2/logstash.yml
sudo sed -i -e 's|/var/lib/logstash|/var/lib/logstash-3|' /etc/logstash-3/logstash.yml
sudo sed -i -e 's|EnvironmentFile=-/etc/default/logstash|EnvironmentFile=-/etc/default/logstash-1|' /etc/systemd/system/logstash-1.service
sudo sed -i -e 's|EnvironmentFile=-/etc/sysconfig/logstash|# EnvironmentFile=-/etc/sysconfig/logstash|' /etc/systemd/system/logstash-1.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-1|' /etc/systemd/system/logstash-1.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-1|' /etc/default/logstash-1
sudo sed -i -e 's|logstash.pid|logstash-1.pid|' /etc/default/logstash-1
sudo sed -i -e 's|EnvironmentFile=-/etc/default/logstash|EnvironmentFile=-/etc/default/logstash-2|' /etc/systemd/system/logstash-2.service
sudo sed -i -e 's|EnvironmentFile=-/etc/sysconfig/logstash|# EnvironmentFile=-/etc/sysconfig/logstash|' /etc/systemd/system/logstash-2.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-2|' /etc/systemd/system/logstash-2.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-2|' /etc/default/logstash-2
sudo sed -i -e 's|logstash.pid|logstash-2.pid|' /etc/default/logstash-2
sudo sed -i -e 's|EnvironmentFile=-/etc/default/logstash|EnvironmentFile=-/etc/default/logstash-3|' /etc/systemd/system/logstash-3.service
sudo sed -i -e 's|EnvironmentFile=-/etc/sysconfig/logstash|# EnvironmentFile=-/etc/sysconfig/logstash|' /etc/systemd/system/logstash-3.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-3|' /etc/systemd/system/logstash-3.service
sudo sed -i -e 's|/etc/logstash|/etc/logstash-3|' /etc/default/logstash-3
sudo sed -i -e 's|logstash.pid|logstash-3.pid|' /etc/default/logstash-3

Add the monitoring config for Logstash so that it reports into Elasticsearch properly. This assumes a local Elasticsearch Client instance running on the Logstash server (localhost:9200)

sudo sed -i '14ixpack.monitoring.elasticsearch.password: "changeme"' /etc/logstash-1/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.username: "logstash_system"' /etc/logstash-1/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.url: "http://localhost:9200"' /etc/logstash-1/logstash.yml
sudo sed -i '14i#' /etc/logstash-31/logstash.yml
sudo sed -i '14i# ------------  xpack monitoring ------------' /etc/logstash-1/logstash.yml
sudo sed -i '14i#' /etc/logstash-1/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.password: "changeme"' /etc/logstash-2/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.username: "logstash_system"' /etc/logstash-2/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.url: "http://localhost:9200"' /etc/logstash-2/logstash.yml
sudo sed -i '14i#' /etc/logstash-2/logstash.yml
sudo sed -i '14i# ------------  xpack monitoring ------------' /etc/logstash-2/logstash.yml
sudo sed -i '14i#' /etc/logstash-2/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.password: "changeme"' /etc/logstash-3/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.username: "logstash_system"' /etc/logstash-3/logstash.yml
sudo sed -i '14ixpack.monitoring.elasticsearch.url: "http://localhost:9200"' /etc/logstash-3/logstash.yml
sudo sed -i '14i#' /etc/logstash-3/logstash.yml
sudo sed -i '14i# ------------  xpack monitoring ------------' /etc/logstash-3/logstash.yml
sudo sed -i '14i#' /etc/logstash-1/logstash.yml

Enable the Logstash service to start on boot and then start it manually.

sudo systemctl enable logstash-1
sudo systemctl enable logstash-2
sudo systemctl enable logstash-3
sudo systemctl start logstash-1
sudo systemctl start logstash-2
sudo systemctl start logstash-3

Your config directory(s) are located at the following paths and are blank by default.

/etc/logstash-1/conf.d/
/etc/logstash-2/conf.d/
/etc/logstash-3/conf.d/

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Close Menu