Stamus Loggers

As a deployment scale is increasing, the first thing to do to ensure a fast shipping of the events is to use an external Elasticsearch cluster. The next bottleneck is the handling of the events by the Stamus Central Server as one single server is handling all the events generated by the flock of probes.

Stamus Loggers have been designed to fix this bottleneck by setting up multiple log forwarders. This is done by converting freshly installed instances of Stamus Central Servers to Stamus Loggers. Once the transformation is done, they can be associated to a Stamus Central Server (SCS).

As mentioned before, this is the second step of scaling the log pipeline so this setup is only possible when an external Elasticsearch cluster is used.

Once Stamus Loggers are setup, the probes will connect to them to ship the events. Each probe will connect to all Stamus Loggers and will load balance the events shipping. This ensures scalability as some probes are sending a really high number of events per second and it also provides failover if ever a part of the Stamus Loggers is not available.

Setting up Stamus Loggers

Enable External Elasticsearch Cluster

Important

Before you begin and in case you already have VPN probe(s), you should have in mind that VPN probes and Loggers are incompatible in v39.0.0. This is due to the fact that the VPN probe(s) will use the VPN tunnel for both management and log shipping via the SCS and the VPN does not have a route to the Loggers.

As said before Stamus Loggers setup will only work with an external elastic cluster. First it needs to be enabled. Click on the drop down menu on the left upper side corner(Stamus Networks logo), click on System Settings -> Main Tab -> select the checkbox Use an external Elasticsearch server Some new fields will open:

Elasticsearch url -> Address/es of the Elasticsearch Cluster nodes. In the form of URL: https://elasticsearch1:9200/,https://elasticsearch2:9200/

Elasticsearch user -> User associated with the cluster (if authorization is set)

Elasticsearch password -> Password associated with the user for the cluster (if authorization is set)

Use elasticsearch with system proxy -> Enable the checkbox if you want to use a proxy

If you need to use TLS certificate authority with the Elasticsearch Cluster: Click on the drop down menu on the left upper side corner(Stamus Networks logo), click on Global Appliance Settings -> click on Certificate Authority tab -> Enable the checkbox Use additional Certificate Authority -> click the browse button and upload your TLS CA file for use with the Elasticsearch cluster. -> click apply.

In the end Apply Changes on the Stamus Central Server.

Setting up SCS

In order to use Stamus Loggers, you first need to enable the feature: from the drop down menu on the left upper side corner(Stamus Networks logo), click on Global Appliance Settings -> go to the Main tab and select the checkbox Enable support for Stamus Logging Server.

Then Apply Changes on the Stamus Central Server.

Register a Stamus Logger

To register a Stamus Logger on the Stamus Central Server, click on Appliances major menu -> Select Add appliance select type -> Stamus Logger. Fill in the needed fields in order to validate the form. Finally click Submit. The appliance that will become a Stamus Logger must be a stock SCS running on the same version as the main SCS. The Stamus Logger appliance page will open. On the left hand side under the Action category, click on -> VPN certificate. Copy the link after Please find the probe VPN configuration here: and download on the Stamus logger with via the shell. wget can be used in this case.

wget https://192.168.2.1/appliances/vpn_cert/<unique key>/vpn-conf-logger1.tar.gz

Register the Stamus Logger with the command: sudo stamus_logger_register vpn-conf-logger1.tar.gz , type YES when asked.

WARNING: this will convert the SCS into a Stamus Logger. The configuration of the SCS will be lost. This operation cannot be reverted.

Go back on the Appliances page, locate the newly create Stamus Logger and click on the View button. Make sure Local information and VPN status information has been updated.

Finally Apply changes

Update the Probes

You need to Apply Changes to all probes after a Stamus Logger is added to ensure that the new Stamus Loggers receive events from the probes.