Deployment Guidelines¶
The proper deployment for Stamus Network Probes is crucial to ensure the correct visibility of network traffic in the environment.
This guide will present recommendations where to capture the traffic from, how to forward it, which requirements need to be addressed and potential issues.
Traffic Capture Points¶
The placement of probes mostly depends on what parts of your network you want to monitor and the capability your network architecture provides. Based on the architecture and targeted coverage, the traffic capture will look differently.
There are several important areas in your network that you might consider being monitored and could result in a priority list. For a full coverage every traffic passing any switch would have to be mirrored, but that is rarely feasible.
One capture point can be either forward to a single probe or multiple capture points could be aggregated before being forwarded to a probe. So the setup is split between the traffic capture spots and the actual place of the probe where the mirrored traffic is investigated.
WAN¶
One of the first points of interest for traffic is the so called North-South traffic that passes from your network to the Internet. Monitoring this traffic would reveal attacks coming from the outside and also requests from your network towards malicious domains or IPs, like Command-and-Control servers.
There are different options where to grab traffic from that all have pros and cons. Sometimes the combination of several provides the best coverage.
Front¶
Traffic captured directly at the WAN uplink will show all traffic coming in and outside the network, which provides a high coverage for North-South but also catches all less interesting traffic like pure scans. One advantage is, that the traffic shows requests after NAT was applied.
Gateway¶
Some firewalls or gateways provide the functionality to forward traffic directly in between the transition from the LAN to the WAN. Depending on the implementation it can provide a look inside not covered by any of the spots in front or the back of the gateway.
Back¶
Capturing the traffic behind the firewalls or gateways provides another view on the North-South angle with less noise in general, since only allowed incoming traffic is monitored. While it might lack some coverage, since blocked attacks towards the firewall or scans are not seen, this spot is one of the most important ones in most scenarios. In deployments with NAT in place, this spot generally shows the traffic before the NAT towards the Internet is applied and makes it easier to spot the original source for outgoing connections.
Misc¶
In more complex setups with multiple uplinks, load-balancing in place or tunnels, the approach should be to follow the different ways a network flow could go and check on which points the traffic will pass by. This could result in challenges like unidirectional traffic forwarding due to asymmetric routing or duplication.
Avoid unidirectional traffic forwarding to a single interface/port on the probe at all costs, the engine is designed to work on a per flow basis and the majority of functionality won’t apply on unidirectional traffic. It’s even worse, since the engine would wait for the other side of the flow to be seen, which would have a big penalty on performance.
Duplication is not as bad as unidirectional traffic, but adds unncessary additions to the processing power and has the potential to result in misordering of the flow information.
LAN¶
Within your local network are several important spots that aim to cover the East-West traffic alongside detection of Lateral Movement. If traffic is only captured on WAN directed paths, the mentioned traffic won’t be seen.
Gathering a full visibility coverage is hard to achieve in most environments, since it would require a lot of resources for the forwarding from each and every switch. Thus the recommendation is to start with central exchanges between different areas of the network. This could be done at Core-Switches or Firewalls that sit between the segregated networks. For example the traffic that passes from the client machines in the office towards the local ActiveDirectory server as seen in the diagram below. In such a scenario the traffic between the clients might not be visibile but access towards and from the AD server would be caught.
Note
The described guideline highly benefits from a proper network architecture with strict segregation
The same concept applies for a datacenter where the production servers would be seperated from the staging servers and the DMZ as well.
DNS Traffic¶
It’s very important to make sure that at least the lookup for DNS entries from the local DNS server is included in the traffic being forwarded.
Traffic Forwarding¶
Once the points of interest in the network are defined, the actual forwarding itself needs to take place. There are several different techniques available to forward traffic.
Port Mirroring¶
Most switches provide a so called Mirroring Port feature where traffic seen on other ports would be forwarded on a dedicated port as a copy. This is one of the less expensive ways but provides no advance features like merging, filtering or load balance traffic. In addiion to that, most switches set a lower priority on the mirror port to avoid issues on the live traffic which could result in less traffic being mirrored.
Network TAP¶
A so called Test Access Point (TAP) is a device that you plug directly in between the cables. Basic TAP devices are purely passive but lack advanced features that active TAP devices would provide. The drawback of passive TAPs is the fact, that they copy RX or TX only one one port ending up in unidirectional flows being forwarded. This needs to be solved either by active TAPs or a packet broker.
Note
It’s important to emphasize that on one port of the Stamus probe it needs to be ensured to have both directions of a flow
Network Packet Broker¶
The most advanced, flexible but also most expensive technique would be a a Network Packet Broker. Those devices can be fed by multiple inputs coming from a variety of sources, like a passive TAP in between the firewall or the port mirror from the core switch. The traffic can afterwards being forwarded to specific output ports, split between probes or load balanced. In addition to that most of the products provide filtering and buffering features.
(ER)SPAN¶
Another alternative are the different ways of Switch Port ANalyzer (SPAN) which are more or less the same like a pure mirroring port. There are the Remote SPAN (RSPAN) and Encapsulated remote SPAN (ERSPAN) allow for more complex traffic forwarding even across multiple switches. Some virtualization environments also support a virtual SPAN as well.
Cloud providers¶
In cloud environments you’re bound to the features the provider supports. The basic concepts described above do also apply here but more advance cloud-native environments make it more complicated to properly decide the network capture points.
AWS¶
In AWS the features is called Traffic Mirroring and an explanation can be found at AWS Traffic Mirroring and some other vendors provide packet broker features in the Marketplace.
GCP¶
In the Google Cloud Platform (GCP) the feature is called Packet Mirroring and the documentation can be found at GCP Packet Mirroring including examples.
Azure¶
The feature in Azure is called Virtual network TAP but is currently on hold, as you can read at Azure Virtual network TAP with the suggestion to use packet broker solutions from partners at the marketplace.
Requirements¶
The Suricata engine has some requirements for the traffic being forwaded, to be met.
Bidirectional¶
One of the most important requirements is, that the engine needs to see the complete flow since it’s a flow aware based detection. While in theory it supports async mode as well, a lot of the detection will not work and performance will suffer as well. It’s also important to note that this requirement is per port and not per probe, since Suricata is listening on the dedicated interface port and the performance optimization is also applied there. So sending one direction to one port of the probe and the other direction to the other port of the probe is NO solution.
Tunneling¶
While Suricata supports ERSPAN (both types), GRE and VLAN, it is recommended to not stack several (tunneling) layers on each other before forwarding it to the probe.
Configuration Recommendations¶
Depending on the environment, there are some recommendations for the configuration to apply.
HOME_NET¶
The variable HOME_NET should be set to internal IPs in most cases but this
can vary based on the actual capture point of the traffic. A good example
would be a setup with a Proxy. The initial connection would have the source
IP of the actual client (ideally included in the HOME_NET variable) while
the outgoing connection would show the Proxy (or Public IP) as the new source
IP and thus should be in the HOME_NET for traffic being seen going out. To
make it more complicated, the destination IP for the client connection would be
the Proxy, so in that case it should be included in EXTERNAL_NET instead of
HOME_NET.
Lateral Movement¶
Depending on your architecture it could make sense to rely on the Lateral Movement configuration including the transformation feature. This would make sure that the variables on signatures are adjusted.
Challenges¶
There are several challenges that one might run into when doing traffic forwarding.
Noise¶
As mentioned earlier, a full coverage could result in very noisy events, especially if traffic is captured on the WAN side before traffic has hit the firewall. Depending on the performance impact it might make sense to even filter the traffic before it hits the probe.
Drops¶
Dropped packets, either on the path towards the probe or on the probe, are a big issue, because the detection is negatively impacted. Depending on the amount of packets missing the engine can’t fully reassemble the stream or specific malicious content was not even seen.
Missing packets¶
If packets are missing it’s very similar to dropped packets and are typical issue with oversubscribed mirror parts. This could also lead to reduced detection capabilities.
Load Balancing¶
While load-balancing in the live traffic can be an advantage, it could be an issue for monitoring the traffic. If the RX and TX direction of a traffic is split between two locations and the traffic forwardind is to local probes, both would just see one side of the flow and thus fail to properly reconstruct the flow and apply the detection patterns.
Therefore a full ordered flow containing RX/TX in the correct order is required for the maximum in detection capabilities.
Traffic¶
There are even more issues, so it highly depends on the traffic. There are more points for consideration:
VLAN QinQ is often implemented in a non-rfc compliant way and thus breaks performance optimizations on the probe
Multilayer Tunneling increases the load on the engine and tends to create confusion on the later analysis
Elephant Flows are typically long lasting high bandwith flows, like backup traffic, that easily saturate a full CPU core on the probe
Although the traffic could still be inspected, it is recommended to not mirror that type of traffic to the probes. Some of those issues could be solved by a Network Packet Broker that could remove some layers or even shunt/cut bigger flows.