Hunting Filters¶

Overview¶

A filter set is basically a hunting idea or concept – translated into criteria based on the selection, negation and wild carding of event metadata values – which results in a filtered query of the SSP data.

Predefined Hunting Filter Sets¶

As of the U39 software release, there are over 120 pre-defined filter sets, accessible from the Enriched Hunting interface and organized around 11 categories. Thus it allows to zoom in for millions of events into a few that can help reveal low hanging fruit and provide better situational awareness with a click.

Adware - provides guided hunting to identify various potentially unwanted programs operating on the network.
Anomaly - provides guided hunting filters for listing Host Insights for assets using traditional services like TLS/SSH/HTTP but using non-traditional ports.
Compliance - provides guided hunting filters for listing out hosts with unusual encryption certificate usage.
Hunt - provides guided hunting filters for broad area of hunting ideas, such as obfuscated executables, suspicious zipped files transfers, suspicious payloads, successful scans, backdoors, exploits, coinminers, cryptominers, base64 functions detected in events and similar to name a few.
Info - provides guided hunting to identify certain groups of user agents operating on the network.
MITRE - provide guided hunting to identify events for which the MITRE technique is identified.
Phishing - provides guided hunting filters for potential successful phishing attempts.
Policy - provides guided hunting filters for potential organisational policy violations such as: older or vulnerable TLS encryption, Dynamic DNS, Abused file sharing, public DNS, possible TOR traffic, clear text passwords usage and more.
Roles - provides guided hunting filters for listing and getting automatic Host Insights critical infrastructure such as domain controllers, DHCP servers, proxies, printers and more.
Services - provides guided hunting filters for different network services observed to and from assets. For example: Apache, Microsoft IIS, Nginx servers detected. HTTP(S) proxy detection and others.
Trojan - provides guided hunting filters for Trojan and PUP (Potentially Unwanted Programs) events.

Detailed use cases and examples of how to use the pre defined filter sets are available in our Threat Hunting blog series. The combined power of the predefined filtersets with local organisational enrichment and knowledge also provides for a proactive approach to minimising the risk of malware and threat actors establishing a foothold.

User Defined Filters¶

Adding Filter Criteria¶

A security analyst may apply additional criteria to any of the predefined filters to advance their hunt. These criteria include IP address, specific network probe, message, not-in-message, port, signature ID, ES filter, protocol, organization-specific network names, and more.

By simply clicking on the magnifier icons next to the metadata field value, a user may apply additional filter criteria to the data using various metadata fields in the events. Any field part of a filter set can further be edited and wild carded if needed.

Hunting Filers allow searching alert’s metadata in various ways such as searching for an IP address, a specific probe, a port number, a signature SID, and more.

Some filters can be negated (NOT filter) and others support wildcard notation for pattern matching. See below.

Filter Type	Input	Negate	Wildcard	Description
IP	IP address	Yes	No	Filter on an IP address, source or destination
Probe	string	Yes	Yes	Filter on designated network probes
Message	string	No	No	Filter on a specific word
Not In Message	string	No	Yes	Exclude matching words from the results
Port	integer	Yes	No	Filter on a port number, source or destination
Signature ID	string	Yes	No	Filter or exclude a specific signature identifier
ES Filter	string	Yes	No	Filter with a custom boolean Elasticsearch filter
Protocol	string	Yes	Yes	Shortcut to access most common protocols and fields used for filtering
Network Def	string	Yes	Yes	Filter alerts from a specific organisation from Network Definitions (ex: Accounting)

IP Filter¶

The IP filter allows searching for an IP address regardless of whether this IP has been observed as the source or the destination of the communication.

Hint

The IP filter also allows typing a subnet such as IP: 10.7.0.0/24 to only get alerts from systems belonging to this subnet. This notation also applies to other IP fields such as src_ip or dest_ip

If you want to limit the results to either a source IP or a destination IP, there are multiple ways to achieve this goal. To illustrate this, let’s see how we can filter on the source IP 10.7.5.5.

Example 1¶

From the Dashboard page, locate the Organizational Information section and the card Sources

If the desired IP is already listed in the TOP5, simply click on the “+” magnifier to add this source IP as a filter. Otherwise, click on the “3 vertical dots” in the card’s title to see the full list. Locate the desired IP and add it as a filter using the “+” magnifier.

Example 2¶

From the Alerts page, set an IP address filter on the desired IP such as IP: 10.7.5.5. As you will see in the results, this IP will appear as either the source or the destination of the communication. Scroll down the alerts list until you find an alert that matches your criteria (i.e. 10.7.5.5 as source IP in this example). Open the alert panel and under the block IP and basic information click on the “+” magnifier to add this source IP as a filter.

The Active Filters are now composed of the 2 filters IP: 10.7.5.5 AND src_ip: 10.7.5.5. At this point, you may decide to remove the IP filter to only have the src_ip filter depending on the search you are trying to build.

Example 3¶

Finally, another way to achieve this goal could be using ES Filters (see below for more details) by typing the expression src_ip: 10.7.5.5

ES Filters¶

Stamus Central Server relies on the ELK stack to allow searching alerts and their associated metadata. With Enriched Hunting, one can perform hunting activities and our software hides the complexity of ELK through the use of our interface that allows selecting desired fields and creating active filters without the need of understanding the underlying data structure nor the ELK search logic.

However, in some situations, relying on Elasticsearch Filters, or ES Filters, may come handy for advanced use cases.

Under The Hood¶

Under the hood, Stamus Central Server uses ELK to store and index the data it receives: alerts and protocol metadata. Each category of data has its own index, for example, alerts will be stored in logstash-alerts, DNS events will be stored into logstats-dns, HTTP events will be stored into logstats-http and so on.

ES Filters allows querying the logstats-alerts index to access alerts’ metadata such as source IP, destination IP, and protocol specific content such as HTTP User-Agent or Payload content for example. ES Filters queries ELK uses Apache Lucene as a query language and simply put, a search is a boolean expression composed of terms, and a term is a word or a pattern.

For example, to search for a source IP of value 10.1.2.3, one would write src_ip: 10.1.2.3. Here, src_ip is the term, followed by a colon and the searched value. The colon indicates an equality such as “src_ip is 10.1.2.3”.

As a search is a boolean expression, it is possible to use operators such as AND, OR, NOT, etc. For example, the query src_ip: 10.7.5.5 OR src_ip: 10.7.5.101 would list all alerts from either sources 10.7.5.101 and 10.7.5.5.

Important

ES Filters aren’t compatible with Hunting Policies as of U38.

Note

Kibana Query Language (KQL) was later introduced as an alternative to Apache Lucene and is currently not supported by Stamus Central Server.

Knowing the Fields to Use¶

When building a query, the first thing we need to know are the fields names available we can use. Simply put, any fields displayed in the Hunting interface can be used and to see the list of all the available fields, simply go under JSON View (Alerts tab).

{
    "flow_id":1587674205492238,
    "stream":1,
    "dest_port":443,
    "geoip":{
        "continent":{
            "code":"EU",
            "geoname_id":6255148,
            "name":"Europe",
        },
        "country_name":"Romania",
        "continent_code":"EU",
        "timezone":"Europe/Bucharest",
        "country":{
            "geoname_id":798549,
            "iso_code":"RO",
            "is_in_european_union":true,
            "name":"Romania",
        }
    },
    "@timestamp":"2021-12-08T04:31:06.157Z",
    "src_port":62227,
    "dest_ip":"-REDACTED-",
    "tls":{
        "ja3s":{
            "string":"769,49172,65281-11",
            "hash":"623de93db17d313345d7ea481e7443cf",
        },
        "fingerprint":"34:f0:60:57:ee:a1:ba:0e:cd:07:34:fb:78:90:e5:b5:4b:3f:89:dc",
        "version":"TLSv1",
    }
}

From the above snippet, we can see different field names we can use to write an ES Filter. Fields that are nested must be concatenated using a dot, for example, "tls": { "ja3s": { "string": "..." }} will be accessed with tls.ja3s.string.

Examples:

dest_port: [1024 TO 65535] will retrieve all alerts from which the destination port of the connection is between 1,024 and 65,535.
dest_port: 443 will retrieve all alerts from which the destination port is 443
geoip.country.iso_code: RO will retrieve all alerts associated to Romania
geoip.country.iso_code: R* will retrieve all alerts associated with any country having a country code starting with the letter “R” such as Romania (RO) or Russia (RU).
payload_printable: *COMCTL32* will retrieve all alerts having the substring COMCTL32 in the payload. This could be “COMCTL32”, “fooCOMCTL32”, “COMCTL32bar”, “fooCOMCTL32bar”, and so on
http.http_response_body_printable: *msdos* will retrieve alerts having the substring MSDOS in the HTTP Response Body
tls.version: tlsv1 will retrieve alerts where TLSv1 was used for SSL

Note

Pattern searches are case insensitive. Searching for geoip.country.iso_code: RO and geoip.country.iso_code: ro will lead to the same results. Fields name are however case sensitive and must be written in lowercase (SRC_IP is invalid for example, src_ip must be used)

Hint

Some of the above examples are really simple and one doesn’t necessarily need to write ES Filters because the web interface allows selecting most of those values in a single click from the alert’s panel

Attention

One of the common error when using wildcards is to have a space between the wildcard and the searched pattern such as payload_printable: * COMCTL32*. In that case, the search will literally return anything!

Note

Enriched Hunting provides FilterSets out of the box and some of them use ES Filters. Besides, some of them use the notation field.keyword: value. The operator “.keyword” from Elasticsearch perform search queries by explicitly stating that the field value shouldn’t be broken into tokens (e.g “New York” could match “New” or “York” otherwise)

Global Searches¶

In some situations, it can also be interesting to search for a pattern regardless of the field in which this pattern may be stored.

Note

Such a search may take a little longer as all fields need to be scanned so make sure to use specific patterns that won’t match on many things!

To search for the word “COMCTL32”, simply put “COMCTL32” in the ES Filter input without giving any field name.

Similarly, to search for a substring, use the wildcard notation, either at the beginning of the pattern or at the end of it, or both.

Finally, both fields searches and global searches can be mixed up to create more complex queries such as *Mozilla* AND NOT http.http_user_agent: *Mozilla* which will search for the substring Mozilla in any fields and will exclude all events having only a match on the HTTP User-Agent.

Specific Filters¶

Note

Any filter specified on those pages will not be carried to the other pages as they only apply on those specific pages. However, they are not removed and so when returning back the initial page, the filter will still be present.

Signatures¶

To scan and search the signatures more easily, the page Signatures extend the default filters with the following ones.

Filter Type	Input	Negate	Wildcard	Description
Alerts min	integer	No	No	Filter signatures that have at least the specified number of matches
Alerts max	integer	No	No	Filter signatures that have at most the specified number of matches
Content	string	No	No	Filter signatures having the specified substring
Not in Content	string	No	No	Exclude signatures having the specified substring

Hint

The substring matching used in Content and Not In Content doesn’t require any special character such as a wildcard to match part of a word. For example, Content: CHA would match CHAT, Charset and so on. This is case insensitive.

Hosts¶

To search the observed hosts in your environment more easily, the page Hosts extend the default filters with the following ones.

Filter Type	Input	Negate	Wildcard	Description
Hosts: HTTP User Agent	string	Yes	Yes	Filter hosts based on observed HTTP User Agents
Hosts: TLS JA3	string	Yes	Yes	Filter hosts based on observed JA3 fingerprints
Hosts: SSH Client Version	string	Yes	Yes	Filter hosts based on observed SSH Client version
Hosts: Username	string	Yes	Yes	Filter hosts based on observed Username
Hosts: Hostname	string	Yes	Yes	Filter hosts based on observed Hostname
Hosts: Services	string	Yes	Yes	Filter hosts based on observed Services
Hosts: Counts	integer	Yes	No	Filter hosts having at least/most the specified count of Services, Hostnames, TLS Agents, HTTP User-Agent, SSH Clients or Usernames

Creating Filter Sets¶

The Hunting interface allows you to create your own filter sets. You can choose between creating a Global Filter Set or a Private Filter Set

Creating Global Filter Sets¶

Global filter sets are usually shared between all authorized and authenticated users on your system. To create such a filter set, you first have to apply the filters you would like to use.

Next, you should click on Save Filter Set under Actions - on the right hand-side of the Hunting interface.

Fill in the Create new Filter Set form and select the Shared checkbox.

Creating Private Filter Sets¶

Private filter sets are available only to the user, who has created them. To create such a filter set, you first have to apply the filters you would like to use.

Next, you should click on Save Filter Set under Actions - on the right hand-side of the Hunting interface.

Fill in the Create new Filter Set form and leave the Shared checkbox unselected.

Loading Filter Sets¶

If you would like to load a filter set, you have to go to any page of the Hunting interface, then click on Load Filter Set from the Actions menu.

Then, you can use the search field to look for a specific filter set within the Global, Private and Stamus Predefined Filter Sets.

Exporting Filter Sets¶

If you would like to export your Global/Private filter sets, you can do this from our Administration interface.

You have to go to Rulesets page -> Import/Export Policies/Filtersets under Actions panel on the left hand-side of the page.

Note

The import/export format is json.

Deleting Filter Sets¶

In order to delete a custom created filter set, you need to click on Load Filter Set from the Actions menu.

Note

Only Global Filter Sets and Private Filter Sets can be deleted.

To delete a filter set - you have to click on the delete icon next to its name.