Warning: You are viewing an older version of this documentation. Most recent is here: 40.0.1

File Extraction

Concept

When turned on, file extraction will save on disk the files observed on the wire by each Stamus Network Probe using Suricata signatures.

The files will be extracted and stored locally on the Network Probe performing the extraction and the REST-API on the manager, Stamus Central Server, allows to centrally retrieve chosen files.

As of U38, the currently supported protocols for file extraction are:

  • HTTP

  • SMTP

  • FTP

  • NFS

  • SMB

File Extraction Activation

To activate file extraction, go under Probe Management, menu Appliances.

Edit the desired Probe, or the desired Template, and go under the Settings tab.

Check the checkbox “Activate file extraction” and a few more options will expand as illustrated by the below screenshot

Activate File Extraction

All extracted files will have their sha256 fingerprint computed, this option cannot be deactivated. This mechanism is used to only store once each and every single file in case of multiple downloads of the same file.

You can then choose to also compute common hashes such as md5 and/or sha1, that is especially useful if you need to integrate with third-party solutions to which you want to send those hashes. This information, as well as the original filename, will be available in the fileinfo structure of the json metadata.

Finally, specify a size limit, in bytes, if you want to disregard large files such as ISO files and Apply changes to make the changes take effect.

Note

When 2 Network Probes, from 2 different capture locations, see the same file, this file will be extracted and stored on each of those 2 Network Probes. The deduplication of files is performed on a per probe basis.

How File Extraction Works

File extraction is performed through the usage of Suricata rules and only the rules using the keyword filestore will perform file extraction.

For example, the following rule will extract and de-duplicate all executables observed on the wire that are downloaded via HTTP from the networks $INFRA_SERVERS and $DC_SERVERS (defined as configuration variables):

alert http [$INFRA_SERVERS,$DC_SERVERS] any -> $EXTERNAL_NET any (msg:"FILE Executable public download from critical infra detected"; flow:established,to_server; http.method; content:"GET"; file.magic; content:"executable"; filestore; sid:1; rev:1;)

Best Practices recommends to always make rules as specific as possible as illustrated by the previous example in order to optimise performances and reduce false positives.

From the Hunting interface, the files will also be accessible directly from the alerts tab and can be downloaded as illustrated below

Downloading an extracted file

Important

Some of those files may be potentially dangerous, proceed with caution when downloading them!

Note

When a file has been extracted, the boolean stored in the fileinfo structure (JSON View / Related Events) will be set to true. Note that this indicates that the file was once stored and may have been purged at the time you are looking at this alert. While this is unlikely for recent alerts, it could be the case for the oldest ones. There is a cleaning process that removes the oldest files extracted when the disk space limit is reached (10% of the /var/log/suricata partition)

Hint

If you need to keep files for longer periods, you need to setup a third-party server on which to export regularly the files using the REST-API for example. The partition on which the files are stored on the Network Probes can only be extended on virtual machines.

Getting Files from the REST-API

Retrieving a file from the REST-API is a 2 step process. First, we need to instruct the manager, SCS, to retrieve the file from the Network Probe. Once this file has been retrieved on SCS, we can download it.

To query this endpoint, we will need 2 components:

  • The hash (sha256) of the file to retrieve

  • The probe name (string) on which the file is stored

First, we can ensure the status of the file using the following cURL command:

curl X GET -k "https://<SCS_ADDRESS>/rest/rules/filestore/<HASH>/status/?host=<PROBE_NAME>" -H "Authorization: Token <TOKEN>" -H 'Content-Type: application/json'

If the file is still present (i.e. not deleted), we will receive the following response:

{"status":"available"}

Now, we first need to instruct SCS to get the file. To do so, we will use the retrieve endpoint:

curl -X GET -k "https://<SCS_ADDRESS>/rest/rules/filestore/<HASH>/retrieve/?host=<PROBE_NAME>" -H 'Authorization: Token <TOKEN>' -H 'Content-Type: application/json'

The file should be downloaded onto SCS and you will get this result

{"retrieve":"done"}

Finally, to download the file, use the download endpoint:

curl -X GET -O -J -k "https://<SCS_ADDRESS>/rest/rules/filestore/<HASH>/download/?host=<PROBE_NAME>" -H "Authorization: Token <TOKEN>" -H 'Content-Type: application/octet-stream'