Conditional Logging

Reducing SIEM Ingestion Costs

Maximize Data Retention Without Sacrificing Security

By leveraging Clear NDR®’s conditional logging and pre-correlation capabilities, security teams can dramatically reduce SIEM data ingestion costs while maintaining comprehensive security visibility. With data volume reductions averaging 97% across various network sizes, organizations can realize significant operational cost savings without compromising security effectiveness.

The approach is particularly valuable for organizations with high-speed networks, where traditional raw data ingestion would make SIEM costs prohibitively expensive. Clear NDR®’s model enables security teams to focus their SIEM budget on extending retention periods and adding additional security data sources rather than processing redundant network traffic data.

The SIEM Cost Challenge

SIEM platforms typically charge based on data ingestion volumes, making the management of log sources a critical cost factor for security teams. Traditional approaches to network security monitoring involve sending raw data from multiple sources to the SIEM:

  • Intrusion Detection Systems (IDS) alerts

  • Network Security Monitor (e.g. Zeek) logs (connection, DNS, HTTP, SSL/TLS, etc.)

  • NetFlow/IPFIX records

  • Packet capture (PCAP) files for investigations

  • Extended detection and response (EDR) event logs

  • Application logs

  • Cloud workflow logs

While comprehensive, this approach results in extremely high data volumes that drive up SIEM costs significantly, often forcing security teams to make difficult tradeoffs between visibility and budget constraints.

Log Volume from Traditional Network Security Tools

When using separate tools to monitor network traffic, organizations typically ingest data from multiple sources, each generating its own set of logs. In this paper, we will focus on three of those tools:

IDS/IPS Systems

  • Alert logs (1-5 GB per day per Gbps of monitored traffic)

  • Rule match details (2-8 GB per day per Gbps)

  • Packet captures of suspicious traffic (varies widely based on configuration)

Network Security Monitor

  • conn.log: Basic TCP/UDP/ICMP connection details (10-20 GB per day per Gbps)

  • dns.log: DNS query and response data (2-5 GB per day per Gbps)

  • http.log: HTTP requests and responses (5-15 GB per day per Gbps)

  • ssl.log: SSL/TLS handshake information (3-8 GB per day per Gbps)

  • files.log: Metadata about files transferred (1-3 GB per day per Gbps)

  • x509.log: Certificate information (0.5-2 GB per day per Gbps)

  • Additional protocol-specific logs (SMB, RDP, SMTP, etc.) adding 5-15 GB per day per Gbps per protocol

NetFlow/IPFIX

  • Flow records (3-8 GB per day per Gbps of monitored traffic)

  • Flow metadata and extensions (additional 1-3 GB per day per Gbps)

For a typical 10 Gbps network, these sources can collectively generate 300-800 GB of log data per day that would need to be ingested into a SIEM for comprehensive visibility. The challenge is compounded by:

  1. Duplication: The same network session may appear in IDS logs, Zeek conn.log, and NetFlow records

  2. Correlation burden: Analysts must manually correlate events across different log formats and timestamps

  3. Varying retention needs: Some logs are more valuable for long-term retention than others

This leads to significant SIEM ingestion costs without providing the pre-correlation and contextual advantages of an integrated NDR solution.

Pre-Correlated, Optimized Data

Clear NDR® offers two key capabilities that directly address SIEM ingestion costs:

  1. Conditional Logging: Instead of capturing every network transaction, Clear NDR® can selectively store only the protocol transactions, flow records, and file transactions associated with critical detection events such as alerts, Sightings™, and policy violations.

  2. Pre-Correlation: Clear NDR® performs correlation at the source, linking relevant network traffic data with security events before sending to the SIEM. This pre-correlation eliminates the need to ingest raw data streams for correlation within the SIEM itself.

Traditional vs. Clear NDR® Approach

The table below compares the data volumes generated by traditional raw data ingestion versus Clear NDR®’s optimized approach:

Conditional Logging

SIEM Cost Savings Across Network Scales

The impact on SIEM ingestion costs scales with network size. The table below demonstrates the potential savings at various network data rates, comparing the data volumes generated by traditional raw data ingestion versus Clear NDR®’s optimized approach:

Conditional Logging

Note

Based on industry average SIEM pricing of $4 per GB/day ingested. Actual savings will vary based on specific SIEM pricing models.

Benefits Beyond Cost Savings

Clear NDR®’s approach delivers additional benefits beyond direct SIEM cost reduction:

  1. Improved Analysis Efficiency: Pre-correlated data means analysts spend less time manually connecting events across multiple data sources.

  2. Faster Query Performance: Reduced data volumes in the SIEM lead to quicker search and investigation times.

  3. Extended Retention: Organizations can afford to retain security data for longer periods within the SIEM.

  4. Preserved Threat Detection: Unlike simply truncating data sources, Clear NDR® maintains full threat detection capabilities while optimizing data volumes.

Implementation Considerations

Security teams can implement this approach through:

  1. Configuring Clear NDR® for conditional logging mode

  2. Setting up SIEM integration using Clear NDR®’s API or direct integration connectors

  3. Eliminating redundant raw data sources from SIEM ingestion

  4. Maintaining a small subset of critical raw logs if required by compliance needs

Optimizing Storage

Maximize Data Retention Without Sacrificing Security

Conditional logging can dramatically reduce the storage requirements and dramatically extend the time for which log data is retained in Clear NDR®.

Clear NDR®’s conditional logging feature solves a critical challenge for organizations deploying network detection and response solutions by dramatically reducing storage requirements while maintaining security effectiveness. Unlike the “verbose” mode that captures extensive metadata from every network transaction, conditional logging selectively stores only data associated with critical detection events like alerts and policy violations. Production deployments demonstrate up to 97% reduction in storage needs, extending data retention periods by up to 46 times with the same storage capacity—all without compromising threat detection capabilities or incurring additional costs. This enables organizations monitoring high-speed networks to balance comprehensive security visibility with practical storage constraints.

In addition to security events, Clear NDR® extracts and stores a wealth of network data logs in their ‘verbose’ mode (full logging). These logs include detailed metadata associated with every network protocol transaction, flow records, file exchanges, and more. The complete data set can be found in the data schema section of the user documentation for Clear NDR® Data fields.

For many organizations, this extensive metadata is crucial to their security practice. For others – particularly those who have deployed Clear NDR® to monitor very high speed networks – the storage requirements associated with this verbose logging make it impractical to retain historical logs for long enough to satisfy their data retention needs.

What is Conditional Logging

Clear NDR® offers a conditional logging option which can dramatically reduce the storage burden while still retaining critical evidence needed for event triage and incident response.

With conditional logging enabled, Clear NDR® stores the protocol transactions, flow records, protocol error/anomaly detection events, and file transactions only when they are associated with critical detection events, such as alerts, Sightings™, DoCs, and DoPVs. This allows users to retain evidence associated with detection events for extended periods without investing in additional storage costs to retain every protocol transaction or flow record. Without conditional logging, every transaction, protocol error/anomaly event, and flow record is stored.

Importantly, conditional logging does not impact Clear NDR®’s ability to detect threats or impact the wealth of data captured associated with threat and policy violation detection events.

Note

Conditional logging should not be confused with the conditional packet capture feature - Conditional PCAP Activation in which Clear NDR® captures all of the raw packets associated with any detection event, logging them as PCAP files. Unlike conditional logging described above, there is no verbose or full logging mode for packet capture.

Reducing Storage and Extending Data Retention

Conditional logging can dramatically reduce the storage requirements, and – by association – dramatically extend the time for which log data is retained in Clear NDR®.

The table below illustrates the impact of conditional logging on an example deployment:

Conditional Logging

And the table below summarizes the savings under several scenarios extrapolated from the above:

Conditional Logging

Licensing

Conditional logging is included in the standard license for Clear NDR® - at no additional cost.