Skip to main content
Splunk Lantern

Expected host not reporting events

You might need to find systems that are not reporting events but should be when doing the following:

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

  • Splunk Enterprise or Splunk Cloud Platform
  • Any data source indexed by Splunk 

Example

Many compliance and regulatory frameworks contain clauses that specify requirements for central logging of event data, as well as retention periods and use of that data to assist in detecting data breaches and investigation and handling of threats. These regulations also specify that a mechanism must exist to notify when critical systems stop forwarding event data: for example, tracking of “initialization, stopping, or pausing of audit logs.” This is directly related to the fact that commonly, the intent behind disruption to event data forwarding is malicious: for example, an attempt to evade a preventive measure or to avoid detection. You need to set up this logging and reporting capability.

To optimize the search shown below, you should specify an index and a time range.  

  1. Identify all relevant IT assets from a data mapping exercise conducted by the Data Privacy Officer’s team. These are all IT assets that are relevant to the full audit trail of data processing activities. This includes not only data stores and repositories that house sensitive personal data (PD) and personally identifiable information (PII), but also any technologies that are involved in the processing, storage, transmission, receipt, rendering, encrypt/decrypt, relaying, and handling of such data in any capacity. 
  2. Ensure that those assets are configured properly to report logging activity to an appropriate central repository. 
  3. Use your data mapping results to build a lookup that associates systems to their system category.
  4. Run the following search:
|tstats prestats=t count(host) where latest=-2h earliest=-7d index=* groupby host 
|tstats prestats=t append=t count where earliest=-2h index=* groupby host 
|stats count(host) AS historical count AS current BY host 
|where current=0 AND isnotnull(historical) AND historical>0 
|lookup <name of lookup you created in step 3> host AS dest OUTPUT category
|search category=*

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

|tstats prestats=t count(host) where latest=-2h earliest=-7d index=* groupby host 

Retrieve a count for all hosts in all indexes starting 2 hours ago and looking back 7 days. This is the "historical data receive frequency".  

|tstats prestats=t append=t count where earliest=-2h index=* groupby host 

Retrieve a count for all hosts in the last 2 hours and append to the results from the first tstats command. This is the "current data send frequency".

|stats count(host) AS historical count AS current BY host 

Calculate the recent count and the historical count per host from the events generated by the two tstats commands. 

|where current=0 AND isnotnull(historical) AND historical>0

Filter for hosts that have sent more than zero events in the historical time frame but are currently sending zero events. 

|lookup <name of lookup you created in step 3> host AS dest OUTPUT category 

Look up the host in the categorization lookup you created. When the host in the lookup matches dest in the event, output the category value from the lookup and add it to the event.

|search category=*

Filter for events that have a value in the category field. These are in scope for GDPR compliance. 

Result

The only known false positive for this search is when a host goes offline, but you are still tracking it as being required and in scope.

When this alert fires, first see if the host is still alive. If it is, you will need to see if Splunk is still running or if there are other interruptions to logging. Stopped logging services can indicate an intrusion, so keep an eye out for any indications of malicious behavior.

Additionally, you should monitor the mapped IT assets changes in logging status, adjust for known outages, and prioritize incident response for any failures to report by hosts that are not scheduled for downtime.

GDPR Relevance: GDPR requires that you collect the full audit trail of data processing activities and involved systems. If you're breached, Article 33 requires you to inform the authorities including details about the nature of the breach, such as how many individuals have been affected. The same requirement is in Article 34 for when your organization has to identify which individuals are affected. If you do not have the logs and can't scope it down, you have to assume the worst case scenario: that all the data was affected. 

In addition, Article 32 requires that you implement proper security controls and test their effectiveness. If you stop seeing logs for a host, then you will not be able to prove the status of the applied security controls on their machine, in case of damage claims (Article 82) or privacy audits (Article 58). If you lack host logs, you can't prove which records have been processed, such as proving that you deleted a record according to an individual's delete request (Article 17) or proving that you complied with a processing restriction requested by an individual (Articles 18 and Article 21). Finally, if a host does not report its event data, a processor can't prove that only authorized individuals have accessed the data (Article 28).

  • Was this article helpful?