Skip to main content
Lantern Home
 
 
Splunk Lantern

Alerting on missing source types

  1. Last updated
  2. Save as PDF

 

With Splunk software, nothing is more important than ensuring that your key data is being properly received. If a source type went missing, dashboards wouldn't populate and alerts wouldn't fire. You want to monitor your data availability by:

  • Creating a state file, or list, of every key source type in your system
  • Automatically updating the state file whenever a new source type appears
  • Alerting when a source type is no longer observed

Solution

  1. Create a source type state file, which is an initial lookup file that contains a list of source types that exist in your environment. You only need to do this one time. To create this, run the following command: 
    | tstats count WHERE index=my* earliest=-24h latest=now BY sourcetype
| eval state="initial"
| outputlookup sourcetype_state.csv
    • Check the permissions of the sourcetype_state.csv lookup file to make sure the search will run correctly.
    • Edit the my* value in index to include any index you want to monitor.
  2. Automatically update your state file. On a designated regular interval (for example, once per week), you'll want your state file to automatically update itself with any new source types that appear. The following looks for all new source types, appends the previous list, then outputs a new lookup. 
    | tstats count WHERE index=my* earliest=-60m latest=now BY sourcetype
| eval state="appended"
| append
    [| inputlookup sourcetype_state.csv]
| dedup sourcetype
| outputlookup sourcetype_state.csv

    It can be useful to manually manage the state file. You might want to delete source types for the file to stop them from being monitored. Alternatively, you may want to add source types manually. The simplest way to do this is to use the Splunk App for Lookup File Editing or, if you have Splunk Enterprise Security, use the managed lookups feature.

  3. Create an alert to regularly search your data for missing source types. The following search looks for recent source types, appends source types from the state file as "missing", and then reduces the results to only missing items. 
    | tstats count WHERE index=my* earliest=-60m latest=now BY sourcetype
| eval state="exists"
| append
    [| inputlookup sourcetype_state.csv
    | eval state="missing"]
| dedup sourcetype
| search state="missing"

Next steps

These additional Splunk resources might help you understand and implement this product tip: