Applying version control to correlation search changes

Splunk Enterprise options

• App: Git Version Control For Splunk
• Maintain a centralized Git repository and make all changes there, using the command line to pull the changes from Git into your deployment.

Splunk Cloud Platform options

• App: ES Choreographer
  • Pros
    • Great at telling you who made a change and when.
    • Done through a query against the _audit index. Here is a sample search to accomplish this:

      index=_audit source=audittrail sourcetype=audittrail 
      host=* succeeded rule ( create_saved_search OR delete_saved_search OR update_saved_search ) ( TERM(action=create_saved_search) OR TERM(action=delete_saved_search) OR TERM(action=update_saved_search) ) action IN ( create_saved_search deleted modified update_saved_search )

    • Allows for easy peer review. You can integrate the app with the incident review dashboard (using Adaptive Response) to get analyst feedback on a correlation search.
  • Cons
    • Doesn’t do a good job of comparing different versions of a correlation search.
    • The interface can be difficult to work with.
    • Doesn’t collect complex fields, such as any adaptive response fields (like notables or risks) beyond the name of the response.
    • Doesn't work with SAML users.
    • index=main is hard-coded in multiple places, rather than using macros.
  
• App: Conf Manager
  • Pros
    • Great at tracking changes to any kind of knowledge object, especially useful for lookup table and macro versioning.
  • Cons
    • Requires a large number of scheduled searches to be fully functional. It uses a search for each type of knowledge object and backs up that knowledge object periodically, which leads to heavy SVC consumption.
    • Doesn’t track adaptive response actions within a correlation search. It only logs the name of the response action, such as notable or risk.
• Internal logging with splunkd_conf
  • Pros
    • Changes to correlation searches are already being logged. You can find them with:

      index=_internal sourcetype=splunkd_conf component=ConfOp "data.asset_uri{}"=*Rule

    • Tracks every field in a correlation search.
  • Cons
    • Short retention time due to the 28-day limit of _internal index. You can't get back to the original value if it's older than that.
    • Only changes are logged. A separate method is needed to track the previous version.
    • Only available when Splunk Enterprise Security is clustered.
• App: Splunk Enterprise Security 8.x
  • Pros
    • Built-in, simple version control. For more information, see Create multiple versions of a detection in Splunk Enterprise Security.
  • Cons
    • Only works with specific apps.
    • Doesn't show a side-by-side comparison.
    • Limited customization capabilities.

Proposed solution

1. Create a lookup table that contains key information about all enabled correlation searches. This table must include a hash of each search and its properties.
2. Create a saved search that:
   1. Queries all enabled correlation searches.
   2. Generates a hash of the search and all its properties.
   3. Compares the freshly generated hash against the lookup table.
   4. Displays the search and properties when a change is detected.
      
3. Enable summary indexing for the saved search.
4. Create a dashboard to display the searches that have been changed, along with the old and new values. You can see a sample dashboard using this JSON file.

The following video shows how this works.