Improving your security detection development processes
If you work in security content development, some of these pain points probably resonate with you:
- A detection fires too often. You have to determine if the data is meant to work with has changed in some way or if the query itself is not good.
- You don't know what content is supposed to do. Hopefully there is documentation somewhere, but you might have to sift through wikis, bug tracking systems, or emails.
- Content works in one environment but not another. You have to spend time replicating both, including versions, technical add-ons and their configurations, macros, etc and then test.
- You can't upgrade for fear that something will break. Your options in this case are to manually test all content in a test environment, delay upgrades, or do the upgrade and hope for the best. Testing is difficult because you might have hundreds of apps and add-ons, and each of their versions might impact other apps and add-ons. However, if you don't upgrade, you might miss out on important bug fixes or updates.
- You don't know who changed something, when, and why. Hopefully, the old version exists somewhere so you can get back to it.
At Splunk we know content development is hard. That's why we we've built some tools to help you address some of these many challenges. These are Attack Range, detections-as-code principles, and ContentCTL.
These tools are not officially supported by Splunk.
How to use Splunk software for this use case
Attack Range
The Splunk Attack Range is an open-source project maintained by the Splunk Threat Research Team. You can run it on a cloud provider, like AWS, or even locally, with some limitations. By running just a few commands, it builds for you:
- an instrumented Splunk cloud server with dozen of apps and TAs installed, and even Splunk Enterprise Security if you want
- a Windows endpoint, possibly a domain controller, with Atomic Red Team tooling and all logging configured to send to the Splunk server and be properly ingested
- additional endpoints like a Kali Linux machine that you can use to launch attacks or a regular Linux machine instrumented with Sysmon
The environment is built in about 20 minutes and is designed to be disposable so you can play around and simulate attacks without risk. You use the environment to develop detections, including referencing specific MITRE attacks, and then test the effectiveness of those detections.
Another way you can benefit from this project is by using the Splunk Attack Data Repo. This repository consists of raw data that was exported from Attack Range simulations and tested in a real Splunk environment. It's a good place to find sample data to test your own detections and see if your environment is set up to ingest it correctly. The YAML files provide information about what is in each data file.
Detection-as-code principles
Software development processes include linting jobs to ensure that code is formatted correctly, compile jobs to make sure code builds correctly, and unit testing. You can bring these same ideas to detection development to ensure a high level of quality and overcome some common challenges. In your team, define the following:
- Detection language: What should your detections look like when they are stored?
- Use a file that is easy to read and edit.
- Include key fields like name, version, description, and search field.
- Test data with raw data and source type so you know how to feed back into a server.
- Version control system: Store detections somewhere like GitHub. Put new or improved detections on a separate branch so they can get tested without interference.
- Automated workflows: Use for building and testing any apps.
- Test-driven development: Using the data sets discussed previously in this article.
Additional resources
The following resources might help you implement the guidance provided in this article:
- Splunk Lantern Article: Applying version control to correlation search changes
- Splunk Lantern Article: Using contentctl to speed up your SOC