Responding to incidents with the Splunk platform and Fox-IT's Dissect
This war story, written by Fox-IT, shows how Splunk's integration can be used with Fox-IT's Dissect in the process of resolving complex and fast-evolving incidents. It breaks down the first 72 hours of a real incident from the perspective of Fox-IT, and shows their average incident response engagement with one of their clients. You can compare this process with your own to gain insights into your incident response process.
In the minutes and hours after detecting that an adversary has established a presence in their organization, many turn to trusted Incident Response (IR) partners like Fox-IT. These IR partners have the expertise, honed by experience, in managing the complexities of investigation, analysis, response, and remediation. They arrive quickly, rapidly deploy the tools they rely on to perform their work, work to understand the scope the incident, formulate a plan to permanently remove the adversary, and provide guidance and insights to the organization they are helping.
During this example engagement, Fox-IT leveraged the Dissect integration with the Splunk platform. Fox-IT's Dissect is a digital forensics and incident response framework and tool set that allows you to quickly access and analyze forensic artifacts from various disk and file formats. Where Dissect is great at gathering the forensic data and showing it on the command-line, the Splunk platform excels in visualizing it in a graphical user interface, performing data correlation, and providing easy search capabilities.
The first 72 hours of a security incident
Friday 18.00. H (Hour) + 0
Fox-IT's incident handler on call receives a heads-up from the account management team that a potentially large incident is incoming. Fifteen minutes later, after rushing home, the incident handler receives a call from the client on the emergency response hotline. The client was notified by law enforcement that they had a security issue. After performing an initial triage by phone, we confirmed we are dealing with a significant incident. We agree with the client on an initial budget and establish an investigation team. Simultaneously, the client uploads the first pieces of investigation data to the Fox-IT forensic lab.
Friday 21:00. H + 3
A kickoff with our team (an incident handler and two analysts) and the client's team takes place. Our team provides initial feedback on the uploaded data and presents a strategy on how to approach the incident.
- The approach is to get EDR (Enterprise Detection and Response) coverage on all systems for live monitoring, and to perform data acquisition on several systems in order to analyze historical events. We use Carbon Black as our preferred EDR platform, and for data acquisition we use the in-house developed Acquire, which is part of the Dissect framework.
- We debate the pros and cons of taking containment measures immediately versus waiting to get a better view of the environment. We usually try to get a better view of the environment if we’re unsure what kind of threat actor we’re dealing with.
- Our data acquisition technique boils down to “give us everything”. We prioritize the acquisition of critical systems (both in importance and likelihood of compromise), but, beyond that, we want to get as much host data as possible. This allows us to do analysis in bulk, instead of having a back-and-forth with the client each time we want to collect a new system. “Everything” is usually still a multi-step process though, and this time was no exception.
Friday 23:00. H + 5
The first Carbon Black EDR agent comes online.
Saturday 01:30. H + 7.5
After ensuring that more Carbon Black EDR agents show up online and that the client team knows how to perform data acquisition and upload that data, our team goes to sleep. We have agreed to resume work around 10:00 CET in the morning. We have also agreed to provide the client with the next update on Saturday at 14.00 CET, which is early morning for the client.
Saturday 09:30. H + 15.5
Using Carbon Black EDR insights, the investigation team discovers concerning live activity of the adversary and decides to wake up the client (in the middle of the night, their time) to prepare for a meeting in 45 minutes. While it generally feels positive when our approach works, telling a client that they likely need to take important and high impact decisions such as shutting down internet connections, and consequently their operations, can feel uncomfortable. Nonetheless, it is important we act based on facts and the evidence currently available to us.
When rolling out the Carbon Black EDR agents, we make sure to also deploy our custom watchlists to monitor the usage of certain tools. The following morning, we noticed that a watchlist had been triggered that monitors for the usage of rclone, a popular data exfiltration tool used by adversaries. What made this process even more suspicious was that it was trying to hide behind the veeam.exe name, the name of a popular backup solution.
Luckily, we're able to match on the actual product name in the binary, which tells us exactly what the actual name of the process should be, no matter how it is renamed by the user, as shown in the example below. The combination of watchlist queries like these with a platform like Carbon Black helps us to make split-second decisions.
Saturday 10:30. H + 16.5
We give a summary presentation to the client and discuss several scenarios based on business impact and effectiveness for containment. The client decides on a specific containment scenario to cut off internet access.
Saturday 14:00. H + 20
As previously agreed on, we have a “start-of-day” meeting with the client and discuss the approach for today. The goal is to do analysis and get a view on initial foothold, scope of compromise, and a timeline.
While investigation data uploads trickle in, we perform our standard processing steps. This consists of running Dissect plugins and artifact parsers against the data and storing the result on disk, followed by sending the exported artifacts to the Splunk platform, our preferred search platform.
To export the Windows event logs in this way, you could run a command like:
target-query /t/hostname.vmdk -f evtx > /h/hostname/evtx rdump /h/hostname/evtx -w splunk://127.0.0.1:1337
You can use optional commands like xargs or tee to speed up the export or parallelize the processing.
In addition to using the Splunk platform for IOC (indicators of compromise) checks and statistical analysis, some of our analysts prefer to use command line tools for a large part of their analysis. For this reason, we export to both files and the Splunk platform, so that analysts with different preferred workflows can efficiently work together on the case data. One analyst can consult the many dashboards we have created and create complex queries, while another can use a combination of Dissect (for example, target-query, rdump) and simple text processing tools (for example, (rip)grep, less).
Saturday 19:00. H + 25
We present the client with findings, including an initial timeline and scope of compromise. Discussions are ongoing concerning how to bring back some of the clients’ core processes in a safe way for the start of the business week.
Sunday 00:00. H + 30
We share additional findings with the client. Our team goes to sleep.
Sunday 10:00. H + 40
FoxCERT management is getting a team together that can provide on-site containment and remediation support for the client. Several calls, chats, and emails take place with remediation, RM&G (Risk Management & Governance) and FoxCERT practices throughout the group. Later that morning, there is confirmation that a containment lead and a pentester will fly to the customer’s premises the next day.
Sunday 14:00. H + 44
Today's “start-of-day” meeting includes a discussion of our approach for today. The investigation team reports a new potential backdoor into the network.
Sunday 19:00. H + 49
We hold a findings meeting with the client. The timeline is expanding. While the meeting is ongoing, the team finds evidence regarding the adversary and initial foothold. The decision is made to share the news later that night when there are additional details and observations regarding the behavior of the attacker. We call this an adversary profile.
Using the different plugins available in the Dissect framework, we triage different artifacts. We find the initial foothold of the actor by checking different sources like PowerShell logging (with the evtx plugin), scheduled tasks present on systems (with the tasks plugin), and an MFT (Master File Table) timeline (with the mft_timeline plugin). Dissect allows us to do this quickly and efficiently across a large number of data sources.
Sunday 23:00. H + 53
We hold an end-of-day meeting. We share information about the (medium confidence) adversary. The TTPs (Tools, Techniques and Procedures) we observed match content in several reputable security blogs. The client and the investigation team use the information on potential goals and TTP of the adversary to improve decision making on how to further mitigate the attack. It turns out that cutting the internet off was probably a very good idea and was taken at just the right time.
Monday 14:00. H + 68
We hold our third “start-of-day” meeting. Each member of the team worked at least 30 hours over the weekend. The team discusses the approach for today and agrees with the client on a shorter working day that day to make up for the weekend's heavy workload.
Monday 18:00. H + 72
We develop a more complete picture of the incident, including the systems and users involved, and how far back the compromise dates. The timeline of the attack will likely extend significantly (multiple months) into the past, and the initial intrusion vector is confirmed. Meanwhile, the on-site Fox containment team arrives and starts their continuous support of the client in advising about mitigation and containment and even how to re-start some business processes in a secure way. This secure way is devised by looking at the techniques used by the attacker that were found by the investigation team and making sure that this process is not sabotaged or disturbed by the attacker.
Wrapping up
Incident response commonly requires the combination of many different artifact sources for triaging purposes. With Dissect, we were able to provide the client with a picture of the incident using the data sources from around 100 systems, while using two host investigators, within 72 hours. During the remainder of the incident, this number reached around 250 systems, but still only two host investigators were involved.
By combining the Dissect framework with the Splunk platform, we were able to create a complete timeline of the path the attacker took within the network, as well as all the pivots and tool sets that were used. With this information, we quickly and accurately answered the questions the client had regarding the scope of the compromise, as well as visually showed the scope. Dissect and the Splunk platform enabled us to effectively analyze historical artifacts. Combining these with an EDR agent, such as Carbon Black, gave us the ability to simultaneously monitor the environment in real-time, with both sides feeding each other with indicators to get more insights into the environment and the attack.
Next steps
You can contact Fox-IT if you'd like more information on their incident response processes, or on how to integrate Dissect and the Splunk platform. Fox-IT is an IT-Security company and dedicated cyber security expert creating special security, intelligence, and forensics solutions.
The user- and community-generated information, content, data, text, graphics, images, videos, documents and other materials made available on Splunk Lantern is Community Content as provided in the terms and conditions of the Splunk Website Terms of Use, and it should not be implied that Splunk warrants, recommends, endorses or approves of any of the Community Content, nor is Splunk responsible for the availability or accuracy of such. Splunk specifically disclaims any liability and any actions resulting from your use of any information provided on Splunk Lantern.