Using Splunk DataSense Navigator

Solution

The Splunk DataSense Navigator (DSNav) addresses this problem, using a data-centric methodology that allows you to explore potential use cases within your environment, using your existing data resources. DSNav integrates the analysis of use case and data preparation requirements right from the start. This organized approach reduces delays and eliminates the need for time-consuming rework.

• DSNav is designed for easy access and speed. For example, it automates the identification of use cases and data models, making setup and management processes more efficient. It also adapts to existing environments, broadening the range of applicable use cases and adapting to your needs.
• DSNav introduces a new standard with event-specific subsampling techniques. Unlike other tools that analyze large datasets in aggregate, the DataSense Navigator looks at each event individually. By using key fields and statistical methods to manage data size, this approach ensures an accurate understanding of whether specific information is present or absent in the data, doing so more efficiently and without the typical computational challenges associated with such precision.
• Finally, DSNav can help with all the following scenarios:
  • Use case development and data onboarding. Ensures efficient validation of new data sources against all or selected use cases, confirming their compatibility and meeting necessary conditions for effective use case implementation.
  • Efficient use case discovery. Accelerates and refines use case discovery, providing recommendations for use case activations based on existing data quality. It also aids in future use case planning, especially when partial prerequisites are met, potentially reducing the time to value compared to use cases with no prerequisites fulfilled.
  • Data quality validation. Examines the presence of events within data models that contribute to Splunk Enterprise Security (ES) use cases, and facilitates a thorough examination of event presence within data models crucial for ES use cases, providing a clear picture of data quality.
  • Priority setting for data quality improvement. Identifies gaps in field extractions or improper configurations from the source, helping to prioritize data quality improvement tasks. For example, if certain use cases are prioritized, any data or tag attribution gaps will be highlighted, aiding in precise effort planning. It also allows tracking modifications at the field/tag level rather than just the data source level, providing a clearer picture of effort needed for different data sources.
  • Post-configuration push impact analysis. Assesses the impact of a configuration push on the use case coverage potential of a particular data source. Variations in coverage post-deployment, assuming consistent data sampling intervals, can indicate success or the need for additional adjustments.
  • Facilitation of a structured workflow to users of different skill levels. Provides a structured yet interactive workflow design which accommodates differently-skilled users, guiding them to similar conclusions on the gaps between onboarded data and target use cases.

Table of contents

• Getting started
• Installation and configuration
• Best practices
• Known issues
• FAQ
• Next steps

Getting started

Installation and configuration

Best practices

• To enhance subsampling speed, it's recommended to narrow down the search time range before adjusting the sampling precision. A practical initial time range is about 1-4 hours, and it's only recommended to extend this if the dataset volume is relatively low, such as less than 100k events in the non-sampled reference.
• When the option to select multiple indexes and source types is available, it's usually better to review only a few source types at a time, ideally those managed by a single add-on. This approach means you can more easily draw inferences from the tool regarding any necessary modifications to the specific TA in question.
• The result calculations in Stage 4, derived from KV store data generated in Stages 1 to 3, allow concurrent analysis in Stage 4 while the context in the previous stages is being updated with new information. For instance, if Linux logs were initially analyzed at a 1-hour interval at the minimal configuration setting, this would expedite the data sampling results for further analysis in Stage 4. At the same time, Stage 3 could be regenerated, either with the same data at a longer interval, such as 4 hours, with a more precise sampling setting such as moderate or advanced, or with a completely different data source. This parallel processing minimizes waiting time for outputs as data generation and analysis steps for each phase can occur simultaneously.
• It's recommended to switch the update KV store option from Update to View after the confirmation message appears. Although neglecting this step usually doesn't affect normal operations, a tool refresh could trigger the regeneration of some KV store contexts which might hinder analysis. This action also helps in reducing impacts on search concurrency.

Known issues

• The subsampling stage tends to take a longer time for large event sizes. This delay is more noticeable with JSON events (like those from Crowdstrike) and XML events (like those from Windows).
  • When working with JSON or XML files, the results returned might be on the lower side under moderate sampling precision settings. This is due to the utilization of "punct" as an indicator of event variation. Given how the punct field is defined at index time, it might not capture all punctuation fields present in a particular event, especially if the event is large as seen with JSON and XML events. Even though the number of subsampled events is modest, tests with live customer examples show that it rarely affects the accuracy of comparative matches. However, for those seeking higher precision, advanced or comprehensive subsampling techniques could alleviate this entirely.
• If the app is set to public and is accessed by multiple users at the same time, the KV stores might get overwritten. This issue stems from the current design which does not support concurrent multi-user interactions on the application. A temporary solution is to use the app in a private mode for additional users who want their own separate instance. This way, the data in the KV store remains intact as interactions from one user instance do not affect the data in another user's KV store. Preliminary tests show that switching to a private version doesn’t hamper the app’s functionality, although more comprehensive testing of concurrent usage is still taking place.

• You might also notice that some events show a full or partial match, but not all required tags for a specific data model appear in the drilldown search. This is by design. The tool identifies a data model association if at least one of the tag requirements of that data model is met. Here are some underlying reasons for why a partial approach for tags was ultimately considered:
  • Some tags, like "Endpoint", are associated with multiple data models (for example, Change and Endpoint) as well as hierarchical data models within a main data model.
  • While tags referenced by the CIM data models are also used by other vendors when TAs are being created, there's generally overlap in how tag definitions get assigned. However, inconsistencies can arise between vendors, especially if tag attributions are defined by a custom user configuration.
  • Certain data models use two to three tags, all of which must be qualified. This is usually where perceived gaps in the DSNavs identification may be observed.
  • Data models like Endpoint and Performance don’t have a base tag, but rather have different tags for different variations.

  The main aim of DSNav is to ensure accurate field qualifications. The tag qualifiers are intentionally broadened so as not to be overly strict, as tags are often defined based on the existence of certain fields. If the tool identifies events with matches or partial matches but some tags are missing, rectifying this is relatively straightforward given the flexibility in how tag definitions are defined.

  Opting not to enforce the existence of all tags also serves to enhance performance. While filtering out base tags in Stage 2 is straightforward, requiring all tags would increase computation times in Stages 3 and 4, especially while preserving the functionality of showing both partial and full matches. Ultimately, it was found that showing more possible matches by reducing the qualifying criteria for data model attributions to partial instead of full, added value. The aim is to provide a broader view, allowing for more flexible and efficient identification and correction where needed.

FAQ

What is the impact on performance when this tool is actively used in an environment?

• The tool primarily relies on four base searches for most of its functions when active. All searches are performant with the exception of the subsampling stage, which tends to use more resources, especially when handling a large number of standard events (around 10 million) or when the individual event sizes are large (as in the case with XML/JSON events and usually noticeable around 1 million events). Importantly, the tool does not run any recurring search processes when it's not in use, ensuring no performance impact when not in active usage.

Are there certain situations where the tool might provide inaccurate results?

• The tool identifies search conditions using a set of regex conditions executed in Stage 1. In some cases, a field referenced by a correlation search might pertain to a calculated field that isn't available when analyzing the indexed event. This scenario may result in a match being labeled as partial when it could have been a full match. While such instances seem to be rare based on preliminary customer testing and are often identified quickly, challenges might arise if there's an attempt to modify the data onboarding of the event to include the calculated field. This could be problematic without understanding that the calculated field is only computed after the index is added to the data model.

Why does this tool only look at indexed events instead of the data model directly?

• The tool is designed to be deployed after data has been ingested into the Splunk environment but before the index housing that data is allow listed as part of the Data Model Set Up process. This sequence is deliberate for several reasons:
  • Data validation. By positioning the tool at this stage, you can validate that the ingested data conforms to the requirements of your specific use cases.
  • Data fidelity. The tool helps to ensure that only high-fidelity data, which is well onboarded, is promoted to the accelerated data models. This avoids the risk of diluting the quality of the data models with low-fidelity or poorly onboarded data.
• By aligning the tool in this manner, you preserve the efficiency and integrity of the data models.

Is it possible to use this tool in environments other than ES (Enterprise Security)?

• The tool's operability isn't confined to ES environments. As long as the environment has CIM data models and some searches formatted in the correlation search format, the tool should function as intended. While ES isn't a mandatory requirement, it naturally meets these prerequisites, making it a conducive environment for the tool.
• Regarding the tool's applicability in other environments like Splunk Enterprise, Splunk ITSI, etc., there are plans to extend compatibility in future iterations of the tool (likely around Q2 of 2024). Currently, the scope is more focused since non-DMA driven searches usually don't reference a large subset of data sources, minimizing the inefficacy often associated with larger data sets. This narrower scope simplifies the validation of prerequisites for a particular use case, as opposed to use cases referencing a data model which might entail a broader set of data sources.
• ES was chosen for initial testing of this solution framework due to its breadth, complexity, and the intensive nature of conforming data quality standards to both the CIM model and the use case prerequisites. On the other hand, non-ES environments typically deal with a smaller subset of datasets and only the use case prerequisites, which often results in fewer inaccuracies and challenges. This distinction became clearer through customer environment evaluations, revealing that even customers with lower security maturity levels usually face no issues creating searches that merely reference indexes and source types.

Why haven't more search performance monitoring aspects been incorporated into the tool to track elements like search concurrency, or additional panels to provide more insight on ES health?

• Currently, there are other tools available that cater to these monitoring needs, along with several initiatives in development slated for future release. While the effectiveness of existing tools and the advantages of an interactive approach (similar to what DSNav employs) could be useful to other suggested applications, the core design philosophy of this tool centers around simplification. Incorporating functions that don't directly align with the core objective of identifying use cases from a sample dataset could potentially distract end users, and might affect adoption especially among users with less proficiency in ES related processes.
• To elaborate further, the perceived gap between an ideal tool monitoring aspects like search concurrency and the current popular tools addressing this need, might be seen as relatively small. On the other hand, the gap between a tool proficient in identifying use case prerequisites with high accuracy and what's available prior to the advent of the DataSense Navigator is considerably larger, hence justifying the development efforts.

Why is there a discrepancy between the number of events displayed in the drilldown step of Stage 4 and the number of matches identified?

• The discrepancy stems from the different processing stages. The sample matches in Stage 4 aren't subjected to the same subsampling reduction and filtering as in Stage 3. Stage 4 aims to offer a wider perspective on the potential matches, thus enabling a quicker search process.

How can I ascertain the tool's proper functionality?

• The event drilldown feature of the tool is designed to fulfill two main purposes. It enables viewing of possible events that either meet or partially meet the requirements of a potential use case, and in this scenario, it helps validate the accuracy of match identification. If clicking the drilldown search returns no results, it could suggest issues encountered during the sampling generation phase. Conducting a visual examination of the drilldown events to check for the presence of field matches can further confirm whether the match conditions have been correctly identified.
• It's important to note that there might be instances where no contributing events are identified during this phase, especially if such events are rare. The match findings in Stage 4 are obtained from the time interval during which the subsample was created. While the data drilldown utilizes this time interval, there could be a time lapse from when the subsample was taken to when the drilldown was conducted. There might be certain time frames at the peripheries outside the overlap between Stage 3 and the Stage 4 drilldown. If the contributing events fall outside this overlap, they may not appear. This issue can be resolved by either taking a new sample and conducting the drilldown immediately afterward or by opening the drilldown in a separate window and extending the search range to encompass a larger data subset.

Next steps

These resources might help you understand and implement this guidance:

• Splunkbase: DataSense Navigator