Skip to main content
Lantern Home

 

Splunk Lantern

Detecting and fixing CIM non-compliance with Self-Healing Pipelines

  1. Last updated
  2. Save as PDF

Maintaining data integrity across security and analytics pipelines is an ongoing challenge. Data sources evolve, formats change, and field mappings drift over time. These changes can cause dashboards and alerts across the Splunk platform, including correlation searches in Splunk Enterprise Security (ES), to produce incomplete results or fail without an obvious warning. The usual cause is a Technology Add-on (TA) that doesn't extract, alias, or normalize fields the way the Common Information Model (CIM) expects. Finding the mismatch means comparing live events against data model definitions, and fixing it means hand-editing props.conf and transforms.conf, which takes Search Processing Language (SPL) and configuration expertise that many teams don't have time for.

The Self-Healing Pipelines feature in the Ingest Monitoring app closes that loop. You create alerts that continuously check your source types against CIM data models. When an alert triggers, an AI-assisted workflow analyzes the add-on configuration, proposes fixes, lets you review, edit, and preview them, and packages the result as an add-on overlay that you deploy to your environment. The overlay layers on top of your existing add-on, so the original add-on is never modified. This article walks through that process end to end.

Self-Healing Pipelines was released as part of the autonomous data pipelines innovations, which help teams get the right data to the right place for the right purpose, with less manual toil and more confidence in the quality of data. It works alongside Guided Onboarding with Auto-schema: use Guided Onboarding with Auto-schema to onboard data faster, and use Self-Healing Pipelines to monitor and maintain its quality.

As one Splunk platform administrator at a higher education institution said:

"Getting data into the Splunk platform is easy, making use of that data is hard. Ensuring your data feeds don't change or have issues is complicated. In a world of interconnected systems and services trying to ensure everything gets updated and nothing gets broken is difficult. With the CIM compliance monitoring it was easy to identify incorrect or missing data. The app automatically tries to provide the needed fixes, this saves time as an administrator making sure that our security data stays accurate and available."

If you're new to CIM normalization, see Normalizing values to a common field name with the Common Information Model. For the full feature reference, see Self-healing data management pipelines in the Ingest Monitoring documentation.

Prerequisites

Before you start, confirm the following in your Splunk Cloud Platform environment:

  • Admin permissions: Sign in with an sc_admin or admin account in the environment where Ingest Monitoring is installed.
  • Ingest Monitoring app, version 1.4.0 or later: The editing, preview, and AI Assistant capabilities described in this article require the latest 1.4 release.
  • Guided setup: The first time you open the Ingest Monitoring app, complete the guided setup and select the indexes to monitor. For details, see Get started.
  • Splunk Common Information Model Add-on: The feature reads CIM data model definitions from this add-on.
  • Data model acceleration: Turn on acceleration for every CIM data model you want to monitor. To check, go to Settings > Data Models, find the data model, and select Edit > Edit Acceleration if acceleration is off. The data model must be shared, and your role needs the accelerate_datamodel capability. For details, see Manage data models. The feature reads the acceleration summary to discover which source types are mapped to a model. If acceleration is off, the source type list stays empty and you can't create an alert for that model.

Self-Healing Pipelines is available on primary search heads and on ES search heads, in six Amazon Web Services (AWS) regions: us-east-1, us-west-2, eu-central-1, eu-west-1, eu-west-3, and eu-south-1. For details, see Feature availability and scope. Alert rules are stored locally on the search head where you create them, so rules created on an ES search head don't appear on the primary search head, and vice versa.

Data required

  • The source types you want to monitor: Each alert rule watches one source type against one CIM data model dataset. To monitor several source types, create one alert rule for each.
  • The CIM data model and dataset each source type should comply with: For example, Authentication.Authentication for sign-in events or Network_Traffic.All_Traffic for firewall and flow data.

How to use Splunk software for this use case

The feature measures compliance in two ways:

  • Field coverage is the percentage of events with a value for a CIM field. If 950 of 1,000 Authentication events have a populated src field, coverage is 95%.
  • CIM value validation is the percentage of a field's values that match what the CIM expects. For example, action must be success, failure, pending, or error.

Create a CIM non-compliance alert

  1. In the Ingest Monitoring app, go to Alerts, click Create alert, and select the CIM non-compliance template.

    clipboard_a8d596cd-78ae-4e02-9ac6-5136e1031b80.png

  2. Name the alert, then choose the data model and dataset and the source type. The source type list shows only source types mapped to the selected model.
  3. Review the CIM fields. All fields are selected by default. Deselect any fields you don't want to monitor.

    clipboard_56dba783-51c1-4fb2-9e75-a44db2c13bee.png

  4. Set the field coverage and CIM value compliance thresholds, the schedule, and the Throttle, which suppresses duplicate alerts within a time window. Both thresholds have a recommended value of 80%.

    clipboard_2387a0ac-a84f-4364-ad91-ca89adac65ad.png

  5. Select a severity and click Create.

For every setting, see Create a CIM non-compliance alert.

Open the triggered alert group

When an alert rule triggers, open Alerts > Triggered alerts and select the alert to see its details.

The Triggered alerts tab lists each triggered alert rule with its severity, the number of alerts, and its AI remediation status. The details panel includes an Investigate button.

clipboard_3490b432-9ae0-40fc-86a4-7cd1386e67a6.png

Alerts from the same rule that haven't been investigated yet are grouped together, so you can fix them in one pass instead of one at a time. Click Investigate to see the trigger time of each alert in the group, sample events, and the non-compliant values for each monitored field.

The Investigation page shows when each alert in the group triggered, how many fields are affected, and sample events with the non-compliant values highlighted.

clipboard_58805398-70e3-43eb-ac2c-150aaadb84f1.png

When you're ready, click Start remediation. Every alert in the group moves to Remediation in progress status. When remediation finishes, the group's status changes to Done. Alerts in a Done group stay visible until the rule's expiry. For the full procedure, see Investigate and remediate CIM non-compliance.

Review the AI analysis

The AI analysis starts automatically. The app collects the alert results (which fields failed, why, and their coverage percentages), gathers sample events that show the actual field values, locates the relevant stanzas in props.conf and transforms.conf, and submits this context to the AI analysis service.

A progress indicator shows the AI's reasoning as it works. When the analysis completes, you see a summary of the non-compliant fields and why they failed, which fields can and can't be fixed, and a confidence level for each fixable field. Review the analysis, then click Continue.

The analysis summarizes each non-compliant field, its issue type, and the confidence level of the proposed fix.

clipboard_ca9315ce-162e-4abb-8b66-b1aab05de240.png

Review, edit, and preview the proposed fixes

In the Recommended remediations step, select the props.conf and transforms.conf tabs to switch between files. A side-by-side editor shows your current configuration on the left and the proposed fix on the right. Expand AI reasoning for suggested recommendations to see why each change was made.

The field extraction preview table shows which monitored fields are compliant with the fix applied, before you install anything.

In an upcoming release, you will be able to edit proposed fixes using the code editor:

  1. Edit the proposed configuration in the right-hand editor.
  2. Click Apply changes. The app checks your edits and refreshes the preview.
  3. When the monitored fields are compliant, click Continue.

If you can't continue, click the warning icon to see why. Fix the listed issues, or click Ask AI Assistant to have AI correct them.

clipboard_ec992d81-0965-40f2-a242-32b48f5b1ee9.png

Refine the fixes with the AI Assistant

In an upcoming release, you will be able to use the AI Assistant to explain or update proposed fixes.

Click Open chat to ask the AI Assistant to explain or change the proposed fixes, or select a suggested prompt such as "What should I review before applying these changes?" When the assistant changes the fixes, the editor and preview refresh.

clipboard_46774786-9194-48d2-b6bd-b6c4860d2087.png

Download and install the add-on overlay

In Download artifacts, preview the generated files and click Download TA package to get the overlay as a .tar.gz file.

The Download artifacts step lists the generated files and previews the selected file.

clipboard_35350a83-dff3-4927-9ff8-0b20cbf4fd61.png

To deploy on Splunk Cloud Platform:

  1. Install the overlay as a private app on the same search head where you ran detection and remediation. An overlay generated on one search head doesn't fix compliance problems on another.
  2. Install the overlay before you run remediation again for the same alert rules, so later fixes build on it.
  3. Check that the app appears on the Uploaded Apps page. Its name and version appear after it passes all Splunk Cloud Platform vetting checks. For upload details, see Deploy and manage private apps.
  4. Click Done to return to the Triggered alerts list.

Things to keep in mind

  • Review before you deploy: The AI sees only props.conf and transforms.conf, not other extraction methods such as automatic lookups. The app validates only EXTRACT, EVAL, and FIELDALIAS in props.conf and REGEX and FIELDS in transforms.conf.
  • Limits: Detection covers only data models and fields defined in the CIM, so custom data models and custom fields aren't supported. Each alert rule filters by source type only, not by source, host, or event type. Thresholds are static, so some environments might see false positives. For details, see Limitations of Self-Healing Pipelines.
  • Data transfer: The AI analysis can involve cross-region inferencing, which might transfer your data to data centers outside your selected storage region.
  • Vendor add-on upgrades: If a vendor update fixes CIM issues natively, your overlay can override the updated files. Check the release notes. Remove the overlay only if every field it patches is now fixed.

Next steps

After the overlay is installed, watch for the alert to stop triggering, or run a tstats search against the data model to confirm the fields now return data. Repeat this workflow for each alert rule.

You might also find the following resources useful: