Skip to main content

Splunk Lantern turned 5 on May 28th. Thank you for being one of our 750,000 annual users!
Click here to join our Slack channel to tell us what you love about the site or what content you'd like to see more of.
Lantern Home

 

Splunk Lantern

Upgrading to Enterprise Security 8.0.x - Walkthrough and validation

  1. Last updated
  2. Save as PDF

 

This article describes validation steps to take after running an upgrade to Splunk Enterprise Security 8.0.x. These steps will help you ensure that the upgrade completed successfully and your deployment is functioning as expected.

This article is part of a comprehensive guide to help you upgrade or migrate pre-8.0.x Splunk Enterprise Security deployments to Splunk Enterprise Security 8.0.x. If you do not feel comfortable completing these steps on your own and would prefer assistance in completing the upgrade, contact our Professional Services experts.

General system state and health

Validate the following, depending on your deployment type.

Stand alone search head

  • The Splunk Enterprise Security (ES) upgrade completed successfully.
  • The supporting and domain add-ons are enabled and on the same version (8.0.x) as ES.
  • The Mission Control app is installed and enabled.
  • All indexes included with ES 8.0.x are now deployed to indexers and available for search.
  • All default ES dashboards can be loaded without any errors.
    • Any panels not populating should be validated with your available data.
    • If data does not exist for a dashboard or the panel, hide the dashboard from view.
  • If your incident review settings were customized before the upgrade, check that existing notable events were not negatively affected or reset by the upgrade. Notable events should retain their current state, status, and associated metadata.
  • Check errors in the essinstaller logs.
  • Check errors inindex=”_internal”. Compare them to any errors and warnings from before the upgrade.
  • Check there are no messages that indicate a failure to migrate ES detections or other components. A failed ES detection state can be verified with the following search: 
    | rest /services/messages | table message
  • Chech errors from modinputs. A modular input error state can be verified with the following search: 
    index=_internal sourcetype=splunkd log_level=ERROR component=ModularInputs 
| table _time host source component log_level event_message
| sort -_time
  • Verify that no restart is necessary. This can be done with the following search: 
    index=_internal sourcetype=splunkd log_level=INFO ("restart required" OR "Restart required" OR "changes detected requiring restart" OR "dynamic update failed") 
| table _time host component message
| sort -_time
    Another way to check if a restart is required is by looking for restart-required flags in the KV store or deployment system: 
    |
rest
/
services
/
messages 

|
search
message
=
"restart required"

|
table
title message severity

Search head cluster

The upgrade documentation for search head cluster environments can help you validate the success of your upgrade. Be sure to check the following: 

  • The Splunk Enterprise Security (ES) upgrade completed successfully.
  • The supporting and domain add-ons are enabled and on the same version (8.0.x) as ES.
  • The Mission Control app is installed and enabled.
  • All indexes included with ES 8.0.x are now deployed to indexers and available for search.
  • All default ES dashboards can be loaded without any errors.
    • Any panels not populating should be validated with your available data.
    • If data does not exist for a dashboard or the panel, hide the dashboard from view.
  • Check data changes made in IR settings, notables don't regress.
  • Check errors in the essinstaller logs.
  • Check errors inindex=”_internal”. Compare them to any errors and warnings from before the upgrade.
  • Check errors from modinputs. A modular input error state can be verified with the following search: 
    index=_internal sourcetype=splunkd log_level=ERROR component=ModularInputs 
| table _time host source component log_level event_message
| sort -_time
  • Validate that no restart is necessary. This can be done with the following search: 
    index=_internal sourcetype=splunkd log_level=INFO ("restart required" OR "Restart required" OR "changes detected requiring restart" OR "dynamic update failed") 
| table _time host component message
| sort -_time
    Another way to check if a restart is required is by looking for restart-required flags in the KV store or deployment system: 
    |
rest
/
services
/
messages 

|
search
message
=
"restart required"

|
table
title message severity

Splunk Enterprise Security navigation

The upgrade inherits any configuration changes and files saved in the app /local and /lookups paths.

  1. The upgrade maintains local changes to the menu navigation.
  2. After the upgrade, configuration changes inherited through the upgrade process might affect or override new settings.
  3. Use the Configuration Health dashboard to review configuration settings that might conflict with new configurations. For more information, see ES Configuration Health.
  4. If you customized your navigation bar in previous versions of ES, you need to reset it back to default in order to see the new navigation bar pages for version 8.0.x. Follow the instructions here. 

Detection versioning

By default detection versioning should be turned off. To validate this status:

  1. Verify that index=cms_main has been created.
  2. Navigate to Configure > General Settings.
  3. Verify that Detection Versioning is turned off post-upgrade by default.
  4. Navigate to Security Content > Content Management and verify that the Version column is empty for all event-based detections.
  5. Click the three dot action menu on the right. Validate there is no "Version History" option available.
  6. Open any event-based detection (out-of-the-box or custom) and verify there is no right-side panel in the Detection Editor.

If you enabled detection versioning, validate that it is enabled and working properly by doing the following:

  1. Verify that index=cms_main has been created.
  2. Navigate to Configure > General Settings.
  3. Verify that detection versioning is enabled. It should say “Versioning is turned on for detections”.
  4. Navigate to Security Content > Content Management and verify the version column for all event-based detections (EBD) and finding-based detections (FBD) is populated with a starting version of ‘1.1’.
  5. Click the three dot action menu to the right of any EBD or FBD. Verify there is a "Version History" option.
  6. Open any out-of-the-box or custom detection, either an EBD or FBD is fine, and verify that there is a right-side panel in the Detection Editor and that it contains version information.

Notables and findings

Validate that notables have been migrated to findings by doing the following:

  1. Validate that index=notable is searchable.
  2. Validate that notables generated prior to the upgrade are now visible in the Mission Control (Analyst Queue) view.
  3. Navigate to Content management, and validate that there are no references to notables in filters at top of page.
  4. Open an event-based detection or finding-based detection in the Detection Editor and review the new features. Validate you now have a “Finding output type” section at the top. Anything you configured to generate notables prior to the upgrade should now have a finding output type of "Finding".
    1. There are a number of out-of-the-box detections that created notables in previous versions of ES that are now configured to only generate a risk event (intermediate finding). If you had enabled these before the upgrade, they should not change (should still generate notables), but you will want to validate these are configured as expected
    2. You can spot-check these using Content Management, or you can run a search against the appropriate REST endpoint using the following search to verify that the notable action is still enabled: | rest /services/saved/searches | search action.notable=1 disabled=0 | fields title action.notable
      . REST endpoints have not yet been fully updated to reflect new language shipped with ES 8.x.
  5. Review key dashboards such as Security Posture and Executive Summary to ensure that these now have panels that reference “Findings”. There should be no reference to notables.

Event-based detections

Select a handful of enabled out-of-the-box and custom detections to use for the following post-upgrade verification steps:

  • Out-of-the-box default and custom correlation searches are converted to event-based detections. You can validate this in Content Management. You should see “Event-Based Detection” on the right of the page.
  • For any previous risk rules (those that only generate risk events, not a notable), the "Finding Output Type" is designated as "Intermediate Finding".
  • For any previous detections that were configured to send a notable, the "Finding Output Type" is designated as "Finding".
  • Event-based detections (EBDs) load properly in Mission Control (Analyst Queue).
  • Detection changes made before the upgrade, including status, description, schedule, search, risk, adaptive response configured, and other parameters, are preserved.
  • Out-of-the-box EBDs can be edited and saved. Test cloning functionality from both the EBD editor and the Content Management page.
  • In the backend for existing searches, action.correlationsearch.detection_type = ebd is set.
  • Any previously created notable event annotations, dispositions, assignments, comments, etc. (anything stored in the KV store) are preserved.
  • New findings continue to be created for out-of-the-box and custom correlation searches.
    • New findings can be created when events match the search.
    • No errors in index=_internal for the scheduled search.

Other (custom) detections

Choose two custom correlation searches with no risk or notable adaptive response actions (ARA) configured. It's fine if they have other ARAs configured, like email.

  1. Open the custom detection and ensure it opens in the event-based detection (EBD) editor.
  2. Verify that Intermediate Finding is the default finding output.
  3. Verify that the adaptive response action (not risk action or notable action) is triggered before risk is configured in the new detection editor.

    In Splunk Enterprise Security 8.0.x, the components within the detection rules run in a specific sequence. If an adaptive response action (such as sending an email, running a script, or adding a notable event) is triggered before risk is assigned, the risk-based action won’t get processed yet.

  4. Edit the detection to populate the risk section required fields, like entity and risk score. Assigning risk is mandatory for any new EBD. After you do this and save, new risk events should get created from the detection.

Previous step

  • Written by Randy Trobock and Ted Skinner
  • Professional Services at Splunk