Upgrading to Enterprise Security 8.0.x - Configuration and customization

Pre-upgrade

1. Create an investigation archive index.
   1. Go to Settings > Indexes > New Index.
   2. Make the index name investigation_archive or a different name according to your naming convention.
   3. Set the size and paths as needed.
2. Collect data from KV stores related to investigations. Run each of these searches once:
   1. | rest splunk_server=local count=0 /services/storage/investigation/investigation
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation
   2. | rest splunk_server=local count=0 /services/storage/investigation/investigation_attachment
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation_attachment
   3. | rest splunk_server=local count=0 /services/storage/investigation/investigation_event
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation_event
   4. | rest splunk_server=local count=0 /services/storage/investigation/investigation_lead
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation_lead
   5. | rest splunk_server=local count=0 /services/storage/investigation/investigative_canvas
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation_canvas
   6. | rest splunk_server=local count=0 /services/storage/investigation/investigative_canvas_entries
| eval _time=create_time
| collect index=investigation_archive sourcetype=investigation_canvas_entries
3. Verify the collected data is present in the investigation_archive index using the following searches. 
   1. In this one, the result count should match the number of investigations you have. index=investigation_archive sourcetype=investigation | stats dc(id)
   2. If this one returns any count > 1 for any unique ID, you have ingested a duplicate investigation or data. You might have accidentally run the commands in step 2 twice. index=investigation_archive | stats count by id | sort -count

      If you have more than 10K results, your results might not display potential duplicates. In this case, look at the raw events to see if the ID field has any values with more than 1 to verify for duplicates.

4. Export a list of file attachment IDs for all investigations as a CSV using the following search: index=investigation_archive sourcetype=investigation_attachment | table id name | eval ID=replace(id, ".*/", "") | dedup ID
5. Export the file as a CSV using the UI, name it "attachments.csv" (or a different name according to your naming convention), and copy it to the ES search head.

   If you have more than 10K results you can append "| outputcsv attachments.csv" to the end of your query and it will export all results of your CSV file (even if it only displays 10K results in the UI) to $SPLUNK_HOME/var/run/*.csv. This can also save you the effort of getting the CSV file onto the ES search head instead of manually copying it over.

6. Use the following script to export attachments and move them to a long term storage location. You will need to move this script to a system that has access to the ES search head or to the ES search head itself. 

   If you do not leverage attachments, or only have a handful, it might be easier to just download them via the UI inside of ES one at a time and manually move them to a convenient location. In that case, you do not need to follow the remainder of the steps in this procedure for using the script. Move on to Post-migration.

    

   #!/bin/bash
   
   # Adjustable variables
   TARGET_DIR="/home/splunk/investigation_attachments" # Directory to save files
   SPLUNK_HOST="myessearchhead.splunk.com" # Change this to your Splunk host
   TARGET_FILE="/home/splunk/attachments.csv" # CSV file containing the IDs and names exported from above step
   
   # Ensure the target directory exists, create if not
   mkdir -p "$TARGET_DIR"
   
   # Check if the target file exists
   if [ ! -f "$TARGET_FILE" ]; then
   echo "Error: $TARGET_FILE not found!"
   exit 1
   fi
   
   # Loop through each line in the CSV (skip header)
   tail -n +2 "$TARGET_FILE" | while IFS=',' read -r ID NAME; do
   
   # Check if ID and NAME are not empty
   if ; then
   
   # Construct the filename
   FILENAME="${ID}_${NAME}"
   
   # Run the curl command to download the file
   echo "Downloading attachment for ID: $ID -> $FILENAME"
   curl -u "$SPLUNK_USERNAME:$SPLUNK_PASSWORD" -k -o "$TARGET_DIR/$FILENAME" "https://$SPLUNK_HOST:8089/services/s...&download=true"
   else
   echo "Skipping row with missing ID or NAME."
   fi
   done
   
   echo "Download process completed. Files saved in: $TARGET_DIR"
   
   

7. Update variables as needed and run the script.
   1. Set your environmental variables for your Splunk username/password.
      1. export SPLUNK_USERNAME="your_username"
      2. export SPLUNK_PASSWORD="your_password"
   2. Update script variables as needed
      1. Update the target directory to save investigation attachments, if needed.
      2. Update the Splunk host name, if needed.
      3. Update the target CSV file to process, if needed.
   3. Ensure the script has appropriate permissions. chmod +x download_attachments.sh
   4. Run the script: /path/to/script/myscript.sh
   5. Verify a successful export and copy your attachments to a secure location for future access. Use this search to compare the number of files versus unique IDs: index=investigation_archive sourcetype=investigation_attachment
   6. Unset your environmental variables.
      1. unset SPLUNK_USERNAME
      2. unset SPLUNK_PASSWORD

Post-migration

Below is a simple XML dashboard that can be used for querying the imported investigation data. You can install this anywhere that has access to the investigation_archive index:

<form version="1.1" theme="dark">
  <label>Legacy ES  Investigations</label>
  <description>Legacy ES Investigation Data</description>
  <fieldset submitButton="false">
    <input type="time" token="time_token" searchWhenChanged="true">
      <label>Time Range</label>
      <default>
        <earliest>0</earliest>
        <latest></latest>
      </default>
    </input>
    <input type="text" token="investigation_id" searchWhenChanged="true">
      <label>Investigation ID</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
    <input type="text" token="query_token">
      <label>Investigation _raw query</label>
      <default>*</default>
      <initialValue>*</initialValue>
    </input>
  </fieldset>
  <row>
    <panel>
      <title>Number of Investigations</title>
      <single>
        <search>
          <query>index=investigation_archive sourcetype=investigation
| stats dc(id)</query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </single>
    </panel>
  </row>
  <row>
    <panel>
      <title>Investigations</title>
      <table>
        <search>
          <query>index=investigation_archive sourcetype=investigation title="$investigation_id$" "$query_token$" 
| eval create_time=strftime(create_time, "%Y-%m-%dT%H:%M:%S")|eval mod_time=strftime(mod_time, "%Y-%m-%dT%H:%M:%S")
| rex field=_raw "collaborators=\"(?&lt;collaborators_json&gt;\[{.*?}\])\"" 
| eval clean_collaborators_json = replace(collaborators_json, "[\\{\\}\\[\\]\"\\\\]", "")
| eval clean_collaborators_json = replace(clean_collaborators_json, ", write", "_write")
| eval collaborators = split(clean_collaborators_json, ", ")
| rex field=_raw "status=.{15}(?&lt;status_value&gt;\w+)"
| eval status=status_value 
| rename title as id 
| table create_time,creator,status,collaborators,name,description,id,mod_time,_raw
| sort 0 -create_time</query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">cell</option>
        <option name="refresh.display">progressbar</option>
        <drilldown>
          <set token="parent_id">$row.id$</set>
        </drilldown>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <table>
        <title>Investigation Events for $parent_id$</title>
        <search>
          <done>
            <set token="investigation_i">$result.sourcetype$</set>
          </done>
          <query>index=investigation_archive sourcetype=investigation_event parent_id=$parent_id$
| rename title as event_id
| table _time,creator,name,description
| sort -_time</query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row>
    <panel>
      <title>Investigation Attachments for $parent_id$</title>
      <table>
        <title>Investigation Attachments</title>
        <search>
          <query>index=investigation_archive sourcetype=investigation_attachment parent_id IN ($event_id$)
| rename title as attachment_id
| table _time content_type, name, attachment_id
| sort 0 -_time</query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
  <row depends="$foo$">
    <panel>
      <table>
        <title>Event Ids</title>
        <search>
          <done>
            <set token="event_id">$result.id$</set>
          </done>
          <query>index=investigation_archive sourcetype=investigation_event parent_id=$parent_id$ | stats list(title) as id | eval id=mvjoin(id, ",")</query>
          <earliest>$time_token.earliest$</earliest>
          <latest>$time_token.latest$</latest>
        </search>
        <option name="drilldown">none</option>
        <option name="refresh.display">progressbar</option>
      </table>
    </panel>
  </row>
</form>