Skip to main content
Lantern Home

 

Splunk Lantern

Masking IP addresses from a specific range

  1. Last updated
  2. Save as PDF

 

Your web team has asked you to mask IP addresses from your internal 10.x.x.x range in your web server data.

Solution

There are multiple ways of achieving this IP masking use case with SPL2 in Splunk Edge Processor, depending on how flexible you want your pipeline to be. Let’s look at two possible methods.

Method 1: Use eval replace

This option uses a simple eval statement to replace the IP address using a regular expression. We will use access_combined logs throughout this example.

  1. Go to the Pipelines page, click New pipeline, select Blank Pipeline, and click Next.
  2. On the pipeline Partition menu click Partition by sourcetype. Click the Value setting and select access_combined, click Apply, and then click Next.
  3. In Add Sample Data click Skip.
  4. In Select a Data Destination, choose where you want this pipeline to send its data. For this example, expand the Default Destination, and click Done
  5. Click the Preview "Play" to see the raw example events.
  6. In the pipeline pane, under Actions, click the + button and select Mask values in _raw…
  7. In the Matching regular expression field, paste the following regular expression: 
    ^10\.(?:[0-9]{1,3}\.){2}[0-9]{1,3}
  8. In the Replace with field, enter [IP_REDACTED].
  9. Leave Match case selected and click Apply. An eval command is automatically added to your SPL2 statement, which should now look like this: 
    $pipeline =
| from $source
| eval _raw=replace(_raw, /^10\.(?:[0-9]{1,3}\.){2}[0-9]{1,3}/, "[IP_REDACTED]")
| into $destination;
  10. (Optional) Add a custom indexed field to your events to help you identify them in Splunk Cloud Platform by adding an eval command to your pipeline. To create a field called "redacted" with the value set to "true": 
    $pipeline =
| from $source
| eval _raw=replace(_raw, /^10\.(?:[0-9]{1,3}\.){2}[0-9]{1,3}/, "[IP_REDACTED]")
| eval redacted="true" 
| into $destination;
  11. Test your masking rule by clicking the blue Preview pipeline button in the top right corner of the screen. You should see a preview of your events in the center of the screen showing the IP addresses redacted.
  12. (Optional) Verify that your custom field appears by clicking the Data tab and selecting the participant check box under the FIELDS section to show the participant field in the preview pane.
  13. Return to the Actions tab and click the +, and select Target index.
  14. Select Specify Index for All Event and enter web for the index name. Click Apply.
  15. In the top right corner of the screen, click Save pipeline.
  16. Give your pipeline a suitable name, such as access_combined_ip_mask_<yourName> or something similar.
  17. Click Yes, then click Apply. Then select the Edge Processors to deploy the new pipeline to and click Save.
  18. Log in to Splunk Cloud Platform and open the Search & Reporting app.
  19. Run the following search over the last 15 minutes and verify that you now see the redacted events: 
    index=web "[IP_REDACTED]"

Method 2: Use rex and cidrmatch

This option leverages the power of SPL2 to build a more complex pipeline that will give you more to adapt your IP masking in future.

  1. Go to the Pipelines page, click New pipeline, select Blank Pipeline, and click Next.
  2. On the pipeline Partition menu, click Partition by sourcetype. Click the Value setting and select access_combined, click Apply, and then click Next.
  3. In Add Sample Data, click Skip.
  4. In Select a Data Destination, choose where you want this pipeline to send its data. For this example, expand the Default Destination, and click Done
  5. Click the Preview "Play" to see the raw example events.
  6. Extract IP addresses from your web events into a new field called "ip" by adding the following rex command to your pipeline after | from $source. 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/

    Splunk Edge Processor has been updated to now use PCRE regular expressions instead of RE2, which was available in earlier releases.

  7. Check that your new field extracts correctly by clicking the blue Preview pipeline button in the top right corner of the screen and selecting the ip field on the Data tab. You should see the ip column displayed in the preview pane.
  8. To check whether the value of ip is in the 10.x.x.x range, add an eval statement and leverage the cidrmatch function to check against the 10.0.0.0/8 CIDR range. If the value matches, the range ip_type will be set to internal. Otherwise, it will be set to external. 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/
| eval ip_type=if(cidrmatch("10.0.0.0/8",ip), "internal", "external")
  9. Add a where statement to filter by only internal IPs, that is, only those that match the 10.0.0.0/8 CIDR range. 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/
| eval ip_type=if(cidrmatch("10.0.0.0/8",ip), "internal", "external")
| where ip_type="internal"
  10. Now that you’re filtering by only the events with IPs based on the 10.0.0.0/8 range, add an eval statement and use the replace function to mask/redact the values in the ip field: 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/
| eval ip_type=if(cidrmatch("10.0.0.0/8",ip), "internal", "external")
| where ip_type="internal"
| eval _raw=replace(_raw, ip, "[IP_REDACTED]")
  11. Click the blue Preview pipeline button in the top right corner of the screen and check that the IPs are redacted from your events.
  12. Now that you’ve masked IPs from _raw, you need to remove the ‘ip’ field you created as that field contains unmasked IP addresses. 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/
| eval ip_type=if(cidrmatch("10.0.0.0/8",ip), "internal", "external")
| where ip_type="internal"
| eval _raw=replace(_raw, ip, "[IP_REDACTED]")
| fields -ip
  13. (Optional) Finally, add a "redacted" field to your events to help you identify them in Splunk Cloud Platform: 
    | from $source
| rex field=_raw /(?P<ip>(([0-9]{1,3}\.){3}[0-9]{1,3}))/
| eval ip_type=if(cidrmatch("10.0.0.0/8",ip), "internal", "external")
| where ip_type="internal"
| eval _raw=replace(_raw, ip, "[IP_REDACTED]")
| fields -ip
| eval redacted="true"
| into $destination;
  14. Test your masking rule by clicking the blue Preview pipeline button in the top right corner of the screen. You should see a preview of your events in the center of the screen showing the IP addresses redacted.
  15. Return to the Actions tab and click the +, and select Target index. Select Specify Index for All Event and enter web for the index name. Click Apply.
  16. In the top right corner of the screen, click Save.
  17. Give your pipeline a suitable name, such as access_combined_ip_cidr_mask_<yourName> or something similar.
  18. Click Yes, then click Apply. Then select the Edge Processors to deploy the new pipeline to and click Save.
  19. Log in to Splunk Cloud Platform and open the Search & Reporting app.
  20. Run the following search over the last 15 minutes and verify that you now see the redacted events: 
      index=web "[IP_REDACTED]"

Next steps

These additional Splunk resources might help you understand and implement this use case: