Skip to main content
Splunk Lantern

New domains accessed by network users

You might need to determine what domains visited by your network users are anomalous when doing the following.

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

Example

You are concerned about employees accidentally accessing malicious domains. You want to create and save a search that will allow you to see what domains employees accessed that they haven't accessed before.

To optimize the search shown below, you should specify an index and a time range. 

Option 1

Use this option if you have not created a baseline of activity in a lookup table. If you have CIM compliant data and populated web data models, you can use Option 3 below instead.

  1. Run the following search:
tag=web url=*
| eval list="mozilla"
| `ut_parse_extended(url,list)`
| stats earliest(_time) AS earliest latest(_time) AS latest BY ut_domain
| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0)
| convert ctime(earliest) ctime(latest)
| where isOutlier=1

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

tag=web 

Search for fields with the "web" tag.

url=*

Return proxy data by searching for results with a value in the URL field.

| eval list="mozilla"

Search the Mozilla catalog for top level domains.

This eval function is required for the next line in the search (ut_parse_extended) to work.

| `ut_parse_extended(url,list)`

Parse the URLs based on the Mozilla top level domain list.

The punctuation in a Splunk macro is always a back tick (`), not a single quote (').

| stats earliest(_time) AS earliest latest(_time) AS latest BY ut_domain

Return the results in a table—grouped by the ut_domain field—that includes columns for the first and last time each domain was seen in the dataset.

| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0)

Create a new column called isOutlier. If the earliest time a domain appears in the dataset is more recent than or equal to 24 hours from when the search was run, return a value of 1 in the isOutlier column. Otherwise, return a value of 0.

Make sure your search time range is set appropriately or all domains might appear to be new. Setting the search to the past 7 days is a good start.

| convert ctime(earliest) ctime(latest)

Convert epoch time for the earliest and latest visits to the domains into a more easily readable format.

| where isOutlier=1

Return only results where the isOutlier value is 1, which indicates a new domain.

Option 2

Use this option if you want to search against a baseline that you previously saved in a lookup table.

  1. Run the following search:
tag=web url=* earliest=-15m
| eval list="mozilla"
| `ut_parse_extended(url,list)`
| stats earliest(_time) AS earliest latest(_time) AS latest BY ut_domain
| inputlookup append=t <name of lookup file ending in .csv>
| stats min(earliest) AS earliest max(latest) AS latest BY ut_domain
| outputlookup <name of lookup file ending in .csv>
| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0)
| convert ctime(earliest) ctime(latest)
| where isOutlier=1

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

tag=web 

Search for fields with the "web" tag.

url=*

Return proxy data by searching for results with a value in the URL field.

earliest=-15m

Return only domains accessed in the last 15 minutes.

| eval list="mozilla"

Search the Mozilla catalog for top level domains.

This eval function is required for the next line in the search (ut_parse_extended) to work.

| `ut_parse_extended(url,list)`

Parse the URLs based on the Mozilla top level domain list.

The punctuation in a Splunk macro is always a back tick (`), not a single quote (').

| stats earliest(_time) AS earliest latest(_time) AS latest BY ut_domain

Return the results in a table—grouped by the ut_domain field—that includes columns for the first and last time each domain was seen in the dataset.

| inputlookup append=t <name of lookup file ending in .csv>

Retrieve the CSV file you previously created and add it to your data set from the last 15 minutes.

| stats min(earliest) AS earliest max(latest) AS latest BY ut_domain

Combine the data from the previous 15 minutes and the baseline domain list, and return the “earliest earliest” and “latest latest” time for each domain in the complete dataset.

| outputlookup <name of lookup file ending in .csv>

Write the updated table to the same CSV lookup file. The latest field may be updated if the domain was previously in the CSV, while new domains previously not in the CSV are added with their earliest and latest seen times.

| eval isOutlier=if(earliest >= relative_time(now(), "-1d@d"), 1, 0)

Create a new column called isOutlier. If the earliest time a domain appears in the dataset is more recent than or equal to 24 hours from when the search was run, return a value of 1 in the isOutlier column. Otherwise, return a value of 0.

Make sure your search time range is set appropriately or all domains might appear to be new. Setting the search to the past 7 days is a good start.

| convert ctime(earliest) ctime(latest)

Convert epoch time for the earliest and latest visits to the domains into a more easily readable format.

| where isOutlier=1

Return only results where the isOutlier value is 1, which indicates a new domain.

Option 3

Use this option if you have CIM compliant data and populated web data models. The tstats command works with accelerated data and is much faster than the search in Option 1.

  1. Run the following search:
| tstats count from datamodel=Web by Web.url _time
| rename "Web.url" as "uri"
| eval list="mozilla"
| `ut_parse_extended(uri,list)`
| stats earliest(_time) as earliest latest(_time) as latest by ut_domain
| eval isOutlier=if(earliest >= relative_time(now(),"-1d@d"), 1, 0)
| convert ctime(earliest) ctime(latest)

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation

| tstats count from datamodel=Web by Web.url _time

Return domains accessed, the date and time each was accessed, and the number of times each was accessed.

You must have CIM compliant data and populated data models for this search to work.

| rename "Web.url" as "uri"

Rename the Web.url column "uri".

| eval list="mozilla"

Search the Mozilla catalog for top level domains.

This eval function is required for the next line in the search (ut_parse_extended) to work.

| `ut_parse_extended(uri,list)`

Parse the URLs based on the Mozilla top level domain list.

The punctuation in a Splunk macro is always a back tick (`), not a single quote (').

| stats earliest(_time) as earliest latest(_time) as latest by ut_domain

Return the results in a table—grouped by the ut_domain field—that includes columns for the first and last time each domain was seen in the dataset.

| eval isOutlier=if(earliest >= relative_time(now(),"-1d@d"), 1, 0)

Create a new column called isOutlier. If the earliest time a domain appears in the dataset is more recent than or equal to 24 hours from when the search was run, return a value of 1 in the isOutlier column. Otherwise, return a value of 0.

Make sure your search time range is set appropriately or all domains might appear to be new. Setting the search to the past 7 days is a good start.

| convert ctime(earliest) ctime(latest)

Convert epoch time for the earliest and latest visits to the domains into a more easily readable format.

Result

You might decide to investigate any returned results to make sure the domains don't pose a security risk. Save this search so you can easily monitor new domains on the network when needed. If you used option 1, you might also want to save the commonly accessed domain information to a lookup file to create a baseline you can use in other searches.

  • Was this article helpful?