Skip to main content
Splunk Lantern

Detect spikes in blocked outbound traffic from AWS

The table below explains the steps of a Splunk Enterprise or Splunk Cloud Platform search to help you detects spikes in blocked outbound network connections originating from within your AWS environment. For more information, review the use case Monitoring AWS for suspicious traffic.

Some commands, parameters, and field names in the searches below may need to be adjusted to match your environment.  In addition, to optimize the searches shown below, you should specify an index and a time range when appropriate.

Splunk Search Explanation

sourcetype=aws:cloudwatchlogs:vpcflow action=blocked (src_ip=10.0.0.0/8 OR src_ip=172.16.0.0/12 OR src_ip=192.168.0.0/16) ( dest_ip!=10.0.0.0/8 AND dest_ip!=172.16.0.0/12 AND dest_ip!=192.168.0.0/16)  [search  sourcetype=aws:cloudwatchlogs:vpcflow action=blocked (src_ip=10.0.0.0/8 OR src_ip=172.16.0.0/12 OR src_ip=192.168.0.0/16) ( dest_ip!=10.0.0.0/8 AND dest_ip!=172.16.0.0/12 AND dest_ip!=192.168.0.0/16)  

Retrieves all the VPC Flow log entries that have recorded a blocked outbound network connection originating from your AWS environment.
| stats count AS numberOfBlockedConnections BY src_ip  Counts the number of blocked outbound connections by each source IP.
| inputlookup baseline_blocked_outbound_connections append=t  Loads the cache file that contains the number of data points, the count from the latest hour, the average blocked connections, and the standard deviation for each source IP.
| fields - latestCount 
| stats values(*) AS * BY src_ip 
Drops the count from the latest hour, since it is not necessary, and merges the rest of the data with the results of the stats command.
| rename numberOfBlockedConnections AS latestCount  Renames numberOfBlockedConnections as latestCount.
| eval newAvgBlockedConnections=avgBlockedConnections + (latestCount-avgBlockedConnections)/720 
| eval newStdevBlockedConnections=sqrt(((pow(stdevBlockedConnections, 2)*719 + (latestCount-newAvgBlockedConnections)*(latestCount-avgBlockedConnections))/720)) 
| eval avgBlockedConnections=coalesce(newAvgBlockedConnections, avgBlockedConnections), stdevBlockedConnections=coalesce(newStdevBlockedConnections, stdevBlockedConnections), numDataPoints=if(isnull(latestCount), numDataPoints, numDataPoints+1) 
 
Calculates the new average value for each source IP with the latest count, weighting the past much more heavily than the current hour. It does the same for the standard deviation, weighting the past more heavily than the current.
| table src_ip, latestCount, numDataPoints, avgBlockedConnections, stdevBlockedConnections Display the results in a table with columns in the order shown.
| outputlookup baseline_blocked_outbound_connections  Updates the cache file with the latest results.
| eval dataPointThreshold = 5, deviationThreshold = 3  Sets the minimum threshold for the number of data points and sets the number of standard deviations away from the mean it must be to be considered a spike.
| eval isSpike=if((latestCount > avgBlockedConnections+deviationThreshold*stdevBlockedConnections) AND numDataPoints > dataPointThreshold, 1, 0)  Makes a determination regarding whether or not the current count is a spike by checking to see if the minimum data-point threshold has been met and the count is a sufficient number of standard deviations away from the average.
| where isSpike=1 
| table src_ip] 
| stats values(dest_ip) AS "Blocked Destination IPs", values(interface_id) AS "resourceId" count AS numberOfBlockedConnections, dc(dest_ip) AS uniqueDestConnections BY src_ip
Filters out anything that it determines is not a spike and returns the list of source IPs to the main search. The main search subsequently gets the list of all destination IPs for which the traffic was blocked, the network interface ID, the number of unique destination IP, and the total number of blocked connections for each of these source IP addresses.
`detect_spike_in_blocked_outbound_traffic_from_your_aws_filter` Looks up the average and standard deviation and returns both the average and the number of standard deviations the spike is from the average.