Skip to main content
Splunk Lantern

Trends in exceptions and stack traces

You might need to detect trends in exceptions and stack traces when doing the following:

Prerequisites 

In order to execute this procedure in your environment, the following data, services, or apps are required:

Example

The presence of a stack trace within application logs is a strong indicator of application errors or problems and is most often emitted when an exception is thrown and not caught. You want to inventory and monitor the stack traces being emitted by an application and to identify and inspect specific stack traces during an investigation.

To optimize the search shown below, you should specify an index and a time range. 

  1. Run the following search:
    host = <host to look at> 
    linecount>3 (unhandled OR exception OR traceback OR stacktrace)
    | rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})"
    | stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype
    | table index sourcetype stacktrace trend count
    | sort - count
    

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation
host = <host to look at>  Search only one specific host.
linecount>3 Search for a line count greater than three.  Stack traces are multiline messages or events. 
(unhandled OR exception OR traceback OR stacktrace) Find events with specific words in them, such as “unhandled’, “exception”, “traceback”, or “stacktrace”.
| rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})" Extract the first and second lines of the stack trace to group them. They have the same number of lines, and the second line is the same between stack traces.
| stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype Add a sparkline chart that shows the event count trend for each listed source type.
| table index sourcetype stacktrace trend count Display the results in a table with columns in the order shown.
| sort - count Sort the results in descending order.

Result

The results of this search include the index where the stacktrace was found, the sourcetype that generated it, the text of the stacktrace, and a sparkline that shows the trend in frequency of occurrence.
 

  • Was this article helpful?