Skip to main content
Splunk Lantern

Trends in exceptions and stack traces

The presence of a stack trace within application logs is a strong indicator of application errors or problems and is most often emitted when an exception is thrown and not caught. You want to inventory and monitor the stack traces being emitted by an application and to identify and inspect specific stack traces during an investigation.

Procedure

Run the following search. You can optimize it by specifying an index and adjusting the time range.

host = <host to look at> 
linecount>3 (unhandled OR exception OR traceback OR stacktrace)
| rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})"
| stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype
| table index sourcetype stacktrace trend count
| sort - count

Search explanation

The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.

Splunk Search Explanation
host = <host to look at>  Search only one specific host.
linecount>3 Search for a line count greater than three.  Stack traces are multiline messages or events. 
(unhandled OR exception OR traceback OR stacktrace) Find events with specific words in them, such as “unhandled’, “exception”, “traceback”, or “stacktrace”.
| rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})" Extract the first and second lines of the stack trace to group them. They have the same number of lines, and the second line is the same between stack traces.
| stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype Add a sparkline chart that shows the event count trend for each listed source type.
| table index sourcetype stacktrace trend count Display the results in a table with columns in the order shown.
| sort - count Sort the results in descending order.

Next steps

The results of this search include the index where the stacktrace was found, the sourcetype that generated it, the text of the stacktrace, and a sparkline that shows the trend in frequency of occurrence.

Additionally, you might need to detect trends in exceptions and stack traces when using stack traces to detect application errors.