Trends in exceptions and stack traces
The presence of a stack trace within application logs is a strong indicator of application errors or problems and is most often emitted when an exception is thrown and not caught. You want to inventory and monitor the stack traces being emitted by an application and to identify and inspect specific stack traces during an investigation.
Data required
Procedure
Run the following search. You can optimize it by specifying an index and adjusting the time range.
host = <host to look at>
linecount>3 (unhandled OR exception OR traceback OR stacktrace)
| rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})"
| stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype
| table index sourcetype stacktrace trend count
| sort - count
Search explanation
The table provides an explanation of what each part of this search achieves. You can adjust this query based on the specifics of your environment.
| Splunk Search | Explanation |
|---|---|
| host = <host to look at> | Search only one specific host. |
| linecount>3 | Search for a line count greater than three. Stack traces are multiline messages or events. |
| (unhandled OR exception OR traceback OR stacktrace) | Find events with specific words in them, such as “unhandled’, “exception”, “traceback”, or “stacktrace”. |
| | rex field=_raw "(?<FirstLine>(.*){1})\n(?<SecondLine>(.*){1})" | Extract the first and second lines of the stack trace to group them. They have the same number of lines, and the second line is the same between stack traces. |
| | stats sparkline(count,1h) AS trend first(_raw) AS stacktrace count BY linecount SecondLine, index, sourcetype | Add a sparkline chart that shows the event count trend for each listed source type. |
| | table index sourcetype stacktrace trend count | Display the results in a table with columns in the order shown. |
| | sort - count | Sort the results in descending order. |
Next steps
The results of this search include the index where the stacktrace was found, the sourcetype that generated it, the text of the stacktrace, and a sparkline that shows the trend in frequency of occurrence.
Additionally, you might need to detect trends in exceptions and stack traces when using stack traces to detect application errors.