Monitoring the system health of Splunk Enterprise

The Splunk Enterprise Monitoring Console is an app included with every Splunk installation. It consists of dashboards, platform alerts, and health checks. It enables Splunk administrators to gain insight into the system health of Splunk Enterprise, including indexing and search performance, OS resource usage, and license usage. But it's not just a stethoscope on system health, the information in the monitoring console provides insight about how your searches are working, and where you can tune them to make them better!

How the Monitoring Console helps promote a healthy Splunk Enterprise deployment

The monitoring console goes beyond just showing if your indexer or search heads are up or down. The monitoring console has a series of dashboards that help you find answers to common problems, for example, why users are getting "peer unresponsive" errors, or why search performance is slow. These diagnostics can also indicate where you may have inefficient searches set up, or if you have too many automated reports running that are affecting system performance.

Metrics in the Monitoring Console can also help you know when to scale. If you notice your system performance consistently running at near-peak levels even after optimizing searches, it may be time to add an indexer.

• Search Usage Statistics: The Search Activity and Search Usage Statistics dashboards can enlighten you on details such as Aggregate Search Runtime, Top 10 Memory-Consuming Searches, and Long-Running Searches. Warning: users of these dashboards have been known to favorite the documentation on how to Write better searches...don't say we didn't warn you!

• Scheduler activity: The Scheduler Activity dashboards monitor the activity and success rate of the search scheduler. This can help you run traffic control on your scheduled searches and ensure they are efficient and make good use of system resources.

• License usage: The License Usage dashboard gives you insight into daily indexing volume, license warnings, and the last 30 days of your license usage directly from the Splunk Web.

• Platform alerts: A platform alert is a saved search in the monitoring console that notifies administrators of conditions that might compromise their Splunk Enterprise environment. Click here for a video that demonstrates how to create alerts.

Beyond the Monitoring Console

There are a plethora of community created apps that take monitoring of Splunk to the next level. Take a peek at the comments of this post to learn more.