You can use historical KPI data and machine learning algorithms in Splunk ITSI to predict an outage 20-30 minutes before it happens. This process works best when a service has more than 5 good KPIs and more than 1 week of historical data.
The machine learning algorithm looks for recognizable/predictable KPI behavior, which comes before the service's aggregate health score changes. You can use machine learning within Splunk ITSI to build a model for the service you want to track.
Build a model
- Open Splunk ITSI and in the top toolbar click Configuration, then Services.
- In the list of services, find the service you want to track. Click the Edit drop-down box to the right of the service name, then click Predictive Analytics.
- On this screen you will train and test different machine learning algorithms to determine which one gives the most accurate prediction. Use the instructions on-screen to select a time, algorithm type and algorithm, and click the Train button.
- After the model has run, investigate the results, and click Save.
- Test out other algorithms by repeating steps 3 and 4.
Review the Predictive Analytics score and add it to a glass table
- Open Splunk ITSI and in the top toolbar click Dashboards, then Predictive Analytics.
- In the list of services, find the service you want to track, and select the recommended algorithm model.
- After you have selected a model, Splunk ITSI will calculate the future Service Health Score. Click the Cause Analysis button to review the suggested KPIs.
- Click the spyglass to review the SPL. Save it to a notepad to copy it into a glass table later.
- In the Splunk ITSI top toolbar, click Glass Tables and select the glass table you'd like to add this score to.
- Click the Edit button and review the Database Future Health Score attributes.
- Select the Future Health for the service you are interested in, then under the Configuration menu, click Set up Primary Data Source.
- Click Create Ad-Hoc Search. Copy the previous Predictive Analytics SPL and click the Run & Save button.