Skip to main content

Splunk Lantern turned 5 on May 28th. Thank you for being one of our 750,000 annual users!
Click here to join our Slack channel to tell us what you love about the site or what content you'd like to see more of.
Lantern Home

 

Splunk Lantern

Getting started with Splunk Artificial Intelligence

  1. Last updated
  2. Save as PDF

 

This article provides a structured, prescriptive approach for organizations to adopt the artificial intelligence/machine learning (AI/ML) capabilities in Splunk software. It outlines a progressive journey through a three-tiered capability model, guiding prerequisites, implementation steps, resource requirements, and expected outcomes at each stage in conjunction with the Splunk Validated Architecture (SVA) for Splunk AI and ML. 

Splunk AI/ML capabilities offer organizations powerful tools to derive deep insights, identify patterns, detect anomalies, and predict outcomes from their data. However, implementing these capabilities requires a structured approach to ensure success. This adoption path is designed to guide organizations from initial exploration to advanced implementation, with clearly defined milestones and considerations at each step.

The Splunk AI/ML Adoption Framework follows the Splunk AI SVA's three-tiered capability model, with each tier representing an increasing level of AI/ML sophistication:

  1. Foundation: Core Splunk AI/ML capabilities within the platform
  2. Advancement: Machine Learning Toolkit (MLTK) implementation 
  3. Innovation: Data Science and Deep Learning (DSDL) deployment

Each tier builds on the skills and technology of the previous one, allowing organizations to progress at their own pace while incrementally developing skills and infrastructure.

Phase 0: Assessment and planning 

While not an implementation phase, this foundational phase is critical to the success of subsequent AI/ML efforts. This phase serves as a strategic planning and readiness assessment step, guiding the organization’s evaluations of its current Splunk environment, identifying impactful use cases, and aligning resources and infrastructure. This phase should be completed before engaging in the first step of any implementation phase outlined in this framework.

Step 1: Current state assessment 

  • Evaluate existing Splunk infrastructure and deployment model. 
  • Inventory data sources and use cases currently in your Splunk environment.
  • Assess team skills in Splunk Search Processing Language (SPL), statistics, and data science. 
  • Document organizational AI/ML objectives and priorities. 

Step 2: AI/ML use case identification 

  • Identify two or three initial use cases that align with business priorities. 
  • Categorize use cases by sophistication (simple statistical analysis to advanced ML). 
  • Determine which tier of the Splunk capability model is required for each use case. 
  • Prioritize use cases based on business value and technical feasibility. 

Step 3: Resource and infrastructure planning 

  • Determine infrastructure requirements for each tier. 
  • Identify skills gaps and training needs. 
  • Develop a timeline for implementation phases. 
  • Create a budget for necessary infrastructure and training investments. 

Phase 1: Foundation - Core Splunk implementation 

This phase marks the beginning of AI/ML exploration within your Splunk environment. It is focused on building foundational skills and validating early use cases using core SPL and statistical commands. The emphasis is on preparing infrastructure, shaping usable data, and applying basic analytical techniques to surface anomalies, trends, and simple predictions. Rather than deploying fully operational models, teams use this stage to experiment, build confidence, and identify areas where more advanced ML approaches might later add value.

Step 1: Infrastructure preparation 

  • Ensure your Splunk deployment meets minimum requirements for AI/ML workloads. 
  • Review and optimize search head performance for analytical workloads. 
  • Implement workload management policies to accommodate ML tasks. 

Step 2: Skill development 

  • Train the team on statistical SPL commands and functions. 
  • Develop an understanding of data preprocessing techniques. 
  • Build skills in results interpretation and validation. 
  • Document best practices and lessons learned. 

Step 3: Data preparation 

  • Ensure consistent data ingestion and field extraction. 
  • Implement field aliases and calculated fields needed for analysis. 
  • Create knowledge objects (lookups, tags, etc.) to enrich data. 
  • Validate data quality and completeness for target use cases. 

Step 4: Core SPL implementation 

  • Develop and test SPL searches using statistical commands. 
  • Implement basic anomaly detection using built-in commands. 
  • Create prediction and trending models with core SPL. 
  • Develop dashboards to visualize results. 

Use case examples

  • Detecting outliers in system performance metrics 
  • Forecasting capacity requirements based on historical usage 
  • Identifying seasonal patterns in business transactions 

Phase 2: Advancement - Machine Learning Toolkit (MLTK) implementation 

In this phase, organizations introduce AI/ML capabilities into their operations. The focus shifts from experimentation to regular use of the Splunk Machine Learning Toolkit (MLTK) for production-grade use cases, including tuning performance limits, developing and refining models, and operationalizing models through scheduled training, alerting, and dashboards. Organizations expand use cases, integrate outputs into workflows, and begin scaling ML infrastructure to support broader, automated insights as adoption matures.

Step 1: MLTK installation and configuration 

  1. Install Python for Scientific Computing (PSC) add-on. 
  2. Install MLTK.
  3. Configure algorithm performance costs and resource limits. 
  4. Implement workload management for ML tasks. 

Step 2: Model development 

  • Use MLTK Showcase to explore relevant algorithms. 
  • Develop and test models for identified use cases. 
  • Use experiments to compare and refine models. 
  • Document model performance and parameters. 

Step 3: Model operationalization 

  • Schedule model training and scoring jobs. 
  • Implement alerts based on model outputs. 
  • Create dashboards to visualize model results. 
  • Establish model monitoring and retraining processes. 

Step 4: Expansion 

  • Identify additional use cases for MLTK implementation. 
  • Evaluate the need for a dedicated search head for ML workloads. 
  • Develop advanced models using multiple algorithms. 
  • Integrate model outputs into operational workflows. 

Use case examples

These use case examples highlight practical machine learning applications and statistical analysis within Splunk software. 

  • User behavior analysis and anomaly detection: help identify unusual patterns that could signal insider threats or compromised accounts. 
  • Predictive maintenance for IT infrastructure: leverage historical performance data to forecast potential system failures before they occur. 
  • Automated classification of security events: enable faster triage by tagging incidents based on learned patterns, improving response times and reducing analyst workload.

Phase 3: Innovation - Data science and deep learning (DSDL) implementation 

This phase introduces advanced AI knowledge and capabilities, expanding on the functionality and experience gained in earlier phases. Typically, custom models are developed with The Splunk app for Data Science and Deep Learning (DSDL) and integrated into operations. This phase requires an understanding of deep learning models, real-time predictions, and custom algorithms.

Step 1: DSDL installation and configuration 

  1. Install DSDL on the search head. 
  2. Configure connections to the container environment. 
  3. Set up MLFlow and TensorBoard integrations. 
  4. Implement data transfer optimizations. 

Step 2: Advanced model development 

  • Develop custom models using JupyterLab. 
  • Implement deep learning models for complex use cases. 
  • Leverage GPUs for model training acceleration. 
  • Utilize standalone large language models (LLMs) or combine with Retrieval-Augmented Generation (RAG) operations. 
  • Integrate with vector or graph databases for advanced applications.
  • Use MLFlow for experiment tracking and model management. 

Step 3: Production deployment 

  • Deploy models to production containers. 
  • Implement model performance monitoring. 
  • Establish a continuous integration/continuous delivery (CI/CD) pipeline for model updates. 
  • Document operational procedures for model maintenance. 

Use case examples

These advanced use cases demonstrate the power of AI in transforming operational intelligence. 

  • Natural language processing for log analysis: enable intuitive log analysis by extracting insights from unstructured text data. 
  • Deep learning for complex pattern recognition: identify complex, nonlinear patterns that traditional methods might miss. 
  • Generative AI for automated root cause analysis: synthesize data from multiple sources to suggest likely causes, accelerating incident resolution and decision-making.

Implementation roadmap

The table below outlines each AI/ML adoption journey phase, including estimated timelines, key deliverables, and success criteria. While timelines reflect typical project durations, actual implementation may vary based on the complexity and maturity of individual use cases.

Phase Timeline Key Deliverables Success Criteria
Assessment and planning 2-4 weeks Use case inventory, infrastructure plan, resource requirements Prioritized use cases, approved resource plan
Foundation 4-8 weeks SPL queries, basic dashboards, baseline metrics Operational insights from statistical analysis
Advancement 8-12 weeks MLTK models, training schedules, alerting workflows Automated anomaly detection and predictions
Innovation 12-16 weeks Custom models, deep learning implementations, and model management framework Advanced AI/ML capabilities for complex use cases 

Best practices for sustainable AI/ML operations

This section outlines key best practices across model management, team structure, and performance monitoring to ensure long-term success with AI/ML in your Splunk environment. Organizations can scale their ML efforts with confidence and control by implementing standardized processes, fostering cross-functional collaboration, and proactively managing system performance.

Team structure and roles 

  • Define roles for Splunk administrators, ML engineers, and data scientists. 
  • Establish collaboration workflows between teams. 
  • Create knowledge-sharing mechanisms. 
  • Develop skills progression plans for team members. 

Model management 

  • Establish model inventory and documentation standards. 
  • Implement version control for models and code. 
  • Create model performance monitoring procedures. 
  • Define model retraining triggers and schedules. 

Performance management 

  • Monitor search and ML workload impact on Splunk infrastructure. 
  • Establish performance baselines and thresholds. 
  • Implement scaling procedures for increasing ML workloads. 
  • Develop capacity planning processes for ML growth. 

Conclusion

This prescriptive adoption path provides a structured approach to implementing Splunk AI/ML capabilities across the organization. By following this progressive implementation framework, organizations can build their AI/ML capabilities effectively while ensuring alignment with business objectives and operational requirements, as well as regulatory compliance models and frameworks.

Next step