Skip to main content
Lantern Home
 
 
Splunk Lantern

Improving data onboarding with props.conf configurations

  1. Last updated
  2. Save as PDF

 

By ensuring that all source types have the required props.conf definitions and stanzas, companies can improve index and search performance through more accurate data parsing. This improvement leads to more efficient data analysis, operational efficiency, enhanced decision-making capabilities, and reduced total cost of ownership (TCO) of the Splunk platform. The following list describes in more detail the benefits of proper data source configuration.

  • Improved Parsing and Data Consistency: The props.conf file defines how the Splunk platform processes and extracts information from the raw data. By including appropriate configuration settings, such as TIME_FORMAT, LINE_BREAKER, or KV_MODE, organizations can normalize data across sources and ensure consistent parsing, enabling seamless analysis and reliable insights.
  • Search Performance Optimization: The correct extraction of fields and timestamps through source type configuration allows the Splunk platform to index data accurately, leading to faster and more accurate search results. Optimized search performance translates into improved operational efficiency, reduced query execution times, and enhanced user productivity.
  • Consistent Field Extraction and Enrichment: By specifying field extractions, transformations, or lookups, organizations can enrich their data with relevant contextual information. This consistency in field extraction and enrichment empowers analysts to perform in-depth analysis and derive valuable insights from the data. This allows the fields to be used in base searches, improving search optimization and data processing.
  • Data Standardization and Normalization: The "Great 8" principles for configuring new source types cover aspects such as timestamp recognition, event line breaking, multivalue field extraction, and more. By adhering to these principles, outline in the next section, organizations can establish consistent data formats and structures across different source types, enabling seamless data integration and cross-source analysis.
  • Data Governance and Compliance: By enforcing consistent parsing and field extraction rules, organizations can adhere to data quality standards and regulatory compliance requirements. Properly defined source types help maintain data integrity, facilitate auditing processes, and ensure that data is properly classified and protected.
  • Improved Data Analysis and Visualization: By correctly parsing and extracting relevant fields, organizations can generate meaningful reports, dashboards, and visualizations. This enables stakeholders to gain insights quickly, make informed decisions, and uncover actionable intelligence from their data.
  • Scalability and Flexibility: By establishing standardized props.conf settings, organizations can efficiently onboard new data sources, ensuring consistent parsing and analysis methodologies. This scalability and flexibility facilitate streamlined data onboarding, reduced maintenance efforts, and improved adaptability to evolving business needs.
  • Knowledge Sharing and Collaboration: When all source types have the required props.conf definitions and stanzas, it becomes easier for analysts, administrators, and developers to understand and work with the data. This promotes effective collaboration, reduces the learning curve, and enables the efficient transfer of expertise within your organization. 

The "Great 8" configurations

The props.conf configuration file is a power configuration option for controlling how data is ingested, parsed, and transformed during the onboarding process. Among other things, props.conf is used for defining field extractions, identifying and capturing specific pieces of information from your raw data.

The Great 8 configurations below provide a standard for transforming raw data into well formatted, searchable events within the Splunk platform. They ensure that events are accurately separated, timestamps are correctly captured, so that fields can be properly extracted for analysis. By adhering to these configurations, you enhance data consistency, accessibility, and reliability, setting the stage for accurate insights and efficient analysis.

The following list only provides a brief explanation of each of these configurations. For complete, hands-on configuration guidance, see Configuring new source types.

  1. SHOULD_LINEMERGE = false (always false): This configuration tells the Splunk platform not to merge multiple lines of data into a single event. This is particularly useful for log files where each line represents a separate event, preventing accidental merging of unrelated lines. For additional context, reference the props.conf spec around line breaking.
  2. LINE_BREAKER = regular expression for event breaks: The LINE_BREAKER configuration specifies a regular expression pattern that indicates where one event ends and another begins. This is essential for parsing multi-line logs into individual events for proper indexing and analysis. For additional context, reference the props.conf spec around line breaking.
  3. TIME_PREFIX = regex of the text that leads up to the timestamp: When data contains timestamps, TIME_PREFIX helps the Splunk platform identify the portion of the data that precedes the actual timestamp. This helps the Splunk platform correctly locate and extract the timestamp for indexing and time-based analysis.
  4. MAX_TIMESTAMP_LOOKAHEAD = how many characters for the timestamp: This configuration sets the maximum number of characters that the Splunk platform will look ahead from the TIME_PREFIX to find the timestamp. It ensures that the Splunk platform doesn't search too far ahead, optimizing performance while accurately capturing timestamps.
  5. TIME_FORMAT = strptime format of the timestamp: TIME_FORMAT specifies the format of the timestamp within the data. The Splunk platform uses this information to correctly interpret and index the timestamp, making it usable for time-based searches and analyses.
  6. TRUNCATE = 999999 (always a high number): TRUNCATE configuration helps prevent overly long events from causing performance issues. It limits the maximum length of an event, ensuring that extremely long lines don't negatively impact the performance of the Splunk platform.
  7. EVENT_BREAKER_ENABLE = true: This configuration indicates whether event breaking should be enabled. Setting it to true ensures that event breaking based on LINE_BREAKER is activated.
  8. EVENT_BREAKER = regular expression for event breaks: EVENT_BREAKER allows you to define an additional regular expression pattern for event breaking. This can be useful for scenarios where more complex event breaking is required.

Next steps

This article is part of the Splunk Outcome Path, Reducing your infrastructure footprint. Click into that path to find more ways you can maximize your investment in Splunk software and achieve cost savings. 

In addition, these resources might help you implement the guidance provided in this article: