Enabling persistent queue for metrics and traces

File storage extension settings

Within the extensions section of the config file, you need to define a file storage extension using the file_storage item. You should give this extension a name so that it can easily be accessed within the exporter sections.

The file storage extension has lots of configuration options that are worth exploring. For this example, we are keeping all the default settings except for the timeout value, which is used as a timeout to grab file locks.

extensions:
  # persistent Queue file storage
  file_storage/persist:
    # default windows directory c:\ProgramData\Splunk\OpenTelemetry Collector\FileStorage
    timeout: 10s
    # lots of additional options for file_storage and worth looking at
     # directory: "./myDirectory"    # Define directory
     # create_directory: true           # Create directory
     # timeout: 1s                      # Timeout for file operations
     # compaction:                      # Compaction settings
     #   on_start: true                 # Start compaction at Collector startup
     #   # Define compaction directory
     #   directory: "./myDirectory/tmp"
     #   # Max. size limit before compaction occurs
     #   max_transaction_size: 65536

After we define the file_storage extension, we need to add it to the service extension list to be enabled. Below is an example of the default extensions with the new file_storage extension added.

service:
  extensions: [health_check, http_forwarder, zpages, smartagent, file_storage/persist]

Exporter helper settings

By default, most exporters have access to the exporter helper. This allows trace and metrics exporters to enable a persistent queue.

The exporter helper provides both the retry_on_failure and sending_queue settings. Both items support additional configuration options that are worth exploring.

Within the retry_on_failure section, the important setting is max_elapsed_time. This setting is the amount of time the collector spends trying to send the batch. It's important to understand that this works more like a time-to-live (TTL) value then a timeout. Each time the batch is attempted, the time spent trying to send it counts towards the max_elapsed_time. On the exporters, the default timeout on a data export operation is 5s, and the default max_elapsed_time is 300s. So, as an example, if the data export fails due to timeout (5s), then the batch will have roughly 60 (5s x 60 = 300s) attempts to send the data before the max_elapsed_time is reached. However, errors such as DNS failures or bad URLs result in a faster failure time, which results in more than 60 data export attempts.

This retry rate is further impacted by the config value max_interval, which is the max pause between retry attempts. The default is 30s. With roughly 60 retry attempts and a 30s pause between attempts you end up with 30 min, meaning that with the default settings the queue holds batches that are roughly 30min old before they hit the max_elapsed_time and are dropped with the "No More Retries" error. However, there is some randomness intentionally added to the max_interval to help prevent all queued batches from sending at the same time. This added randomness makes this calculation approximate.

To force the retry to be indefinite, set the max_elapsed_time to 0.

For our example of handling an 8-hour outage, we want to ensure that all data is eventually received, so we set the max_elapsed_time to 0 to force infinite retries. However, infinite retries are not ideal in a production environment, so as an alternative (and to ensure you still have enough retries to handle the outage), you can set the max_elapsed_time to the same time frame as the outage you need to handle. For our example of handling an 8-hour outage, you can set max_elapsed_time to 8h. Note that since this is a duration field it, understands s for seconds, m for minutes, and h for hours.

This will be the upper bound that is required. Using the details above, you can calculate a value that matches closer to your needs.

Queue size

Getting the queue_size right is important as we need to ensure that we have a large enough queue to hold all the batches that we can expect during the outage duration we are trying to handle. Queue_size is a batch count and is not a size such as kB. We can calculate an approximate size based on the config values timeout and send_batch_size.

Both values, timeout and send_batch_size, are used to trigger a batch export operation, and its the total count of these batch export operations over a duration that is needed for the queue_size. For more details, see the preceding section on Batches. If queue_size is too low, then, when connectivity is restored, you will see data backfilled from when the outage started until the point that the queue was full and no more data could be persisted. From the point the queue was full until the end of the outage, there will be a gap in the data. If you are filling your queue, you will see errors in your collector logs and event viewer like “Sender Error...Sending queue is full”.

For our example of handing an 8-hour outage, we can perform some simple math to calculate the upper bound of the queue_size.

The default timeout is 200ms, meaning that it will send a data batch every 200ms. This equates to five data batches per second, which equals 300 batches per minute, which equals 18,000 batches per hour. To cover an 8-hour outage, you need a queue_size of 8 hours x 18,000 = 144,000. This calculation is meant only as a guide. Many factors can impact queue_size, so you should perform testing to ensure the coverage you require.

Below is a sample of a config showing these settings for the otlp/http exporter (traces) and the signalFX exporter (metrics). These exporters also must be in the traces and metrics section of the data pipeline to be enabled.

exporters:
  # Traces
  otlphttp:
    traces_endpoint: https://ingest.us1.signalfx.com/v2/trace/otlp
    headers:
      X-SF-Token: "${SPLUNK_ACCESS_TOKEN}"
    retry_on_failure:
      max_elapsed_time: 0
    sending_queue:
      storage: file_storage/persist
      queue_size: 144000 #calculated value to handle 8 hour outage
# Metrics + Events
signalfx:
  access_token: "${SPLUNK_ACCESS_TOKEN}"
  api_url: "${SPLUNK_API_URL}"
  ingest_url: "${SPLUNK_INGEST_URL}"
  # Use instead when sending to gateway
  #api_url: http://${SPLUNK_GATEWAY_URL}:6060
  #ingest_url: http://${SPLUNK_GATEWAY_URL}:9943
  sync_host_metadata: true
  correlation:
  retry_on_failure:
    max_elapsed_time: 0
    sending_queue:
      storage: file_storage/persist
      queue_size: 144000 #calculated value to handle 8 hour outage