Monitoring applications using OpenAI API and GPT models with OpenTelemetry and Splunk APM

Building the application with Flask

from flask import (
   Flask,
   render_template,
   request,
   Response,
   stream_with_context,
   jsonify,
)

import openai

client = openai.OpenAI()

app = Flask(__name__)

chat_history = [
   {"role": "system", "content": "Hello, I'm Shelly's Assistant; I (actually) run The Splunk T-Shirt Company. AMA"},
]


@app.route("/", methods=["GET"])
def index():
   return render_template("index.html", chat_history=chat_history)


@app.route("/chat", methods=["POST"])
def chat():
   content = request.json["message"]
   chat_history.append({"role": "user", "content": content})
   return jsonify(success=True)


@app.route("/stream", methods=["GET"])
def stream():...


@app.route("/reset", methods=["POST"])
def reset_chat():...
   global chat_history
   chat_history = [
       {"role": "system",
        "content": "Hello, I'm Shelly's Assistant; I (actually) run The Splunk T-Shirt Company. AMA"},
   ]
   return jsonify(success=True)

Instrumenting with OpenTelemetry

Next, integrate OpenTelemetry into the Flask app to capture traces and spans.

The Splunk distribution of OpenTelemetry Python instrumentation provides a Python agent that automatically instruments your Python application to capture and report distributed traces to Splunk Application Performance Monitoring.

$ pip install splunk-opentelemetry[all]

# otel imports
from opentelemetry import trace
from opentelemetry.metrics import MeterProvider
from opentelemetry.sdk.resources import SERVICE_NAME, Resource
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter
# setup otel tracing
resource = Resource(attributes={
   SERVICE_NAME: "splunk-shelly-AI-Assistant"
})
provider = TracerProvider(resource=resource)
trace.set_tracer_provider(provider)
tracer = trace.get_tracer(__name__)
otlp_exporter = OTLPSpanExporter(endpoint="http://localhost:4317", insecure=True)
span_processor = BatchSpanProcessor(otlp_exporter)
provider.add_span_processor(span_processor)
# Adding custom attributes to span in the stream() function 
@app.route("/stream", methods=["GET"])
def stream():
  def generate():...
    with tracer.start_as_current_span("call_gpt_model") as span:
      with client.chat.completions.create(
                    messages=chat_history,
                    stream=True,
                    model=GPT_model,
                    temperature=GPT_temperature,
                    top_p=GPT_top_p,
            ) as stream:...


      span.set_attribute("id", chunk.id)
      span.set_attribute("GPT-model", chunk.model)
      span.set_attribute("GPT-temperature", GPT_temperature)
      span.set_attribute("GPT-top_p", GPT_top_p)
      span.set_attribute("prompt", prompt)
      span.set_attribute("response", result)
      span.set_attribute("latency", duration)
      span.set_attribute("tokens_used", tokens_used)      span.set_attribute("completion_tokens", completion_tokens)

Configuring the Splunk Distribution of the OpenTelemetry Collector

Configure the Splunk Distribution of the OpenTelemetry Collector to receive and export metrics and traces to Splunk Observability Cloud.

First, install the collector with one of the following methods:

• If you plan to run the collector on Linux, Windows, or Kubernetes, installing the collector is straightforward and all instructions are available here.
• If you’re developing on a Mac, you have to build the custom binary. All steps to build and execute are available here. The agent and gateway configuration can be found in this GitHub repo. Modify the parameters as you see fit.

Next, set the following environment variables:

OPENAI_API_KEY=<OPENAI-API-KEY>
OTEL_EXPORTER_OTLP_ENDPOINT=http://localhost:4317
OTEL_RESOURCE_ATTRIBUTES=deployment.environment=dev,service.version=0.0.1
OTEL_SERVICE_NAME=Splunk-Shelly-AI-Assistant

After you have configured the collector and specified the correct Splunk Observability Cloud endpoint URL and the API token, you’re ready to start your collector. Start the collector with the proper configuration:

sudo SPLUNK_ACCESS_TOKEN=<TOKEN> SPLUNK_REALM=<REALM> ./otelcol --config=/etc/otel/collector/agent_config.yaml

Prefix your service run command splunk-py-trace to enable instrumentation:

splunk-py-trace python -m flask run --host=0.0.0.0 --port=5000