You are a new Splunk APM customer and want to determine the best method of sending traces to APM from your services.
Instrumentation is the process through which application code is extended to capture and report trace spans. Splunk APM is instrumentation agnostic; as long as spans are sent in a recognized format, it doesn’t matter how you instrument your code. APM supports instrumentation libraries such as OpenTelemetry, Jaeger, and Zipkin, and has default instrumentation for languages such as Java, Ruby, Python, Node, Go, .NET, and PHP. However, you can manually instrument applications for other languages. Before doing so, you should understand when to use automatic and when to use manual instrumentation.
Automatic instrumentation is a run-time automated process that automatically identifies the frameworks and libraries that are in use. It instruments those libraries to capture tracing, without making any code changes. If your library is supported by auto-instrumentation, helpful operational data can be obtained. For example, you might want to know when:
- a Python application made an HTTP request to another service
- a Java application called a MySQL database to create a new user
Auto-instrumentation covers most cases and adds Splunk APM instrumentation to your application or service. However, you might want to use manual instrumentation if any of the following apply to you:
- auto-instrumentation does not capture relevant distributed traces
- there is missing metadata relevant to your business logic
- you don't have expected level of tracing data
Manual (custom) instrumentation
With manual instrumentation, you modify application code to selectively capture trace spans in interesting areas. You can also add relevant span metadata tags around blocks of code that you want to instrument. Manually adding spans in this way gives you flexibility of capturing what you need and allows for capturing custom metadata. This method is the only way to add instrumentation and capture detail in first-party business logic.
You can mix automatic and manual instrumentation. Even when your library is supported by auto-instrumentation, manual instrumentation can be added to get more insight. Auto instrumentation data can be augmented by the addition of manually captured spans and tags that add more detail.
An example of the kind of information that could be captured by a hybrid approach could be where a Java application makes a call to a MySQL database to create a new user (auto), and the new user belongs to a certain customer (manual).
Do not mix automatic instrumentation from two different vendors.
If you found this article useful and want to advance your skills, Splunk Education offers a 3-hour, instructor-led course on Manual Instrumentation with Splunk APM. The hands-on labs in the course will teach you how to:
- Describe what instrumentation is
- Describe what traces and spans are
- Describe instrumentation options
- Manually create spans
- Add span tags
- Describe interoperability
- Enable interoperability between applications
Click here for the course catalog where you can read the details about this and other Splunk APM courses, as well as register.