Skip to main content

 

Splunk Lantern

Getting started with Hyperledger Besu

 

Hyperledger Besu is an Enterprise mainnet-compatible Ethereum client. Hyperledger Besu can send logs directly to Splunk which enables complex search, visualization, and analysis, and exposes OpenTelemetry metrics and traces.

Configuration

There are several different ways to set up Splunk Enterprise so it can interact with Besu. You can use Quickstart, which has the advantage of also running Besu, a Docker Compose environment, a Docker container, or a local Splunk installation. The Quickstart is great if you’re just getting started. You can pick the solution that fits your use case and security requirements the best.

Developer Quickstart with Splunk

To view the Quickstart network logs in Splunk:

  1. Start the Developer Quickstart with Besu. When you run the Quickstart script, you will be offered the option to select a monitoring solution as part of a step by step wizard. Please make sure to select splunk monitoring in the list offered.
  2. Open the Splunk UI.

Splunk Connect for Ethereum Docker Compose

With this method you can run a development Besu node and connect it to Splunk Enterprise, using the Splunk Connect for Ethereum demonstration Docker Compose environment provided by Splunk. You can set up the example by following these steps:

Requirements

A Splunk license is not required to use the trial version of the Splunk Docker image. The image is not suitable for production use and has restrictions on daily log volume.

 

Procedure

  1. Clone the Splunk Connect for Ethereum repository:

    git clone https://github.com/splunk/splunk-connect-for-ethereum.git
    cd splunk-connect-for-ethereum/examples/besu
    docker-compose up -d
    
  2. Start the demonstration environment by following the Splunk Connect for Ethereum repository README. Splunk Enterprise may take some time to start.

    Run docker ps and wait for the STATUS of the 3 containers to be Up [number] seconds (healthy).

    CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS                    PORTS                                                                            NAMES
    127600dd1173        splunkdlt/ethlogger:latest   "ethlogger"              53 seconds ago      Up 51 seconds (healthy)                                                                                    ethlogger
    88dfcee683c4        splunk/splunk:latest         "/sbin/entrypoint.sh…"   53 seconds ago      Up 52 seconds (healthy)   8065/tcp, 8088-8089/tcp, 8191/tcp, 9887/tcp, 9997/tcp, 0.0.0.0:18000->8000/tcp   splunk
    111b0c6d6072        hyperledger/besu:1.4.4       "besu"                   53 seconds ago      Up 52 seconds (healthy)   8545-8547/tcp, 30303/tcp                                                         besu
    

Use Splunk Enterprise as a Docker container

Requirements

Procedure

  1. Start the Splunk Enterprise container:

    docker run \
    -e SPLUNK_START_ARGS=--accept-license \
    -e SPLUNK_HEC_TOKEN=11111111-1111-1111-1111-1111111111113 \
    -e SPLUNK_PASSWORD=changeme \
    --rm \
    -p8080:8000 -p8088:8088 \
    -d \
    --name splunk-demo \
    splunk/splunk:latest
    

    Once the service is started, connect on http://localhost:8080/ and login as the admin user with a password of changeme.\

To follow the logs of the Splunk container: docker logs -f splunk-demo

2. Create the Besu index:

3. In the Splunk Web interface, navigate to the index list in the settings.

  1. Create an event index with an Index Name of besu.
  2. Leave other fields with the default values.
  3. Save the besu index.

4. Run Besu. To start a Besu node running in development mode, run the following command:

LOGGER=Splunk \
SPLUNK_URL=https://localhost:8088 \
SPLUNK_TOKEN=11111111-1111-1111-1111-1111111111113 \
SPLUNK_SKIPTLSVERIFY=true \
besu \
--network=dev \
--miner-coinbase=0xfe3b557e8fb62b89f4916b721be55ceb828dbd73 \
--miner-enabled \
--logging=trace

The environment variables specified send the Besu logs to Splunk. Only LOGGERSPLUNK_URLSPLUNK_TOKEN and SPLUNK_SKIPTLSVERIFY are required in this example. The complete list of options is in the Splunk options reference table.

5. In the Splunk Web interface, navigate to the search page. Type index="besu" in the search field. Log events sent by Besu are displayed.

You can now play with the search and other Splunk features to explore your Besu logs.

Splunk search page

 

Stop Besu with Ctrl+C. Stop the Splunk container with docker stop splunk-demo.

Run a Splunk Enterprise instance

Procedure

  1. Follow the steps in the Splunk Enterprise documentation to download, install, and run Splunk Enterprise.

  2. After logging into the Splunk Enterprise Web interface, navigate to the settings to:

    1. Create an HTTP Event Collector.
    2. Create an event index named besu.
  3. Run Besu as in step 3 in using Splunk on Docker. Set the SPLUNK_URL value to match the HTTP Event Collector address and port.

    You can display logs and use the search engine as in step 4 in using Splunk on Docker.

Splunk options reference

Name Description Required
LOGGER Set to Splunk to activate sending logs to Splunk. Yes
HOST Current host. If in a Docker environment, the default value is the docker container ID. Otherwise, the default value is localhost. No
SPLUNK_URL URL of the Splunk HTTP Event Collector. For example, use https://localhost:8088 Yes
SPLUNK_TOKEN Authentication token, usually of the form 11111111-1111-1111-1111-111111111111 Yes
SPLUNK_INDEX Index to store logs. Defaults to besu No
SPLUNK_SOURCE Source of the logs. Defaults to besu No
SPLUNK_SOURCETYPE Source type of the logs. Defaults to besu No
SPLUNK_BATCH_SIZE_BYTES Size of a log batch in bytes. Defaults to 65536 No
SPLUNK_BATCH_SIZE_COUNT Size of a log batch in number of events. Defaults to 1000 No
SPLUNK_BATCH_INTERVAL Interval at which to send log batches. Defaults to 500 No
SPLUNK_SKIPTLSVERIFY Whether to check the Splunk instance TLS certificate when sending data. Defaults to false No

Use the OpenTelemetry monitoring and tracing service

To enable OpenTelemetry to access Hyperledger Besu, use the --metrics-enabled and --metrics-protocol=opentelemetry options. Use Splunk APM or Splunk Platform to visualize the collected data. A Besu Sync example is available.

You can use OpenTelemetry to monitor the sync time of your Besu node and show where time is spent internally and over the JSON-RPC interface.

This office hours recording shows examples of monitoring Hyperledger Besu.

Download and install the OpenTelemetry Collector to get started.

You can also install exporters that send system metrics to OpenTelemetry to monitor non-Besu-specific items such as disk and CPU usage. The OpenTelemetry Collector can connect to additional applications, and may be deployed in Kubernetes environments as a daemonset.

Procedure

  1. Configure OpenTelemetry to accept data from Besu. For example, use the following configuration for your otel-collector-config.yml file, and send data to Splunk and Splunk APM:

receivers:
     otlp:    
          protocols:             
          grpc:             
          http: 
          
exporters:
     splunk_hec/traces:
              # Splunk HTTP Event Collector token.         
              token: "11111111-1111-1111-1111-1111111111113"         
              # URL to a Splunk instance to send data to.         
              endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"         
              # Optional Splunk source: https://docs.splunk.com/Splexicon:Source         
              source: "besu:traces"         
              # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype         
              sourcetype: "otlp"         
              # Splunk index, optional name of the Splunk index targeted.         
              index: "traces"         
              # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.         
              max_connections: 20         
              # Whether to disable gzip compression over HTTP. Defaults to false.         
              disable_compression: false         
              # HTTP timeout when sending data. Defaults to 10s.         
              timeout: 10s         
              # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.         
              # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.         
              insecure_skip_verify: true     
         splunk_hec/metrics:     
              # Splunk HTTP Event Collector token.         
              token: "11111111-1111-1111-1111-1111111111113"         
              # URL to a Splunk instance to send data to.         
              endpoint: "https://<SPLUNK INSTANCE>:8088/services/collector"         
              # Optional Splunk source: https://docs.splunk.com/Splexicon:Source         
              source: "besu:metrics"         
              # Optional Splunk source type: https://docs.splunk.com/Splexicon:Sourcetype         
              sourcetype: "prometheus"         
              # Splunk index, optional name of the Splunk index targeted.         
              index: "metrics"         
              # Maximum HTTP connections to use simultaneously when sending data. Defaults to 100.         
              max_connections: 20         
              # Whether to disable gzip compression over HTTP. Defaults to false.         
              disable_compression: false         
              # HTTP timeout when sending data. Defaults to 10s.         
              timeout: 10s         
              # Whether to skip checking the certificate of the HEC endpoint when sending data over HTTPS. Defaults to false.         
              # For this demo, we use a self-signed certificate on the Splunk docker instance, so this flag is set to true.         
              insecure_skip_verify: true     
          # Traces     
          sapm:         
              access_token: "${SPLUNK_ACCESS_TOKEN}"         
              endpoint: "https://ingest.${SPLUNK_REALM}.signalfx.com/v2/trace"     
          # Metrics + Events  
          signalfx:         
              access_token: "${SPLUNK_ACCESS_TOKEN}"         
              realm: "${SPLUNK_REALM}" 
              
 processors:     
     batch: 
     
 extensions:     
     health_check:     
     pprof:     
     zpages: 
     
 service:     
     extensions: [pprof, zpages, health_check]     
     pipelines:    
          traces:      
              receivers: [otlp]             
              exporters: [splunk_hec/traces, sapm]             
              processors: [batch]         
          metrics:             
              receivers: [otlp]             
              exporters: [splunk_hec/metrics, signalfx]             
              processors: [batch]

It is easiest to run the OpenTelemetry collector with Docker with the following command:

Syntax Example

docker run -d \ -v ./otel-collector-config.yml:/etc/otel/config.yaml \ -e SPLUNK_ACCESS_TOKEN=<access token> -e SPLUNK_REALM=<realm> \-p 4317:4317 \ otel/opentelemetry-collector-contrib:latest

docker run -d \ -v ./otel-collector-config.yml:/etc/otel/config.yaml \ -e SPLUNK_ACCESS_TOKEN=abcdefg654-e SPLUNK_REALM=us1-p 4317:4317 \ otel/opentelemetry-collector-contrib:latest

  1. You can also refer to this Docker-compose example.

  2. Start Besu with the --metrics-enabled and --metrics-protocol=opentelemetry options. For example, run the following command to start a single node:

Syntax Example
OTEL_EXPORTER_OTLP_ENDPOINT=https://<host>:<port> besu --network=dev --miner-enabled --miner-coinbase <COINBASE ADDRESS> --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry OTEL_EXPORTER_OTLP_ENDPOINT=https://localhost:4317 besu --network=dev --miner-enabled --miner-coinbase fe3b557e8fb62b89f4916b721be55ceb828dbd73 --rpc-http-cors-origins="all" --rpc-http-enabled --metrics-enabled --metrics-protocol=opentelemetry

The OpenTelemetry SDK mandates how to configure the OpenTelemetry gRPC client, so data flows to the collector from Besu.

You can use the following environment variables:

Name Description Required
OTEL_EXPORTER_OTLP_ENDPOINT OpenTelemetry Collector endpoint, of the form https://host:port. The default value is https://localhost:4317 Yes
OTEL_EXPORTER_OTLP_INSECURE Whether to allow insecure connections for OpenTelemetry data. False by default. No