Grafana Series Part III: Tempo - Pushing Spans Using HTTP

👉️URL: https://grafana.com/docs/tempo/latest/api_docs/pushing-spans-with-http/

📝Description:

Sometimes, using a tracking system can be daunting because it seems to require complex application instrumentation or SPAN ingestion pipelines in order to …

Sometimes using a tracking system can be daunting because you seem to need complex application instrumentation or span ingestion pipelines to push spans. This guide aims to demonstrate an extremely basic technique, i.e. use Zipkin Sink, push span with http/json from a Bash script.

Start Tempo

First, let’s start Tempo with the Zipkin receiver configured. To do this, create a configuration file, like this:

server:
  http_listen_port: 3200

distributor:
  receivers:
    zipkin:

storage:
  trace:
    backend: local
    local:
      path: /tmp/tempo/blocks

And run Tempo:

docker run -p 9411:9411 -p 3200:3200 -v $(pwd)/config.yaml:/config.yaml grafana/tempo:latest -config.file /config.yaml

Push Spans

Now Tempo is running and listening on port 9411 Zipkin spans, let’s usecurlPush a span to it.

curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -d '[{
 "id": "1234",
 "traceId": "0123456789abcdef",
 "timestamp": 1608239395286533,
 "duration": 100000,
 "name": "span from bash!",
 "tags": {
    "http.method": "GET",
    "http.path": "/api"
  },
  "localEndpoint": {
    "serviceName": "shell script"
  }
}]'

Please note thattimestampThe fields are based on microsecondFor units, it is run throughdate +%s%6NObtained.durationThe fields are also measured in microseconds, so 100000 is 100 milliseconds.

Receive Traces

The easiest way to get a trace is to execute a simple curl command on Tempo. The format returned is OTLP。

curl http://localhost:3200/api/traces/0123456789abcdef | jq

{
  "batches": [
    {
      "resource": {
        "attributes": [
          {
            "key": "service.name",
            "value": {
              "stringValue": "shell script"
            }
          }
        ]
      },
      "instrumentationLibrarySpans": [
        {
          "spans": [
            {
              "traceId": "AAAAAAAAAAABI0VniavN7w==",
              "spanId": "AAAAAAAAEjQ=",
              "name": "span from bash!",
              "startTimeUnixNano": "1608239395286533000",
              "endTimeUnixNano": "1608239395386533000",
              "attributes": [
                {
                  "key": "http.path",
                  "value": {
                    "stringValue": "/api"
                  }
                },
                {
                  "key": "http.method",
                  "value": {
                    "stringValue": "GET"
                  }
                }
              ]
            }
          ]
        }
      ]
    }
  ]
}

However, staring at a json blob in bash is not very interesting. Let’s start the Tempo query so we can visualize our traces. Tempo query is Jaeger Query of one GRPC Plugin, which can be used to query Tempo.

docker run --env BACKEND=localhost:3200 --net host grafana/tempo-query:latest

and open in the browser of your choicehttp://localhost:16686/trace/0123456789abcdefto view:

More Spans

Now that we have the basics, it’s easy to move on to building our tracks. By specifying the same trace ID and a parent span ID, we can start building a trace.

curl -X POST http://localhost:9411 -H 'Content-Type: application/json' -d '[{
 "id": "5678",
 "traceId": "0123456789abcdef",
 "parentId": "1234",
 "timestamp": 1608239395316533,
 "duration": 100000,
 "name": "child span from bash!",
  "localEndpoint": {
    "serviceName": "shell script"
  }
}]'

Now, the user interface displays:

Spans from everything

Tracking is not limited to corporate languages with complex frameworks. As you can see, it’s easy to store and track events from your JS, Python, or Bash scripts. Today you can use Tempo/distributed tracing to trace CI pipelines, long-running bash processes, python data processing flows, or anything else you can think of.

Good luck with tracing!

Grafana series of articles

Grafana series of articles