Go Web Applications Tracing Database and HTTP Client Calls with OpenTelemetry

Tracing Go Web Applications Database and HTTP Client Calls

In today's complex, distributed systems, understanding the flow of requests and pinpointing performance bottlenecks is paramount. Go web applications, often interacting with databases and external HTTP services, are no exception. Without proper observability, debugging issues can become a time-consuming and frustrating endeavor, leading to increased mean time to resolution (MTTR) and customer dissatisfaction. This is where distributed tracing comes into play. By instrumenting our applications, we can gain invaluable insights into the lifetime of a request as it traverses various components. This article will delve into the practical aspects of manually integrating OpenTelemetry into Go web applications to trace database queries and HTTP client calls, offering a granular view of our application's behavior.

Understanding the Pillars of Distributed Tracing

Before diving into the code, let's establish a common understanding of the core OpenTelemetry concepts that will underpin our tracing efforts.

Trace: A trace represents a single, end-to-end transaction or request within a distributed system. It's a collection of spans, causally related, that collectively describe the entire operation.
Span: A span is a single, atomic operation within a trace. It represents a unit of work, such as an incoming HTTP request, a database query, or an outgoing API call. Each span has a name, a start time, and an end time, along with attributes that provide contextual information. Spans can be nested, forming a parent-child relationship.
Tracer: A tracer is an interface used to create spans. It's typically obtained from a TracerProvider.
TracerProvider: A TracerProvider manages and configures Tracer instances. It's responsible for setting up the tracing pipeline, including span processors and exporters.
Span Processor: A SpanProcessor is an interface that allows for processing spans before they are exported. It can perform tasks like sampling, enriching spans with additional attributes, or buffering spans.
Exporter: An Exporter sends collected telemetry data (spans, metrics, logs) to a backend system, such as Jaeger, Zipkin, or OpenTelemetry Collector.
Context Propagation: The mechanism by which trace context (containing trace ID and span ID) is passed across service boundaries. This is crucial for linking spans together to form a complete trace. In HTTP, this is typically done via request headers.

The principle behind tracing is to create a new span for each significant operation, linking it to its parent span using context propagation. This forms a directed acyclic graph (DAG) of spans, providing a detailed timeline of the request flow.

Manually Integrating OpenTelemetry for Database and HTTP Client Tracing

Our goal is to manually instrument a Go web application to trace calls to a database (e.g., PostgreSQL) and outgoing HTTP requests. We will set up a basic OpenTelemetry pipeline to export traces to a local OpenTelemetry Collector, which can then forward them to a tracing backend like Jaeger.

Let's start by setting up our OpenTelemetry collector and Jaeger locally using Docker Compose for demonstration purposes.

# docker-compose.yaml
version: '3.8'
services:
  jaeger:
    image: jaegertracing/all-in-one:1.35
    ports:
      - "6831:6831/udp" # Agent UDP
      - "16686:16686" # UI
      - "14268:14268" # Collector HTTP
    environment:
      COLLECTOR_ZIPKIN_HOST_PORT: 9411
      COLLECTOR_OTLP_ENABLED: true
  otel-collector:
    image: otel/opentelemetry-collector:0.86.0
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4317:4317" # OTLP gRPC receiver
      - "4318:4318" # OTLP HTTP receiver
    depends_on:
      - jaeger

And the OpenTelemetry Collector configuration:

# otel-collector-config.yaml
receivers:
  otlp:
    protocols:
      grpc:
      http:

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      exporters: [jaeger]

Run these with docker-compose up -d. You can then access Jaeger UI at http://localhost:16686.

Now, let's write our Go application. This example will feature a simple HTTP server that handles a request by fetching data from a PostgreSQL database and then making an external HTTP call.

First, we need to initialize our OpenTelemetry TracerProvider and configure it to export spans.

package main

import (
	"context"
	"fmt"
	"log"
	"net/http"
	"time"

	"github.com/gin-gonic/gin"
	"github.com/jackc/pgx/v5"
	"go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
	"go.opentelemetry.io/otel/trace"
	"google.golang.org/grpc"
	"google.golang.org/grpc/credentials/insecure"
)

var tracer = otel.Tracer("my-go-web-app")

// InitTracerProvider initializes the OpenTelemetry TracerProvider
func InitTracerProvider() *sdktrace.TracerProvider {
	ctx := context.Background()

	// Configure the OTLP exporter to send traces to the collector
	conn, err := grpc.DialContext(ctx, "otel-collector:4317",
		grpc.WithTransportCredentials(insecure.NewCredentials()),
		grpc.WithBlock(),
	)
	if err != nil {
		log.Fatalf("failed to dial gRPC: %v", err)
	}

	exporter, err := otlptrace.New(
		ctx,
		otlptracegrpc.WithGRPCConn(conn),
	)
	if err != nil {
		log.Fatalf("failed to create OTLP trace exporter: %v", err)
	}

	// Create a new trace processor that will export spans
	bsp := sdktrace.NewBatchSpanProcessor(exporter)

	// Define the resource that identifies this service
	res, err := resource.New(ctx,
		resource.WithAttributes(
			semconv.ServiceNameKey.String("go-web-app"),
			semconv.ServiceVersionKey.String("1.0.0"),
		),
	)
	if err != nil {
		log.Fatalf("failed to create resource: %v", err)
	}

	// Create a new TracerProvider
	tp := sdktrace.NewTracerProvider(
		sdktrace.WithBatchSpanProcessor(bsp),
		sdktrace.WithResource(res),
	)

	// Register the global TracerProvider
	otel.SetTracerProvider(tp)

	return tp
}

// simulateDBQuery simulates a database query.
func simulateDBQuery(ctx context.Context, userID string) (string, error) {
	// Create a new span for the database query
	ctx, span := tracer.Start(ctx, "db.get_user_data",
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(
			attribute.String("db.system", "postgresql"),
			attribute.String("db.statement", "SELECT * FROM users WHERE id = $1"),
			attribute.String("db.user_id", userID),
			attribute.String("database.connection_string", "user=postgres password=password dbname=test host=localhost port=5432 sslmode=disable"),
		),
	)
	defer span.End()

	// Simulate database connection and query execution
	conn, err := pgx.Connect(ctx, "postgresql://postgres:password@localhost:5433/test?sslmode=disable")
	if err != nil {
		span.RecordError(err)
		span.SetStatus(trace.StatusError, "Failed to connect to database")
		return "", fmt.Errorf("failed to connect to database: %w", err)
	}
	defer conn.Close(ctx)

	// Simulate a query delay
	time.Sleep(100 * time.Millisecond)

	var username string
	err = conn.QueryRow(ctx, "SELECT username FROM users WHERE id = $1", userID).Scan(&username)
	if err != nil {
		span.RecordError(err)
		if err == pgx.ErrNoRows {
			span.SetStatus(trace.StatusError, "User not found")
			return "", fmt.Errorf("user %s not found", userID)
		}
		span.SetStatus(trace.StatusError, "Failed to query database")
		return "", fmt.Errorf("failed to query database: %w", err)
	}

	span.SetStatus(trace.StatusOK, "Successfully retrieved user data")
	return username, nil
}

// makeExternalAPIRequest simulates an HTTP client call to an external service.
func makeExternalAPIRequest(ctx context.Context, endpoint string) (string, error) {
	// Create a new span for the HTTP client call
	ctx, span := tracer.Start(ctx, fmt.Sprintf("http.client.get_%s", endpoint),
		trace.WithSpanKind(trace.SpanKindClient),
		trace.WithAttributes(
			attribute.String("http.method", "GET"),
			attribute.String("http.url", endpoint),
		),
	)
	defer span.End()

	req, err := http.NewRequestWithContext(ctx, "GET", endpoint, nil)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(trace.StatusError, "Failed to create HTTP request")
		return "", fmt.Errorf("failed to create HTTP request: %w", err)
	}

	// Propagate trace context to the outgoing request
	otel.GetTextMapPropagator().Inject(ctx, otel.HeaderCarrier(req.Header))

	client := http.DefaultClient
	resp, err := client.Do(req)
	if err != nil {
		span.RecordError(err)
		span.SetStatus(trace.StatusError, "Failed to make HTTP request")
		return "", fmt.Errorf("failed to make HTTP request: %w", err)
	}
	defer resp.Body.Close()

	span.SetAttributes(
		attribute.Int("http.status_code", resp.StatusCode),
	)

	if resp.StatusCode != http.StatusOK {
		span.SetStatus(trace.StatusError, fmt.Sprintf("HTTP request failed with status: %d", resp.StatusCode))
		return "", fmt.Errorf("external API call failed with status: %d", resp.StatusCode)
	}

	// Simulate processing the response
	time.Sleep(50 * time.Millisecond)

	span.SetStatus(trace.StatusOK, "Successfully made external API request")
	return "External API response for: " + endpoint, nil
}

func main() {
	// Initialize OpenTelemetry TracerProvider
	tp := InitTracerProvider()
	defer func() {
		if err := tp.Shutdown(context.Background()); err != nil {
			log.Printf("Error shutting down tracer provider: %v", err)
		}
	}()

	// Database setup (for demonstration)
	// You need to have a PostgreSQL instance running,
	// e.g., with a Docker container:
	// docker run --name some-postgres -p 5433:5432 -e POSTGRES_PASSWORD=password -d postgres
	// Then connect and create a 'users' table:
	// CREATE TABLE users (id VARCHAR(255) PRIMARY KEY, username VARCHAR(255));
	// INSERT INTO users (id, username) VALUES ('123', 'john_doe');
	// INSERT INTO users (id, username) VALUES ('456', 'jane_doe');

	router := gin.Default()
	router.Use(otelgin.Middleware("my-go-web-app")) // Use Gin OpenTelemetry middleware for incoming requests

	router.GET("/user/:id", func(c *gin.Context) {
		ctx := c.Request.Context() // Get context from Gin, which already contains the trace context
		userID := c.Param("id")

		// Trace database call
		username, err := simulateDBQuery(ctx, userID)
		if err != nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
			return
		}

		// Trace an external HTTP client call
		externalAPIResponse, err := makeExternalAPIRequest(ctx, "http://jsonplaceholder.typicode.com/todos/1")
		if err != nil {
			c.JSON(http.StatusInternalServerError, gin.H{"error": err.Error()})
			return
		}

		c.JSON(http.StatusOK, gin.H{
			"message":             fmt.Sprintf("User %s found: %s", userID, username),
			"external_api_result": externalAPIResponse,
		})
	})

	log.Fatal(router.Run(":8080")) // Listen and serve on 0.0.0.0:8080
}

To run this example, you'll need the following Go modules:

go get github.com/gin-gonic/gin
go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/attribute
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace
go get go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc
go get go.opentelemetry.io/otel/sdk/resource
go get go.opentelemetry.io/otel/sdk/trace
go get go.opentelemetry.io/otel/semconv/v1.21.0
go get google.golang.org/grpc
go get go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin
go get github.com/jackc/pgx/v5

Explanation of the Code:

InitTracerProvider():
- This function sets up the OpenTelemetry tracing pipeline.
- It creates a gRPC client to connect to the OpenTelemetry Collector on otel-collector:4317.
- otlptracegrpc.New() creates an OTLP trace exporter, which sends spans over gRPC.
- sdktrace.NewBatchSpanProcessor() buffers spans and sends them in batches, improving performance.
- A resource is defined with ServiceNameKey and ServiceVersionKey to identify our application in the tracing backend.
- sdktrace.NewTracerProvider() creates the TracerProvider with the configured processor and resource.
- otel.SetTracerProvider(tp) registers this provider globally, allowing other parts of the application to obtain a Tracer.
main() function:
- Calls InitTracerProvider() at the start and ensures tp.Shutdown() is called on exit to flush any buffered spans.
- Gin Integration: router.Use(otelgin.Middleware("my-go-web-app")) leverages the OpenTelemetry Gin middleware. This middleware automatically creates a root span for each incoming HTTP request, extracts the trace context from incoming headers (if present), and injects the context into the Gin Request.Context(). This is crucial for context propagation across HTTP requests.
- The tracer variable is obtained using otel.Tracer("my-go-web-app"), picking up the globally configured TracerProvider.
simulateDBQuery() function:
- tracer.Start(ctx, "db.get_user_data", ...) manually creates a new span for the database operation.
- trace.WithSpanKind(trace.SpanKindClient) indicates that this span represents a client-side call (our application acting as a client to the database).
- trace.WithAttributes(...) adds semantic conventions and custom attributes to the span, providing rich context about the database call (system, statement, user ID). This is vital for filtering and analyzing traces in Jaeger.
- defer span.End() ensures the span is always ended, recording its duration.
- Error handling includes span.RecordError(err) and span.SetStatus(trace.StatusError, ...) to indicate errors within the trace.
makeExternalAPIRequest() function:
- Similar to the database function, a new span is created for the HTTP client call.
- otel.GetTextMapPropagator().Inject(ctx, otel.HeaderCarrier(req.Header)) is a critical step for distributed context propagation. It takes the current trace context from ctx and injects it into the outgoing HTTP request headers (e.g., traceparent). If the external service is also instrumented with OpenTelemetry, it will extract this context and continue the trace, forming a single end-to-end trace.
- Attributes like http.method and http.url provide details about the HTTP request. The HTTP status code from the response is also added as an attribute.

Observing the Traces

After running the docker-compose up -d and then starting your Go application:

Make a request to your application: curl http://localhost:8080/user/123.
Open the Jaeger UI at http://localhost:16686.
Select "go-web-app" from the service dropdown and click "Find Traces".

You should see a trace similar to this:

GET /user/ (Gin HTTP Server Span - from otelgin.Middleware)
- db.get_user_data (Our manually created database span)
- http.client.get_http://jsonplaceholder.typicode.com/todos/1 (Our manually created HTTP client span)

Each span will have detailed attributes, including the ones we set, providing a clear picture of what happened during the request. You can inspect the duration of each operation and identify potential bottlenecks.

Conclusion

Manually integrating OpenTelemetry for tracing database and HTTP client calls in Go web applications provides granular visibility into your system's behavior. By understanding core concepts like traces, spans, and context propagation, along with careful instrumentation using the OpenTelemetry SDK, you can construct a robust observability framework. This approach empowers developers to efficiently diagnose performance issues and understand the complex interactions within their distributed services, ultimately leading to more stable and performant applications. Adopting OpenTelemetry is a significant step towards achieving comprehensive system observability.