Distributed tracing records network communication between application components. You add code to your components to identify the transactions you want to trace, using a client library which sends details to a tracing collector. The collector stores sample data and ties together transactions which span multiple components - so you can visualize the communication in a user flow.
Jaeger is a CNCF project for which supports the OpenTracing standard. You run Jaeger collectors in containers, and scale the tracing components with a message queue and a separate data store. The Jaeger UI shows you the traffic in your app, giving you an overview of how your components connect, and letting you drill down to see durations for different parts of a transaction.
In this episode we’ll walk through the dev work you need to add tracing to your app, how you run Jaeger in a simple environment and how you run it in production with Kubernetes.
Here it is on YouTube - ECS-V3: Distributed Tracing with Jaeger and Kubernetes
Docker for .NET Apps - my new Udemy course
OpenTelemetry - standards for observability
Docker Desktop - with Kubernetes enabled (Linux container mode if you’re running on Windows).
Jaeger runs as a distributed app itself, but for a simple non-prod environment you can use the all-in-one image on Docker Hub.
We’ll run a demo app using Compose - v1.yml.
Run Jaeger on its own to start with:
docker-compose -f demo1/v1.yml up -d jaeger
Browse to the UI at http://localhost:16686
The Jaeger UI component makes requests to the query component, which records tracing information.
Now run the rest of the app:
docker-compose -f demo1/v1.yml up -d
There are two back-end services and a website:
The web app records tracing information with OpenTelemetry and Jaeger client libraries.
Refresh the Jaeger UI and search for:
The traces show the client service calls but not dependent services
Open the system architecture in Jaeger http://localhost:16686/dependencies
The demo services have code for tracing, it just needs to be enabled with a feature flag.
The v2.yml spec turns on tracing for the APIs.
Update the deployment to enable distributed tracing:
docker-compose -f demo2/v2.yml up -d
Browse to the app again at http://localhost:8080
Check the traces in Jaeger at http://localhost:16686
The spans are recorded across the services and you can map out the dependency graph.
You’ll see the services could use some caching - v3.yml turns it on for the stock API.
Update the stock API:
docker-compose -f demo2/v3.yml up -d
Try the app again and check the traces and logs for the stock calls
In production you’ll use the Jaeger Operator to deploy to Kubernetes; that requires an ingress controller.
Deploy an Nginx ingress controller:
kubectl apply -f ./demo3/ingress-controller/ kubectl get all -n ingress-nginx
Deploy the Jaeger Operator:
kubectl apply -f ./demo3/jaeger-operator/crds/ kubectl apply -f ./demo3/jaeger-operator/
The Operator creates Jaeger instances when you deploy a custom resource - jaeger.yaml is as simple as it gets.
Create a Jaeger instance:
kubectl apply -f ./demo3/jaeger/ kubectl get all kubectl get ingress
Browse to http://localhost
It’s the standard Jaeger UI - the backend is now distributed in a production configuration.
The demo app works with the same Docker images and app configuration in Kubernetes.
Deploy the demo app:
kubectl apply -f ./demo3/widgetario/ kubectl get pods kubectl get ingress
Browse to http://widgetario.local/
Refresh the app a few times, then check the traces in Jaeger at http://localhost
Hit the Reset Kubernetes button in Docker Desktop :)