Elton's Container Show

Logo

Elton's Container Show - resources for the YouTube broadcast

View the Project on GitHub sixeyed/ecs

ECS-V3: Distributed Tracing with Jaeger and Kubernetes

Distributed tracing records network communication between application components. You add code to your components to identify the transactions you want to trace, using a client library which sends details to a tracing collector. The collector stores sample data and ties together transactions which span multiple components - so you can visualize the communication in a user flow.

Jaeger is a CNCF project for which supports the OpenTracing standard. You run Jaeger collectors in containers, and scale the tracing components with a message queue and a separate data store. The Jaeger UI shows you the traffic in your app, giving you an overview of how your components connect, and letting you drill down to see durations for different parts of a transaction.

In this episode we’ll walk through the dev work you need to add tracing to your app, how you run Jaeger in a simple environment and how you run it in production with Kubernetes.

Here it is on YouTube - ECS-V3: Distributed Tracing with Jaeger and Kubernetes

Pre-reqs

Docker Desktop - with Kubernetes enabled (Linux container mode if you’re running on Windows).

Demo 1 - Jaeger and application tracing

Jaeger runs as a distributed app itself, but for a simple non-prod environment you can use the all-in-one image on Docker Hub.

We’ll run a demo app using Compose - v1.yml.

Run Jaeger on its own to start with:

docker-compose -f demo1/v1.yml up -d jaeger

Browse to the UI at http://localhost:16686

The Jaeger UI component makes requests to the query component, which records tracing information.

Now run the rest of the app:

docker-compose -f demo1/v1.yml up -d

There are two back-end services and a website:

The web app records tracing information with OpenTelemetry and Jaeger client libraries.

Refresh the Jaeger UI and search for:

The traces show the client service calls but not dependent services

Open the system architecture in Jaeger http://localhost:16686/dependencies

Nothing.

Demo 2 - Distributed tracing

The demo services have code for tracing, it just needs to be enabled with a feature flag.

The v2.yml spec turns on tracing for the APIs.

Update the deployment to enable distributed tracing:

docker-compose -f demo2/v2.yml up -d

Browse to the app again at http://localhost:8080

Check the traces in Jaeger at http://localhost:16686

The spans are recorded across the services and you can map out the dependency graph.

You’ll see the services could use some caching - v3.yml turns it on for the stock API.

Update the stock API:

docker-compose -f demo2/v3.yml up -d

Try the app again and check the traces and logs for the stock calls

Demo 3 - Jaeger and Kubernetes

In production you’ll use the Jaeger Operator to deploy to Kubernetes; that requires an ingress controller.

Deploy an Nginx ingress controller:

kubectl apply -f ./demo3/ingress-controller/

kubectl get all -n ingress-nginx

Deploy the Jaeger Operator:

kubectl apply -f ./demo3/jaeger-operator/crds/

kubectl apply -f ./demo3/jaeger-operator/

The Operator creates Jaeger instances when you deploy a custom resource - jaeger.yaml is as simple as it gets.

Create a Jaeger instance:

kubectl apply -f ./demo3/jaeger/

kubectl get all

kubectl get ingress

Browse to http://localhost

It’s the standard Jaeger UI - the backend is now distributed in a production configuration.

The demo app works with the same Docker images and app configuration in Kubernetes.

Deploy the demo app:

kubectl apply -f ./demo3/widgetario/

kubectl get pods

kubectl get ingress

Browse to http://widgetario.local/

Refresh the app a few times, then check the traces in Jaeger at http://localhost

Teardown

Hit the Reset Kubernetes button in Docker Desktop :)

Coming next