Hands-On Guide to Distributed Tracing: Encore vs OpenTelemetry

Let's say you're managing an application with multiple moving parts, like an e-commerce app. In this application, you have services such as user authentication, product catalog, payment processing, order management, etc. Each of these services is deployed across different servers and potentially even different regions for redundancy and performance.

A simple workflow in your e-commerce app could be when a user logs in, browses various products, adds a few items to their cart, and completes a purchase. This single workflow triggers multiple interactions amongst the different services of your e-commerce app.

Say, for instance, the user reports that their order confirmation was delayed. With so many moving parts, how do you start examining them to find the root cause of this issue? Did the payment service take too long to respond? Could there be a delay in the order management service?

With so many services involved, understanding how they all interact can get challenging—unless, of course, you're using tracing. With tracing, you can be sure of what happened and where it was happening. How? Well, that's the focus of this article.

In this article, we'll give a quick overview of distributed tracing and demonstrate through a demo how it works in Opentelemetry and Encore. By the end of this article, you can decide from a practical standpoint what works best for you when it comes to distributed tracing.

Prerequisites

Although the demos are going to have a beginner-level approach, here are some things you'll need in order to get started:

• Go: Both demos will require you to have a basic understanding of Go and an installation of its compiler. For this, you can refer to the official Go installation documentation.
• Docker: For the Opentelemetry demo, you'll need Docker. If you don't have Docker installed, you can refer to the official Docker installation documentation.
• Familiarity with Encore and Opentelemetry: So not to get a bit confused by specific terms, it'll be helpful to know the basics of Encore and Opentelemetry.

What is distributed tracing?

Tracing in a distributed architecture refers to the process of tracking the flow of requests as they travel through various services and components within a system.

It provides a comprehensive view of the request's entire life cycle, including the time taken at each step, the services involved, and any errors or bottlenecks encountered.

With tracing, each request in your application is assigned a unique identifier, often called a trace ID. As the request moves from one service to another, this trace ID is passed along, allowing the system to track the entire journey of the request.

When you trace a request's flow through a distributed system, each step in that flow is captured as a span. Together, the spans form the complete trace, showing the request's journey across various services and operations.

Think of a trace as a map of a journey, and each span as a specific stop or action taken along the way. Each span gives detailed information about what happened at that particular step, such as the time it took, any errors that occurred, and how it fits into the overall sequence of operations.

Although you can trace manually by adding an ID to a structured log and making sure that it propagates across the network, it's often easier to use distributed tracing tools like OpenTelemetry and Encore.

Distributed tracing with Opentelemetry

Opentelemetry is a CNCF project that is focused on managing telemetry data such as traces, metrics, and logs. It provides a standardized way to instrument applications and collects trace data across various languages and platforms.

In the demo, we'll work with a basic HTTP service that makes a gRPC call to a checkout service while capturing traces with Jaeger.

To follow along with this demo, you can get the source code from GitHub by cloning the repository, courtesy of Pablo Morelli:

git clone https://github.com/pmorelli92/open-telemetry-go.git

After cloning, navigate to the gateway folder and inspect the main.go file. This contains the code for the HTTP service, and we'll break down the steps below.

Step 1: Setting up the Environment

To get started, you'll need to configure the environment variables necessary for OpenTelemetry, Jaeger, and your service addresses:

jaegerEndpoint := utils.EnvString("JAEGER_ENDPOINT", "localhost:4318")
checkoutAddress := utils.EnvString("CHECKOUT_SERVICE_ADDRESS", "localhost:8080")
httpAddress := utils.EnvString("HTTP_ADDRESS", ":8081")

This step points OpenTelemetry to the correct Jaeger backend and ensures that the services are correctly networked. If you miss this step, your traces won’t be recorded.

Step 2: Initializing the tracer

Next, you need to set up the global tracer that OpenTelemetry will use to monitor requests throughout the system:

err := utils.SetGlobalTracer(context.TODO(), "gateway", jaegerEndpoint)
if err != nil {
    log.Fatalf("failed to create tracer: %v", err)
}

With this step, you initialize the tracer with the correct backend, in this case, Jaeger.

Step 3: Setting up gRPC with tracing

To enable tracing for your gRPC client calls, you must integrate OpenTelemetry's gRPC interceptors:

conn, err := grpc.Dial(
    checkoutAddress,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()))

These interceptors automatically attach trace information to each gRPC request. This ensures that the distributed trace captures the entire request flow, including interactions with the gRPC service.

Step 4: Handling HTTP requests with tracing

In the checkoutHandler function, we trace the HTTP request and the subsequent gRPC call to the checkout service:

tr := otel.Tracer("http")
ctx, span := tr.Start(r.Context(), fmt.Sprintf("%s %s", r.Method, r.RequestURI))
defer span.End()

In this step, we'll create a new span to trace the HTTP request. This span will include the details of the HTTP method and URI, giving you visibility into incoming requests.

Step 5: Capturing errors and completing the trace

When making the gRPC call, any errors are captured and added to the trace:

_, err := c.DoCheckout(ctx, &pb.CheckoutRequest{ItemsID: []int32{1, 2, 3, 4}})
rStatus := status.Convert(err)
if rStatus != nil {
    span.SetStatus(codes.Error, err.Error())
    w.WriteHeader(http.StatusInternalServerError)
    return
}

This step ensures that if something goes wrong, the error is logged within the span, allowing you to identify and resolve issues faster.

Now that's out of the way, let's run this demo using Docker to access Jaeger via the browser. To do so, navigate back to the root folder and run the following command:

docker compose up -d --build

This command will kick things off by starting the services and setting up Jaeger as well as RabbitMQ. Now, you can access Jaeger via the browser through the following URL:

http://localhost:16686/search

It should look like this on your browser:

To test, let's produce a trace. We'll send a POST request to the /api/checkout endpoint using the curl command:

curl -X POST localhost:8081/api/checkout

Although it'll produce a 500 error response, a trace will be captured in Jaeger. To see this trace, select the gateway service and the POST /api/checkout operation on the left-hand side of the screen.

If you click on the trace, you'll see the full details.

Although this is very comprehensive, it took a couple of steps to set up. Imagine using this much code to instrument just two services. What if you had five, ten, or more services, which is not strange for large teams? Well, that’ll be a pain.

If you care about speed, then this solution might not be the most efficient one for you. Let's see how many steps it'll take us to set up tracing using Encore.

Distributed tracing with Encore

Encore is an easy-to-use platform for building distributed systems. It simplifies the process of creating, deploying, and managing cloud-native applications by automating a ton of its underlying infrastructure. It has several features, but for this demo, we'll focus on its built-in tracing and metrics.

Unlike Opentelemetry where you'll need a visual dashboard like Jaegar to see your traces, Encore offers you tracing, all while giving you a user-friendly dashboard.

For this demo, we'll be working with two services: an order service (Service A) and a notification service (Service B). Both services will communicate with each other to process orders and send notifications easily.

To get the code for this project, you can clone it on GitHub using the following command:

git clone https://github.com/Aahil13/Order-processing-services-with-Encore.git

Once you're done cloning, you can inspect the .go files available in the serviceA and serviceB folders. These files contain the code for the two services, which we'll break down in the steps below.

Step 1: Setting up the order service (Service A)

In the serviceA.go file, you'll find the PlaceOrder function. This function handles requests to the /serviceA/placeorder endpoint and simulates order processing.

This is what the code looks like:

package serviceA

import (
    "context"
)

// OrderRequest defines the structure for an order request
type OrderRequest struct {
    ProductID string
    Quantity  int
}

// OrderResponse defines the structure for an order response
type OrderResponse struct {
    OrderID   string
    Message   string
}

// PlaceOrder handles requests to /serviceA/placeorder.
//encore:api public method=POST path=/serviceA/placeorder
func PlaceOrder(ctx context.Context, req *OrderRequest) (*OrderResponse, error) {
    // Simulate order processing and generating an Order ID
    orderID := "ORDER12345"
    message := "Order placed successfully!"

    return &OrderResponse{OrderID: orderID, Message: message}, nil
}

This function is straightforward. It takes an OrderRequest, processes it, and returns an OrderResponse with a generated OrderID and a success message.

Step 2: Setting up the notification service (Service B)

Next, in the serviceB.go file, you'll find the SendOrderNotification function. This service communicates with Service A to place an order and then sends a notification.

Here's how it looks:

package serviceB

import (
    "context"
    "encore.app/serviceA"
)

// NotificationResponse defines the response structure
type NotificationResponse struct {
    NotificationMessage string
}

// SendOrderNotification handles requests to /serviceB/sendnotification.
//encore:api public method=POST path=/serviceB/sendnotification
func SendOrderNotification(ctx context.Context, req *serviceA.OrderRequest) (*NotificationResponse, error) {
    // Call Service A to place an order
    orderRes, err := serviceA.PlaceOrder(ctx, req)
    if err != nil {
        return nil, err
 }

    notificationMessage := "Notification: " + orderRes.Message + " with Order ID: " + orderRes.OrderID

    return &NotificationResponse{NotificationMessage: notificationMessage}, nil
}

This function calls PlaceOrder from Service A, processes the response, and returns a notification message. And that's it!

Step 3: Running the services

With both services set up, running them is as simple as using the Encore command:

encore run

This command starts both services and automatically sets up the necessary tracing. You can test the services either via the command line or directly through Encore's dashboard.

Step 4: Viewing traces in Encore's dashboard

One of Encore's most compelling features is its built-in local dashboard, where you can view traces without any additional setup.

After running the services and making requests, you can access the Encore dashboard by going to http://localhost:4000.

For example, you can send a POST request to the /serviceB/sendnotification endpoint:

curl -X POST http://localhost:4000/serviceB/sendnotification -d '{"ProductID": "123", "Quantity": 2}'

This will process the order through Service A and send a notification through Service B, with the entire flow captured in Encore's tracing system.

The trace details, including the interaction between Service A and Service B, can be viewed directly in the dashboard.

Features of Encore Beyond Tracing

From both demos, it's evident that setting up tracing in Encore takes less time and a lot less code than OpenTelemetry. It's basically done automatically.

However, distributed tracing is but one feature that Encore provides. Below, we'll look at some other features of Encore:

Internal developer support

Encore's local developer dashboard is a comprehensive support platform designed to streamline your development process.

The API Explorer tab allows you to view all your API interactions. It also offers the ability to view all traces and effortlessly filter between API endpoints. This level of visibility makes debugging and optimizing your APIs a breeze.

Take a look at the Service Catalog tab. Here, you'll find a list of all your services. A simple click on any service reveals an in-depth description, including detailed information about each request and response. This helps you understand the behavior of your services without leaving the dashboard.

But that’s not all. The Flow tab automatically generates architectural diagrams for your services. This visual representation of your service architecture is not just convenient but essential for grasping the flow of your application.

Encore delivers all this with an emphasis on simplicity, ensuring that even complex systems are easy to understand and manage.

Simplified infrastructure management

Encore's Go Backend Framework simplifies infrastructure management. Forget about manually setting up and maintaining different environments—Encore automates the whole process.

Whether you're working locally or deploying to the cloud on AWS or GCP, Encore ensures everything is in sync. No more writing endless Infrastructure as Code (IaC) scripts like Terraform.

Instead, your application code becomes the single source of truth, and Encore takes care of the rest, letting you focus on what truly matters—building features.

Secrets management

Storing sensitive data like API keys or database passwords in your source code is a bad idea, but Encore makes it easy to keep them secure. With Encore's built-in secrets manager, you can define secrets in your code as unexported structs. Encore ensures these secrets are set before running or deploying your app, preventing any security mishaps.

Managing secrets is flexible—you can use the Encore Cloud Dashboard for a visual setup or the CLI for command-line control. Secrets are encrypted with Google Cloud's Key Management Service and synced across environments, whether you’re working locally, in development, or in production. If needed, you can even override secrets locally without impacting others.

Encore makes using any of its features as easy as possible with its easy-to-understand user experience and documentation. You can get started right away using the Encore quick start guide.

Conclusion

In this article, you went through a general overview of what distributed tracing is. You saw a demo of how it works in Opentelemetry and how easy it is to replicate with Encore. Finally, we looked at some of the features of Encore that make it easier to work with distributed tracing.

If speed and efficiency are top priorities for you and your team, then using a tool that does just that is the best solution. Encore automatically handles tracing, which can come in handy when you're in a time crunch.

Like this article? Sign up for our newsletter below and become one of over 1000 subscribers who stay informed on the latest developments in the world of DevOps. Subscribe now!