### What logs and metrics cannot tell you Your payment service is slow. Error rate is 4%. Users are complaining. You open Grafana and the dashboard confirms it — p99 latency is at 6 seconds instead of the normal 200ms. You open the logs and see timeouts. You check CPU and memory — both normal. You know *something* is wrong. You do not know *where*. The payment service makes calls to three other services — auth, inventory, and the payment gateway. One of those three is causing the slowdown. But your logs only show what happens inside the payment service. Your metrics tell you the aggregate is bad. Neither one tells you which downstream call is eating 5.8 seconds of that 6-second response time. This is the gap that distributed tracing fills. ### What distributed tracing adds **Distributed tracing** records the full journey of a request as it travels through your services. Instead of seeing what happened inside one service, you see what happened across all of them — in sequence, with exact timing. Here is what a trace shows you for that slow payment request: ```text POST /payments/process (6,020ms total) ├── auth.validate_token 45ms ✓ ├── inventory.check_stock 38ms ✓ ├── payment-gateway.charge 5,890ms ← THIS ONE └── db.save_transaction 47ms ✓ ``` In 10 seconds you know the answer. The payment gateway call is taking 5.8 seconds. Everything else is fine. You do not need to check logs from four different services or guess which one is slow. This is why distributed tracing has become a standard part of every production ops stack. ### The four signals and when to use each Distributed tracing is one of four observability signals. Each one answers a different question. Understanding when to reach for which one prevents wasted time during incidents. | Signal | Question it answers | Best used for | |--------|-------------------|--------------| | **Metrics** | Is something wrong right now? How bad is it? | Alerts, dashboards, trend analysis | | **Logs** | What happened at this moment in the code? | Debugging specific decisions, audit trails | | **Traces** | Which service call was slow or broken? | Latency debugging, dependency mapping | | **Errors** | What exception was thrown and where? | Crash investigation, error grouping | The Sentry engineering team describes it well: metrics tell you the rate is wrong, logs tell you what the code decided, traces tell you where the time went. None of these replaces the others. Used together they give you the complete picture. A real incident workflow looks like this: Metric alert fires (something is wrong) | v Check traces (which service call is slow/broken) | v Check logs at the failing service (what did the code decide and why) | v Root cause found > 📌 **Remember:** Traces are sampled — you do not keep every single one. Logs are not sampled — you keep all of them. This is why you use traces to find the slow request and logs to understand why it made a specific decision. Do not try to use traces as a substitute for logs. ---
### What a trace is A **trace** is the complete record of one request traveling through your system, from the moment it arrives to the moment the response is returned. Think of it like a receipt that follows a customer's order through a Swiggy restaurant — it records every station the order passed through, how long each station took, and in what order. When something goes wrong with the order, you can read the receipt instead of asking every station what happened. In software: one user clicks "Pay Now" on Razorpay → one trace records everything that happens until the payment confirmation is shown. ### What a span is A **span** is one unit of work within a trace. Every time a service does something meaningful — receives a request, calls another service, queries a database — it creates a span. A span records: * What operation it represents (name) * When it started and how long it took (timestamp and duration) * Whether it succeeded or failed (status) * Any additional context (attributes like user ID, HTTP method, DB query) The trace from earlier has five spans: ```text Span 1: POST /payments/process (the whole request) Span 2: auth.validate_token (child of span 1) Span 3: inventory.check_stock (child of span 1) Span 4: payment-gateway.charge (child of span 1) Span 5: db.save_transaction (child of span 1) ``` Span 1 is the **root span** — it represents the entire request. Spans 2-5 are **child spans** — they represent work that happens as part of the root request. ### How trace ID connects everything across services Each trace has a unique **trace ID** — a random string like `a3ce929d0e0e4736f03067aa0ba902b7`. This ID is passed along with every request across service boundaries. When the payment service calls the auth service, it includes the trace ID in the request header. The auth service creates its own span and attaches that same trace ID to it. When both spans land in the tracing backend, they are connected by the shared trace ID into one unified trace. This is the key mechanism. Without the trace ID being passed between services, you would have isolated spans with no way to connect them. ``` Payment Service Auth Service | | Creates span (trace_id=abc123) | | | Calls auth with trace_id=abc123 ---> | Creates span (trace_id=abc123, parent=payment) | Returns response | Trace backend connects spans by trace_id=abc123 ``` > **Note:** The trace ID travels in an HTTP header called `traceparent`. The W3C TraceContext standard defines its format: `version-trace_id-parent_span_id-flags`. For example: `00-a3ce929d0e0e4736f03067aa0ba902b7-f03067aa0ba902b7-01`. You rarely need to work with this directly — the OpenTelemetry SDK handles it automatically. ### Parent and child spans — the waterfall The parent-child relationship between spans creates the waterfall view you see in tracing tools like Jaeger or Grafana Tempo. ``` Root span: POST /payments/process ████████████████████████ 6020ms auth.validate_token ██ 45ms inventory.check_stock ██ 38ms payment-gateway.charge ██████████████████ 5890ms db.save_transaction ██ 47ms ``` Each bar starts at the moment the operation began (relative to the root span start) and ends when it completed. Overlapping bars mean operations ran in parallel. Sequential bars mean they ran one after another. This waterfall view is how you instantly see which span is slow — it is the longest bar in the chart. ---
### The mechanism that connects spans across services **Context propagation** is how trace information travels from one service to another. Without it, each service creates isolated spans that cannot be connected into a trace. The mechanism is straightforward: when Service A calls Service B, it injects the current trace context into the outgoing request headers. Service B extracts that context from the incoming headers, creates a new span as a child of the incoming context, and does the same when it calls Service C. ``` Service A Service B Service C | | | Creates root span | | (trace_id=xyz, span_id=001) | | | | | HTTP call with header: | | traceparent: 00-xyz-001-01 -> | | Extracts context | Creates child span | (trace_id=xyz, span_id=002, | parent_id=001) | | | HTTP call with header: | traceparent: 00-xyz-002-01 -------> | Extracts context Creates child span (trace_id=xyz, span_id=003, parent_id=002) ``` All three spans share trace_id=xyz. The tracing backend reconstructs the full trace from these three spans. ### What breaks when propagation fails If any service in the chain does not pass the `traceparent` header forward, the trace breaks. Downstream spans cannot connect to the root, and you see orphaned spans in your tracing backend instead of a complete trace. Common reasons propagation breaks: * A service uses an HTTP client that does not auto-propagate headers * An async job or message queue does not carry the trace context * A load balancer or proxy strips custom headers * A new service was added without OTel instrumentation When you see broken or incomplete traces in Jaeger or Tempo, the first thing to check is where the `traceparent` header stops being passed. > 🔴 **Common Mistake:** Forgetting to propagate context through message queues and async jobs. When the payment service puts a job on a Kafka queue and a worker picks it up, the worker is part of the same logical operation — but it runs in a separate process. Without explicit context injection into the Kafka message and extraction in the worker, the trace breaks here and you lose visibility into async operations. ---
### Installing the SDK OpenTelemetry for Python is split into packages. You install what you need: ```bash ## Core SDK — needed for all instrumentation pip install opentelemetry-api opentelemetry-sdk ## Exporter to send traces to a collector or Jaeger pip install opentelemetry-exporter-otlp ## Auto-instrumentation for Flask web framework pip install opentelemetry-instrumentation-flask ## Auto-instrumentation for outgoing HTTP requests pip install opentelemetry-instrumentation-requests ## Semantic conventions — standard attribute names pip install opentelemetry-semantic-conventions ``` > **Note:** The OTel Python SDK is split into `opentelemetry-api` and `opentelemetry-sdk`. The API defines the interfaces. The SDK implements them. Your application code imports from the API. The SDK is configured at startup. If no SDK is configured, all API calls are no-ops — this means you can instrument a library with the OTel API without forcing its users to use OTel if they do not want to. ### Setting up a tracer and creating your first spans ```python # tracing_setup.py # Configure OpenTelemetry once at application startup # Then use the tracer anywhere in your code from opentelemetry import trace from opentelemetry.sdk.trace import TracerProvider from opentelemetry.sdk.trace.export import BatchSpanProcessor from opentelemetry.exporter.otlp.proto.grpc.trace_exporter import OTLPSpanExporter from opentelemetry.sdk.resources import Resource def setup_tracing(service_name: str, otlp_endpoint: str = "http://localhost:4317"): """ Initialize OpenTelemetry tracing for a service. Call this once at application startup before any requests are handled. service_name: identifies this service in the tracing backend otlp_endpoint: where to send spans (OTel Collector or Jaeger) """ # Resource describes this service to the tracing backend # service.name is the most important attribute — it appears in Jaeger/Tempo resource = Resource.create({ "service.name": service_name, "service.version": "1.0.0", "deployment.environment": "production" }) # TracerProvider is the central object that manages tracing provider = TracerProvider(resource=resource) # OTLPSpanExporter sends spans to the OTel Collector (or directly to Jaeger) # BatchSpanProcessor buffers spans and sends them in batches # — much more efficient than sending one span at a time # insecure=True disables TLS — required for local Jaeger/Collector without certificates # Remove insecure=True in production and configure proper TLS exporter = OTLPSpanExporter(endpoint=otlp_endpoint, insecure=True) provider.add_span_processor(BatchSpanProcessor(exporter)) # Set as the global provider so trace.get_tracer() works anywhere in the app trace.set_tracer_provider(provider) return trace.get_tracer(service_name) ``` Using the tracer in your service code: ```python # payment_service.py # Example: instrumenting a payment processing function from opentelemetry import trace from opentelemetry.trace import Status, StatusCode # Get a tracer — the name identifies which part of the code created the span tracer = trace.get_tracer("payment-service") def process_payment(user_id: str, amount: float, card_token: str) -> dict: """Process a payment request and return the result.""" # start_as_current_span creates a span for this block of code # The span automatically closes when the with block exits # even if an exception is raised with tracer.start_as_current_span("process_payment") as span: # Attributes add context to the span — visible in Jaeger/Tempo # Use semantic conventions for standard fields (http.method, db.system) # Use your own names for business-specific attributes span.set_attribute("user.id", user_id) span.set_attribute("payment.amount", amount) span.set_attribute("payment.currency", "INR") try: # Each significant sub-operation gets its own child span # This is what creates the waterfall view with tracer.start_as_current_span("validate_card") as validate_span: result = validate_card_token(card_token) validate_span.set_attribute("card.valid", result["valid"]) with tracer.start_as_current_span("charge_gateway") as gateway_span: charge = call_payment_gateway(amount, card_token) gateway_span.set_attribute("gateway.transaction_id", charge["txn_id"]) gateway_span.set_attribute("gateway.response_code", charge["code"]) # Mark span as successful span.set_status(Status(StatusCode.OK)) return {"status": "success", "transaction_id": charge["txn_id"]} except Exception as e: # Record the exception so it appears in the trace span.record_exception(e) span.set_status(Status(StatusCode.ERROR), str(e)) raise ``` ### Adding attributes and events to spans Attributes and events make spans rich with context. Without them, you know a span was slow but not why. ```python from opentelemetry import trace tracer = trace.get_tracer("order-service") def check_inventory(product_id: str, quantity: int) -> bool: """Check if sufficient inventory exists for an order.""" with tracer.start_as_current_span("check_inventory") as span: # Attributes: key-value pairs that describe the operation # These are searchable in Jaeger and Grafana Tempo span.set_attribute("product.id", product_id) span.set_attribute("requested.quantity", quantity) available = get_stock_from_db(product_id) span.set_attribute("available.quantity", available) # Events: timestamped markers for significant moments within a span # Think of them as lightweight log lines attached to the span if available < quantity: span.add_event("insufficient_stock", { "product_id": product_id, "requested": quantity, "available": available }) return False span.add_event("inventory_confirmed") return True ``` > **Note:** The difference between attributes and events: attributes describe the span as a whole (what was the user ID, what was the HTTP method). Events mark specific moments within the span's lifetime (when did the cache miss happen, when did the retry trigger). Use attributes for facts about the operation, events for things that happened during it. ### Auto-instrumentation for Flask and outgoing requests Manual instrumentation gives you full control but takes time. Auto-instrumentation wraps popular frameworks automatically so you get traces with no extra code. ```python # app.py # Flask application with automatic OTel instrumentation from flask import Flask, request, jsonify from opentelemetry.instrumentation.flask import FlaskInstrumentor from opentelemetry.instrumentation.requests import RequestsInstrumentor import requests from tracing_setup import setup_tracing app = Flask(__name__) # Initialize tracing BEFORE instrumenting # The service name appears in Jaeger and Tempo tracer = setup_tracing( service_name="payment-api", otlp_endpoint="http://otel-collector:4317" ) # FlaskInstrumentor automatically creates a span for every HTTP request # — you get route, method, status code, and duration without any extra code FlaskInstrumentor().instrument_app(app) # RequestsInstrumentor automatically creates a span for every outgoing # HTTP call made with the requests library, and propagates trace context # so downstream services see the correct parent span RequestsInstrumentor().instrument() @app.route("/payments/process", methods=["POST"]) def process_payment(): data = request.json # This span is created automatically by FlaskInstrumentor # You can enrich it by getting the current span # trace.get_current_span() is the correct way — do not call it on the tracer object current_span = trace.get_current_span() # This outgoing call automatically gets a child span with traceparent header # The auth service receives the header and creates its span as a child auth_response = requests.post( "http://auth-service/validate", json={"token": data.get("token")} ) if not auth_response.json().get("valid"): return jsonify({"error": "unauthorized"}), 401 # Process payment logic here return jsonify({"status": "success"}) ``` ---
### What the collector does and why you need it The **OpenTelemetry Collector** is a standalone service that sits between your instrumented applications and your observability backends (Jaeger, Grafana Tempo, Prometheus, etc.). Without the collector, every service needs to know about every backend — if you have 20 services and want to send traces to Jaeger and metrics to Prometheus, each service needs both exporters configured. Change backends and you update 20 services. With the collector: ``` Service A \ Service B > ---> OTel Collector ---> Jaeger (traces) Service C / |----------> Prometheus (metrics) |----------> Grafana Loki (logs) ``` Every service sends data to the collector in one standard format (OTLP). The collector handles routing, filtering, and sending to the right backends. Change backends by updating one collector config. ### Receivers, processors, and exporters explained The collector has three stages. Data flows through them in order: **Receivers** — how data gets in. Your services push OTLP data to the receiver. The collector can also receive data from Prometheus, Jaeger, Zipkin, and other formats. **Processors** — what happens to data in transit. You can batch spans for efficiency, filter out noise, add attributes like the environment or cluster name, or sample high-volume data before it reaches the backend. **Exporters** — where data goes. One pipeline can fan out to multiple destinations simultaneously. The data pipeline looks like this: ``` Application (sends OTLP) | v [Receiver] receives OTLP on port 4317 (gRPC) or 4318 (HTTP) | v [Processor] batch, filter, add attributes | v [Exporter] send to Jaeger, Tempo, Prometheus, etc. ``` ### A complete collector config for an AIOps ops stack ```yaml ## otel-collector-config.yaml ## A production-ready collector config for an AIOps environment ## Receives traces and metrics from services, sends to Jaeger and Prometheus receivers: otlp: protocols: grpc: ## Services send traces and metrics to this port endpoint: 0.0.0.0:4317 http: ## Alternative HTTP endpoint for services that prefer it endpoint: 0.0.0.0:4318 ## Also scrape Prometheus metrics from the collector itself ## Useful for monitoring collector health prometheus: config: scrape_configs: - job_name: otel-collector scrape_interval: 10s static_configs: - targets: [localhost:8888] processors: ## Batch spans before sending — much more efficient than one at a time ## send_batch_size: send when this many spans accumulate ## timeout: send even if batch isn't full after this long batch: send_batch_size: 1000 timeout: 10s ## Add metadata to every span — useful for identifying source ## This adds cluster and environment to every span and metric resource: attributes: - key: cluster value: prod-mumbai action: insert - key: environment value: production action: insert ## Memory limiter prevents the collector from OOM crashing ## under high load — essential for production memory_limiter: check_interval: 5s limit_mib: 512 ## restart if collector uses more than 512MB spike_limit_mib: 128 exporters: ## Send traces to Jaeger (for development/internal use) otlp/jaeger: endpoint: jaeger:4317 tls: insecure: true ## remove this in production — use proper TLS ## Send traces to Grafana Tempo (for production) otlp/tempo: endpoint: tempo:4317 tls: insecure: true ## Expose metrics in Prometheus format for scraping prometheus: endpoint: 0.0.0.0:8889 namespace: otel ## Debug exporter — prints spans to stdout, useful during development ## Remove this in production debug: verbosity: basic extensions: ## Exposes a health endpoint at http://localhost:13133 ## Required for the curl health check below and for Kubernetes liveness probes health_check: endpoint: 0.0.0.0:13133 service: pipelines: ## Traces pipeline: receive from services, add metadata, send to Jaeger and Tempo traces: receivers: [otlp] processors: [memory_limiter, resource, batch] exporters: [otlp/jaeger, otlp/tempo] ## Metrics pipeline: receive from services and self-monitoring metrics: receivers: [otlp, prometheus] processors: [memory_limiter, resource, batch] exporters: [prometheus] ## Extensions must be listed here to be active — defining them above is not enough extensions: [health_check] ``` Run the collector with Docker during development: ```bash ## Run the OTel Collector with your config docker run -d \ --name otel-collector \ -p 4317:4317 \ ## gRPC OTLP receiver -p 4318:4318 \ ## HTTP OTLP receiver -p 8889:8889 \ ## Prometheus metrics exporter -v $(pwd)/otel-collector-config.yaml:/etc/otelcol/config.yaml \ otel/opentelemetry-collector-contrib:latest ## Check the collector is healthy curl http://localhost:13133/ ``` ---
### Jaeger for local development **Jaeger** is an open source distributed tracing backend originally built by Uber. It is the standard choice for local development and small to medium production setups. Run Jaeger locally in one command: ```bash ## Run Jaeger all-in-one — includes UI, collector, and storage ## The UI is at http://localhost:16686 ## Traces are received on port 4317 (OTLP) or 14268 (Jaeger native) docker run -d \ --name jaeger \ -p 16686:16686 \ ## Jaeger UI -p 4317:4317 \ ## OTLP receiver -p 14268:14268 \ ## Jaeger native receiver jaegertracing/all-in-one:latest ``` Open `http://localhost:16686`, select your service from the dropdown, click "Find Traces", and you see all traces from your service. Jaeger stores traces in memory by default — data disappears when the container restarts. For production use with Jaeger, add a persistent backend (Cassandra or Elasticsearch). ### Grafana Tempo for production **Grafana Tempo** is the production-grade tracing backend from Grafana Labs. Its key advantage is storage efficiency — it only requires object storage (S3, GCS, or a local disk), which is far cheaper than Elasticsearch or Cassandra. Tempo integrates tightly with Grafana dashboards and Grafana Loki (log storage). This means you can jump from a slow trace directly to the logs from that time window in the same UI. For teams already using the Grafana stack (Prometheus + Grafana + Loki), Tempo is the natural choice for traces. The complete Grafana observability stack covers all three pillars in one place. ### What to look for in any backend | Requirement | Why it matters | |------------|---------------| | Native OTLP support | No translation layer needed — send directly from collector | | Trace to logs linking | Jump from a slow trace to the logs from the same time | | Search by attributes | Find traces by user ID, error type, service name | | Retention policy | How long traces are kept — balance cost vs debugging needs | | Sampling support | High-volume environments need tail-based sampling at the backend | ---
What logs and metrics cannot tell you Your payment service is slow. Error rate is 4%. Users are complaining. You open Gr...
What a trace is A trace is the complete record of one request traveling through your system, from the moment it arrives ...
The mechanism that connects spans across services Context propagation is how trace information travels from one service ...
Installing the SDK OpenTelemetry for Python is split into packages. You install what you need: > Note: The OTel Python S...
What the collector does and why you need it The OpenTelemetry Collector is a standalone service that sits between your i...
Jaeger for local development Jaeger is an open source distributed tracing backend originally built by Uber. It is the st...
Why you cannot trace everything In a high-traffic system, tracing every single request is not practical. A service handl...
Reading a trace waterfall When you open a trace in Jaeger or Grafana Tempo, you see the waterfall view — a timeline of a...
Why standardized attribute names matter When you add attributes to spans, you could name them anything: All three record...
What you will build A simple Flask payment API fully instrumented with OpenTelemetry. Traces sent to a local Jaeger inst...
Why correlation matters A trace on its own tells you one request was slow. Connecting it to metrics tells you how many r...
OTel command and concept reference Concept What it is TracerProvider Central object that manages tracing — configure onc...
Aligns directly with DevOps, Site Reliability (SRE), and Platform Engineering job descriptions.