Distributed Tracing (Deep Workflow)#

Traces answer which hop consumed time and where errors surfaced across services. Success requires consistent propagation, meaningful spans, and sampling that preserves signal without bankrupting storage.

When to Offer This Workflow#

Trigger conditions:

• Microservices “unknown latency” between A and B
• Adopting OpenTelemetry, Jaeger, Zipkin, X-Ray, Cloud Trace
• Need service map and dependency insights
• High cardinality or cost concerns from traces

Initial offer:

Use six stages: (1) define goals & SLOs, (2) instrumentation plan, (3) propagation & context, (4) sampling strategy, (5) analysis workflows, (6) governance & cost. Confirm languages and infra (K8s, service mesh).

---

Stage 1: Goals & SLOs#

Goal: Know why tracing exists— latency, errors, dependency discovery, or customer journey mapping.

Questions#

1. Top p95/p99 pain routes?
2. Compliance or PII constraints on span attributes?
3. Cardinality tolerance— user IDs on every span?

Exit condition: Success metrics: e.g., “reduce unknown time in checkout to <5% of trace duration.”

---

Stage 2: Instrumentation Plan#

Goal: Spanness where it helps— not every function.

Layers#

• HTTP server middleware: span per request, route name normalized
• HTTP clients: outgoing spans with peer service
• DB: client spans with statement type—not raw SQL text in prod by default
• Queues: produce/consume spans with message correlation
• Background jobs: separate spans with job type

Naming#

• Span names stable ( GET /orders/{id} patterns) vs high-cardinality raw paths

Attributes#

• service.name, deployment.environment, http.status_code, db.system—follow semantic conventions (OTel)

Exit condition: Inventory of frameworks auto-instrumented vs manual spans needed.

---

Stage 3: Propagation & Context#

Goal: Trace ID crosses async boundaries— no broken traces.

Practices#

• W3C Trace Context headers for HTTP; messaging propagators for Kafka/AMQP
• Async tasks: attach context when scheduling (executor, asyncio, Promise)
• Batch processing: link spans or baggage carefully—avoid leaking PII

Service mesh#

• Sidecar tracing vs library tracing—avoid double counting; configure one source of truth

Exit condition: Broken trace rate measurable; top 5 causes documented (missing propagation, etc.).

---

Stage 4: Sampling Strategy#

Goal: Representative traces without storing everything.

Head-based#

• Fixed percentage; always sample errors (tail sampling often still needed)

Tail-based#

• Interesting traces (high latency, errors) retained— complexity but better signal

Cost controls#

• Attribute limits; span limits per trace; drop health checks

Exit condition: Written policy: baseline rate + error always + latency outliers.

---

Stage 5: Analysis Workflows#

Goal: Engineers use traces in incidents and perf work.

Workflows#

• Trace view: critical path, longest child span
• Compare releases: same route, different p99 span
• Service map from edges—validate unexpected dependencies

Anti-patterns#

• Only looking at averages— trace is about specific slow requests

Exit condition: Runbook snippet: “How to find slowest span in checkout.”

---

Stage 6: Governance & Cost#

Goal: PII controlled; budget predictable.

Practices#

• PII redaction processors; secrets never in attributes
• Retention policies per env; export to cheap storage for long-term if needed
• Ownership of semantic conventions in org

---

Final Review Checklist#

• Instrumentation covers critical paths and async boundaries
• Propagation validated; broken trace rate monitored
• Sampling policy balances cost vs signal
• Semantic conventions applied consistently
• PII/secrets not in spans

Tips for Effective Guidance#

• Prefer OpenTelemetry as the single API with vendor exporters—avoid vendor lock-in at instrumentation.
• DB spans: recommend query shape (normalized) not raw SQL in prod.
• Logs ↔ traces: inject trace_id in logs for correlation.

Handling Deviations#

• Monolith: single-process traces still valuable— async and thread hops still break.
• High cardinality crisis: drop labels first, then sampling— never drop error visibility blindly.