Observability & SLOs (Deep Workflow)#

SLOs connect engineering work to user-perceived reliability. SLIs must be measurable from systems but grounded in user journeys.

When to Offer This Workflow#

Trigger conditions:

• Defining 99.9% without defining for what
• Too many pages or none; need error budget discipline
• Product wants features while stability degrades

Initial offer:

Use six stages: (1) pick user journeys, (2) define SLIs, (3) set SLO targets & windows, (4) error budget policy, (5) alerting on budget burn, (6) review & iterate). Confirm metric stack and dependency SLOs from vendors.

---

Stage 1: User Journeys#

Goal: Critical paths that matter if broken—checkout, login, API sync, not “CPU low”.

Output#

3–10 journeys ranked by business impact and frequency.

Exit condition: One paragraph per journey: user intent + failure symptom.

---

Stage 2: Define SLIs#

Goal: Ratio of good events over total over a window— implementation explicit.

Examples#

• Availability: successful requests / valid requests (define “valid”)
• Latency: proportion of requests faster than T ms

Good SLIs#

• Objective, low-cardinality enough to measure reliably

Exit condition: SLI formula + data source (metrics, logs, probes).

---

Stage 3: SLO Targets & Windows#

Goal: Target (e.g., 99.9% monthly) implies allowed bad minutes—make it explicit.

Practices#

• Rolling 30d common; align with release cadence
• Tier services: not everything needs same SLO

Exit condition: Published table: journey → SLI → target → window.

---

Stage 4: Error Budget Policy#

Goal: What we do when budget is healthy vs exhausted.

Policy ideas#

• Budget healthy → ship features; low → freeze risky changes, focus on reliability
• Escalation when budget burns fast (multi-window alerts)

Exit condition: Written policy with product sign-off.

---

Stage 5: Alerting on Burn#

Goal: Page on budget burn rate, not every blip— multi-window multi-burn-rate pattern when using Google-style SLO alerting.

Practices#

• Fast burn = page soon; slow burn = ticket/track

Exit condition: Alert rules linked to runbooks.

---

Stage 6: Review & Iterate#

Goal: SLOs drift with architecture— quarterly review; adjust targets with data.

---

Final Review Checklist#

• Journeys and SLIs tied to real user pain
• Targets realistic vs dependencies and cost
• Error budget policy agreed with product
• Alerts on burn, not noisy symptom spam
• Review cadence scheduled

Tips for Effective Guidance#

• Translate 99.9% to minutes/month of allowed badness.
• SLA (contract) vs SLO (internal)—don’t confuse.
• Dependency SLO caps what you can promise—surface that early.

Handling Deviations#

• No metrics yet: start with proxy SLI (synthetic probes) and improve instrumentation.
• Batch systems: event processing lag as SLI instead of HTTP.