SRE Practices (Deep Workflow)#

SRE is not “ops with a fancy title”—it is engineering reliability with explicit trade-offs between velocity and stability, measured with SLOs and managed through error budgets and toil budgets.

When to Offer This Workflow#

Trigger conditions:

• Defining or revisiting SLOs; too many pages or too few alerts
• “We need five nines” without user-visible meaning
• High toil: manual deploys, ticket-driven scaling, runbooks that never shrink
• Post-incident push for “more reliability” without cost discussion

Initial offer:

Walk through six stages: (1) user journeys & SLIs, (2) SLO targets & windows, (3) error budgets & policy, (4) alerting & on-call, (5) toil & automation, (6) continuous improvement. Confirm service tiering and business criticality.

---

Stage 1: User Journeys & SLIs#

Goal: Measure what users actually experience, not only server uptime.

Activities#

• List critical journeys: signup, pay, search, API sync, etc.
• For each, pick SLI types: availability, latency, freshness, correctness (where measurable)
• Define SLI implementation: e.g., “successful HTTP 2xx from LB / all requests excluding health checks” vs deeper synthetic probes

Good SLIs#

• Specific, measurable, aligned with pain—avoid vanity metrics

Exit condition: SLI definitions documented with data sources (metrics, logs, probes).

---

Stage 2: SLO Targets & Windows#

Goal: Set achievable targets with explicit consequences.

Process#

• Choose window: rolling 30d common; align with release cadence
• Set target (e.g., 99.9% availability) from error budget math: allowed downtime per month
• Tier services: not everything needs 99.99%

Realism#

• Account for dependencies you don’t control (public cloud, third-party APIs)—SLO cannot exceed dependency SLO unless architecture isolates failures.

Exit condition: Published SLO document per service or journey with measurement method.

---

Stage 3: Error Budget Policy#

Goal: Decide how to spend budget—feature velocity vs reliability work.

Policy Examples#

• Budget healthy → ship aggressively; budget low → freeze risky changes, focus on reliability
• Exceptions process: who can override, with what review

Communication#

• Product/engineering shared ownership of budget—not “SRE says no” in the dark

Exit condition: Written policy: what happens when budget burns at 25/50/100%.

---

Stage 4: Alerting & On-Call#

Goal: Pages are symptom-based, actionable, low noise.

Principles#

• Alert on user pain or imminent SLO threat, not every blip
• Severity maps to response: SEV1 customer-wide vs warning
• Runbooks linked; ownership clear

On-Call Health#

• Limit pages per engineer per week; track toil hours
• Post-incident follow-through to reduce repeat pages

Exit condition: Alert inventory reviewed; tuning backlog for noisy alerts.

---

Stage 5: Toil & Automation#

Goal: Reduce manual, repetitive, automatable work with measurable toil budgets.

Identify Toil#

• Frequent tickets, manual scaling, click-ops deploys, data fixes without guardrails

Remediate#

• Eliminate > automate > document—in that preference order when safe
• Self-service platforms with guardrails beat hero scripts

Exit condition: Toil reduction roadmap with owners; ideally 50% toil cap aspiration per team norm (Google SRE guideline—adapt to org).

---

Stage 6: Continuous Improvement#

Goal: Reliability work is prioritized like features.

Loops#

• Incident → action items with tracking
• Game days / failure injection where mature
• Quarterly SLO review—targets drift with product changes

---

Final Review Checklist#

• SLIs tied to user-visible outcomes
• SLO targets realistic vs dependencies
• Error budget policy agreed with product
• Alerts actionable; noise tracked
• Toil identified with automation path

Tips for Effective Guidance#

• Translate 99.9% to minutes of downtime per month—makes trade-offs concrete.
• Never promise zero incidents; promise learning and measurable improvement.
• Separate SLI (measurement) from SLO (target) from SLA (contract)—terms get confused.

Handling Deviations#

• Early startup: start with basic monitoring + incident reviews before full SLO program.
• No SRE role: practices still apply—relabel “production excellence” if needed.