MLOps (Deep Workflow)#

MLOps connects research velocity to production reliability: version data, code, and artifacts together; monitor behavior after deploy.

When to Offer This Workflow#

Trigger conditions:

• First production model; batch or online serving
• Drift, bias, or latency SLO misses
• Compliance needs for lineage and explainability

Initial offer:

Use six stages: (1) problem & risk class, (2) data & reproducibility, (3) training & evaluation, (4) packaging & deployment, (5) monitoring & feedback, (6) governance & rollback). Confirm batch vs real-time and regulatory tier.

---

Stage 1: Problem & Risk Class#

Goal: Align ML to decision risk (credit, health vs recommendation).

Exit condition: Offline and online success metrics defined.

---

Stage 2: Data & Reproducibility#

Goal: Snapshot training data; deterministic pipelines; PII handling.

Practices#

• Feature stores optional but valuable for consistency
• Secrets not in notebooks; orchestrated jobs

Exit condition: Run id reproduces artifact hash within agreed bounds.

---

Stage 3: Training & Evaluation#

Goal: Train/val/test without leakage; time-series splits careful.

Practices#

• Model card with limits and metrics
• Fairness slices where policy requires

---

Stage 4: Packaging & Deployment#

Goal: Immutable artifacts; canary or shadow before full cutover.

Practices#

• Model + preprocessing code version pinned together

Exit condition: Rollback to previous artifact id documented.

---

Stage 5: Monitoring & Feedback#

Goal: Data drift, concept drift, latency; business KPIs tied to model decisions.

Practices#

• Human review queue for low-confidence predictions when needed

---

Stage 6: Governance & Rollback#

Goal: Approvals for retrain/deploy; audit trail; A/B for big changes.

---

Final Review Checklist#

• Offline metrics aligned with business risk
• Data and code reproducibility
• Packaged artifacts with versioning and rollback
• Online monitoring and drift strategy
• Governance and approval path

Tips for Effective Guidance#

• Training-serving skew is a top bug—feature parity tests help.
• Offline accuracy ≠ online business outcome.
• Fairness needs explicit slices—not one headline number.

Handling Deviations#

• LLM-heavy products: lean on eval harnesses and prompt versioning (see llm-evaluation).
• Tiny teams: start with artifact registry + dashboards before a full feature store.