> workflow / monitoring
Monitoring
Observe production behavior, model quality drift, and incident signals with actionable alerts — not vanity dashboards.
What should I do next?
Complete the steps and evidence below, clear the human gates, then continue to Production.
When to use
- Service is in stage/prod and needs SLOs on the glass
- Agent/tool paths lack tracing
- You need alert routes and on-call runbooks before GA traffic
- Eval drift or cost anomalies must page someone
Step-by-step
- Confirm golden signals and AI-specific metrics (latency, errors, tokens, tool failure rate).
- Instrument tracing across agent tool calls and model providers.
- Build dashboards that map to SLOs and abort criteria — delete unused charts.
- Define alert policies with runbooks; eliminate pages without owners.
- Wire Sentry/OTel (or equivalent) for errors and traces.
- Rehearse an incident with incident.prompt; fix gaps in paging.
- Schedule drift monitors for evals and prompt versions.
Decisions to make
- What pages vs tickets vs dashboards-only
- Retention and PII redaction in logs/traces
- SEV definitions for AI quality regressions
- Who owns model-provider outage response
Evidence to collect
- Dashboards linked from runbooks
- Alert policies with owners
- Tracing enabled on agent/tool paths
- On-call runbooks for top failure modes
Role-specific next actions
- PM: Define customer-impact language for SEVs and comms triggers.
- Architect: Ensure telemetry covers new boundaries and third-party deps.
- Engineer: Add instrumentation with features; fix noisy logs.
- DevOps: Own alert routing, retention, and on-call integrations.
Inputs
- SLOs and abort criteria
- Existing dashboards and log pipelines
- On-call roster and escalation paths
Outputs
- Production dashboards
- Alert policies and runbooks
- Drift monitors for evals/prompts
Required prompts
- Incident
Structures live incident response.
- Root cause analysis
Evidence-bound post-incident analysis.
Required skills
- Observability
Primary skill for telemetry and alerting quality.
Required MCP servers
- OpenTelemetry MCP
Query traces and metrics during diagnosis.
- Sentry MCP
Error and issue triage.
- Datadog MCP
Optional if Datadog is the org standard.
- Grafana MCP
Optional dashboard operations.
Tooling & handbook
- Tracing
Agent/tool tracing standards.
- Root cause pattern
Evidence-bound incident learning.
- Observability agent
Assists dashboard/alert drafting.
- Incident response agent
Live incident assist under IC authority.
Recommended agents & patterns
- Root cause pattern
Primary pattern after incidents.
- Observability agent
Telemetry co-pilot.
- Incident response agent
SEV response assist.
Human approval gates
- Incident commander required for SEV-1/2
- Alert changes that mute pages require dual review
Automation opportunities
- Tracing for agent tool calls by default
- Eval drift monitors on prompt/version changes
- Auto-open incidents from SLO burn alerts
Enterprise best practices
- Retention and PII controls on prompts/logs
- Mandatory post-incident reviews with actions
- Vendor outage playbooks for model providers
Common mistakes
- Alert fatigue from unowned pages
- No tracing on agent/tool paths
- Dashboards without SLO linkage