First-pass investigation when an alert pages on-call
An AI SRE agent pulls metrics, logs, deploys and past incidents when a page fires, and posts likely causes for the responder.
The use case
When an alert wakes someone at 3am, the first twenty minutes go to gathering context: what changed, which service, have we seen this before. AI SRE agents from Datadog, incident.io and AWS start that work as soon as the alert fires. They query telemetry, look at recent deploys and config changes, compare with past incidents, and post ranked hypotheses with links to the evidence in the incident channel. incident.io notes its agent can draft a fix but will not ship changes on its own. The responder still confirms the cause, chooses the mitigation, and runs any rollback or production change.
Example workflow
- Connect telemetry, deploy history and chat to the agent
- Agent starts investigating when an alert triggers
- Agent posts hypotheses with evidence links
- Responder checks the evidence and picks a mitigation
- Responder runs the fix; agent notes go into the timeline
Tools
- Datadog Bits AI
- incident.io
- AWS DevOps Agent
- Slack
Risks and guardrails
- Confident but wrong hypotheses can send responders down the wrong path
- Read-only access by default; production changes need a human
- Telemetry can include customer data; review what the agent can read
Sources
Written by Agents Lookup. Credits and official pages for the tools this use case names:
- Datadog docs: Bits AI investigations: docs.datadoghq.com/bits_ai/bits_investigation
- incident.io: AI SRE: incident.io/solution/ai-sre
- AWS: DevOps Agent: aws.amazon.com/devops-agent
- incident.io (official page): docs.incident.io/ai/remote-mcp (checked 2026-11-19)
- AWS DevOps Agent (official page): aws.amazon.com/devops-agent (checked 2026-11-21)
- Slack MCP server (official page): docs.slack.dev/ai/slack-mcp-server (checked 2026-11-09)