{
"version": "https://jsonfeed.org/version/1.1",
"title": "Yurii Serhiichuk β Blog",
"home_page_url": "https://serhiichuk.dev/",
"feed_url": "https://serhiichuk.dev/blog/feed.json",
"description": "Articles on cloud architecture, DevOps, FinOps, and AI engineering.",
"authors": [
{
"name": "Yurii Serhiichuk",
"url": "https://serhiichuk.dev/"
}
],
"language": "en",
"items": [
{
"id": "https://serhiichuk.dev/blog/building-an-sre-agent-with-adk-and-antigravity/",
"url": "https://serhiichuk.dev/blog/building-an-sre-agent-with-adk-and-antigravity/",
"title": "Building an Autonomous SRE Agent with Google ADK and the Antigravity SDK",
"summary": "A runnable autonomous SRE agent: Google ADK for multi-agent diagnosis, the Antigravity SDK for a deny-by-default runtime. Runs locally, zero GCP credentials.",
"content_text": "One of the most stressful parts of being an on-call engineer is triaging a production incident in\nthe middle of the night. Modern distributed systems amplify the pain with the extra cognitive\noverload of, well, the distributed systems: logs scattered across dozens of microservices, deeply\nnested trace paths, and half a dozen observability dashboards you have to correlate by hand.\n\nUsually the setup makes total sense for the SREs who built it (maybe it's the cost, maybe it's\nwhichever tool the team knows best, maybe it's just optimizing for one particular concern), but it\npiles a lot of load onto whoever is on call.\n\nSo in the era of AI, it makes sense to get some help from an autonomous, smart system that's\nfine-tuned for your setup and remembers all the little bits and pieces specific to your\ninfrastructure.\n\nThis post is a complete, runnable blueprint for exactly that: an autonomous SRE agent on Google\nCloud you can host next to your main stack. I'm combining two Google frameworks that solve two very\ndifferent problems:\n\n- the [**Agent Development Kit (ADK)**](https://adk.dev/) for the multi-agent diagnostic _reasoning_, and\n- the [**Google Antigravity SDK**](https://antigravity.google/product/antigravity-sdk) for the agent _runtime_ β tool wiring, deny-by-default safety\n policies, and local simulation.\n\nThe whole stack runs locally with **zero GCP credentials** thanks to a mock-telemetry mode, so you\ncan try it in under a minute. And everything below is verified against the code in the repo, no\nhand-waving.\n\n---\n\n## The Core Architecture: Reasoning + Safety\n\nA proper SRE assistant has to get two things right at the same time: **reasoning orchestration**\n(which diagnostic step happens when) and **environmental safety** (the agent must never be able to\nmutate production while it pokes around).\n\nThe blueprint splits those concerns across four small FastAPI services that talk to each other over\nan [Agent-to-Agent (A2A) protocol](https://a2a-protocol.org/latest/), with results streamed back as\n[Server-Sent Events (SSE)](https://developer.mozilla.org/en-US/docs/Web/API/Server-sent_events).\n\n```mermaid\nflowchart TB\n User([\"π€ On-call engineer\"])\n Orch[\"π‘οΈ Orchestrator Β· Cloud Run: sre-agent
Antigravity runtime
policy = deny('*'), allow('diagnose_sre')
(sandboxed β its only move is to delegate)\"]\n SRE[\"π¬ SRE sub-agent Β· Cloud Run: sre-sub-agent
SSE endpoint /v1/agents/sre/messages
ADK workflow: TraceAnalyzer β LogCorrelator\"]\n Inv[\"π Inventory agent Β· inventory-agent\"]\n FS[(\"Firestore
topology cache + sessions\")]\n Obs[(\"βοΈ Cloud Trace Β· Logging Β· Monitoring\")]\n App[\"π Target app 'chaos monkey' Β· sre-chaos-monkey\"]\n\n User -->|\" GET /chat Β· POST prompt (SSE) \"| Orch\n Orch -->|\" diagnose_sre β A2A HTTP + SSE \"| SRE\n SRE -->|\" streamed Markdown report \"| Orch\n Orch -->|\" A2UI render + π₯ download button \"| User\n SRE -->|\" GET topology (A2A) \"| Inv\n Inv --> FS\n SRE -->|\" read-only queries \"| Obs\n App -->|\" write-only: spans + logs \"| Obs\n```\n\nI'm using the Antigravity SDK for the Orchestrator, the front-facing agent. It gives you solid\nsafety gates out of the box and is smart enough to handle the user's requests directly. Its only\nreal capability is to delegate to the read-only SRE sub-agent, and that's enforced by a\ndeny-by-default policy:\n\n```python\nsafety_policies = [\n deny(\"*\"), # nothing is allowed by default\n allow(\"diagnose_sre\") # β¦except delegating to the SRE sub-agent\n]\n```\n\nThe config above basically gates whatever tools we want to allow for the agent to use.\n\nThe actual diagnostic work is done by Google ADK agents. ADK is a code-first library for\nmulti-agent graphs, and the SRE sub-agent runs a two-node graph:\n\n- **TraceAnalyzer** scans recent trace summaries, filters for transactions that errored or breached\n the latency budget (>5000 ms), and isolates the single failing `traceId`.\n- **LogCorrelator** pulls the spans and the logs tagged with that `traceId`, then reasons over them\n with a small toolbelt (metric queries, cascade analysis, post-mortem generation) to produce the\n root-cause report.\n\nThere's also a third agent: an inventory agent that gives the SRE aggregated knowledge of the\nresources available in the GCP project. This grounds the diagnosis and keeps the agent focused on\nthe actual resources instead of wandering around too much.\n\n> **Runs anywhere, credentials optional.** Every cloud dependency (`google-adk`, `google-antigravity`,\n`google-cloud-*`, `opentelemetry`) is imported behind a `try/except ImportError` with a mock\nfallback, and a `MOCK_GCP` flag swaps live API calls for local JSON fixtures. Same code path for the\nlocal simulation and the Cloud Run deployment.\n\n---\n\n## Anatomy of a Diagnosis\n\nWhen an alert fires, here's what actually happens end to end, from the on-call prompt to a finished\npost-mortem. One thing worth calling out is the **two-tier reasoning**: with a real model key the\nfull ADK graph runs; offline it falls back to a deterministic simulated workflow that produces an\nidentically-structured report. So the demo behaves the same whether or not you have a Gemini key.\n\n```mermaid\nsequenceDiagram\n autonumber\n actor U as Engineer\n participant O as Orchestrator (sre-agent)\n participant S as SRE sub-agent (sre-sub-agent)\n participant I as Inventory agent\n participant T as Trace/Log/Metric tools\n participant M as Gemini (ADK)\n U ->> O: \"Diagnose the latency spikes and write a post-mortem\"\n O ->> S: diagnose_sre() Β· A2A POST /v1/agents/sre/messages\n S -->> O: SSE: \"π§ fetching topologyβ¦\"\n S ->> I: GET project topology\n I -->> S: services + databases (Firestore cache)\n S ->> T: query_traces(limit=10)\n T -->> S: recent trace summaries\n\n alt HAS_ADK and GEMINI_API_KEY present\n S ->> M: TraceAnalyzer β pick failing traceId\n M -->> S: traceId\n S ->> M: LogCorrelator β reason over spans + logs\n M -->> S: root-cause narrative\n else offline / no key\n S ->> S: _run_simulated_diagnostics() (deterministic)\n end\n\n S ->> T: analyze_trace_cascade() + generate_post_mortem()\n T -->> S: bottleneck table + post-mortem markdown\n S -->> O: SSE chunks β final report\n O -->> U: A2UI dashboard + π₯ Download post-mortem\n```\n\nThe Orchestrator does the one thing its policy allows: it hands the problem to the SRE sub-agent and\nsteps back. From there the sub-agent streams its progress back as Server-Sent Events, so the on-call\nengineer watches the investigation unfold live instead of staring at a spinner.\n\nIt starts by pulling the project topology (which services exist, how they call each other, and where\ntheir databases sit) from the Inventory agent's Firestore-backed cache. That map tells the diagnosis\nwhere to look. Then it fetches the recent traces and runs them through the two-node graph:\nTraceAnalyzer collapses thousands of spans down to the single failing trace worth investigating, and\nLogCorrelator pulls every span and log line sharing that trace's ID to name the actual root cause.\nNot just _\"the database was slow,\"_ but which span failed, with which error, and why.\n\nOnly then does it run the cascade analysis and draft the post-mortem, streaming the finished report\nback through the Orchestrator to the chat UI, where it lands as a rendered dashboard with a one-click\ndownload.\n\nThe nice thing about this setup is how extensible it is. Adding a new sub-agent or expanding an\nexisting tool is easy with ADK and the A2A protocol, so the blueprint grows with your infrastructure\ninstead of against it.\n\n---\n\n## Deep Dive: Cascade Latency & Bottleneck Analysis\n\nIn order to showcase the agent, I have also built another FastAPI service that emulates real\ndatabase errors with timeout exceptions and telemetry spans. As it is just a simulation, quite a lot\nof actual problems are quite similar to this one, and frequently the initial gateway/backend latency\nor errors are hidden downstream. But of course this is a textbook example still: a gateway request\nthat looks 10-second-slow, but where 99% of the time is actually trapped in a database call three\nlevels down.\n\n```mermaid\ngantt\n title Trace b49dβ¦ β Gateway request timeline (ms)\n dateFormat x\n axisFormat %Lms\n section /api/gateway\n inclusive 10270ms (self 20ms): active, 0, 10270\n section /api/backend\n inclusive 10250ms (self 50ms): active, 10, 10260\n section /api/database β timeout\n inclusive 10200ms (self 10200ms): crit, 30, 10230\n```\n\nThe cascade-analysis tool builds the span parent/child map and computes, for every span:\n\n- **Inclusive duration** β wall-clock time of the span including its children.\n- **Exclusive (self) duration** β the active time spent _in that span alone_:\n\n $$\\text{ExclusiveTime}(s) = \\text{InclusiveTime}(s) - \\sum_{c \\in \\text{children}(s)} \\text{InclusiveTime}(c)$$\n\nThe span with the largest exclusive time is the true bottleneck. Here's the actual, verified output\nfrom `uv run simulate_incident.py` (no edits, no GCP):\n\n```text\n### π Span Latency Breakdown\n| Service / Span Name | Span ID | Parent ID | Status | Inclusive Time | Exclusive (Self) Time | Contribution |\n| :--- | :--- | :--- | :--- | :--- | :--- | :--- |\n| /api/gateway | span-gateway-111 | None | ERROR | 10270 ms | 20 ms | 0.2% |\n| βββ /api/backend | span-backend-222 | span-gateway-111 | ERROR | 10250 ms | 50 ms | 0.5% |\n| βββ /api/database | span-database-333 | span-backend-222 | ERROR | 10200 ms | 10200 ms | 99.3% |\n\n### π¨ Identified Bottleneck\n* Bottleneck Span: /api/database (span-database-333)\n* Self-Execution Time: 10200 ms (99.3% of total trace)\n* Status: ERROR\n* Error Message: ConnectionTimeoutError: Failed to connect to db-primary.gcp.internal:5432 after 10000ms\n```\n\nThe gateway and backend both look \"slow\" at 10 s inclusive, but their _self_ time is a rounding\nerror. The agent ignores the noise and points straight at `/api/database`: 99.3% of the budget,\nburned in a single connection timeout.\n\n---\n\n## Automated Post-Mortem & One-Click Export\n\nDiagnosis is only half the job. The deliverable on-call engineers actually need is a post-mortem.\nAfter the cascade analysis, a post-mortem generator drafts a complete Markdown document with an\nIncident Overview (time, root service, Trace ID, impact duration), a Timeline, a Root Cause\nAnalysis, and a Prevention Plan, all populated from the real trace and log data.\n\nThat Markdown is then rendered through the [A2UI protocol](https://a2ui.org/):\n\n```mermaid\nflowchart TB\n R[\"SRE report markdown
# π¨ Incident Post-Mortem\"] --> T[\"translate_markdown_to_a2ui()\"]\n T --> C[\"A2UI components:
alert Β· preview Β· download_button\"]\n C --> B[\"Web chat renders
.download-pm-btn\"]\n B --> D[\"π₯ Client-side Blob download
post_mortem.md\"]\n```\n\nA server-side translator spots the post-mortem heading and appends a download-button component; the\nbrowser renders it as a styled button that builds the file entirely client-side (no server\nround-trip) from the Markdown it already holds.\n\nOne click exports `post_mortem.md`, ready to drop into your incident-review wiki.\n\nAnd here's how it actually looks in the deployed chat UI:\n\n\n\n---\n\n## Least-Privilege IAM on Cloud Run\n\nHanding an autonomous agent unrestricted cloud access is a non-starter. The deploy pipeline gives\neach Cloud Run service its **own service account** with the narrowest role set I could get away with:\n\n```mermaid\nflowchart TB\n subgraph W[\"βοΈ Write-only β emits telemetry\"]\n direction LR\n A[\"sre-chaos-monkey-sa
(target app)\"] --- AR[\"roles/cloudtrace.agent
roles/logging.logWriter\"]\n end\n O[(\"Cloud Trace /
Logging /
Monitoring\")]\n subgraph R[\"ποΈ Read-only β consumes telemetry\"]\n direction LR\n S[\"sre-agent-sa
(SRE diagnostics)\"] --- SR[\"roles/cloudtrace.user
roles/logging.viewer
roles/monitoring.viewer
roles/datastore.user\"]\n end\n A -. \"spans + logs\" .-> O\n O -. \"queries\" .-> S\n```\n\nThe split is the whole point: the app that _generates_ the chaos can only ever **write** telemetry,\nand the agent that _investigates_ it can only ever **read**. Neither can act on the other's plane.\n(The deploy also provisions an `inventory-agent-sa` for topology discovery and a dedicated\n`sre-build-sa` for Cloud Build, each scoped just as tightly.)\n\n---\n\n## Try It Yourself in 60 Seconds\n\nThe whole scan β correlate β analyze β post-mortem loop runs locally, with **no GCP account or\ncredentials required.**\n\n```bash\n# 1. Clone the repo\ngit clone https://github.com/xSAVIKx/sre-agent.git\ncd sre-agent\n\n# 2. Install uv and sync the workspace (app, agent, sre_agent, inventory_agent, sre_common)\npip install uv\nuv sync --all-packages\n\n# 3. Run the incident simulation\nuv run simulate_incident.py\n```\n\nThe simulation triggers a database-timeout incident in the target app, writes mock traces and logs\nto `mock_telemetry_data/`, boots the Orchestrator in mock mode, runs the diagnostic workflow\nin-process, and prints the report to your terminal. You'll see the structured telemetry logs (gateway\nβ backend β database, ending in a `CRITICAL ConnectionTimeoutError`) followed by the full diagnosis:\nthe **99.3% `/api/database` bottleneck table** and the complete `# π¨ Incident Post-Mortem` shown\nabove.\n\nWant the full multi-service experience with the web chat UI? `docker-compose up --build` brings up\nthe Orchestrator, SRE agent, Inventory agent, a Firestore emulator, and the target app together, then\nyou open the chat at `/chat`.\n\n---\n\n## Project Resources\n\nThe complete, runnable source, plus a step-by-step [**CODELAB**](https://github.com/xSAVIKx/sre-agent/blob/master/CODELAB.md) that builds this from\nscratch, lives\non GitHub:\n\n**π [github.com/xSAVIKx/sre-agent](https://github.com/xSAVIKx/sre-agent)**",
"date_published": "2026-07-09T00:00:00.000Z",
"tags": [
"SRE",
"AI Agents",
"Google Cloud",
"Google ADK",
"DevOps",
"Observability"
],
"image": "https://serhiichuk.dev/_astro/og.DyRRkI16.png"
}
]
}