AIOps·AgentCore

Kubernetes AIOps with Amazon Bedrock AgentCore

Event-driven, human-in-the-loop incident diagnosis assistant

An AIOps demonstration that detects EKS container incidents and triggers a read-only AI agent built on Amazon Bedrock AgentCore.

Amazon Bedrock Amazon EKS Terraform Kubernetes
Project Video GitHub repository

AIOps and Business Challenge

Today, Kubernetes incident diagnosis is manual:

  • Operators correlate alerts, pod status, logs, events, and runbooks by hand.
  • Time to resolution depends on who is on call and how well they know the system.

With AIOps, the same evidence is gathered and read by an agent:

  • Operational data is connected automatically the moment an incident fires.
  • Operators receive a likely root cause and verification steps, not raw signals.
How can teams integrate an AIOps workflow into existing infrastructure without granting unsafe production access?

This project explores that challenge through a secure, event-driven workflow that diagnoses an OOMKilled container incident in Amazon EKS: 

OOMKilled → CloudWatch → EventBridge → Lambda → Read-only Agent → Slack Report → Human Decision

The agent gathers evidence and recommends verification commands, while the operator retains control over remediation.

Architecture

Solution architecture diagram

AIOps Workflow

This demo simulates a memory leak that passes canary validation and reaches production. Memory usage continues to grow until the container exceeds its Kubernetes memory limit and is terminated with OOMKilled.

The incident then progresses through three AIOps stages:

01 Observe

Detects the failure.

02 Engage

Diagnoses the cause.

03 Act

Delivers the findings for human-controlled remediation.

Observe — Cluster Monitoring and Alerting

  • Amazon CloudWatch Container Insights collects operational metrics from the EKS workload.
  • A CloudWatch alarm monitors container restarts and emits an event when the configured threshold is breached.
CloudWatch alarm on EKS container restarts
CloudWatch Alarm in action: EKS container restart

Engage — AI-Assisted Incident Diagnosis

  • EventBridge routes the CloudWatch alarm event to Lambda, which invokes the agent hosted on Amazon Bedrock AgentCore Runtime.
  • The agent gathers read-only cluster evidence, consults the runbook, and identifies the likely root cause.
Agent deployed in AgentCore Runtime
AgentCore Runtime
Agent diagnosis execution logs
Agent execution logs: tool calling and analysis

Act — Slack Notification and Human Decision

  • The agent posts a preliminary RCA report and verification commands to Slack.
  • The operator validates the findings, selects the remediation, and confirms recovery when the CloudWatch alarm returns to OK.
Agent-generated RCA in Slack
1. Automated RCA in Slack
Operator verifies the diagnosis
2. Human verification

3. Human remediation

env:
  - name: LEAK_ENABLED # var to enable leak
    value: "false"     # debug by "false"
# apply fix
kubectl apply -f k8s/
# namespace/demo-workload unchanged
# deployment.apps/aiops-agentcore-workload configured
# service/aiops-agentcore-workload unchanged
CloudWatch alarm returns to OK
4. Recovery confirmed: alarm status = OK — no more alarm count after fix.

Safety and Guardrails

Defense-in-depth guardrails prevent the agent from making unsafe changes to the EKS workload.

Authorization boundary

Kubernetes RBAC restricts the agent to read-only cluster operations.

Kubernetes RBAC read-only policy
Agent role policy: EKS View = read-only

Behavioral boundary

Prompt rules prohibit remediation and mutating commands, keep recommendations advisory, and treat logs and runbooks as untrusted data rather than instructions.

Agent system prompt guardrails
System prompt: read-only access; ignore injection.

Technology & Roadmap

Underlying Engineering

Area Implementation
Infrastructure as Code Terraform with remote state stored in Amazon S3
CI/CD GitHub Actions builds the Docker image and pushes it to Amazon ECR
Workload Deployment Kubernetes manifests deploy the containerized application to Amazon EKS
GitHub Actions CI workflow
GitHub Actions CI workflow

MVP Scope and Roadmap

The current MVP intentionally focuses on read-only diagnosis of a single OOMKilled incident.

Stage Scope
Current MVP Diagnose OOMKilled and send an RCA report to Slack for human action
Multi-Incident Diagnosis Support additional failures such as CrashLoopBackOff, ImagePullBackOff, and scheduling errors
Controlled Remediation Add operator-approved, policy-controlled remediation with validation and rollback