Why I Built SREK3S: An AI Incident Responder That Cannot Write to Your Cluster

Most “AI SRE agents” are catastrophic control failures dressed as features. By binding service accounts to cluster-admin and mounting raw credentials, they turn container stdout—which routinely leaks AWS keys and Postgres passwords—into an exfiltration pipeline to third-party LLMs. Worse, granting the model write authority turns a log-line prompt injection into a full cluster mutation requiring zero vulnerabilities in the model itself.

I built SREK3S to take the opposite approach: remove the capability entirely. SREK3S is a zero-trust, read-only AI incident responder. Because it holds no cluster write authority anywhere in its architecture, the question of whether the model would do something dangerous never arises—it physically can’t.

What SREK3S Fixes

Instead of relying on prompt hardening, SREK3S enforces strict, structural security boundaries:

• In-Memory Redaction: No secret crosses the network egress boundary. The Go Sentinel uses a hermetic package to scrub secrets in-memory, on the node, before any socket is opened.
• Ordered, Normative Rules: Redaction follows 11 strict rules (targeting PEM blocks, JWTs, AWS keys), executing the cross-line pass first to prevent partial unmasking.
• Zero-Write RBAC: The Sentinel is bound to a namespaced Role limited strictly to get, list, and watch. There is no ClusterRole or binding.
• Credential-Free Agent: The Python Agent runs with automountServiceAccountToken: false as UID 10001 with a read-only root filesystem and all capabilities dropped.

The Rigorous Engineering (Not Just Another Vibecoded Wrapper)

Instead of blindly piping model hallucinations to kubectl apply, SREK3S treats LLM output with extreme suspicion.

• YAML AST Validation: The verify_yaml_ast function parses original and patched documents into Abstract Syntax Trees to prove exactly one semantic field changed, catching type errors simple diffs miss.
• True GitOps Failsafes: It runs git apply --check in a materialized throwaway repo against the exact bytes provided, refusing to normalize or repair malformed output.
• Disposable POSIX Sandboxing: Analysis executes in a child process bound by strict limits (256 MiB memory, 1 CPU-second, 0 core dumps). No state survives the investigation.
• Fail-Closed Law: Incident routing is deterministic. If evidence is ambiguous (e.g., a generic CrashLoopBackOff), SREK3S refuses to guess, defaulting to a Tier-2 architectural review that dispatches a Markdown war room report without proposing a patch.

We Threw Everything At It: The Verification Gauntlet

We didn’t just write happy-path tests; we tried to break this system in every way imaginable. The codebase passes every gate we could throw at it (go vet, gofmt, -race, black, flake8, mypy, pytest), currently sitting at 1087 passing tests (179 in Go).

• The Scrubber Corpus: The memory scrubber is validated against a 46-case corpus spanning 8 groups. This includes 32 maskable secrets and a dedicated negative_controls group of 6 cases that must survive untouched—because a redaction tool that just blanks out everything is completely useless.
• Performance Bounding: We enforce a throughput budget of ≥20,000 lines/sec/core. This ensures the masking stays well inside the 2-second detection budget, with rules compiled exactly once and never recompiled in the hot path.
• Prose-to-Code Assertions: We have a build gate (test_scrubber_manifest_spec.py) that parses the normative rule table straight out of CONTRIBUTING.md and fails the build if the code’s rule IDs, order, or patterns drift from the documentation.
• RBAC Enforcement: TestSentinelRoleGrantsNoMutatingVerb parses our deployment YAML and fails the build if any verb other than get, list, or watch ever sneaks into the Sentinel’s Role.
• Catching Real Defects: We document our failures in lessons-learned.md. Our exhaustive testing caught edge cases like a CrashLoop fixture rendered undetectable by restartPolicy: Never, a Service selector that routed nowhere despite 16 green manifest tests, and a terminationGracePeriodSeconds block mistakenly placed at the container level—which text-matching tests missed, but the actual API server rejected.

SREK3S extracts the genuinely useful capabilities of LLMs—reading crash evidence and forming hypotheses—without handing over the keys to the cluster.

References & Resources

• GitHub Repository: duckiec/SREK3S: A fail-closed, AI-powered Site Reliability Engineer for your Kubernetes cluster.
• Security Architecture: Review the deep dive into the system’s threat model in the Security Invariants documentation.
• Redaction Specifications: The 11 normative masking rules are detailed in CONTRIBUTING.md.
• Lessons Learned: Explore the bugs and edge cases caught during development in lessons-learned.md.