[{"data":1,"prerenderedAt":2223},["ShallowReactive",2],{"article-\u002Finsights\u002Ffailure-modes-parallel-agent-sessions":3,"related-\u002Finsights\u002Ffailure-modes-parallel-agent-sessions":818},{"_path":4,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":8,"description":9,"date":10,"author":11,"image":19,"category":20,"tags":22,"body":27,"_type":811,"_id":812,"_source":813,"_file":814,"_stem":815,"_extension":816,"sitemap":817},"\u002Finsights\u002Ffailure-modes-parallel-agent-sessions","insights",false,"","Failure Modes of Parallel Agent Sessions","Running several coding agents against one codebase breaks in ways that have nothing to do with model quality. Worktrees are not an isolation boundary, a container on a Mac does not stop what you think it stops, deny rules do not survive a plumbing command, and an agent cannot be the source of evidence about its own work.","2026-10-20",{"name":12,"headshot":13,"role":14,"contact":15},"Levente Simon","\u002Fheadshots\u002FLS.jpeg","creator of dethernety",{"linkedin":16,"email":17,"twitter":18},"https:\u002F\u002Fwww.linkedin.com\u002Fin\u002Flevente-simon\u002F","levente.simon@dether.net","https:\u002F\u002Fx.com\u002FLevente_Simon","\u002Fimages\u002Fdiagram-worktree-sharing.svg",[21],"Tech Deep-Dives",[23,24,25,26],"agents","AI-native development","security architecture","DevSecOps",{"type":28,"children":29,"toc":800},"root",[30,38,48,53,58,65,70,166,176,181,190,195,203,208,216,221,234,246,251,257,278,283,288,321,377,425,444,457,463,476,484,495,508,516,552,578,584,589,610,629,666,672,677,682,687,692,698,703,708,713,719,724,734,744,754,764,774,784],{"type":31,"tag":32,"props":33,"children":35},"element","h1",{"id":34},"failure-modes-of-parallel-agent-sessions",[36],{"type":37,"value":8},"text",{"type":31,"tag":39,"props":40,"children":41},"p",{},[42],{"type":31,"tag":43,"props":44,"children":45},"em",{},[46],{"type":37,"value":47},"What I can demonstrate breaks, and where the enforcement has to live",{"type":31,"tag":39,"props":49,"children":50},{},[51],{"type":37,"value":52},"One human, several coding agent sessions, one codebase. Everything below is either something I verified on a machine while writing this, or something that follows from how the tools are built. Where I could not check a claim, it is not here.",{"type":31,"tag":39,"props":54,"children":55},{},[56],{"type":37,"value":57},"The thing that shapes all of it: you can revoke a person's access and they carry that forward. They remember, they adjust, they have a reputation that survives the incident. An agent carries nothing between sessions. It cannot be held responsible, cannot be fired, and will not learn from the consequence, so the consequence lands on whoever delegated. Delegation to an agent is a loan of your own authority with no return path, which is why the authority has to be bounded on the way out rather than corrected on the way back.",{"type":31,"tag":59,"props":60,"children":62},"h2",{"id":61},"worktrees-are-not-an-isolation-boundary",[63],{"type":37,"value":64},"Worktrees are not an isolation boundary",{"type":31,"tag":39,"props":66,"children":67},{},[68],{"type":37,"value":69},"This is the assumption I see most often and it is wrong. A git worktree per session feels like separation, and it separates almost nothing.",{"type":31,"tag":39,"props":71,"children":72},{},[73,80,82,88,90,95,97,103,105,111,112,118,120,126,127,133,134,140,142,148,150,156,158,164],{"type":31,"tag":74,"props":75,"children":77},"code",{"className":76},[],[78],{"type":37,"value":79},"git worktree add",{"type":37,"value":81}," writes a ",{"type":31,"tag":74,"props":83,"children":85},{"className":84},[],[86],{"type":37,"value":87},".git",{"type":37,"value":89}," ",{"type":31,"tag":43,"props":91,"children":92},{},[93],{"type":37,"value":94},"file",{"type":37,"value":96}," pointing back into the main repository. What you get per worktree is the pseudorefs (",{"type":31,"tag":74,"props":98,"children":100},{"className":99},[],[101],{"type":37,"value":102},"HEAD",{"type":37,"value":104},", ",{"type":31,"tag":74,"props":106,"children":108},{"className":107},[],[109],{"type":37,"value":110},"MERGE_HEAD",{"type":37,"value":104},{"type":31,"tag":74,"props":113,"children":115},{"className":114},[],[116],{"type":37,"value":117},"ORIG_HEAD",{"type":37,"value":119}," and friends), the index, the HEAD reflog, and three ref namespaces: ",{"type":31,"tag":74,"props":121,"children":123},{"className":122},[],[124],{"type":37,"value":125},"refs\u002Fbisect",{"type":37,"value":104},{"type":31,"tag":74,"props":128,"children":130},{"className":129},[],[131],{"type":37,"value":132},"refs\u002Fworktree",{"type":37,"value":104},{"type":31,"tag":74,"props":135,"children":137},{"className":136},[],[138],{"type":37,"value":139},"refs\u002Frewritten",{"type":37,"value":141},". Everything else is shared: every ref under ",{"type":31,"tag":74,"props":143,"children":145},{"className":144},[],[146],{"type":37,"value":147},"refs\u002F",{"type":37,"value":149},", meaning branches, tags, remotes, notes and the stash, plus the object database. ",{"type":31,"tag":74,"props":151,"children":153},{"className":152},[],[154],{"type":37,"value":155},".git\u002Fconfig",{"type":37,"value":157},". ",{"type":31,"tag":74,"props":159,"children":161},{"className":160},[],[162],{"type":37,"value":163},".git\u002Fhooks",{"type":37,"value":165},".",{"type":31,"tag":39,"props":167,"children":169},{"align":168},"center",[170],{"type":31,"tag":171,"props":172,"children":175},"img",{"src":19,"alt":173,"width":174},"Three worktrees each holding only HEAD, the index, the HEAD reflog and the bisect, worktree and rewritten ref namespaces, all pointing into a single shared .git directory holding the object database, every branch, tags, remotes, the stash, config and hooks",1000,[],{"type":31,"tag":39,"props":177,"children":178},{},[179],{"type":37,"value":180},"You can confirm it in any repository in one command:",{"type":31,"tag":182,"props":183,"children":185},"pre",{"code":184},"$ git -C some-worktree rev-parse --git-path hooks\n\u002Fpath\u002Fto\u002Fmain\u002F.git\u002Fhooks\n",[186],{"type":31,"tag":74,"props":187,"children":188},{"__ignoreMap":7},[189],{"type":37,"value":184},{"type":31,"tag":39,"props":191,"children":192},{},[193],{"type":37,"value":194},"Note what is not in the per-worktree list: branches. So one session can move or delete a branch another session has checked out. Git appears to stop you, which is how the assumption survives:",{"type":31,"tag":182,"props":196,"children":198},{"code":197},"$ git branch -f peer HEAD\nfatal: cannot force update the branch 'peer' used by worktree at '..\u002Fwt-peer'\n",[199],{"type":31,"tag":74,"props":200,"children":201},{"__ignoreMap":7},[202],{"type":37,"value":197},{"type":31,"tag":39,"props":204,"children":205},{},[206],{"type":37,"value":207},"That refusal is porcelain politeness. The plumbing has no such scruples:",{"type":31,"tag":182,"props":209,"children":211},{"code":210},"$ git update-ref refs\u002Fheads\u002Fpeer HEAD     # exit 0, silent\n$ git update-ref -d refs\u002Fheads\u002Fpeer       # exit 0, silent\n",[212],{"type":31,"tag":74,"props":213,"children":214},{"__ignoreMap":7},[215],{"type":37,"value":210},{"type":31,"tag":39,"props":217,"children":218},{},[219],{"type":37,"value":220},"After the second one, the peer's checkout is sitting on a branch that no longer exists. Nothing warned either session.",{"type":31,"tag":39,"props":222,"children":223},{},[224,226,232],{"type":37,"value":225},"The shared stash is worse than it sounds, because ",{"type":31,"tag":74,"props":227,"children":229},{"className":228},[],[230],{"type":37,"value":231},"git stash pop",{"type":37,"value":233}," takes no argument and indexes from the top. Session A stashes, session B stashes, session A pops, and B's work lands in A's tree. No error on either side.",{"type":31,"tag":39,"props":235,"children":236},{},[237,239,244],{"type":37,"value":238},"And because ",{"type":31,"tag":74,"props":240,"children":242},{"className":241},[],[243],{"type":37,"value":163},{"type":37,"value":245}," is shared, a hook written by one session runs in every other session in that worktree set, including the main checkout.",{"type":31,"tag":39,"props":247,"children":248},{},[249],{"type":37,"value":250},"If you want sessions isolated from each other, give each one its own clone. A local clone copies neither hooks nor local config, which worktrees do share. Use worktrees for cheap disk and fast branch switching, which is what they are for.",{"type":31,"tag":59,"props":252,"children":254},{"id":253},"get-the-sessions-off-your-host-and-check-what-that-bought-you",[255],{"type":37,"value":256},"Get the sessions off your host, and check what that bought you",{"type":31,"tag":39,"props":258,"children":259},{},[260,262,268,270,276],{"type":37,"value":261},"If sessions run as your own user on your own machine, the controls you reach for are not boundaries. Not because of anything subtle: a boundary the agent can opt out of is not a boundary. A harness deny rule governs the harness's own tools and the agent writes the file with ",{"type":31,"tag":74,"props":263,"children":265},{"className":264},[],[266],{"type":37,"value":267},"sed",{"type":37,"value":269},". A pre-commit hook is ",{"type":31,"tag":74,"props":271,"children":273},{"className":272},[],[274],{"type":37,"value":275},"--no-verify",{"type":37,"value":277},". Enforcement material cannot be read-only to a process running as the user who owns it.",{"type":31,"tag":39,"props":279,"children":280},{},[281],{"type":37,"value":282},"So put the sessions in a container or a VM that your host is not in. That one boundary is what stops a session reading your cloud credentials, your kube contexts, your SSH agent socket and your git credential store. It is cheap and it is the highest-value thing on this page.",{"type":31,"tag":39,"props":284,"children":285},{},[286],{"type":37,"value":287},"Then check what you actually bought, because the defaults give most of it back.",{"type":31,"tag":39,"props":289,"children":290},{},[291,297,299,305,307,313,314,320],{"type":31,"tag":292,"props":293,"children":294},"strong",{},[295],{"type":37,"value":296},"The Docker socket.",{"type":37,"value":298}," Mounting ",{"type":31,"tag":74,"props":300,"children":302},{"className":301},[],[303],{"type":37,"value":304},"\u002Fvar\u002Frun\u002Fdocker.sock",{"type":37,"value":306}," is host root in one command. It is also the most common ask for an agent that runs compose or Testcontainers. Same family: ",{"type":31,"tag":74,"props":308,"children":310},{"className":309},[],[311],{"type":37,"value":312},"--privileged",{"type":37,"value":104},{"type":31,"tag":74,"props":315,"children":317},{"className":316},[],[318],{"type":37,"value":319},"--pid=host",{"type":37,"value":165},{"type":31,"tag":39,"props":322,"children":323},{},[324,329,331,337,339,345,346,352,354,360,362,367,369,375],{"type":31,"tag":292,"props":325,"children":326},{},[327],{"type":37,"value":328},"The repository mount itself.",{"type":37,"value":330}," This one is easy to miss. The agent writes ",{"type":31,"tag":74,"props":332,"children":334},{"className":333},[],[335],{"type":37,"value":336},"package.json",{"type":37,"value":338}," scripts, a Makefile target, ",{"type":31,"tag":74,"props":340,"children":342},{"className":341},[],[343],{"type":37,"value":344},".envrc",{"type":37,"value":104},{"type":31,"tag":74,"props":347,"children":349},{"className":348},[],[350],{"type":37,"value":351},".vscode\u002Ftasks.json",{"type":37,"value":353},", a git hook, a ",{"type":31,"tag":74,"props":355,"children":357},{"className":356},[],[358],{"type":37,"value":359},".gitattributes",{"type":37,"value":361}," filter driver. Then ",{"type":31,"tag":43,"props":363,"children":364},{},[365],{"type":37,"value":366},"your host",{"type":37,"value":368}," executes it the next time you build, open the editor or ",{"type":31,"tag":74,"props":370,"children":372},{"className":371},[],[373],{"type":37,"value":374},"cd",{"type":37,"value":376}," into the directory. A writable repo mount is a code-execution path back across the boundary you just built.",{"type":31,"tag":39,"props":378,"children":379},{},[380,385,387,393,395,401,403,408,410,416,418,424],{"type":31,"tag":292,"props":381,"children":382},{},[383],{"type":37,"value":384},"Networking, which on a Mac does not work the way the advice says.",{"type":37,"value":386}," I tested this while writing, on Docker 29.8.1 under macOS. A service bound to the Mac's ",{"type":31,"tag":74,"props":388,"children":390},{"className":389},[],[391],{"type":37,"value":392},"127.0.0.1:38421",{"type":37,"value":394},", reachable from the host. From a container on ",{"type":31,"tag":74,"props":396,"children":398},{"className":397},[],[399],{"type":37,"value":400},"--network host",{"type":37,"value":402},": nothing, because host networking hands over the ",{"type":31,"tag":43,"props":404,"children":405},{},[406],{"type":37,"value":407},"Linux VM's",{"type":37,"value":409}," loopback, not yours. From a container on the plain default bridge: HTTP 200, via ",{"type":31,"tag":74,"props":411,"children":413},{"className":412},[],[414],{"type":37,"value":415},"host.docker.internal",{"type":37,"value":417},", which resolves to ",{"type":31,"tag":74,"props":419,"children":421},{"className":420},[],[422],{"type":37,"value":423},"192.168.65.254",{"type":37,"value":165},{"type":31,"tag":39,"props":426,"children":427},{},[428,430,435,437,442],{"type":37,"value":429},"So on macOS, declining ",{"type":31,"tag":74,"props":431,"children":433},{"className":432},[],[434],{"type":37,"value":400},{"type":37,"value":436}," buys you nothing against loopback exposure, and the default already reaches your local services. The same structure applies to Colima, Rancher Desktop and OrbStack, which are all Linux VMs with a host gateway. If that matters to you, block the gateway address from the container, and note that blocking the ",{"type":31,"tag":43,"props":438,"children":439},{},[440],{"type":37,"value":441},"name",{"type":37,"value":443}," does nothing because the raw IP works.",{"type":31,"tag":39,"props":445,"children":446},{},[447,449,455],{"type":37,"value":448},"Last thing worth knowing about one shared container: it separates the sessions from you, not from each other. Inside it they share a uid, a filesystem, a loopback interface and a PID namespace, so a session can read a peer's ",{"type":31,"tag":74,"props":450,"children":452},{"className":451},[],[453],{"type":37,"value":454},"\u002Fproc\u002F\u003Cpid>\u002Fenviron",{"type":37,"value":456}," and recover whatever was passed to it in the environment. A container per session is what makes session-to-session controls real.",{"type":31,"tag":59,"props":458,"children":460},{"id":459},"deny-rules-are-a-speed-bump",[461],{"type":37,"value":462},"Deny rules are a speed bump",{"type":31,"tag":39,"props":464,"children":465},{},[466,468,474],{"type":37,"value":467},"Denying ",{"type":31,"tag":74,"props":469,"children":471},{"className":470},[],[472],{"type":37,"value":473},"push",{"type":37,"value":475}," at the tool layer is worth doing and is not a control. The verb moves:",{"type":31,"tag":182,"props":477,"children":479},{"code":478},"$ git -C \u002Fpath\u002Fto\u002Fworktree push origin main     # push is not argv[1]\n$ git -c alias.yolo='push origin main' yolo     # the verb is in config\n$ git send-pack \u002Fpath\u002Fto\u002Fremote.git main        # no \"push\" token at all\n",[480],{"type":31,"tag":74,"props":481,"children":482},{"__ignoreMap":7},[483],{"type":37,"value":478},{"type":31,"tag":39,"props":485,"children":486},{},[487,493],{"type":31,"tag":74,"props":488,"children":490},{"className":489},[],[491],{"type":37,"value":492},"send-pack",{"type":37,"value":494}," is the plumbing behind push. No configuration, no flags, no mention of the word.",{"type":31,"tag":39,"props":496,"children":497},{},[498,500,506],{"type":37,"value":499},"The config surface is worse, because ",{"type":31,"tag":74,"props":501,"children":503},{"className":502},[],[504],{"type":37,"value":505},"git -c",{"type":37,"value":507}," reaches arbitrary execution directly:",{"type":31,"tag":182,"props":509,"children":511},{"code":510},"$ git -c alias.pwn='!id' pwn\nuid=501(...) gid=20(staff) ...\n",[512],{"type":31,"tag":74,"props":513,"children":514},{"__ignoreMap":7},[515],{"type":37,"value":510},{"type":31,"tag":39,"props":517,"children":518},{},[519,521,527,529,535,537,542,544,550],{"type":37,"value":520},"So a deny-list of config keys is not a control either. ",{"type":31,"tag":74,"props":522,"children":524},{"className":523},[],[525],{"type":37,"value":526},"core.fsmonitor",{"type":37,"value":528}," and ",{"type":31,"tag":74,"props":530,"children":532},{"className":531},[],[533],{"type":37,"value":534},"diff.external",{"type":37,"value":536}," execute the same way, through ",{"type":31,"tag":74,"props":538,"children":540},{"className":539},[],[541],{"type":37,"value":505},{"type":37,"value":543}," or ",{"type":31,"tag":74,"props":545,"children":547},{"className":546},[],[548],{"type":37,"value":549},"GIT_CONFIG_*",{"type":37,"value":551},", with no file write anywhere.",{"type":31,"tag":39,"props":553,"children":554},{},[555,557,562,564,569,571,576],{"type":37,"value":556},"What holds instead is two things. ",{"type":31,"tag":292,"props":558,"children":559},{},[560],{"type":37,"value":561},"Credential scope",{"type":37,"value":563},", so a push fails because the token cannot reach that remote. And ",{"type":31,"tag":292,"props":565,"children":566},{},[567],{"type":37,"value":568},"server-side authorization",{"type":37,"value":570},": rulesets, required checks, pre-receive hooks. That is the only layer a session cannot ",{"type":31,"tag":43,"props":572,"children":573},{},[574],{"type":37,"value":575},"modify",{"type":37,"value":577},". One precondition on that, and it is load-bearing: branch protection exempts administrators by default, and rulesets have bypass actors that can include apps. A ruleset with a non-empty bypass list is advisory, exactly like everything else here.",{"type":31,"tag":59,"props":579,"children":581},{"id":580},"an-agent-cannot-be-the-evidence-for-its-own-work",[582],{"type":37,"value":583},"An agent cannot be the evidence for its own work",{"type":31,"tag":39,"props":585,"children":586},{},[587],{"type":37,"value":588},"\"The tests pass\" is a claim, and a transcript is written by the thing making the claim. A session can print the sentence. A run ID pasted into a transcript is no better, unless somebody fetches that run from the API.",{"type":31,"tag":39,"props":590,"children":591},{},[592,594,600,602,608],{"type":37,"value":593},"The genuine anchors are a check run attributed to a required GitHub App, which is enforced server-side, and artifact attestations (",{"type":31,"tag":74,"props":595,"children":597},{"className":596},[],[598],{"type":37,"value":599},"actions\u002Fattest",{"type":37,"value":601},", verified with ",{"type":31,"tag":74,"props":603,"children":605},{"className":604},[],[606],{"type":37,"value":607},"gh attestation verify",{"type":37,"value":609},"), which are the cryptographic version. Bind them to the commit, and remember that an amend or rebase produces new SHAs, so a real green no longer covers the new head.",{"type":31,"tag":39,"props":611,"children":612},{},[613,615,620,622,627],{"type":37,"value":614},"There is a specific way this goes wrong that is worth knowing, because it inverts the obvious guess. A renamed or deleted required check does not pass; the pull request hangs forever waiting for a name that never reports. What passes is a ",{"type":31,"tag":292,"props":616,"children":617},{},[618],{"type":37,"value":619},"skipped",{"type":37,"value":621}," job. GitHub's own documentation: ",{"type":31,"tag":43,"props":623,"children":624},{},[625],{"type":37,"value":626},"\"A job that is skipped will report its status as 'Success'. It will not prevent a pull request from merging, even if it is a required check.\"",{"type":37,"value":628}," Required checks are satisfied by success, skipped or neutral.",{"type":31,"tag":39,"props":630,"children":631},{},[632,634,640,642,648,650,656,658,664],{"type":37,"value":633},"So the way an agent makes its own gate green is a job-level ",{"type":31,"tag":74,"props":635,"children":637},{"className":636},[],[638],{"type":37,"value":639},"if:",{"type":37,"value":641}," condition, not a rename. And since workflows for ",{"type":31,"tag":74,"props":643,"children":645},{"className":644},[],[646],{"type":37,"value":647},"pull_request",{"type":37,"value":649}," events run from the merge ref, an agent that can edit ",{"type":31,"tag":74,"props":651,"children":653},{"className":652},[],[654],{"type":37,"value":655},".github\u002Fworkflows\u002F",{"type":37,"value":657}," controls the gate that judges it. The fix is to make the CI definition a path that needs human review, source the actual checks from a workflow in a repository no session can write, and make the single required check an aggregator that runs with ",{"type":31,"tag":74,"props":659,"children":661},{"className":660},[],[662],{"type":37,"value":663},"if: always()",{"type":37,"value":665}," and fails unless its dependencies succeeded.",{"type":31,"tag":59,"props":667,"children":669},{"id":668},"an-approval-is-not-a-credential",[670],{"type":37,"value":671},"An approval is not a credential",{"type":31,"tag":39,"props":673,"children":674},{},[675],{"type":37,"value":676},"A coordinator telling a worker that the owner approved something is passing a description of an approval. Nothing in current harnesses lets the receiving session verify audience, scope or expiry.",{"type":31,"tag":39,"props":678,"children":679},{},[680],{"type":37,"value":681},"The property it needs has been specified for decades. Macaroon caveats narrow a credential and cannot be removed from it. OAuth token exchange, RFC 8693, issues downscoped tokens the authorization server enforces. There are IETF drafts now on attenuated agent tokens and on delegation chains for agent tool calls. None of it is wired into the harnesses, which is the vocabulary to use when asking a vendor what they intend to do about it.",{"type":31,"tag":39,"props":683,"children":684},{},[685],{"type":37,"value":686},"Until then, be honest about the status: there is no control for relay fidelity, only containment. A decision register written by the coordinator cannot evidence that the coordinator relayed correctly. What does hold is keeping outward actions with the person, because that is enforced somewhere else.",{"type":31,"tag":39,"props":688,"children":689},{},[690],{"type":37,"value":691},"One mechanism is available now. Bind the approval to the artifact it was granted over and have the gate refuse a mismatch. Branch protection does a version of this already when it dismisses approvals on a new push. Pick your hash deliberately, though: a message-only amend leaves the tree SHA unchanged and changes the commit SHA, so binding to the tree lets a rewritten commit through on an old approval, and binding to the commit costs a re-approval for every trivial amend.",{"type":31,"tag":59,"props":693,"children":695},{"id":694},"rules-decay-so-stop-writing-rules",[696],{"type":37,"value":697},"Rules decay, so stop writing rules",{"type":31,"tag":39,"props":699,"children":700},{},[701],{"type":37,"value":702},"Sessions compact their context and start fresh. Any rule enforced by recall will eventually meet a session that does not have it.",{"type":31,"tag":39,"props":704,"children":705},{},[706],{"type":37,"value":707},"A rule that breaks once gets written down. A rule that breaks twice becomes a mechanism: a hook, a deny rule, a gate, a wrapper. Then test the mechanism the way you test code, including a test that it rejects what it should reject. A hook that has only been checked for allowing the right things has not been tested.",{"type":31,"tag":39,"props":709,"children":710},{},[711],{"type":37,"value":712},"The same logic caps delegation. If workers can delegate, delegation recurses, so allow it on the coordinator only and cap the depth explicitly. CrewAI's own guidance is to enable its delegation flag on the manager alone for exactly this reason.",{"type":31,"tag":59,"props":714,"children":716},{"id":715},"what-this-does-not-cover",[717],{"type":37,"value":718},"What this does not cover",{"type":31,"tag":39,"props":720,"children":721},{},[722],{"type":37,"value":723},"Each of these deserves its own treatment by someone who has checked it properly, and that is not this article.",{"type":31,"tag":39,"props":725,"children":726},{},[727,732],{"type":31,"tag":292,"props":728,"children":729},{},[730],{"type":37,"value":731},"Collisions outside the repository.",{"type":37,"value":733}," Two sessions generating database migrations, allocating the same topic name, or writing the same shared cache. The shape differs by framework and the mechanisms differ by resource, and it is the failure that costs most in practice.",{"type":31,"tag":39,"props":735,"children":736},{},[737,742],{"type":31,"tag":292,"props":738,"children":739},{},[740],{"type":37,"value":741},"Supply chain.",{"type":37,"value":743}," Lockfile regeneration smuggling dependency bumps through a conflict resolution, pinning actions to commit SHAs, vetting MCP servers and plugins.",{"type":31,"tag":39,"props":745,"children":746},{},[747,752],{"type":31,"tag":292,"props":748,"children":749},{},[750],{"type":37,"value":751},"Test integrity.",{"type":37,"value":753}," An agent that makes tests pass by widening an assertion or mocking the thing under test produces a genuine green run, which the evidence rule above does not catch.",{"type":31,"tag":39,"props":755,"children":756},{},[757,762],{"type":31,"tag":292,"props":758,"children":759},{},[760],{"type":37,"value":761},"Untrusted input.",{"type":37,"value":763}," A session reading a dependency README, an issue body or a web page is consuming attacker-controlled content while holding credentials.",{"type":31,"tag":39,"props":765,"children":766},{},[767,772],{"type":31,"tag":292,"props":768,"children":769},{},[770],{"type":37,"value":771},"Provenance.",{"type":37,"value":773}," Showing which code in production was agent-generated and what control it passed. A ledger in the repository is a working record, not an audit trail.",{"type":31,"tag":39,"props":775,"children":776},{},[777,782],{"type":31,"tag":292,"props":778,"children":779},{},[780],{"type":37,"value":781},"More than one human.",{"type":37,"value":783}," Everything here assumes one owner.",{"type":31,"tag":39,"props":785,"children":786},{},[787,789,798],{"type":37,"value":788},"The longer version, with controls stated as what they prevent and what they cost, is in ",{"type":31,"tag":790,"props":791,"children":795},"a",{"href":792,"rel":793},"https:\u002F\u002Fgithub.com\u002Flevente-simon\u002Fgoverned-semi-autonomous-development",[794],"nofollow",[796],{"type":37,"value":797},"governed semi-autonomous development",{"type":37,"value":799},". CC BY 4.0, and a draft.",{"title":7,"searchDepth":801,"depth":801,"links":802},4,[803,805,806,807,808,809,810],{"id":61,"depth":804,"text":64},2,{"id":253,"depth":804,"text":256},{"id":459,"depth":804,"text":462},{"id":580,"depth":804,"text":583},{"id":668,"depth":804,"text":671},{"id":694,"depth":804,"text":697},{"id":715,"depth":804,"text":718},"markdown","content:insights:failure-modes-parallel-agent-sessions.md","content","insights\u002Ffailure-modes-parallel-agent-sessions.md","insights\u002Ffailure-modes-parallel-agent-sessions","md",{"loc":4},[819,1349],{"_path":820,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":821,"description":822,"date":823,"author":824,"image":826,"category":827,"tags":828,"body":835,"_type":811,"_id":1345,"_source":813,"_file":1346,"_stem":1347,"_extension":816,"sitemap":1348},"\u002Finsights\u002Farchitecture-overview","Designing for compromise: a multi-tenant SaaS architecture on AWS","Per-customer infrastructure isolation on AWS, where a cross-tenant breach means handing an attacker the blueprint to an organization's defenses.","2026-02-20",{"name":12,"headshot":13,"role":14,"contact":825},{"linkedin":16,"email":17,"twitter":18},"\u002Fimages\u002Fdesigned_for_compromise.jpg",[21],[829,830,25,831,832,833,834],"AWS","multi-tenant","infrastructure","Terraform","Fedora CoreOS","dethernety",{"type":28,"children":836,"toc":1334},[837,842,847,852,857,862,867,873,878,883,888,893,906,911,917,934,939,944,949,954,959,965,970,979,984,989,994,999,1004,1009,1015,1020,1025,1033,1045,1056,1061,1067,1080,1088,1098,1108,1118,1128,1133,1139,1147,1152,1157,1162,1168,1173,1184,1195,1200,1210,1220,1230,1240,1245,1251,1256,1262,1313,1317],{"type":31,"tag":32,"props":838,"children":840},{"id":839},"designing-for-compromise-a-multi-tenant-saas-architecture-on-aws",[841],{"type":37,"value":821},{"type":31,"tag":39,"props":843,"children":844},{},[845],{"type":37,"value":846},"Multi-tenant SaaS on AWS has a gravitational pull. The defaults are well-known: a shared VPC, a central management plane with broad IAM permissions, one Cognito pool with custom attributes for tenant isolation, a single Terraform state file that describes everything. It works and it ships fast, but it creates a system where the compromise of one component can reach every customer.",{"type":31,"tag":39,"props":848,"children":849},{},[850],{"type":37,"value":851},"This is the first article in a series about an architecture that takes a different path — and about why this particular platform demands it.",{"type":31,"tag":39,"props":853,"children":854},{},[855],{"type":37,"value":856},"The platform is a threat modeling tool. Customers use it to map their attack surfaces, model attack paths through their infrastructure, catalog their defenses, and analyze where their security architectures are weakest. The data inside each workspace is, by definition, a detailed roadmap of that organization's vulnerabilities. Where the entry points are. Which assets are reachable from which attack paths. What's defended and what isn't.",{"type":31,"tag":39,"props":858,"children":859},{},[860],{"type":37,"value":861},"A cross-tenant data breach on a typical SaaS platform exposes user records or business data. A cross-tenant breach on a security platform hands one organization's complete threat model to another party. The stakes are categorically different, and they demand a level of infrastructure isolation that shared-everything multi-tenancy cannot provide.",{"type":31,"tag":39,"props":863,"children":864},{},[865],{"type":37,"value":866},"The series will go deep on every component: Terraform modules, IAM policies, container configurations, cost breakdowns. This article starts from the top: the motivation, the design philosophy, what the resulting system looks like, and how a customer goes from sign-up to a running workspace.",{"type":31,"tag":59,"props":868,"children":870},{"id":869},"the-problem-with-shared-everything",[871],{"type":37,"value":872},"The problem with shared everything",{"type":31,"tag":39,"props":874,"children":875},{},[876],{"type":37,"value":877},"Infrastructure isolation isn't absent in modern cloud architectures, but it's typically functional. AWS multi-account strategies, landing zones, and hub-and-spoke models separate workloads along functional boundaries: a security account, a logging account, a networking hub, separate environments for dev, staging, and production. This is valuable. It means a compromised application or database doesn't take down visibility: monitoring remains intact in its own account. A dev environment can't touch production resources.",{"type":31,"tag":39,"props":879,"children":880},{},[881],{"type":37,"value":882},"But within the production workload, all customers still share the same VPC, the same database engine, the same compute, the same management plane. The isolation between Customer A and Customer B happens in the application layer: tenant ID filters on database queries, middleware that checks ownership, row-level security policies. Infrastructure isolation per customer is rare.",{"type":31,"tag":39,"props":884,"children":885},{},[886],{"type":37,"value":887},"For most SaaS products, application-level tenant isolation is entirely adequate. And at scale (millions of users, thousands of tenants), per-customer infrastructure isolation isn't an option. The operational overhead, the cost, and the AWS service limits themselves (VPCs per region, Cognito pools per account, subnets per VPC) make it prohibitive. Application-level isolation is the only viable model at that scale, and it serves most platforms well.",{"type":31,"tag":39,"props":889,"children":890},{},[891],{"type":37,"value":892},"But the data on this platform isn't profile information or project files. It's security architectures, attack graphs, defense mappings, and vulnerability analyses. The cost of a cross-tenant leak isn't a privacy incident. It's handing an attacker the blueprint to an organization's defenses.",{"type":31,"tag":39,"props":894,"children":895},{},[896,898,904],{"type":37,"value":897},"A compromised provisioner doesn't check tenant IDs before accessing an S3 bucket. A leaked IAM credential doesn't respect row-level security. When the application layer is bypassed, the infrastructure's boundaries are what remain, and in a shared-everything architecture each shared layer is a path between customers. A shared VPC means network-level reachability. A shared Cognito pool means a misconfiguration can leak tokens across tenants. A shared management plane means the most privileged component in the system can touch every customer's resources. A single Terraform state file means an operational error (an accidental ",{"type":31,"tag":74,"props":899,"children":901},{"className":900},[],[902],{"type":37,"value":903},"terraform destroy",{"type":37,"value":905},", a corrupted state during concurrent apply) affects everyone.",{"type":31,"tag":39,"props":907,"children":908},{},[909],{"type":37,"value":910},"The conventional response is to add more application-level checks, more middleware, more filters. For a platform that holds threat models and attack surfaces, this architecture takes a different approach: remove the shared paths entirely.",{"type":31,"tag":59,"props":912,"children":914},{"id":913},"design-for-compromise",[915],{"type":37,"value":916},"Design for compromise",{"type":31,"tag":39,"props":918,"children":919},{},[920,922,927,929],{"type":37,"value":921},"The question this architecture starts from is not ",{"type":31,"tag":43,"props":923,"children":924},{},[925],{"type":37,"value":926},"how do we prevent unauthorized access between tenants?",{"type":37,"value":928}," It is: ",{"type":31,"tag":43,"props":930,"children":931},{},[932],{"type":37,"value":933},"if a component is compromised, what can it reach?",{"type":31,"tag":39,"props":935,"children":936},{},[937],{"type":37,"value":938},"When the answer to that question could be \"another organization's complete threat model,\" the architecture cannot afford to rely on application-level enforcement alone.",{"type":31,"tag":39,"props":940,"children":941},{},[942],{"type":37,"value":943},"Prevention is about making the walls strong. Containment is about making the walls narrow. Prevention fails when any single layer is bypassed. Containment holds even when a layer is fully compromised, because the compromised component simply cannot reference another customer's resources. The reference doesn't exist.",{"type":31,"tag":39,"props":945,"children":946},{},[947],{"type":37,"value":948},"The strategy is structural isolation. Each customer gets their own network segment, their own identity provider, their own storage, their own compute, their own deployer, their own databases. Not through application logic that filters by tenant, but through infrastructure boundaries that make cross-tenant access impossible. Separate subnets. Separate IAM roles that reference only one customer's resources by ARN. Separate Cognito pools. Separate Terraform state files. Separate deployers that can assume only one customer's provisioner role.",{"type":31,"tag":39,"props":950,"children":951},{},[952],{"type":37,"value":953},"The cost is more of everything: more IAM policies, more Cognito pools, more Terraform state paths, more deployers. The benefit is that a compromise is contained, structurally, to one customer.",{"type":31,"tag":39,"props":955,"children":956},{},[957],{"type":37,"value":958},"Everything else in the architecture follows from this premise.",{"type":31,"tag":59,"props":960,"children":962},{"id":961},"what-each-customer-gets",[963],{"type":37,"value":964},"What each customer gets",{"type":31,"tag":39,"props":966,"children":967},{},[968],{"type":37,"value":969},"When a customer deploys their workspace, the platform provisions a complete, isolated environment:",{"type":31,"tag":39,"props":971,"children":972},{"align":168},[973],{"type":31,"tag":171,"props":974,"children":978},{"src":975,"alt":976,"width":977},"\u002Fimages\u002Fdiagram-per-customer-boundary.svg","diagram per customer boundary",800,[],{"type":31,"tag":39,"props":980,"children":981},{},[982],{"type":37,"value":983},"Each customer gets a \u002F28 private subnet within the customer VPC. No instance has a public IP. Inbound traffic arrives exclusively through a dedicated CloudFront distribution, which connects to the subnet over the AWS backbone via a VPC Origin. Authentication runs through a dedicated Cognito User Pool with its own password policy, MFA configuration, and token settings. A bug or misconfiguration in one pool cannot leak data to another.",{"type":31,"tag":39,"props":985,"children":986},{},[987],{"type":37,"value":988},"Each customer gets dedicated compute. On the Consultant tier, a single EC2 instance. On higher tiers, a K3s cluster per customer with rolling updates and redundancy. The blast radius boundary is the same.",{"type":31,"tag":39,"props":990,"children":991},{},[992],{"type":37,"value":993},"The Consultant tier has the harder engineering problem: delivering the same isolation and manageability at a price point where Kubernetes isn't justifiable. The instance must be reproducible, immutable, and updatable without SSH or manual intervention.",{"type":31,"tag":39,"props":995,"children":996},{},[997],{"type":37,"value":998},"Fedora CoreOS solves this — a container-optimized Linux with a read-only root filesystem, no package manager, and no SSH. The entire OS configuration is defined in a Butane YAML file, compiled to Ignition JSON, uploaded to S3, and fetched by the instance at first boot. The containers run under systemd on a Podman bridge network (an isolated container network on the host). When the configuration changes, Terraform replaces the entire instance. The persistent EBS data volume is detached and reattached; databases survive. The root volume is destroyed and recreated from scratch. There is no configuration drift.",{"type":31,"tag":39,"props":1000,"children":1001},{},[1002],{"type":37,"value":1003},"Two Route53 zones for the same domain create a split-horizon DNS: a public zone pointing at CloudFront and a private zone for internal service discovery. The instance can write TLS challenge records to the public zone but cannot touch routing records or the private zone. Storage is two S3 buckets (frontend assets and internal configuration) with public access blocked and encryption enabled, plus the persistent EBS data volume for databases.",{"type":31,"tag":39,"props":1005,"children":1006},{},[1007],{"type":37,"value":1008},"Each customer's Fargate deployer has an IAM role that can do exactly one thing: assume this customer's provisioner role. That role references only this customer's resources by hardcoded ARN, never wildcards. Where AWS supports condition keys (record types, resource tags, source IPs), they further narrow what each role can do. The Terraform state file is also per-customer.",{"type":31,"tag":59,"props":1010,"children":1012},{"id":1011},"the-architecture-from-above",[1013],{"type":37,"value":1014},"The architecture from above",{"type":31,"tag":39,"props":1016,"children":1017},{},[1018],{"type":37,"value":1019},"The full deployment includes the expected functional isolation (separate accounts for security, logging, and monitoring) but this overview focuses on the workload architecture: the infrastructure that runs customer environments.",{"type":31,"tag":39,"props":1021,"children":1022},{},[1023],{"type":37,"value":1024},"The workload runs in two VPCs within a single AWS region.",{"type":31,"tag":39,"props":1026,"children":1027},{"align":168},[1028],{"type":31,"tag":171,"props":1029,"children":1032},{"src":1030,"alt":1031,"width":977},"\u002Fimages\u002Fdiagram-two-vpc-architecture.svg","diagram-two-vpc-architecture",[],{"type":31,"tag":39,"props":1034,"children":1035},{},[1036,1038,1043],{"type":37,"value":1037},"The ",{"type":31,"tag":292,"props":1039,"children":1040},{},[1041],{"type":37,"value":1042},"management VPC",{"type":37,"value":1044}," hosts shared platform functions: the registration and payment Lambdas, account provisioning, the Fargate deployers that run Terraform, bastion access, and monitoring.",{"type":31,"tag":39,"props":1046,"children":1047},{},[1048,1049,1054],{"type":37,"value":1037},{"type":31,"tag":292,"props":1050,"children":1051},{},[1052],{"type":37,"value":1053},"customer VPC",{"type":37,"value":1055}," hosts customer workloads. Each customer's \u002F28 subnet is allocated atomically via a DynamoDB counter; indices are never reused. AWS service limits cap the number of customers per account, but the architecture doesn't try to maximize that number. It scales through a cell-based model: each cell is a separate AWS account with its own pair of VPCs, its own service limits, and its own blast radius boundary. Cells isolate groups of tenants from each other at the account level. This is a second containment boundary, on top of the per-customer isolation within each cell.",{"type":31,"tag":39,"props":1057,"children":1058},{},[1059],{"type":37,"value":1060},"VPC peering connects the two networks within a cell, with DNS resolution enabled in both directions. The split means a compromised customer instance has no network path to the management plane, and a compromised deployer has no persistent presence in the customer VPC.",{"type":31,"tag":59,"props":1062,"children":1064},{"id":1063},"from-sign-up-to-running-workspace",[1065],{"type":37,"value":1066},"From sign-up to running workspace",{"type":31,"tag":39,"props":1068,"children":1069},{},[1070,1072,1078],{"type":37,"value":1071},"The shared platform infrastructure (VPCs, peering, NAT, container registries, shared IAM roles) is provisioned once by the platform operator. What's specific to this architecture is what happens when a customer arrives. The provisioning pipeline is a sequence of Lambda functions, each running as a single compiled binary inside a minimal ",{"type":31,"tag":74,"props":1073,"children":1075},{"className":1074},[],[1076],{"type":37,"value":1077},"FROM scratch",{"type":37,"value":1079}," container: no runtime, no shell, no dependencies beyond the binary itself. Each invocation starts clean, runs one function, and exits. The pipeline has four stages, with gates between them.",{"type":31,"tag":39,"props":1081,"children":1082},{"align":168},[1083],{"type":31,"tag":171,"props":1084,"children":1087},{"src":1085,"alt":1086,"width":977},"\u002Fimages\u002Fdiagram-deployment-flow.svg","diagram-deployment-flow.drawio.svg",[],{"type":31,"tag":39,"props":1089,"children":1090},{},[1091,1096],{"type":31,"tag":292,"props":1092,"children":1093},{},[1094],{"type":37,"value":1095},"Registration",{"type":37,"value":1097}," is itself a queue-based, multi-step flow. The sign-up form creates a DynamoDB record and sends a verification email. Nothing else. Email verification pushes a job to SQS, which triggers a provisioner Lambda that creates the Cognito pool. Each step is isolated: the registration Lambda can write to DynamoDB and send email but cannot create Cognito pools; the provisioner Lambda can create Cognito pools but is triggered only through SQS, not directly. Failed provisioning retries through the queue, with a dead letter queue and CloudWatch alarms as a backstop. A bot submitting 10,000 fake sign-ups creates 10,000 DynamoDB records (negligible cost) and they auto-delete within two hours. No real AWS resources are created until a human verifies their email.",{"type":31,"tag":39,"props":1099,"children":1100},{},[1101,1106],{"type":31,"tag":292,"props":1102,"children":1103},{},[1104],{"type":37,"value":1105},"Payment",{"type":37,"value":1107}," gates the transition from a database record to real resources. The payment flow is itself isolated: Stripe webhook → API Gateway → Lambda (signature verification) → SQS → processing Lambda → DynamoDB. Each step authenticates independently. The webhook Lambda verifies the Stripe signature, the processing Lambda validates the SQS message and cross-references the session. The webhook Lambda can only push to SQS; it has no direct access to DynamoDB or any infrastructure creation capability. Until payment is confirmed, the customer exists only in DynamoDB.",{"type":31,"tag":39,"props":1109,"children":1110},{},[1111,1116],{"type":31,"tag":292,"props":1112,"children":1113},{},[1114],{"type":37,"value":1115},"Account provisioning",{"type":37,"value":1117}," requires three independent factors before any infrastructure is created. A triple gate: a valid JWT (proving identity through Cognito authentication), an active payment status in DynamoDB (proving Stripe confirmation), and a valid provisioning token from a magic link email (proving email access, time-bound, hashed). All three must pass independently. Only then does the platform create the customer's Cognito pool, \u002F28 subnet, split-horizon DNS zones, per-customer IAM roles with hardcoded ARNs, and an internal S3 bucket.",{"type":31,"tag":39,"props":1119,"children":1120},{},[1121,1126],{"type":31,"tag":292,"props":1122,"children":1123},{},[1124],{"type":37,"value":1125},"Infrastructure deployment",{"type":37,"value":1127}," is triggered by the customer. They click \"Deploy Infrastructure\" in the management console. This launches a Fargate task whose IAM role can do exactly one thing: assume this customer's provisioner role. The provisioner role references only this customer's resources by ARN. It cannot discover or touch another customer's infrastructure. Terraform runs against a per-customer state file, so even a corrupted state or an accidental destroy is contained to one customer. The deployer creates the EC2 instance, data volume, CloudFront distribution, frontend bucket, TLS certificate, DNS records, and management API. The instance boots from a clean Ignition configuration fetched from S3 and stands up the full container stack.",{"type":31,"tag":39,"props":1129,"children":1130},{},[1131],{"type":37,"value":1132},"Each stage is idempotent — re-running produces the same result.",{"type":31,"tag":59,"props":1134,"children":1136},{"id":1135},"how-traffic-reaches-the-backend",[1137],{"type":37,"value":1138},"How traffic reaches the backend",{"type":31,"tag":39,"props":1140,"children":1141},{"align":168},[1142],{"type":31,"tag":171,"props":1143,"children":1146},{"src":1144,"alt":1145,"width":977},"\u002Fimages\u002Fdiagram-request-flow.svg","diagram-request-flow",[],{"type":31,"tag":39,"props":1148,"children":1149},{},[1150],{"type":37,"value":1151},"A user request to their workspace hits CloudFront first. Static assets go to S3 through Origin Access Control. API and GraphQL requests go to the VPC Origin. CloudFront connects to an ENI in the customer's subnet over the AWS backbone, injecting a secret header that proves the request came through the CDN.",{"type":31,"tag":39,"props":1153,"children":1154},{},[1155],{"type":37,"value":1156},"The EC2 instance has no public IP, but traffic between CloudFront and the backend is still encrypted end-to-end. The instance runs a Let's Encrypt certificate obtained through DNS-01 challenge validation. No inbound port 80, no public endpoint. nginx terminates TLS, validates the secret header (rejecting anything that didn't arrive through CloudFront), and reverse-proxies to the backend. The backend connects to the graph database, vector store, and policy engine over a Podman bridge network. Nothing is exposed to the host except nginx on port 443.",{"type":31,"tag":39,"props":1158,"children":1159},{},[1160],{"type":37,"value":1161},"Admin operations take a separate path through API Gateway and Lambda. Every step in the chain independently verifies the user's JWT and authenticates the upstream component that forwarded the request. No layer trusts that a previous one already validated.",{"type":31,"tag":59,"props":1163,"children":1165},{"id":1164},"deliberate-trade-offs-and-tiering",[1166],{"type":37,"value":1167},"Deliberate trade-offs and tiering",{"type":31,"tag":39,"props":1169,"children":1170},{},[1171],{"type":37,"value":1172},"This architecture optimizes for blast radius isolation. That has consequences, and the trade-offs differ by tier.",{"type":31,"tag":39,"props":1174,"children":1175},{},[1176,1177,1182],{"type":37,"value":1037},{"type":31,"tag":292,"props":1178,"children":1179},{},[1180],{"type":37,"value":1181},"Consultant tier",{"type":37,"value":1183}," runs a single EC2 instance per customer, Podman with systemd, and instance replacement for updates. It's designed for independent consultants and small teams running threat modeling sessions, where brief downtime during updates is acceptable and single-node simplicity keeps cost and operational complexity low.",{"type":31,"tag":39,"props":1185,"children":1186},{},[1187,1188,1193],{"type":37,"value":1037},{"type":31,"tag":292,"props":1189,"children":1190},{},[1191],{"type":37,"value":1192},"Team and Enterprise tiers",{"type":37,"value":1194}," move from a single node to a self-managed Kubernetes cluster (K3s) per customer. The isolation model stays the same: per-customer subnet, per-customer IAM roles, per-customer state. What changes is the compute layer inside that boundary: one instance with Podman becomes a multi-node cluster with rolling updates and no downtime. The Terraform modules are different, but the blast radius boundary is identical. The series will cover these tiers separately.",{"type":31,"tag":39,"props":1196,"children":1197},{},[1198],{"type":37,"value":1199},"Within the Consultant tier, the trade-offs are explicit:",{"type":31,"tag":39,"props":1201,"children":1202},{},[1203,1208],{"type":31,"tag":292,"props":1204,"children":1205},{},[1206],{"type":37,"value":1207},"No high availability at the instance level.",{"type":37,"value":1209}," A single EC2 instance. If it fails, the instance is replaced and the data volume survives, but there are 2-5 minutes of downtime. For threat modeling sessions, not real-time transactions, this is acceptable.",{"type":31,"tag":39,"props":1211,"children":1212},{},[1213,1218],{"type":31,"tag":292,"props":1214,"children":1215},{},[1216],{"type":37,"value":1217},"No Kubernetes.",{"type":37,"value":1219}," Multiple containers on one node, managed by Podman and systemd. Kubernetes would add control plane and networking complexity without corresponding benefit at single-node scale.",{"type":31,"tag":39,"props":1221,"children":1222},{},[1223,1228],{"type":31,"tag":292,"props":1224,"children":1225},{},[1226],{"type":37,"value":1227},"No shared database.",{"type":37,"value":1229}," Each customer runs their own graph database and vector store as containers. More expensive than a shared managed cluster, but customer data never shares a database engine with another customer. Blast radius isolation extends to the data layer.",{"type":31,"tag":39,"props":1231,"children":1232},{},[1233,1238],{"type":31,"tag":292,"props":1234,"children":1235},{},[1236],{"type":37,"value":1237},"No in-place updates.",{"type":37,"value":1239}," Updates have two speeds. Container image updates are fast: pull the new image, restart the service. OS-level changes (kernel, systemd units, nginx config) replace the entire instance, which means 2-5 minutes of downtime. Either way, there is no configuration drift. Every update starts from a known, clean state.",{"type":31,"tag":39,"props":1241,"children":1242},{},[1243],{"type":37,"value":1244},"Per-customer isolation is inherently more expensive than shared infrastructure. The architecture manages this through deliberate choices across the platform: an open-source NAT instance instead of the AWS NAT Gateway, CloudFront edge locations limited to the target market, and an S3 Gateway Endpoint to avoid data transfer costs.",{"type":31,"tag":59,"props":1246,"children":1248},{"id":1247},"what-comes-next",[1249],{"type":37,"value":1250},"What comes next",{"type":31,"tag":39,"props":1252,"children":1253},{},[1254],{"type":37,"value":1255},"This article described the shape of the architecture: the motivation, the isolation model, and how the pieces fit together. The rest of the series goes inside each component: IAM policies, Terraform modules, queue-based provisioning, split-horizon DNS, the immutable instance lifecycle, and cost. Each article will include working code, configuration, and the reasoning behind the trade-offs.",{"type":31,"tag":59,"props":1257,"children":1259},{"id":1258},"series",[1260],{"type":37,"value":1261},"Series",{"type":31,"tag":1263,"props":1264,"children":1265},"ol",{},[1266,1277,1286,1295,1304],{"type":31,"tag":1267,"props":1268,"children":1269},"li",{},[1270,1275],{"type":31,"tag":790,"props":1271,"children":1272},{"href":820},[1273],{"type":37,"value":1274},"Architecture overview",{"type":37,"value":1276}," (this article)",{"type":31,"tag":1267,"props":1278,"children":1279},{},[1280],{"type":31,"tag":790,"props":1281,"children":1283},{"href":1282},"\u002Finsights\u002Fblast-radius",[1284],{"type":37,"value":1285},"Customer isolation from the infrastructure up",{"type":31,"tag":1267,"props":1287,"children":1288},{},[1289],{"type":31,"tag":790,"props":1290,"children":1292},{"href":1291},"\u002Finsights\u002Fcustomer-deployment-flows",[1293],{"type":37,"value":1294},"Automating isolation: the self-service deployment pipeline",{"type":31,"tag":1267,"props":1296,"children":1297},{},[1298],{"type":31,"tag":790,"props":1299,"children":1301},{"href":1300},"\u002Finsights\u002Fcloudfront-vpc-origins",[1302],{"type":37,"value":1303},"CloudFront VPC Origins: what breaks and how to fix it",{"type":31,"tag":1267,"props":1305,"children":1306},{},[1307],{"type":31,"tag":790,"props":1308,"children":1310},{"href":1309},"\u002Finsights\u002Fcompute-stack",[1311],{"type":37,"value":1312},"The compute stack: Fedora CoreOS, Podman quadlets, and systemd-native container orchestration",{"type":31,"tag":1314,"props":1315,"children":1316},"hr",{},[],{"type":31,"tag":39,"props":1318,"children":1319},{},[1320],{"type":31,"tag":43,"props":1321,"children":1322},{},[1323,1325,1332],{"type":37,"value":1324},"This architecture is implemented in ",{"type":31,"tag":790,"props":1326,"children":1329},{"href":1327,"rel":1328},"https:\u002F\u002Fdether.net",[794],[1330],{"type":37,"value":1331},"dether.net",{"type":37,"value":1333},", a graph-native threat modeling platform. If you're interested in seeing these patterns applied to security architecture analysis, that's where they run in production.",{"title":7,"searchDepth":801,"depth":801,"links":1335},[1336,1337,1338,1339,1340,1341,1342,1343,1344],{"id":869,"depth":804,"text":872},{"id":913,"depth":804,"text":916},{"id":961,"depth":804,"text":964},{"id":1011,"depth":804,"text":1014},{"id":1063,"depth":804,"text":1066},{"id":1135,"depth":804,"text":1138},{"id":1164,"depth":804,"text":1167},{"id":1247,"depth":804,"text":1250},{"id":1258,"depth":804,"text":1261},"content:insights:architecture-overview.md","insights\u002Farchitecture-overview.md","insights\u002Farchitecture-overview",{"loc":820},{"_path":1282,"_dir":5,"_draft":6,"_partial":6,"_locale":7,"title":1285,"description":1350,"date":1351,"author":1352,"image":1354,"category":1355,"tags":1356,"body":1360,"_type":811,"_id":2219,"_source":813,"_file":2220,"_stem":2221,"_extension":816,"sitemap":2222},"How per-customer blast radius containment works across layers of AWS infrastructure — IAM, network, DNS, identity, storage, state, compute, and ingress — each enforcing isolation on its own.","2026-02-22",{"name":12,"headshot":13,"role":14,"contact":1353},{"linkedin":16,"email":17,"twitter":18},"\u002Fimages\u002Fdiagram-isolation-layers.png",[21],[829,1357,1358,830,25,1359,834],"IAM","blast radius","isolation",{"type":28,"children":1361,"toc":2205},[1362,1367,1378,1383,1389,1410,1415,1423,1429,1450,1469,1474,1479,1489,1499,1509,1519,1532,1595,1600,1629,1635,1640,1645,1650,1655,1661,1673,1684,1773,1778,1789,1794,1800,1805,1810,1815,1821,1826,1847,1852,1886,1891,1896,1902,1914,1919,1931,1937,1942,1947,1952,2040,2045,2051,2056,2061,2068,2085,2095,2100,2106,2111,2116,2121,2127,2132,2137,2142,2147,2151,2183,2186,2199],{"type":31,"tag":32,"props":1363,"children":1365},{"id":1364},"customer-isolation-from-the-infrastructure-up",[1366],{"type":37,"value":1285},{"type":31,"tag":39,"props":1368,"children":1369},{},[1370,1371,1376],{"type":37,"value":1037},{"type":31,"tag":790,"props":1372,"children":1373},{"href":820},[1374],{"type":37,"value":1375},"previous article",{"type":37,"value":1377}," described an architecture where each customer gets their own subnet, Cognito pool, S3 buckets, Terraform state, and deployer. This article explains the constraint behind those decisions and traces it through the infrastructure, layer by layer. Later articles go deeper into the implementation of each component.",{"type":31,"tag":39,"props":1379,"children":1380},{},[1381],{"type":37,"value":1382},"The constraint is blast radius. Every boundary in the architecture exists to answer one question: if this component is compromised, what can it reach?",{"type":31,"tag":59,"props":1384,"children":1386},{"id":1385},"the-organizing-principle",[1387],{"type":37,"value":1388},"The organizing principle",{"type":31,"tag":39,"props":1390,"children":1391},{},[1392,1394,1400,1402,1408],{"type":37,"value":1393},"Application-level isolation — tenant ID checks, middleware, row-level security — fails when any check is wrong: a missing filter, a misconfigured middleware, a bypassed code path. Infrastructure-level isolation fails differently. A provisioner role that contains ",{"type":31,"tag":74,"props":1395,"children":1397},{"className":1396},[],[1398],{"type":37,"value":1399},"arn:aws:s3:::{prefix}-acme-{suffix}-internal",{"type":37,"value":1401}," and not ",{"type":31,"tag":74,"props":1403,"children":1405},{"className":1404},[],[1406],{"type":37,"value":1407},"arn:aws:s3:::{prefix}-corp-{suffix}-internal",{"type":37,"value":1409}," cannot access Corp's bucket regardless of what the application does. The reference doesn't exist in the policy document. AWS denies the request before it reaches any application code.",{"type":31,"tag":39,"props":1411,"children":1412},{},[1413],{"type":37,"value":1414},"The principle is to push isolation down the stack — into IAM policies, subnet boundaries, DNS zones, Cognito pools, S3 bucket policies, Terraform state paths — so that each layer enforces containment on its own. The blast radius of a compromise is the intersection of all layers, not the weakest one. A compromise must bypass all of them to reach another customer.",{"type":31,"tag":39,"props":1416,"children":1417},{"align":168},[1418],{"type":31,"tag":171,"props":1419,"children":1422},{"src":1420,"alt":1421,"width":977},"\u002Fimages\u002Fdiagram-isolation-layers.svg","diagram isolation layers",[],{"type":31,"tag":59,"props":1424,"children":1426},{"id":1425},"iam-hardcoded-arns-no-wildcards",[1427],{"type":37,"value":1428},"IAM: hardcoded ARNs, no wildcards",{"type":31,"tag":39,"props":1430,"children":1431},{},[1432,1434,1440,1442,1448],{"type":37,"value":1433},"IAM determines what stolen credentials can reach. In a shared-everything architecture, a compromised role with ",{"type":31,"tag":74,"props":1435,"children":1437},{"className":1436},[],[1438],{"type":37,"value":1439},"s3:GetObject",{"type":37,"value":1441}," on ",{"type":31,"tag":74,"props":1443,"children":1445},{"className":1444},[],[1446],{"type":37,"value":1447},"arn:aws:s3:::platform-*",{"type":37,"value":1449}," can read every customer's bucket. The wildcard is the blast radius.",{"type":31,"tag":39,"props":1451,"children":1452},{},[1453,1455,1460,1462,1467],{"type":37,"value":1454},"This architecture eliminates wildcards from per-customer policies. The provisioner role for customer Acme contains ",{"type":31,"tag":74,"props":1456,"children":1458},{"className":1457},[],[1459],{"type":37,"value":1399},{"type":37,"value":1461},". The role for Corp contains ",{"type":31,"tag":74,"props":1463,"children":1465},{"className":1464},[],[1466],{"type":37,"value":1407},{"type":37,"value":1468},". Terraform templates the customer ID into every ARN at creation time.",{"type":31,"tag":39,"props":1470,"children":1471},{},[1472],{"type":37,"value":1473},"The same applies to every other AWS service in the stack. Route53 zone IDs, subnet ARNs, Cognito pool ARNs, Lambda function ARNs, CloudWatch log group paths. All customer-specific, all hardcoded.",{"type":31,"tag":39,"props":1475,"children":1476},{},[1477],{"type":37,"value":1478},"The roles form a chain that narrows scope at each step:",{"type":31,"tag":39,"props":1480,"children":1481},{},[1482,1487],{"type":31,"tag":292,"props":1483,"children":1484},{},[1485],{"type":37,"value":1486},"Bootstrap",{"type":37,"value":1488}," creates S3 state buckets and the platform IAM user. Runs once, by the operator.",{"type":31,"tag":39,"props":1490,"children":1491},{},[1492,1497],{"type":31,"tag":292,"props":1493,"children":1494},{},[1495],{"type":37,"value":1496},"Platform",{"type":37,"value":1498}," creates shared infrastructure: VPCs, container registries, the parent DNS zone, shared IAM roles. Broad permissions within the platform account, but no per-customer resources.",{"type":31,"tag":39,"props":1500,"children":1501},{},[1502,1507],{"type":31,"tag":292,"props":1503,"children":1504},{},[1505],{"type":37,"value":1506},"Provisioner",{"type":37,"value":1508}," is per-customer. Inline policies containing hardcoded ARNs for this customer's resources only. A permission boundary caps the maximum scope. Even if an inline policy is misconfigured, the boundary prevents escalation beyond this customer's resource set.",{"type":31,"tag":39,"props":1510,"children":1511},{},[1512,1517],{"type":31,"tag":292,"props":1513,"children":1514},{},[1515],{"type":37,"value":1516},"Runtime",{"type":37,"value":1518}," is the EC2 instance profile. Narrower still: S3 read\u002Fwrite for the customer's own buckets, ECR pull for shared container images, CloudWatch scoped to one log group, and Route53 restricted to TXT records in one public zone (for certificate renewal). No permissions to create or destroy infrastructure.",{"type":31,"tag":39,"props":1520,"children":1521},{},[1522,1524,1530],{"type":37,"value":1523},"Each customer's Fargate deployer has a role with exactly one permission: ",{"type":31,"tag":74,"props":1525,"children":1527},{"className":1526},[],[1528],{"type":37,"value":1529},"sts:AssumeRole",{"type":37,"value":1531}," on that customer's provisioner role. The trust policy requires the customer ID as an external ID, preventing confused deputy scenarios:",{"type":31,"tag":182,"props":1533,"children":1537},{"className":1534,"code":1535,"language":1536,"meta":7,"style":7},"language-hcl shiki shiki-themes github-dark github-dark","# The deployer task role — one permission, one customer\n{\n  \"Effect\": \"Allow\",\n  \"Action\": \"sts:AssumeRole\",\n  \"Resource\": \"arn:aws:iam::${account_id}:role\u002F${prefix}-${customer_id}-infra-provisioner\"\n}\n","hcl",[1538],{"type":31,"tag":74,"props":1539,"children":1540},{"__ignoreMap":7},[1541,1552,1560,1569,1577,1586],{"type":31,"tag":1542,"props":1543,"children":1546},"span",{"class":1544,"line":1545},"line",1,[1547],{"type":31,"tag":1542,"props":1548,"children":1549},{},[1550],{"type":37,"value":1551},"# The deployer task role — one permission, one customer\n",{"type":31,"tag":1542,"props":1553,"children":1554},{"class":1544,"line":804},[1555],{"type":31,"tag":1542,"props":1556,"children":1557},{},[1558],{"type":37,"value":1559},"{\n",{"type":31,"tag":1542,"props":1561,"children":1563},{"class":1544,"line":1562},3,[1564],{"type":31,"tag":1542,"props":1565,"children":1566},{},[1567],{"type":37,"value":1568},"  \"Effect\": \"Allow\",\n",{"type":31,"tag":1542,"props":1570,"children":1571},{"class":1544,"line":801},[1572],{"type":31,"tag":1542,"props":1573,"children":1574},{},[1575],{"type":37,"value":1576},"  \"Action\": \"sts:AssumeRole\",\n",{"type":31,"tag":1542,"props":1578,"children":1580},{"class":1544,"line":1579},5,[1581],{"type":31,"tag":1542,"props":1582,"children":1583},{},[1584],{"type":37,"value":1585},"  \"Resource\": \"arn:aws:iam::${account_id}:role\u002F${prefix}-${customer_id}-infra-provisioner\"\n",{"type":31,"tag":1542,"props":1587,"children":1589},{"class":1544,"line":1588},6,[1590],{"type":31,"tag":1542,"props":1591,"children":1592},{},[1593],{"type":37,"value":1594},"}\n",{"type":31,"tag":39,"props":1596,"children":1597},{},[1598],{"type":37,"value":1599},"The chain — deployer task credential → provisioner role → customer resources — narrows at each step. No step can widen it. A compromised deployer can assume the provisioner role for its customer and nothing else. A compromised provisioner can touch its customer's resources and nothing else.",{"type":31,"tag":39,"props":1601,"children":1602},{},[1603,1605,1611,1613,1619,1621,1627],{"type":37,"value":1604},"Where resource ARNs aren't known at creation time (EC2 instances, for example), tag-based conditions provide equivalent scoping. The provisioner applies a ",{"type":31,"tag":74,"props":1606,"children":1608},{"className":1607},[],[1609],{"type":37,"value":1610},"Customer",{"type":37,"value":1612}," tag at creation time through an ",{"type":31,"tag":74,"props":1614,"children":1616},{"className":1615},[],[1617],{"type":37,"value":1618},"aws:RequestTag",{"type":37,"value":1620}," condition, and subsequent operations are restricted to resources carrying that tag via ",{"type":31,"tag":74,"props":1622,"children":1624},{"className":1623},[],[1625],{"type":37,"value":1626},"aws:ResourceTag",{"type":37,"value":1628},". The tag is set once, at creation, and the condition is evaluated by AWS, not by application code.",{"type":31,"tag":59,"props":1630,"children":1632},{"id":1631},"network-one-subnet-no-cross-tenant-paths",[1633],{"type":37,"value":1634},"Network: one subnet, no cross-tenant paths",{"type":31,"tag":39,"props":1636,"children":1637},{},[1638],{"type":37,"value":1639},"The workload runs in two VPCs within a single cell. The broader account structure (security, logging, account-level monitoring) is outside this article's scope. The management VPC hosts deployers and workload monitoring. The customer VPC hosts customer environments. VPC peering connects them, but routing rules prevent customer instances from initiating connections to the management plane.",{"type":31,"tag":39,"props":1641,"children":1642},{},[1643],{"type":37,"value":1644},"Each customer gets a \u002F28 subnet in the customer VPC (sixteen addresses, eleven usable after AWS reservations). No instance has a public IP. Inbound traffic arrives through two paths only: a CloudFront VPC Origin and a per-customer Lambda, both routed over internal AWS networks, not the public internet.",{"type":31,"tag":39,"props":1646,"children":1647},{},[1648],{"type":37,"value":1649},"The security group on a customer's instance allows HTTPS ingress from exactly two sources: the CloudFront VPC Origin security group and a per-customer Lambda security group (for the management API). No security group rule allows traffic between customer subnets.",{"type":31,"tag":39,"props":1651,"children":1652},{},[1653],{"type":37,"value":1654},"A compromised instance in Acme's \u002F28 has no network path to Corp's \u002F28. No security group rule provides it. No route table entry enables it. The instance can reach the internet through a shared NAT instance for outbound calls, but it cannot initiate connections to other customer subnets, to the management VPC, or to any other internal resource. There is nothing to discover at the network layer.",{"type":31,"tag":59,"props":1656,"children":1658},{"id":1657},"dns-two-zones-two-permission-sets",[1659],{"type":37,"value":1660},"DNS: two zones, two permission sets",{"type":31,"tag":39,"props":1662,"children":1663},{},[1664,1666,1672],{"type":37,"value":1665},"Each customer has two Route53 hosted zones for the same domain, for example ",{"type":31,"tag":74,"props":1667,"children":1669},{"className":1668},[],[1670],{"type":37,"value":1671},"acme.dethernety.io",{"type":37,"value":165},{"type":31,"tag":39,"props":1674,"children":1675},{},[1676,1677,1682],{"type":37,"value":1037},{"type":31,"tag":292,"props":1678,"children":1679},{},[1680],{"type":37,"value":1681},"public zone",{"type":37,"value":1683}," is internet-resolvable. It holds the CloudFront ALIAS record that routes user traffic, the ACM validation CNAME for the CloudFront TLS certificate, and Let's Encrypt DNS-01 challenge TXT records for the backend TLS certificate. The EC2 instance can write to this zone for certificate renewal, but its IAM policy restricts modifications to TXT records only:",{"type":31,"tag":182,"props":1685,"children":1687},{"className":1534,"code":1686,"language":1536,"meta":7,"style":7},"{\n  Effect   = \"Allow\"\n  Action   = [\"route53:ChangeResourceRecordSets\"]\n  Resource = \"arn:aws:route53:::hostedzone\u002F${var.public_zone_id}\"\n  Condition = {\n    \"ForAllValues:StringEquals\" = {\n      \"route53:ChangeResourceRecordSetsRecordTypes\" = [\"TXT\"]\n    }\n  }\n}\n",[1688],{"type":31,"tag":74,"props":1689,"children":1690},{"__ignoreMap":7},[1691,1698,1706,1714,1722,1730,1738,1747,1756,1765],{"type":31,"tag":1542,"props":1692,"children":1693},{"class":1544,"line":1545},[1694],{"type":31,"tag":1542,"props":1695,"children":1696},{},[1697],{"type":37,"value":1559},{"type":31,"tag":1542,"props":1699,"children":1700},{"class":1544,"line":804},[1701],{"type":31,"tag":1542,"props":1702,"children":1703},{},[1704],{"type":37,"value":1705},"  Effect   = \"Allow\"\n",{"type":31,"tag":1542,"props":1707,"children":1708},{"class":1544,"line":1562},[1709],{"type":31,"tag":1542,"props":1710,"children":1711},{},[1712],{"type":37,"value":1713},"  Action   = [\"route53:ChangeResourceRecordSets\"]\n",{"type":31,"tag":1542,"props":1715,"children":1716},{"class":1544,"line":801},[1717],{"type":31,"tag":1542,"props":1718,"children":1719},{},[1720],{"type":37,"value":1721},"  Resource = \"arn:aws:route53:::hostedzone\u002F${var.public_zone_id}\"\n",{"type":31,"tag":1542,"props":1723,"children":1724},{"class":1544,"line":1579},[1725],{"type":31,"tag":1542,"props":1726,"children":1727},{},[1728],{"type":37,"value":1729},"  Condition = {\n",{"type":31,"tag":1542,"props":1731,"children":1732},{"class":1544,"line":1588},[1733],{"type":31,"tag":1542,"props":1734,"children":1735},{},[1736],{"type":37,"value":1737},"    \"ForAllValues:StringEquals\" = {\n",{"type":31,"tag":1542,"props":1739,"children":1741},{"class":1544,"line":1740},7,[1742],{"type":31,"tag":1542,"props":1743,"children":1744},{},[1745],{"type":37,"value":1746},"      \"route53:ChangeResourceRecordSetsRecordTypes\" = [\"TXT\"]\n",{"type":31,"tag":1542,"props":1748,"children":1750},{"class":1544,"line":1749},8,[1751],{"type":31,"tag":1542,"props":1752,"children":1753},{},[1754],{"type":37,"value":1755},"    }\n",{"type":31,"tag":1542,"props":1757,"children":1759},{"class":1544,"line":1758},9,[1760],{"type":31,"tag":1542,"props":1761,"children":1762},{},[1763],{"type":37,"value":1764},"  }\n",{"type":31,"tag":1542,"props":1766,"children":1768},{"class":1544,"line":1767},10,[1769],{"type":31,"tag":1542,"props":1770,"children":1771},{},[1772],{"type":37,"value":1594},{"type":31,"tag":39,"props":1774,"children":1775},{},[1776],{"type":37,"value":1777},"The instance cannot modify A records, CNAME records, or any routing record. A compromised instance with Route53 credentials cannot redirect traffic to a different origin.",{"type":31,"tag":39,"props":1779,"children":1780},{},[1781,1782,1787],{"type":37,"value":1037},{"type":31,"tag":292,"props":1783,"children":1784},{},[1785],{"type":37,"value":1786},"private zone",{"type":37,"value":1788}," is VPC-associated only — not resolvable from the internet. It holds the instance's private IP for internal service discovery. CloudFront's VPC Origin resolves the backend through this zone. Only the provisioner role can write to it; the instance profile has no permissions for the private zone at all.",{"type":31,"tag":39,"props":1790,"children":1791},{},[1792],{"type":37,"value":1793},"DNS is a privilege escalation vector. Control over DNS records means the ability to redirect traffic, intercept requests, and steal tokens. Split-horizon DNS with scoped IAM policies limits what a compromised instance can do: renew its own TLS certificate (TXT records in the public zone), but not redirect traffic (A\u002FCNAME records) and not modify internal routing (private zone).",{"type":31,"tag":59,"props":1795,"children":1797},{"id":1796},"identity-one-cognito-pool-per-customer",[1798],{"type":37,"value":1799},"Identity: one Cognito pool per customer",{"type":31,"tag":39,"props":1801,"children":1802},{},[1803],{"type":37,"value":1804},"Each customer gets a separate Cognito User Pool — not a shared pool with tenant-ID attributes, but a distinct AWS resource with its own ARN, password policy, MFA configuration, and PKCE app client.",{"type":31,"tag":39,"props":1806,"children":1807},{},[1808],{"type":37,"value":1809},"A shared pool is simpler to manage and works for most platforms. But it creates a shared authentication surface. A misconfigured Lambda trigger, a broken attribute mapping, or a token-handling bug in a shared pool affects every customer's authentication at once. The blast radius of an identity misconfiguration is every tenant.",{"type":31,"tag":39,"props":1811,"children":1812},{},[1813],{"type":37,"value":1814},"With per-customer pools, the blast radius is one customer. The app client's callback URLs point to this customer's subdomain. Self-registration is disabled. The JWT tokens issued by one pool are validated against that pool's JWKS endpoint. A token from Acme's pool is meaningless to Corp's backend: the issuer doesn't match, the audience doesn't match, the signing keys are different.",{"type":31,"tag":59,"props":1816,"children":1818},{"id":1817},"storage-no-shared-data-layer",[1819],{"type":37,"value":1820},"Storage: no shared data layer",{"type":31,"tag":39,"props":1822,"children":1823},{},[1824],{"type":37,"value":1825},"Each customer has two S3 buckets and a dedicated EBS volume. No shared database engine. No shared storage backend.",{"type":31,"tag":39,"props":1827,"children":1828},{},[1829,1831,1837,1839,1845],{"type":37,"value":1830},"Bucket names include a random suffix generated at provisioning time: ",{"type":31,"tag":74,"props":1832,"children":1834},{"className":1833},[],[1835],{"type":37,"value":1836},"{prefix}-acme-{random}-internal",{"type":37,"value":1838}," rather than the predictable ",{"type":31,"tag":74,"props":1840,"children":1842},{"className":1841},[],[1843],{"type":37,"value":1844},"{prefix}-acme-internal",{"type":37,"value":1846},". S3 bucket names are globally unique, and predictable names are a known attack vector: an adversary who can guess the name can pre-create the bucket in another account and intercept data written to it before the legitimate owner provisions. The suffix makes the name unguessable.",{"type":31,"tag":39,"props":1848,"children":1849},{},[1850],{"type":37,"value":1851},"The frontend bucket serves static assets through CloudFront Origin Access Control. The bucket policy allows access only from one CloudFront distribution, identified by ARN. No other distribution, and no direct access.",{"type":31,"tag":39,"props":1853,"children":1854},{},[1855,1857,1863,1864,1870,1871,1877,1878,1884],{"type":37,"value":1856},"The internal bucket holds Ignition configuration, Terraform outputs, backups, and state files. Public access is blocked at every level: ",{"type":31,"tag":74,"props":1858,"children":1860},{"className":1859},[],[1861],{"type":37,"value":1862},"block_public_acls",{"type":37,"value":104},{"type":31,"tag":74,"props":1865,"children":1867},{"className":1866},[],[1868],{"type":37,"value":1869},"block_public_policy",{"type":37,"value":104},{"type":31,"tag":74,"props":1872,"children":1874},{"className":1873},[],[1875],{"type":37,"value":1876},"ignore_public_acls",{"type":37,"value":104},{"type":31,"tag":74,"props":1879,"children":1881},{"className":1880},[],[1882],{"type":37,"value":1883},"restrict_public_buckets",{"type":37,"value":1885},", all enabled. Versioning is on for recovery. Lifecycle rules manage retention: backups archive to Glacier after seven days, state versions expire after ninety days.",{"type":31,"tag":39,"props":1887,"children":1888},{},[1889],{"type":37,"value":1890},"The EBS volume stores the graph database, vector store, and supporting services. Encrypted at rest, tagged with the customer ID, persistent across instance replacements. The instance profile's IAM policy restricts volume operations to volumes carrying this customer's tag. A compromised instance can access its own data volume but cannot discover or attach another customer's.",{"type":31,"tag":39,"props":1892,"children":1893},{},[1894],{"type":37,"value":1895},"The cost is higher than a multi-tenant RDS cluster or a shared S3 prefix. The tradeoff is that every storage boundary (bucket names, bucket policies, IAM ARNs, volume tags) is enforced by AWS, not by application logic in the request path.",{"type":31,"tag":59,"props":1897,"children":1899},{"id":1898},"state-one-terraform-state-file-per-customer",[1900],{"type":37,"value":1901},"State: one Terraform state file per customer",{"type":31,"tag":39,"props":1903,"children":1904},{},[1905,1907,1912],{"type":37,"value":1906},"Terraform state files contain infrastructure metadata: resource IDs, ARNs, configuration values, and sometimes sensitive outputs. In a shared-state architecture, a corrupted state file or an accidental ",{"type":31,"tag":74,"props":1908,"children":1910},{"className":1909},[],[1911],{"type":37,"value":903},{"type":37,"value":1913}," affects every customer provisioned from that state.",{"type":31,"tag":39,"props":1915,"children":1916},{},[1917],{"type":37,"value":1918},"This architecture isolates state at every level. Bootstrap uses local state; the S3 bucket doesn't exist yet. Platform state lives in a dedicated S3 object. Each customer's account state lives in a shared bucket under a customer-specific key path. Each customer's infrastructure state lives in that customer's own internal S3 bucket.",{"type":31,"tag":39,"props":1920,"children":1921},{},[1922,1924,1929],{"type":37,"value":1923},"The provisioner role for Acme can read and write Acme's state. It cannot read Corp's. The ARN isn't in the policy. Concurrent provisioning operations run against separate state files with separate DynamoDB locks. An accidental ",{"type":31,"tag":74,"props":1925,"children":1927},{"className":1926},[],[1928],{"type":37,"value":903},{"type":37,"value":1930}," against Acme's state tears down Acme's infrastructure and nothing else.",{"type":31,"tag":59,"props":1932,"children":1934},{"id":1933},"compute-immutable-replaceable-contained",[1935],{"type":37,"value":1936},"Compute: immutable, replaceable, contained",{"type":31,"tag":39,"props":1938,"children":1939},{},[1940],{"type":37,"value":1941},"The EC2 instances run Fedora CoreOS: a read-only root filesystem, no package manager, no SSH by default. The entire OS configuration is defined in a Butane YAML file, compiled to Ignition JSON, uploaded to S3, and fetched at first boot. The instance is not configured after launch. It boots into its final state.",{"type":31,"tag":39,"props":1943,"children":1944},{},[1945],{"type":37,"value":1946},"Updates replace the entire instance. Terraform detects a change in the Ignition configuration, terminates the old instance, launches a new one, and reattaches the persistent EBS data volume. No in-place patching, no configuration drift, and no persistence across replacements. An attacker who compromises an instance loses that access when the next deployment replaces it. The root volume is destroyed. The new instance boots from a clean configuration.",{"type":31,"tag":39,"props":1948,"children":1949},{},[1950],{"type":37,"value":1951},"Inside the instance, containers run on a Podman bridge network under systemd. The isolation model follows the same principle: minimize what each component can reach.",{"type":31,"tag":1953,"props":1954,"children":1955},"ul",{},[1956,1982,1992,2002,2020,2030],{"type":31,"tag":1267,"props":1957,"children":1958},{},[1959,1964,1966,1972,1974,1980],{"type":31,"tag":292,"props":1960,"children":1961},{},[1962],{"type":37,"value":1963},"Capabilities",{"type":37,"value":1965}," are dropped wholesale (",{"type":31,"tag":74,"props":1967,"children":1969},{"className":1968},[],[1970],{"type":37,"value":1971},"DropCapability=ALL",{"type":37,"value":1973},"), then the minimum set is re-added per container. nginx gets ",{"type":31,"tag":74,"props":1975,"children":1977},{"className":1976},[],[1978],{"type":37,"value":1979},"NET_BIND_SERVICE",{"type":37,"value":1981}," to bind port 443. Application containers get nothing beyond the default.",{"type":31,"tag":1267,"props":1983,"children":1984},{},[1985,1990],{"type":31,"tag":292,"props":1986,"children":1987},{},[1988],{"type":37,"value":1989},"NoNewPrivileges",{"type":37,"value":1991}," is set on every container. Escalation through setuid binaries is blocked.",{"type":31,"tag":1267,"props":1993,"children":1994},{},[1995,2000],{"type":31,"tag":292,"props":1996,"children":1997},{},[1998],{"type":37,"value":1999},"Root filesystems",{"type":37,"value":2001}," are read-only, with tmpfs for runtime data (cache, PID files).",{"type":31,"tag":1267,"props":2003,"children":2004},{},[2005,2010,2012,2018],{"type":31,"tag":292,"props":2006,"children":2007},{},[2008],{"type":37,"value":2009},"SELinux",{"type":37,"value":2011}," runs in enforcing mode. Volume mounts use ",{"type":31,"tag":74,"props":2013,"children":2015},{"className":2014},[],[2016],{"type":37,"value":2017},":Z",{"type":37,"value":2019}," labels for mandatory access control. Each container's files are labeled for that container's context only.",{"type":31,"tag":1267,"props":2021,"children":2022},{},[2023,2028],{"type":31,"tag":292,"props":2024,"children":2025},{},[2026],{"type":37,"value":2027},"Resource limits",{"type":37,"value":2029}," are set per container (memory hard and soft limits). A runaway process can't starve other services.",{"type":31,"tag":1267,"props":2031,"children":2032},{},[2033,2038],{"type":31,"tag":292,"props":2034,"children":2035},{},[2036],{"type":37,"value":2037},"Application processes",{"type":37,"value":2039}," run as unprivileged users (a dedicated UID), not root.",{"type":31,"tag":39,"props":2041,"children":2042},{},[2043],{"type":37,"value":2044},"A container escape still faces the read-only host filesystem, SELinux context boundaries, and an instance profile scoped to one customer's resources. The layers above (IAM, network, DNS, storage) don't care what happened inside the container.",{"type":31,"tag":59,"props":2046,"children":2048},{"id":2047},"ingress-two-paths-separate-secrets",[2049],{"type":37,"value":2050},"Ingress: two paths, separate secrets",{"type":31,"tag":39,"props":2052,"children":2053},{},[2054],{"type":37,"value":2055},"Many platforms implement management as routes within the application: same process, same authentication middleware, same database connection. A bug in the app exposes the management surface. A bug in management exposes user data. Here, the management function is not part of the application. It runs on separate infrastructure: a separate entry point, separate compute, separate authentication, separate credentials.",{"type":31,"tag":39,"props":2057,"children":2058},{},[2059],{"type":37,"value":2060},"Traffic reaches a customer's environment through two paths. The data path serves user requests through CloudFront. The admin path handles management operations through API Gateway and Lambda. The two share no secrets.",{"type":31,"tag":39,"props":2062,"children":2063},{"align":168},[2064],{"type":31,"tag":171,"props":2065,"children":2067},{"src":1144,"alt":2066,"width":977},"diagram ingress paths",[],{"type":31,"tag":39,"props":2069,"children":2070},{},[2071,2076,2078,2083],{"type":31,"tag":292,"props":2072,"children":2073},{},[2074],{"type":37,"value":2075},"The data path",{"type":37,"value":2077}," runs through a dedicated CloudFront distribution that accepts one hostname (",{"type":31,"tag":74,"props":2079,"children":2081},{"className":2080},[],[2082],{"type":37,"value":1671},{"type":37,"value":2084},"), serves static assets through Origin Access Control (restricted to this distribution's ARN), and forwards API requests through a VPC Origin to an ENI in the customer's \u002F28 subnet. CloudFront injects a customer-specific secret header into every forwarded request. nginx validates this header before processing anything. A request that didn't arrive through the correct distribution is rejected.",{"type":31,"tag":39,"props":2086,"children":2087},{},[2088,2093],{"type":31,"tag":292,"props":2089,"children":2090},{},[2091],{"type":37,"value":2092},"The admin path",{"type":37,"value":2094}," runs through a per-customer API Gateway and an ephemeral Lambda function in the customer's subnet. API Gateway validates the JWT against this customer's Cognito pool before the request reaches any code. CORS is restricted to the customer's domain. The Lambda runs with a security group that allows egress only to the backend. It starts clean from a container image, processes the request, injects its own secret header (distinct from the CloudFront header), and forwards to nginx. The management service on the instance re-validates the JWT against Cognito's JWKS endpoint from scratch, trusting nothing upstream.",{"type":31,"tag":39,"props":2096,"children":2097},{},[2098],{"type":37,"value":2099},"The two paths share no infrastructure and no credentials. An attacker who obtains the CloudFront origin header can forge data-path requests to one customer's backend but knows nothing about the Lambda secret. An attacker who obtains the Lambda secret can reach the management service but knows nothing about the CloudFront header.",{"type":31,"tag":59,"props":2101,"children":2103},{"id":2102},"where-the-boundaries-intersect",[2104],{"type":37,"value":2105},"Where the boundaries intersect",{"type":31,"tag":39,"props":2107,"children":2108},{},[2109],{"type":37,"value":2110},"What matters for blast radius is what an attacker actually reaches when a component falls. Take the worst case within a customer boundary: a compromised EC2 instance.",{"type":31,"tag":39,"props":2112,"children":2113},{},[2114],{"type":37,"value":2115},"The attacker gets the instance profile credentials, scoped to one customer's buckets, one log group, TXT records in one DNS zone. They get one \u002F28 subnet with no routes to other subnets or the management VPC. They get a read-only filesystem with SELinux enforcing.",{"type":31,"tag":39,"props":2117,"children":2118},{},[2119],{"type":37,"value":2120},"The containment holds because each boundary is enforced by a different AWS mechanism. IAM won't resolve cross-customer ARNs. Security groups have no cross-subnet rules. The Cognito pool is a separate AWS resource with different signing keys. The root volume is read-only (Fedora CoreOS), so there is nowhere to write a persistent backdoor, and the next deployment destroys the instance entirely and boots from a clean Ignition configuration. An IAM exploit doesn't open network routes. A container escape doesn't produce valid Cognito tokens.",{"type":31,"tag":59,"props":2122,"children":2124},{"id":2123},"trade-offs",[2125],{"type":37,"value":2126},"Trade-offs",{"type":31,"tag":39,"props":2128,"children":2129},{},[2130],{"type":37,"value":2131},"Per-customer isolation creates more resources. Each customer adds approximately six IAM resources, two Route53 zones, two S3 buckets, a Cognito pool, a \u002F28 subnet, a CloudFront distribution, an API Gateway with a Lambda function, a Fargate deployer, and a dedicated EC2 instance. The Terraform modules are more complex because they template customer IDs, zone IDs, and subnet ARNs into every resource.",{"type":31,"tag":39,"props":2133,"children":2134},{},[2135],{"type":37,"value":2136},"AWS service limits bound the number of customers per account. The architecture scales through cells: each cell is a separate AWS account with its own VPC pair, its own service limits, and its own blast radius boundary. At tens to low hundreds of customers per cell, the limits are not a constraint. Each cell is itself an isolation boundary, a second containment layer on top of the per-customer isolation within it.",{"type":31,"tag":39,"props":2138,"children":2139},{},[2140],{"type":37,"value":2141},"Each individual resource is simpler to audit than its shared equivalent. Acme's provisioner policy contains Acme's ARNs. No wildcards to reason about. The complexity is in the quantity of resources, not in understanding what any single policy allows. An auditor reviewing Acme's provisioner role sees exactly what it can touch. There is no need to trace application logic to determine whether a shared role might reach another customer's data under certain conditions.",{"type":31,"tag":39,"props":2143,"children":2144},{},[2145],{"type":37,"value":2146},"The operational cost is real: more state files, more deployers, more policies to manage. The architecture manages this through Terraform modules and automated provisioning pipelines. A new customer is a new set of module invocations with a different customer ID. Same modules, different parameters.",{"type":31,"tag":59,"props":2148,"children":2149},{"id":1258},[2150],{"type":37,"value":1261},{"type":31,"tag":1263,"props":2152,"children":2153},{},[2154,2161,2169,2176],{"type":31,"tag":1267,"props":2155,"children":2156},{},[2157],{"type":31,"tag":790,"props":2158,"children":2159},{"href":820},[2160],{"type":37,"value":1274},{"type":31,"tag":1267,"props":2162,"children":2163},{},[2164,2168],{"type":31,"tag":790,"props":2165,"children":2166},{"href":1282},[2167],{"type":37,"value":1285},{"type":37,"value":1276},{"type":31,"tag":1267,"props":2170,"children":2171},{},[2172],{"type":31,"tag":790,"props":2173,"children":2174},{"href":1291},[2175],{"type":37,"value":1294},{"type":31,"tag":1267,"props":2177,"children":2178},{},[2179],{"type":31,"tag":790,"props":2180,"children":2181},{"href":1300},[2182],{"type":37,"value":1303},{"type":31,"tag":1314,"props":2184,"children":2185},{},[],{"type":31,"tag":39,"props":2187,"children":2188},{},[2189],{"type":31,"tag":43,"props":2190,"children":2191},{},[2192,2193,2198],{"type":37,"value":1324},{"type":31,"tag":790,"props":2194,"children":2196},{"href":1327,"rel":2195},[794],[2197],{"type":37,"value":1331},{"type":37,"value":1333},{"type":31,"tag":2200,"props":2201,"children":2202},"style",{},[2203],{"type":37,"value":2204},"html .default .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .shiki span {color: var(--shiki-default);background: var(--shiki-default-bg);font-style: var(--shiki-default-font-style);font-weight: var(--shiki-default-font-weight);text-decoration: var(--shiki-default-text-decoration);}html .dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}html.dark .shiki span {color: var(--shiki-dark);background: var(--shiki-dark-bg);font-style: var(--shiki-dark-font-style);font-weight: var(--shiki-dark-font-weight);text-decoration: var(--shiki-dark-text-decoration);}",{"title":7,"searchDepth":801,"depth":801,"links":2206},[2207,2208,2209,2210,2211,2212,2213,2214,2215,2216,2217,2218],{"id":1385,"depth":804,"text":1388},{"id":1425,"depth":804,"text":1428},{"id":1631,"depth":804,"text":1634},{"id":1657,"depth":804,"text":1660},{"id":1796,"depth":804,"text":1799},{"id":1817,"depth":804,"text":1820},{"id":1898,"depth":804,"text":1901},{"id":1933,"depth":804,"text":1936},{"id":2047,"depth":804,"text":2050},{"id":2102,"depth":804,"text":2105},{"id":2123,"depth":804,"text":2126},{"id":1258,"depth":804,"text":1261},"content:insights:blast-radius.md","insights\u002Fblast-radius.md","insights\u002Fblast-radius",{"loc":1282},1792508656765]