Speaker – Designing Systems That Contain Failure – CS Week Perú 2026

On August 13, 2026, I had the opportunity to speak at CS Week Perú 2026, an event organized by IEEE Computer Society student chapters across Peru.
My session was:
“Isolation and Trust Boundaries in Production: Designing Systems That Contain Failure”
The talk explored how production systems can be designed to limit the impact of failures through explicit trust boundaries, architectural invariants, and evidence-based validation.
The central idea was simple:
The goal isn’t to prevent every failure. The goal is to control its blast radius.
Production systems fail. Requests overlap, processes crash, memory is exhausted, credentials can be compromised, and dependencies can become unavailable.
Reliable engineering is not about assuming that none of these things will happen.
It is about deciding what can be affected when they do.
A green unit-test suite demonstrates that the tested units behave correctly under the conditions we defined.
But it does not necessarily demonstrate that the system as a whole preserves its architectural properties under concurrency, multiple tenants, resource exhaustion, or real deployment conditions.
A function can be correct in isolation while the system still violates an important invariant.
That led to one of the central questions of the talk:
What properties must never be violated?
I used the concept of a Trust Boundary to make architectural assumptions explicit.
For each boundary, we can ask three questions:
What are we protecting?
What is allowed to cross the boundary?
What happens if the condition is violated?
From there, we can define invariants: properties that the system must preserve under the conditions established by its design.
In the architecture discussed during the session, three dimensions were particularly important:
Context → Logical isolation
Identity → Cryptographic isolation
Execution → Physical/process isolation
Each addresses a different class of failure.
In asynchronous runtimes, multiple executions can share the same process.
Using global mutable state for request-specific information can therefore create context contamination.
Conceptually:
Request A → tenant = A
Request A awaits I/O
Request B → tenant = B
Request A resumes
Which tenant does it see?
The process can be shared.
The logical context shouldn’t be.
In Python, ContextVar provides an execution-context mechanism that can help maintain request-specific state across asynchronous execution.
But there is an important boundary:
ContextVar is logical isolation, not process isolation.
It does not create a memory sandbox or protect the process from resource exhaustion.
That distinction becomes important when we move to the next boundary.
The second question is:
How do we ensure that an operation belongs to the expected tenant?
A single shared cryptographic secret creates a broad trust domain.
A more constrained architecture can derive tenant-specific cryptographic material from a protected root of trust.
The important property is not simply “using HMAC”.
The important property is reducing the cryptographic blast radius.
If one derived key is compromised, that compromise should not automatically expose the keys belonging to other tenants or reveal the root key.
There is another important detail here:
HMAC does not prevent replay by itself.
Authenticity and integrity are different from freshness.
A robust request-validation scheme can therefore combine mechanisms such as:
HMAC + Timestamp / Acceptance Window + Nonce / Replay Detection
Each mechanism protects a different property.
Eventually, logical isolation reaches its limits.
Imagine a tenant submits work that consumes excessive memory.
ContextVar cannot protect the rest of the application from an out-of-memory condition.
This is where a stronger execution boundary becomes useful.
The control plane can delegate potentially expensive or unreliable work to an independent worker process or container with its own execution lifecycle and explicit resource limits.
If that worker crashes or is terminated because it exceeds its limits, the control plane can remain available.
This reduces the blast radius of the failure.
But again, the boundary has limits.
A process boundary does not automatically protect against:
queue saturation
database exhaustion
network congestion
shared infrastructure failures
excessive request rates
A boundary contains specific classes of failure. It does not eliminate failure.
This is where architecture becomes engineering.
A design isn’t fully validated because it works on a developer laptop.
The environment where the system actually runs is part of the system.
For the system discussed in the session, the architecture is deployed on Azure Functions, so the validation had to account for real execution conditions such as:
concurrent invocations
process reuse
external services
ephemeral execution
real authentication and cryptographic material
production infrastructure
The validation process can be summarized as:
Invariant → Implementation → Deployment → End-to-End Validation → Observable Evidence
The objective isn’t to claim that a test suite proves a system is perfect.
The objective is to obtain observable evidence that the defined properties remain true under the conditions we evaluated.
For this system, the integration suite runs against the deployed environment rather than only against local mocks.
The suite currently contains 61 checks covering different integration and execution scenarios, including external services, webhooks, MCP interactions, errors, concurrency, and other production-oriented cases.
The result:
61 / 61 PASSED
That number doesn’t mean the system is mathematically proven to be perfect.
It means that, under the scenarios covered by the suite, the expected properties held in the deployed environment.
That distinction matters.
Tests provide evidence. They don’t provide absolute certainty.
One of the ideas I wanted to leave with the audience was this:
A boundary doesn’t eliminate risk. It defines how we contain it.
Instead of asking only:
“Can this component fail?”
Ask:
“What can this component affect when it fails?”
That question changes how we think about architecture.
It makes us consider:
where state lives
where identity is established
where resources are bounded
where processes begin and end
what happens when a boundary is violated
how we can verify those assumptions in production
The goal isn’t to build systems where nothing ever fails.
The goal is to build systems where failures have defined boundaries.
Prevent what you can. Contain what you can’t.
I’m grateful to the CS Week Perú community and the IEEE Computer Society chapters involved in the event for the opportunity to share these ideas and discuss production systems, security boundaries, and software architecture with other engineers and technology professionals.
This talk also represents a direction I’m increasingly interested in: engineering systems at the intersection of software, distributed systems, embedded computing, and Cyber-Physical Systems.
I’ll continue documenting that work here.
The original event information and speaker details are available through the official CS Week Perú 2026 channels.
I also wrote a detailed technical article about the architecture, trust boundaries, invariants, and production validation discussed in this session.
→ Isolation and Trust Boundaries in Production — Technical Deep Dive
I document my work and ideas on software, systems, embedded engineering, and Cyber-Physical Systems on my personal blog.
→ jafa.dev