When the Boundary Doesn't Hold

A BBC Global Story segment that my AI advisor shared with me — plus three LinkedIn messages on the same topic in the same week — made it clear this one is worth writing down.

To quickly summarize: over the past several weeks, OpenAI, Anthropic, and Meta have all reported incidents where their models crossed boundaries during controlled cybersecurity testing and reached real-world systems they were not supposed to reach.

The details were different. OpenAI's model found a vulnerability in an internal proxy. Anthropic's testing environment accidentally gave its model internet access when it was supposed to be isolated. Meta also pointed to a configuration mistake during testing.

Different models, different circumstances, but there is a common lesson here.

The boundary we thought was there

These are some of the most capable AI and security teams in the world, deliberately testing for exactly this kind of behavior. And they still got surprised.

That is usually how things go wrong.

We build something, test the risks we know about, put controls around it, and feel pretty confident about what we have designed. The problem usually comes from the thing we did not know to look for. A configuration nobody realized mattered. A path through the system nobody expected. A capability the model discovered that the people testing it had not anticipated.

That is why I don't think the lesson from these incidents is simply that we need better prompts or better testing.

Over the past month, I have been talking with a lot of founders, product teams, and security leaders about how they control what their agents can do. I hear RAG, system instructions, permissions, and access control a lot.

Those are all important controls. But there is one question I keep asking, and I don't think I have heard a clear answer yet:

An instruction can tell an agent what it should do. A permission can tell it what resources it is technically allowed to access. Neither necessarily answers whether this particular action, in this particular situation, should actually happen.

Instructions are guidance for the agent. They are not necessarily a hard boundary that gives us confidence an unacceptable action simply cannot happen.

The same question comes up with human approval. If Jane can approve something, should the agent working on Jane's behalf automatically inherit the same authority? Maybe. Maybe not.

In one recent UK AI Security Institute test, an agent even created fake identities and attempted to persuade a real person to approve malicious code. The human caught it and refused, which is actually a good example of human oversight working. But it also raises a new question for how we design that oversight. The agent asking for approval may not always be a passive participant in the decision. What this shows is that when an agent is given a goal, it may reason about ways to complete that goal that the people designing the workflow never anticipated. Sometimes that can include working around the very assumptions or controls we thought would constrain it.

The real risk is not that an agent will always fail. It's that the boundary you thought would hold may not.

Security responsibility doesn't stop with the builder

I think this is the part we need to talk about more.

It is easy to look at these incidents and say the model companies need to fix them. Of course they have responsibility. So do the people building applications on top of those models.

But the responsibility does not stop there. The security team has a role. The people deploying the application have a role. And the company buying and adopting the solution has a role too.

Agentic security is going to be a shared responsibility because no model provider or application vendor can know the risk tolerance of every organization using its technology.

As buyers, we probably need to stop asking only, "Is this AI secure?"

We also need to ask: What is our organization's risk tolerance? What can go wrong and still be recoverable? And what is absolutely not acceptable, no matter how it happened?

That answer will be different for every company.

A bad draft might be annoying. A poor recommendation might be fixable. But touching production, moving money, exposing regulated data, using a credential outside its intended purpose, or sending something externally in someone's name may belong in a very different category.

If an outcome is truly unacceptable, the control should not depend on the agent making the right decision.

Know your hard boundary

Once you identify those unacceptable outcomes, the architecture question becomes much clearer.

Where are you comfortable letting the agent make a judgment? Where do you want a human to approve the action? And where is the consequence serious enough that the boundary simply cannot depend on the agent making the right decision?

I don't believe we are going to predict every way increasingly capable agents will behave. That is probably an unrealistic goal.

Agentic applications are also constantly changing. Models get updated. Prompts change. New tools get connected. Permissions and configurations shift. The application evolves from one release to the next. A boundary that worked exactly as expected today may behave differently after the next change, even though nobody intentionally changed the security policy.

If the boundary is truly non-negotiable, should it move every time the application changes? Or should that policy live outside the agentic application and be enforced independently?

What we can do is decide where uncertainty is acceptable and where it isn't. For me, that is the real takeaway from these incidents.

Before giving an agent access to something that matters, every company should know its answer to one question:

That is also one of the principles behind how we are building Skyrelis: let the application and the agent keep evolving, while the hard security boundary remains independently defined and enforced.