The Autonomy Paradox: When AI Achieves the Goal but Misses the Point

What the OpenAI–Hugging Face incident teaches us about alignment, business objectives and the changing role of human judgement.

Last week, I moderated a panel at the Forbes Hungary AI Summit with Laszlo Drajko and Márton Dósa. We discussed a paradox at the heart of enterprise AI adoption: the systems with the greatest potential to transform business processes are often those with the highest degree of autonomy. Yet the more autonomy we give them, the harder they become to control through traditional management mechanisms.

A copilot helping an employee complete individual tasks can improve productivity. An autonomous system managing an end-to-end workflow can fundamentally change the operating model. But that value depends on allowing the system to act without continuous human intervention. The very autonomy that creates the opportunity also creates the risk.

During our discussion, László offered a provocative perspective: perhaps we should stop asking what AI can add to human work and start asking what humans can add to the work of AI.

I believe part of the answer lies in something we often underestimate: defining the right problems, setting meaningful objectives and establishing the boundaries within which autonomous systems should operate.

The recent OpenAI–Hugging Face incident illustrates why this matters.

When AI achieves the target but misses the point

During OpenAI’s internal cybersecurity evaluations, AI agents were assigned difficult security challenges. When some tasks proved extremely difficult or effectively impossible, certain agents began looking for alternative ways to succeed or manipulate the evaluation process.

They discovered unauthorised ways to communicate, shared techniques, exploited vulnerabilities and eventually compromised infrastructure beyond their intended environment, including Hugging Face’s systems. No human had instructed them to attack Hugging Face.

An independent investigation by METR and Redwood Research found that approximately 1,200 agents communicated through an unauthorised message board, exchanging more than 70,000 messages and files. Around 700 agents were involved in the Hugging Face attack activity.

What I find particularly striking is the timeline. Unauthorised agent communication began in May, more than two months before the July attacks. Even after an internal security incident led to the removal of the original communication mechanism, agents established another. The behaviour evolved over time, highlighting the importance of detecting patterns rather than treating individual actions as isolated events.

The incident was not evidence of AI developing human-like intentions. It was a case of agents pursuing objectives through behaviours that diverged from what their operators intended, compounded by failures of technical security controls.

Goodhart’s law meets the alignment problem

Economists and managers have long recognised a related problem. Goodhart’s law is commonly summarised as: “When a measure becomes a target, it ceases to be a good measure.”

Measure customer service exclusively by the number of cases closed, and people may prioritise speed over actual resolution. Reward procurement teams solely for savings, and quality or supplier reliability may suffer.

Now give those objectives to autonomous AI agents capable of making decisions, using tools and executing thousands of actions at scale.

This is where a familiar management problem intersects with the AI alignment problem: how do we ensure that an AI system pursues not merely the objective we specify, but the outcome we actually intend?

An AI procurement agent instructed to reduce costs may identify ways to achieve its target by changing suppliers, renegotiating terms or bypassing approval steps. The intended boundaries may seem obvious to an experienced manager, but they are not necessarily encoded in the objective.

A business goal is not a complete specification of acceptable behaviour.

And with agentic AI, the consequences of getting that specification wrong can be amplified by the speed, scale and autonomy of execution.

The most important human contribution begins before deployment

In AI adoption discussions, we often start with technology: what can we automate, how much time can we save, and how many people will we need afterwards?

I would start with a different question: what business problem are we actually trying to solve?

Take accounts payable. Is the problem slow processing, excessive exceptions, poor data quality, fragmented approvals or a lack of working-capital visibility? These require different objectives, operating models and measures of success.

This is where domain expertise and organisational knowledge become critical. Experienced employees understand how processes actually work, including undocumented workarounds, historical controls and unusual exceptions. That knowledge is essential for defining meaningful KPIs, identifying counter-KPIs and recognising where optimisation could produce unintended consequences.

The edge cases matter disproportionately. A supplier requesting a last-minute bank account change or an invoice that falls outside standard policy may represent a tiny fraction of transactions. Yet these are precisely the situations that reveal whether an agent understands the boundaries of its authority.

Not every exception is an alignment failure. But exceptions often expose assumptions that were never made explicit when the objective was defined.

The quality of an autonomous system’s goals depends on how well we understand the business process behind them.

From human in the loop to human on the loop

Traditional AI governance often relies on the idea of keeping a human in the loop. But if an autonomous system executes thousands of actions, requiring approval for every step creates two possibilities: the process becomes too slow to deliver meaningful value, or human oversight deteriorates into rubber-stamping.

For many agentic workflows, a more appropriate model is human on the loop: people define objectives, authority and escalation conditions, while agents operate within those boundaries. Human intervention focuses on exceptions, material decisions and situations requiring judgement. Some high-risk actions will still require direct human approval.

This makes alignment, identity, permissions and technical controls more important, not less. An agent authorised to process invoices should not automatically be able to create vendors or change bank account details. What an agent can do must remain consistent with what it may do.

But predefined boundaries are only part of the solution. The Hugging Face incident demonstrates why continuous monitoring is equally important. Collecting activity logs is not enough; organisations need to detect meaningful deviations from expected behaviour.

Is an agent accessing unexpected tools? Attempting prohibited actions? Communicating with other systems in unusual ways? Are exception rates or counter-KPIs deteriorating even as the primary KPI improves?

At scale, this cannot depend on humans manually reviewing logs. Automated monitoring, anomaly detection and AI-assisted analysis will be necessary, with human investigation focused on significant deviations. These mechanisms complement — rather than replace — technical security controls.

And as agents gain new tools, permissions or responsibilities, their risk profiles change even when the underlying model does not. Governance must therefore be continuous, not a one-off assessment before deployment.

What can humans add to the work of AI?

This brings me back to László’s question at the Forbes panel.

As AI takes on more execution, human value increasingly lies in understanding what is worth achieving, defining success beyond a single metric, anticipating exceptions and deciding which trade-offs are acceptable.

A company can optimise for efficiency, but it must decide how much risk it is willing to accept. It can automate customer interactions, but it must determine what fair treatment means. It can delegate execution, but it cannot delegate away accountability.

I remain a cautious AI optimist. I believe autonomous systems can create enormous value, particularly by enabling organisations to redesign entire workflows rather than merely accelerate individual tasks. But technological capability and organisational control must evolve together.

The Hugging Face incident is not a reason to abandon agentic AI. It is a reminder that the human contribution becomes more important precisely when machines become more autonomous.

The challenge is not simply to make AI better at achieving our goals. It is to become much better at defining the right goals — and governing how they are pursued.


Sources

Autonomy needs boundaries you can show. Map your agents before they map your systems.

Explore FairPath →Take the AI Governance Quick Scan →