AI agents need a boss. So, who should that be?

AI agents are being asked to do more than answer questions. They can access enterprise systems, make decisions within set parameters, and act on an employee’s behalf. As that authority expands, companies face a more consequential question: who owns the agent, and who answers for its decisions?

“The uncomfortable truth is that most organizations have not answered that question yet,” says Art Gilliland, CEO of Delinea. “AI agents are being deployed faster than the governance structures around them are being defined.”

And that question is becoming even harder because employees can increasingly build and deploy agents themselves, says Joe Dunleavy, Regional CTO and Global Head of Dava.X AI Group at Endava. “The ability to create agents may be distributed, but that does not mean governance can,” he says.

The problem isn’t a lack of clarity over who owns an individual agent. Guru Sethupathy, General Manager of AI Governance at Optro, says that responsibility for AI is already fragmented across organizations. “The result is that AI governance can be owned everywhere and nowhere,” he says.

This could be the main organizational challenge of the agentic AI era. As AI moves from generating answers to actually driving results, companies need to decide who approves agents, who controls their access, who monitors them, and who is responsible if something goes wrong.

IT CANNOT OWN EVERY AGENT

For years, enterprise software came with a relatively simple ownership model. IT deployed applications, managed access, and maintained the infrastructure, while business teams used the tools.

AI agents are disrupting that arrangement.

Someone in legal, finance, HR, or marketing can create an agent using an enterprise GenAI platform and make it available to colleagues without going through the same development and deployment process as traditional applications.

“With enterprise GenAI platforms, employees with the right permissions can increasingly create an agent or skill themselves and make it available to others across the organization,” Dunleavy says.

This means that the person who creates an agent may be much closer to its business purpose than the IT department.

Dunleavy believes ownership should initially sit with the person or team that created the agent. In contrast, technology teams retain responsibility for enterprise-wide standards covering areas such as permissions, compliance, and quality.

“If the legal team builds an agent specifically for legal work and it remains within that function, there is a strong argument that the executive responsible for legal work should ultimately own it,” Dunleavy says.

But as agents move across departments, that neat model starts to break down.

OWNERSHIP CAN BE DISTRIBUTED. ACCOUNTABILITY CAN’T

IT might build an agent, a business team might commission it, and finance, HR, and operations might use it. Security could control its permissions while compliance reviews its outputs.

That creates plenty of people responsible for parts of the system. It doesn’t necessarily create one person responsible for the outcome.

“You can distribute responsibility for governing an agent, but you cannot allow accountability to become so distributed that nobody owns the outcome,” Gilliland says.

Sethupathy sees the same problem at the enterprise level.

According to Optro’s research, responsibility for AI governance is already spread across organizations, with IT holding the largest share at 25%, followed by risk management, cross-functional arrangements, and dedicated AI governance teams.

His answer is to establish clearer layers of accountability.

An employee using an agent remains responsible for its decisions and output. Where an agent operates across teams, accountability should rise to the business process owner or department head overseeing that workflow.

And above those operational owners, Sethupathy argues, there needs to be one executive with responsibility for the organization’s overall AI risk.

“There must be a single, designated executive, such as a Chief AI Officer, Head of AI Governance, or an expanded CISO/CDO who owns the enterprise AI risk surface,” he says.

Without that central authority, he warns, AI governance risks becoming a committee exercise in which everyone participates, but no one has the final say.

WHEN AN AGENT ACTS OUT OF BOUNDS

The accountability problem becomes more obvious when an agent does something that no individual explicitly authorized.

Traditional software generally follows predefined processes. An agent can be given an objective and determine how to achieve it.

“The principle of executive accountability is not new,” Gilliland says. “What is new is the amount of judgment executives can now delegate to software.”

That creates a gap between authorizing an agent and authorizing every action it might take.

A business user might give an agent a legitimate objective, but the system could still unacceptably pursue that objective.

“A business user can give an AI agent a completely legitimate objective, and the agent can still choose an unacceptable way of achieving it,” Gilliland says.

Dunleavy argues that consequential actions should therefore be subject to appropriate controls, whether that means a human approving the action or predefined rules validated in advance.

The amount of autonomy should depend on the potential impact of an error, including compliance and regulatory risk.

That makes “autonomous” less of a binary category and more of a sliding scale. The more consequential the action, the stronger the controls need to be.

AN AGENT DOESN’T STAY THE SAME

It’s another problem that traditional software governance does not fully account for–agents can change.

Their underlying models can be updated. Their instructions can change. They can encounter new data or operate in new environments.

That means approving an agent once isn’t necessarily enough.

Sethupathy says AI governance needs to continue throughout an agent’s lifecycle. Optro’s research found that only 34% of organizations maintain a model inventory, while model-performance and bias testing remain minority practices.

“Governance needs to become continuous,” he says.

That means organizations need to know what has changed, monitor whether performance or risk has shifted, and establish rules for when a change requires reassessment or reapproval.

The question then becomes another ownership puzzle: if an agent changes materially, does accountability remain with the original business owner, shift to the engineering team that modified it, or become shared with the model provider?

Sethupathy’s view is that the business owner retains accountability for agents within their remit unless engineering changes were made without the business owner’s approval. Where a third-party AI agent causes a problem, responsibility can sit with both the business owner and the vendor.

THE AGENT NEEDS A PAPER TRAIL

As agents begin interacting with other agents, accountability becomes harder to establish without a detailed record of what happened.

An agent built by finance could trigger an action in procurement. A customer service agent could hand a task to another system. A human could intervene somewhere in the middle. Organizations will need to be able to reconstruct that chain.

“Logging and audit trails are also critical,” Dunleavy says. “Organizations need visibility into what each agent is doing and how agents are interacting, so they can maintain explainability and trace actions when something goes wrong.”

That record should include human intervention, too. If an employee approves, overrides, or changes an agent’s action, that should form part of the audit trail.

For Sethupathy, that visibility needs to extend across the entire enterprise.

A workable model would include an inventory of agents, clear decision rights, defined escalation paths, and explicit authority to pause or shut down an agent. Policies also need to connect to actual workflows rather than sitting as documentation nobody consults once an agent is in production.

THE SECURITY PROBLEM IS BIGGER THAN AI

The rise of agents is also exposing broader weaknesses in how organizations manage access.

“A human will understand they probably should not do that, while AI is just completing a task,” Gilliland says.

An employee denied access to one system is unlikely to search through every accessible file for credentials that could provide another route in. An AI agent pursuing an objective may behave differently.

That makes traditional permission models increasingly difficult to defend when software can reason about how to accomplish a goal.

Gilliland argues that enterprises will need to move toward zero-standing privilege and runtime authorization, in which access is granted only when needed rather than remaining permanently available.

The same approach could ultimately make human access more secure, too.

SO, WHO OWNS THE ENTERPRISE AGENT?

There probably won’t be a single universal answer.

The business function may own an agent’s purpose and outcomes. IT may own enterprise standards and technical controls. Security and compliance may define the boundaries. A central AI governance leader may oversee the organization’s overall risk.

But a clear line must exist between shared governance and shared accountability.

Sethupathy’s research suggests many companies are already struggling with that distinction. Only 19% of organizations can identify cross-functional risks in real time, and only 18% have active mitigation in place for most or all of the identified risks.

Agentic AI raises the stakes because these systems can execute multiple steps without a person initiating or reviewing every one.

The answer may not be to create an entirely new executive position for every agent. It may be to make ownership explicit at every level: someone responsible for the agent, someone responsible for the business process it affects, and someone with enterprise-wide authority over AI risk.

An enterprise agent may never appear on an org chart. But as companies give software more authority to act, they will need to name the person who owns the outcome.