An AI agent got it wrong. Who answers for it in your company?

AI agents already approve requests and change systems. New studies show their mistakes reach the board before the rules do. What to define before you switch yours on.

A year ago, the question in board meetings was “should we use AI?”. Today, in many companies, the question has changed without anyone noticing: “the AI did something wrong. So whose fault is it?”

That shift shows up clearly in a study IBM released last week. And it matters to anyone in Brazil, even though the study didn’t survey Brazilian companies. Let me explain why.

What the IBM study showed

IBM commissioned Censuswide to survey 3,221 executives at companies with more than a thousand employees, across eight countries in Europe and the Middle East. The fieldwork was done in September and the results came out on October 14, on IBM’s newsroom.

The numbers that stand out most:

  • 80% say an incident involving AI reached the board or the C-suite in the past year.
  • Only 40% of those companies changed their AI usage rules afterward.
  • 35% already have AI agents running in production.
  • 75% expect these agents to make some kind of decision on their own by 2027, but only 37% say their current governance was designed for that.

In plain English: the problem has already landed on the CEO’s desk, and most companies still haven’t changed anything about how they work.

The IBM executive who commented on the study, Ana Paula Assis, summed up the moment like this: the debate has moved from adoption to “governance and accountability”. I agree. And I’d add: whoever doesn’t define this beforehand will end up defining it in the middle of a crisis.

An agent is not a chatbot

It’s worth separating the two, because many companies still mix them up.

A chatbot answers. If it’s wrong, someone reads the wrong answer and, with luck, notices. An AI agent acts. It approves a request, changes a customer record, sends an email to a client, posts an amount in the finance system. When it’s wrong, the mistake has already happened before any person looks at it. It’s no coincidence that Microsoft started treating agents as an insider risk, as we covered here on the blog.

A Kore.ai survey, carried out in May with more than 400 IT leaders at large US companies, shows the scale of this. According to the company’s own survey, 79% have already had to undo an action taken by an agent, and 70% faced a failure the team couldn’t trace. And 26% of these companies’ agents already approve or deny decisions.

A caveat: Kore.ai sells an agent platform, so it has a stake in the subject. Even so, the picture matches IBM’s and what I see day to day. The technology became ready to act before companies became ready to answer for its actions.

The gap nobody wants to own

The IBM figure that bothered me most was a different one. Asked who should answer when an agent causes a loss, the executives reached no agreement at all. The most cited role was CIO or CTO, and even then only 17% chose that option.

In practice, this means that in most companies the answer to “whose fault is it?” will be decided after the problem, in a witch-hunt atmosphere.

Then there’s the question of suppliers. More than half of respondents (54%) admit their rules don’t properly cover AI used by partners and service providers. Only 21% monitor this risk continuously. If your outsourced customer service put an agent to talk to your customers, its mistakes are yours too.

And in Brazil, what applies today

Here, many people are waiting for Brazil’s AI Legal Framework (Marco Legal da IA) to “see what to do”. Bill PL 2338/2023 remains stalled in a special committee of the Chamber of Deputies, and the rapporteur himself said in August that the vote would be left until after the elections, according to the legislative tracking by LCF Consulting. There’s no date.

Waiting for the new law is a mistake, because an old law already covers much of the problem. The LGPD, Brazil’s data protection law, in its Article 20, guarantees data subjects the right to request a review of decisions made solely on the basis of automated processing that affect their interests. That includes credit, consumer and professional profiles. The company also has to explain, when asked, the criteria used in the decision.

Translating that to the world of agents: if your AI denies credit, rejects a candidate or changes a customer’s commercial terms, you need to be able to explain how it got there. And the same law’s sanctions include fines of up to 2% of revenue in Brazil, capped at R$ 50 million per violation.

An agent nobody can trace isn’t just an operational risk. It’s a legal risk that already exists.

What I would define before switching on any agent

You don’t need a committee, a six-month consulting project or a hundred-page document. You need clear answers to five questions, in writing, before the first agent goes into production.

1. Who owns this agent? A person with a name and a job title, not “IT”. They answer for what the agent does, the same way a manager answers for their team.

2. How far can it go on its own? Set concrete limits. For example: it can answer questions and open tickets, but any discount above a certain amount or any change to a contract goes through a person. The IBM survey shows executives themselves want this: 32% cite clear limits for escalating a decision to a human as a need by 2027.

3. Can you see what it did? Every action has to be logged: what it received, what it decided and why. Without that history, you can’t fix the mistake, you can’t learn from it and you can’t respond to an LGPD review request.

4. Can you shut it down quickly? It sounds obvious, but many deployments have no stop button. If the agent starts making mistakes in a row on a Friday night, someone needs to be able to halt everything within minutes, without depending on the vendor.

5. What does the vendor contract say? Who pays if the partner’s agent causes a loss? What data does it use? Most software contracts still say nothing about this. It’s worth renegotiating before renewing.

Notice that none of these questions is technical. They are management decisions. Technology’s role is to make the answers possible: logs, limits, a stop button.

My view on this moment

I don’t think the answer is to hit the brakes. IBM itself shows that most companies prefer to keep deploying within agreed limits while they adjust governance. That makes sense. Whoever stops everything will lose productivity to the competitor that learned to use it safely.

The mistake is switching the agent on thinking it’s just another tool, like a new spreadsheet. A spreadsheet doesn’t make decisions. An agent does, at scale and without asking permission. The right question isn’t whether it’s good, it’s what happens on the day it gets something wrong.

Where experience makes a difference

In 29 years of programming, I’ve followed the arrival of the first management systems, of e-commerce and of process automation. In every one of them, the pattern repeated: the technology arrives, everyone rushes to use it, and only later does someone ask who takes care of it when things go wrong. The companies that did well were the ones that asked that question first.

That’s how we work at IWA7. Before putting an agent to work, we sit down with the people who know the operation to design the limits, the logs and the exact point where a person steps into the decision. The technical part comes later, and it gets simpler precisely because of that.

If your company already has agents running, or is about to, start with the five questions above. If any of them goes unanswered, that’s where your next problem is.


Originally published on IWA7 Insights, IWA7 Digital’s blog, which is now part of Escovando Bytes.

Keep readingSee all →