Who is accountable when the colleague is an agent?

Financial services is going to need AI it can trust. Several bodies have set out what that might mean: the OECD through its AI principles, NIST through its risk management framework, and UNESCO through a substantial body of work on the ethics of AI.

This is one of the questions at the centre of my current postgraduate study in AI ethics at the London School of Economics. I am still forming my own view of what a trustworthy AI framework needs to contain, and I will develop and publish that work as it takes shape.

What I have reached so far is eight elements that play a role worth assessing in establishing trust. I am not ready to set all eight out yet. One, though, is a current focus, in large part because it keeps coming up with clients, and that is accountability.

This is not new territory for our sector. The three lines of defence rest on knowing who is accountable for what. Regimes such as the Senior Managers and Certification Regime in the UK and the Financial Accountability Regime in Australia and many similar regulations globally, exist because clear accountability changes behaviour, and changes it in the right direction for good conduct. So the real question is what happens to that clarity once we put a ‘human in the loop’ (a term I would love to see changed), alongside an AI agent. This is happening more and more.

Deciding who is accountable for what is the relatively straightforward part, although an essential component. The harder part is what accountability does to the motivation of the human in that relationship. When two people work together, responsibility is shared, even if proportionately, and the consequences of poor work look broadly similar for both: a difficult appraisal, a bonus conversation that goes the wrong way, feedback delivered in front of peers. When a person works with an agent, the consequences are asymmetrical. An agent cannot lose its bonus, sit through an uncomfortable review, or feel the weight of a meeting that goes badly. The exposure sits almost entirely with the person.

There is research that speaks directly to this. In a study titled AI teammates and human performance: evidence for commitment deficits, Jonathan Winter found that human effort fell by around 26% when people worked alongside an AI teammate rather than another person. The pattern has a name, a commitment deficit. People do not expect the agent to reciprocate their effort or to carry the consequences of weak work, and their own sense of ownership erodes. Less commitment, less pride in the output, a looser hold on the outcome.

For risk and compliance, this is not a side issue. It will shape how we build hybrid teams, where accountability sits, and how the business runs once agents are handed real work and, increasingly, real decisions.

So, I propose a few questions worth considering as these teams take shape.

• Is the accountability captured and genuinely clear?

• Can a poor agent outcome be escalated easily, without the human becoming a scapegoat for something they did not control?

• Do our policies reflect the new reality, for example, does our code of conduct or speak-up process recognise agents as actors that carry tasks, decisions and outcomes?

• Do our stated behaviours, our values and our training reflect it too?

• Is there enough psychological safety for people to raise problems with an agent, so that adoption stays safe and the commitment deficit never takes hold?

We already know that clear accountability works. The task now is to keep it clear when one of the parties is not human.

Further reading: Jonathan Winter, AI teammates and human performance: evidence for commitment deficits https://bit.ly/4g07BfO