AI Governance And The Employee In The Room

Most of the conversation around AI in financial services has focused on model risk. Is the output accurate? Is the decision explainable? Can the firm demonstrate to a regulator how the model reached its conclusion?

These are the right questions. But they are not the only ones.

There is a quieter set of ethical obligations beginning to surface, and I don't think the industry has properly reckoned with them yet. When AI is used to monitor employee conduct, flag behavioural patterns, or assist in surveillance decisions, something more than model accuracy is at stake. The question is not just whether the system works. It is what the system does to the people it watches.

This matters to me partly because of where my work sits, at the intersection of surveillance, conduct risk, and ethics, and partly because I am currently completing postgraduate research on Ethical AI. What I am noticing is a gap between where regulatory frameworks are focused and where the real ethical weight lies.

The FCA, ASIC, and others have all signalled attention to AI governance in recent years. The frameworks emerging from these signals are largely oriented toward consumer outcomes and model explainability. For firms operating outside EU AI Act jurisdiction, there may be a gap: as few regulators are asking what duties a firm owes to its own employees when AI shapes decisions about their conduct, their careers, or their continued employment.

Employee surveillance is not new. What is new is the scale, the subtlety, and the degree to which patterns identified by a system can inform consequential decisions before a human has properly interrogated the output. That is a different kind of accountability question. And right now, most firms are not asking it.

I am not suggesting firms stop using these tools. The operational case for AI-assisted surveillance is real and regulators are responding with interest. What I am suggesting is that responsible use requires more than technical governance. It requires a set of ethical commitments about how people are treated when they are on the receiving end of algorithmic judgement.

I will be returning to this topic over the coming months as my research develops. For now, I am curious: is your firm asking these questions? And if so, where is the conversation sitting? In compliance, in HR, in legal? Or nowhere in particular?