The Loan Officer and the Algorithm

A teaching story, a papal encyclical and two practitioners on the same point: decide deliberately which decisions an agent settles, and which a human must own.

Hello from Madrid.

The Pope is in my city this weekend, celebrating Corpus Christi in Madrid. Today more than a million people filled the streets around Cibeles for an open-air Mass, all week there has been a feeling of warmth and peace. Whatever your faith, the atmosphere was undeniable and a good backdrop for today’s post on the Ethics of AI.

In 20 seconds

Pope Leo XIV's first encyclical, Magnifica Humanitas, is about artificial intelligence. For me personally and for Trusted Agents it’s one of the most helpful things written on the subject, an advocacy for human compassion in an increasingly algorithmic world. Simon Willison, an engineer many of you will recognise, called it "some of the clearest writing I've seen on the ethics of integrating AI into modern society."

Anthropic's co-founder shared the stage at its launch. You do not need to be Catholic to use this. Read as a working document, it is a governance vocabulary, and it arrives exactly as agentic systems start making decisions on your customers' behalf.

Ths week's shift

The most practical AI governance text of the year has gone into wide circulation, and it draws its brightest line precisely where agentic commerce is most tempted to cross it: the point where a machine decides something that changes a person's life.

What happened.

​Pope Leo XIV released his first encyclical, centred entirely on AI, and arrived in Madrid this week to record crowds. The AI industry was in the room for the launch.

Why it matters to you.

It gives non-technical leaders a plain test for where an agent may act on its own, and where a named human has to stay accountable.

The decision it forces

For every delegated customer journey, decide what an agent may settle by itself, and where a person signs.

Why this should matter to you

Let me tell you a story I use with banks and other enterprise organizations we work with.

Loan approval looks like a perfect candidate for agentic automation. It is heavy with paperwork, policy and regulation. For the bank, the answer can look black and white, a risk calculation with a clear threshold. Hand it to an agent and the queue clears overnight.

So here is what I ask lenders to do. Use AI to pre-process the application, by all means. Then, for the final signature, ask the applicant to come into the branch and sit with a loans officer. A human, and ideally one from the same community.

Why give up the efficiency at the last step? Because the person across the desk may have spent the month choosing between food and the mortgage. They may have a parent approaching retirement with too little put away. They may have been foreclosed on, or watched it happen to someone they love. A human loans officer could put themselves in their shoes. They can also see the one thing an algorithm is built to discount, which is that people change.

I’m elated that the encyclical says it even more plainly than I can, and with the widest audience. Sensitive decisions about credit and livelihood risk being handed to systems that do not know "compassion, mercy, forgiveness, and above all, the hope that people are able to change." {}

That is the agentic commerce question, stated early and without any theology attached.

Agentic commerce is the industrialisation of delegation, software acting and transacting on a person's behalf, thousands of times a day.

The loan desk is only the visible version. The same pattern runs quietly through pricing, claims, eligibility, collections, and the small approvals an agent will soon make with nobody watching. Keeping a human in every loop is neither possible nor the point. The real task is to decide, deliberately, which decisions a human must own.

Do you want Situation Room updates delivered to your inbox?

The Trusted Agents Triangle

We use this triangle to frame the basic prerequisites for an AI Agent that can be trusted by consumers and organizations alke:

The map

The encyclical's concerns mirror to the three sides of the Trusted Agents Triangle.

Sensitive decisions risk being fully delegated to systems that cannot weigh a life.

• Delegation. Which decisions an agent settles alone, and which a named human owns. A decision-rights map. Mark the calls that are agent-final and the ones that need a human signature. Someone must be able to account for a decision, justify it, challenge it, and remedy any harm.
• Trust and Identity. When an agent acts, who is the accountable human, and can the customer contest it. A named owner per decision type, an audit trail, and a route to appeal. Data is the product of many, and models carry "the cultural assumptions of those who designed and trained them."
• Context. The preferences and data that drive an agent's choices, where they came from, and whether the customer agreed. Consent in plain language, data provenance, and bias testing on the outputs. Read again down the right-hand column. Every item is a governance question you can put to a team this week, whatever you believe about the source.

Two practitioners show what those controls look like in practice, one in the architecture and one in the boardroom. Start with the build.

Build it

​Alok Sharma, an enterprise AI architect, paints a simple picture. A model without a control layer around it is coffee without a cup. You can technically drink it. You will burn your hands and lose most of it. The cup is what makes the coffee usable. In his terms that layer is the AI harness (a control layer wrapped around the model), and it is where the real work happens.

His warning to leaders is specific. If you wire a raw foundation AI model straight into your core business logic, you are building on something you do not control. The provider can change the model's behaviour overnight without telling you. Your costs move, your outputs drift, and errors you never saw in testing start reaching customers. The harness is what enforces your safety and compliance boundaries, manages context to cut waste (trimming what the model carries between steps), and runs checks to catch a wrong answer before it leaves the building. "The model is a commodity engine," he writes.

Here is the link back to the loan desk. In that story the human signature is the control. For the thousands of decisions where no one is watching, the harness is the signature. It is where the delegation boundary stops being a good intention and becomes something a system actually enforces.

Govern it

If the harness is the engineering scaffolding then needs organisational scaffolding around it, or it is a set of brakes with nobody trained to use them.

​Dr Joanna Michalska, who works on AI risk and governance in financial services, puts the discipline plainly.

Her argument is that safe speed comes from structure. Clear accountability, active oversight, and decision-makers with enough AI literacy (knowing these systems well enough to question them) to recognise when an output looks wrong. The people who sign off on where the delegation boundary sits have to understand the system well enough to move it, or defend it, under challenge.

The model is not the system

Put the two together and they say in an operator's language what the encyclical says in its own. The model is not the system. Around it sits a harness that enforces your limits and a governance structure that names who is answerable. Skip the harness and your policy has no teeth. Skip the governance and your harness has no owner.

Do you want to have a quick conversation about how you're thinking of deploying Agentic AI to your customers?

One honest caveat

A fair reader will ask the obvious question. Why take a lesson on AI ethics from a document launched with an AI company's co-founder on the stage?

It is worth sitting with that. Anthropic's Chris Olah spoke at the launch, and not everyone read it as goodwill. Some called it "Vatican-washing," a borrowed halo for a sector that builds the very tools the text warns about. Timnit Gebru argued the Church should have stood with "the exploited data workers fighting for their rights, the people whose water is polluted fighting data centers."

Others took the opposite view. Pete Furlong of the Center for Humane Technology noted that the encyclical openly conflicts with what these companies do, and treated that friction as a good sign rather than a stitch-up.

Since 2022 I have been teaching Responsible AI, built on IBM's frameworks. The longer I do it, the clearer one thing becomes. We are at the precipice of handing AI a great deal of control over decisions that shape people's lives and wellbeing, at scale. So I have started telling corporations something that tends to land as a half-joke until they sit with it: hire trained ministers. Not for the faith, but for the training. They have spent years studying the theory and application of morals and ethics, and that discipline is about to matter a great deal more than it does today.

For an operator, the useful part is what the text says about the people you never see. Every fast, clean agent response rests on a long chain: data labelling and content moderation done for low pay, often by young women, and the mining of rare earths, at times by children in dangerous conditions. The encyclical calls this what it is and asks for supply chains transparent enough that no advantage is built on hidden harm. That is a procurement question, not a sermon. When you buy or build agentic capability, the consent, the data provenance and the labour behind it are now part of your trust story, and a customer or a regulator can ask you to account for any of them.

Three questions to brief your team with:

1. For our highest-stakes journey, where exactly is the line between agent-final and human-owned, and who drew it?
2. If a customer challenged an agent's decision tomorrow, who answers, and what can they actually show?
3. Do the people signing off on that line understand the system well enough to defend it under pressure?

Where Trusted Agents comes in

Trusted Agents helps organisations work out where agents can act on a customer's behalf and where a human has to stay accountable, across the full delegated journey from discovery to transaction and after. We work closely with Dr Joanna Michalska, so governance sits in the design from the start rather than arriving as an afterthought once something has already gone wrong.

If you want to push on agentic AI without losing control of what matters, start here and book a 30 minute conversation with us.

Before I sign off today

Here's a question for you, in your business, which decision are you most tempted to fully delegate to an agent, and what would it take for you to keep a human on the final signature?

See you next week.

Gam

​
​

​The Trusted Agents Ltd.​
​Unsubscribe · Preferences​