What AI should and should not decide in verification work

In his essay "AI will not transform your business, but it can stop you losing points", Terrance Thabang Baloyi draws a clear line: AI drafts and flags, and a person confirms every figure that goes to verification. That principle is easy to agree with. It is harder to apply on an ordinary afternoon, when a system has read several hundred supplier certificates and someone has to decide which of its outputs to accept.

This piece is about applying it. Rather than a single line between machine and person, we find it more useful to think in four levels, each with its own rules.

Level 1: AI does it, people sample it

Some tasks are mechanical and can be checked objectively. Reading the expiry date on a B-BBEE certificate. Extracting the level and Black ownership percentage from a sworn affidavit. Matching an invoice number to a payment. Checking that a learnership agreement carries every required signature. Comparing two dates.

Here, AI can do the work at scale and the result can be relied on, provided people check a sample regularly and the error rate is measured and recorded. If the sample shows errors, the task moves up a level until the cause is found and fixed.

Level 2: AI suggests, people confirm

Other tasks involve interpretation but follow clear rules. Working out a supplier's enterprise size from its documents. Mapping a job title to an occupational level for the EE report. Matching a supplier in the procurement system to the right certificate when the names differ. Deciding which scorecard indicator a training course belongs under.

AI can propose the answer and show its reasons. A person confirms it, once, and the confirmed answer is stored and reused. The system should place its reasoning and the source document side by side, so that confirming is quick and informed rather than a reflex click.

Level 3: AI flags, people investigate

Some patterns deserve attention but prove nothing on their own. A newly registered supplier receiving a large share of spend. Several suppliers sharing an address or a director. An affidavit whose figures sit oddly with the supplier's size. A change in a beneficiary's ownership shortly before year-end.

Patterns like these can point to errors, to weak evidence or occasionally to something worse, including indicators of fronting, which is an offence under the B-BBEE Act. AI is well suited to noticing them across thousands of records. It is not suited to deciding what they mean.

A flag should open an inquiry, assigned to a person, with the outcome recorded. It should never quietly remove a supplier from your claim, and it should never attach a label to a person or a business without someone having looked.

Level 4: people decide, AI assists

Finally, there are decisions that belong to people however capable the technology becomes. That essay names the main ones: whether an individual is recognised in a category, what a possible fronting indicator means, whether an ambiguous contribution qualifies, and anything that affects a person's job, bursary or programme place. Two more belong on that list in day-to-day verification work:

• How to respond to a query or finding from the verification agency.
• The final sign-off of the figures and the evidence file.

At this level, AI can still help: gathering the evidence, summarising a long document, drafting a response for a person to edit. But the decision, and the name attached to it, is a person's.

There is a legal reason for this as well as a practical one. Section 71 of POPIA protects people from decisions that have legal consequences for them, or affect them to a substantial degree, where those decisions are based solely on automated processing intended to profile them. The exceptions are narrow, and where they rely on protective measures, those must include the chance to make representations. Any process that decides something about an individual should be designed with that in mind from the start.

Making the levels auditable

A verification agency tests your evidence and your claims, not your software. Still, if AI is part of how you prepare, you should be able to show how it was controlled. Four practices make that possible.

• Record the level. For each task, document which level it sits at and who is responsible for it.
• Record the human step. Every confirmation, inquiry and decision is logged with a name, a date and, where relevant, a reason.
• Record overrides. When a person disagrees with the system, capture the disagreement. Overrides are some of the most useful information you have for improving the system.
• Measure accuracy. Sample the AI's work on a schedule and keep the results. A claim that a system is accurate is only as good as the checking behind it.

Moving tasks between levels

The levels are not fixed. A task might start at level 2 while a system is new and, after months of confirmed accuracy, move to level 1 with sampling. It should move the other way the moment something goes wrong. That decision, too, belongs to a person and should be written down.

What should never happen is drift: a confirmation step that slowly becomes a habit of clicking "accept", until level 2 has quietly turned into level 1 without anyone deciding it should. Watching for that drift is part of governing an AI system, and it is the part most easily forgotten.

A practical way to watch for it is to look at the confirmation records themselves. If a reviewer has accepted every suggestion for three months without a single change, either the system is exceptionally accurate or the review has stopped happening. A sample will tell you which.

If you are planning to use AI in your verification preparation and want to work out which of your tasks belong at which level, write to us at sales@bsvision.co.za. We are happy to map it with your team.