A recurring discussion at our dinner table is whether you have to understand how AI works in order to use it. My husband votes nay and bases his argument on the fact that I have a high success rate at turning on the bathroom light, without the slightest shred of insight or ability to explain how electricity works.
My counterargument is that while it is true that I have no clue about electricity (beyond magic!), I have, through many years of daily testing, built considerable trust in the light coming on when I turn the switch and can, through my experiential knowledge, debug the rare instances when it does not.
Also, the light in our bathroom would never gang up with the neighbor’s to reach singularity and start an uprising.
Joking (mostly) aside, my point is that we are in the important phase of building trust in what AI can do for us and that entails both testing, verification, and maybe a little more comprehension than just magic.
The Atlantic published an article about where AI’s reasoning skills originate, and lo and behold, gamers playing online Dungeons & Dragons have contributed a lot. It turns out that building your prompts as a conversational, step-by-step quest really brings out the best in your chatbot.
It makes sense, given that LLMs are built on, well, language, and if you speak the same language and are specific about what you are asking, you get better results.
The Atlantic article was also a reminder that when we talk about AI reasoning, it is not the same as AI being able to think through a mathematical equation and return a precise answer. In fact, AI is not that great at math. What it can do is give you an answer that is the most likely to be true based on what it has seen before, and if you have shown it enough math problems and solutions, and you break it down into small chunks, it can come indistinguishably close to being correct. On the other hand, adding unrelated information to the prompt can lead to hallucinations.
That leads to another point FT recently made, namely that while autonomy and speed are how agentic AI does its magic, there are fields where “almost right isn’t good enough”.
Steve Hasker, the CEO of Thomson Reuters, continues to say that, for example, in law, banking, and accounting:
“The output [needs to be] authoritative, traceable and accountable. Speed alone isn’t the differentiator, trust is.”
And to have trust, we need experience and/or the help of someone with domain expertise. In recent studies, both from the Federal Reserve of New York and the NY Times, the direction we are going in is that AI, both generative and agentic, is a productivity tool, a collaborator, to do the fast, heavy lifting, and we still get to do the discernment and thinking parts that ensure accuracy, quality, and innovation.
I think that means that I don’t need to know how electricity works as long as I have my lights on.
Regitze Ladekarl, FRM, is FRG’s Director of Company Intelligence. She has 25-plus years of experience where finance meets technology.
This article is part of the FRG Risk Report, published weekly on the FRG blog. To read other entries of the Risk Report, visit frgrisk.com/category/risk-report/.