Evaluating potential risks and their relevance to your specific business is crucial; this process can determine whether technology serves as a time-saving asset or results in significant recovery efforts following a cyber-attack.
AI agents are being used increasingly, promising ever greater automation of work processes. We hear about their positive impacts in the workplace daily – and rightly so. However, before charging ahead to adopt this latest exciting technology, I want to urge organisations to take a beat and think about the risks involved. Afterall, as fast as AI Agents are being adopted, it’s guaranteed that mistakes will happen faster. Evaluating potential risks and their relevance to your specific business is crucial; this process can determine whether technology serves as a time-saving asset or results in significant recovery efforts following a cyber-attack.
What Are AI Agents?
Simply explained, AI agents are a system that autonomously performs tasks by designing workflows using available tools. Large language models (LLMs) are at the core of AI agents, but it is no longer just about answering questions; it is about decision-making, problem-solving, and executing actions. To accomplish all of this, AI agents have access to all internal tools. If that is not enough to complete a task, they also draw on external data sets, web searches, APIs, or other agents. They collect and combine all the information to recognise patterns and perform a task.
Most people are accustomed to using LLMs such as ChatGPT in their daily routines. And to a certain extent we are aware of the potential risks: we avoid inputting confidential information into these systems, and we verify outputs to avoid mistakes and inaccuracies because we know the training data might not always be reliable.
Where is the Risk with AI Agents?
However, the risks don’t stop there. Something else which is often forgotten are the costs. It is not enough to buy the licenses and get started; companies face many hidden costs. This is especially true when AI agents suddenly fail to do what they are supposed to do and companies have to invest even more in human resources to fix the mistakes.
- Average AI project failure costs exceed $500,000 (Source: RAND).
- 17% of AI failures lead to revenue loss over $1M (Source: Deloitte).
- 29% of firms report reputational damage after AI incident (Source: PwC).
The first major risk for companies is the eagerness to participate because AI is supposedly faster, more cost-effective, and allows for a reduction in workforce.
This is incorrect.
AI agents will accelerate the decline of poor company processes rather than improve them. There is no way around data qualification and process determination. You need to consider whether you need AI at all and, if so, where and how. Many people forget that AI agents are software, not magical intelligence, and they will not (yet) think for us and correct our mistakes.
Don’t Be Fooled by All Singing All Dancing Demos
Furthermore, many companies are misled by demos. Just because something looks amazing in a presentation doesn’t mean it will work the same way in the company. The beautiful demos you are given works so well in the closed environment in which it is presented. It is a whole different story in a real corporate environment.
Organisations need to consider who will think about things like the visibility of agent reasoning, error handling, silent recursion paths, cost limits, minimal observability, and non-existent operational governance? How many companies actually test applications in a secure environment before rolling them out? It’s like building a fast car without a steering wheel or brakes!
- 18% average accuracy drop in real-world vs benchmark testing (Source: Stanford HELM).
- 27% error rate in autonomous task execution without human oversight (Source: MIT CSAIL).
- 30% of AI agents misinterpret ambiguous instructions (Source: Stanford HAI).
Continuous Feedback is the Engine of Continuous Improvement
AI agents don’t pause and ask, “Can this result actually be correct?” They work with the available data, possibly even replacing missing data, and confidently carry out tasks based on this. If there is no feedback loop, things will go wrong.
AI agents utilize a range of internal and external tools, which increases the potential for errors from multiple sources. To minimise sources of error, we need the right configurations, the right rules, and the right feedback.
What is more, feedback from humans is important too. It is a misconception that we no longer need humans and that we can control AI with AI (we can, but maybe it’s not the best idea… a topic for a different blog post!)
- 85% of AI projects fail due to poor data quality or lack of data governance (Source: Gartner).
- 60% of organizations report AI underperformance compared to initial expectations (Source: BCG).
- 28% of AI systems produce outputs requiring human correction over 25% of the time (Source: MIT Sloan).
What’s a Prompt Injection Attack?
Cyber risks are also different with AI agents. Have you ever heard of prompt injection attacks? They are a type of cyberattack where malicious users input deceptive text into AI systems, particularly large language models (LLMs), to manipulate their outputs. This exploitation occurs because the AI cannot distinguish between legitimate instructions and harmful inputs, potentially leading to unauthorised actions or data breaches. (Source: IBM)
The consequences are data leakage, misinformation and security breach. To find out more, you can read about Shadow AI in another of my articles here. With increasing complexity, changing market appetite, and detailed compliance requirements, the importance of an experienced marine insurance broker has never been greater.
Real World Examples of AI Gone Wrong
From banks to fast food chains, there are countless examples of AI costing businesses time, money and resources. Here are just a few:
Chatbot gives Unauthorised 80% discount
Taco Bell rethinks AI ordering
Security researchers discover a loophole within a McDonald’s AI chatbot
AI coding solution, Replit, goes rogue
Commonwealth Bank of Australia (CBA) believed the AI voicebots could reduce call volume
And the Solution?
Think first, then act. Many CEOs are blinded by the word “innovation” and don’t want to miss the trend, without properly understanding what AI or AI agents actually are. The first step is to understand your own business processes and tasks to understand where and how you need AI. If you only think about saving costs or reducing resources, then you are definitely on the wrong track. The aim should be to make work easier: AI, AND humans together.
Anita Molitor
Cyber Specialist
T +43 664 962 40 08
