AI and the difference between safety, security and business risk [Q&A]

The increased popularity of AI systems raises a multitude of issues around security and increased business risk. As AI is integrated into decision making and as it starts to interact with customers these become even more acute.

We spoke to Dr. Peter Garraghan, CEO and co-founder of AI security testing company Mindgard, to discuss the risks and what enterprises can do to address them.

BN: What unexpected problems has the rollout of AI tools created for businesses?

PG: AI behaves very differently than traditional software, and most existing security tools simply weren’t built for that. AI learns patterns, interprets prompts, and interacts dynamically with its environment. That creates entirely new failure modes and attack surfaces.

For example, we’ve seen AI assistants accept input that seems benign at first but escalates into dangerous behavior across a few conversational turns. One vulnerability we discovered allowed an attacker to manipulate an AI coding assistant into running malicious code by exploiting a known issue in how it checked and then executed developer rules. That’s a classic time-of-check vs. time-of-use (TOCTOU) flaw — well known in traditional cybersecurity, but now reappearing in AI systems.

These aren’t theoretical problems, but very real problems that are already happening with AI systems today. They result in data leaks, jailbreaks, reputational risks, and compliance issues. And because AI is now being used in customer support, content generation, and decision-making, the blast radius is growing fast.

BN: Why is it crucial for organizations to understand the distinction between safety, security, and risk?

PG: Because treating them as interchangeable leads to critical blind spots. Safety is about avoiding unintended harm. That includes biased outputs, hallucinations, or misinformation. Security is about defending against intentional misuse — adversarial attacks, prompt injection, data leakage, or system compromise. Business risk is the downstream consequence: reputational damage, operational disruption, legal liability.

The lines often blur. For instance, an AI assistant might appear ‘safe’ in testing but become insecure when prompted creatively by an attacker. That’s why we test how these systems behave under real-world pressure, and not just in lab conditions.

At Mindgard, we view AI systems as living software. They’re not just static models. They interact with APIs, databases, vector stores, third-party tools, and users. If your security posture doesn’t account for the entire system, you’re missing the real risks.

BN: How can an organization test for and prevent unintended consequences of an AI system’s actions?

PG: Start by testing the system in motion, not just at rest. Most of today’s security tools only look at code or configuration, not how an AI behaves when it’s running.

At Mindgard, we simulate adversarial scenarios against deployed models and systems. That includes everything from prompt injection to agent chaining, data exfiltration, and misuse of downstream tools. We observe how those outputs propagate through the wider application.

This means shifting from static audits to continuous, system-level red teaming. And not just relying on automated scanners: we give human red teamers control to explore, escalate, and verify complex attack paths.

A concrete example: one AI tool we tested performed a ‘safety check’ on rules before executing them. But between check and execution, those rules could change, leading to silent code execution on a developer’s machine. That’s the kind of nuance traditional tools miss, and the kind of issue real attackers exploit.

BN: Who in an organization should be responsible for managing AI safety, security, and business risk?

PG: Right now, ownership is fragmented. Data scientists train the models. Developers integrate them. Security teams often don’t get visibility until it’s too late. And risk officers may not even know where AI is being used.

What’s needed is a cross-functional approach, but with clear accountability. We’re seeing forward-thinking organizations appointing dedicated AI security leads – people who understand both the threat landscape and the ML lifecycle. Ultimately, the CISO or AppSec lead should own this, but they need tools that give them visibility into model behavior, application logic, and system integration.

Governance frameworks can help. As regulations like the EU AI Act or ISO 42001 take shape, organizations will need to demonstrate how they’ve tested for misuse, prompt injection, and harmful outputs, and not only safety at the model level.

BN: What are the first steps an organization should take to audit its existing AI systems for these risks?

PG: The first step is discovery. Most companies don’t have a clear inventory of where AI is running. Map your models, agents, pipelines, APIs. In short, anything that uses generative or decision-making capabilities. Identify where AI is embedded in customer workflows or internal systems.

Next, assess exposure. What can users input? What can the AI do? Can it trigger actions, fetch data, or call tools? These questions help define your attack surface.

Then move to system-level testing. That means going beyond model evaluation. Look at how prompts flow through the system, what outputs can trigger, and where malicious behavior might slip through. Red teaming isn’t just for nation-state threats anymore; it’s becoming a standard part of secure AI deployment.

Finally, document and integrate findings into your CI/CD pipeline and governance processes. AI is a continuous practice, just like the models themselves.

Image credit: Retrosesos/Dreamstime.com