When NVIDIA unveiled its “Guardian Angel” concept for self-driving cars, the idea sounded almost too simple: alongside the powerful, complex AI that actually drives the car, run a second, much smaller and much more predictable piece of software whose only job is to watch the first one and slam on the brakes if it starts doing something dangerous. It is not a NVIDIA-only idea — under different names, this two-tier structure is close to how most serious automated-driving and advanced driver-assistance systems handle the one problem that terrifies engineers more than any single sensor failure: an AI system making a confident, decisive, and wrong decision.
Why one clever system isn’t enough
The AI stack that perceives the road, predicts what other traffic will do, and plans a path — the same “sense, think, act” pipeline we’ve covered in a dedicated series — is necessarily complex: deep neural networks trained on enormous datasets, making probabilistic judgement calls in situations no engineer explicitly programmed for. That complexity is exactly what makes it powerful, and exactly what makes it hard to certify as safe by traditional means. A classical safety-critical system (say, an aircraft autopilot) can often be exhaustively tested and formally proven correct because its logic is explicit and bounded. A deep-learning perception or planning system, by contrast, is a black box trained on examples — brilliant at generalising to situations like the ones it learned from, and unpredictable in the rare situations that fall outside them. Regulators and engineers have converged on the same answer: don’t try to make the primary AI provably perfect. Instead, wrap it in a second, independent layer — deliberately simple enough to be formally verified — whose sole job is deciding, moment to moment, whether the primary system’s output is safe to act on.
How the supervisor actually works
A safety-supervisor layer typically doesn’t try to drive better than the primary AI — it tries to know less, on purpose, and be right about it. It checks the primary system’s proposed action (a steering angle, a braking request, a lane change) against a set of hard, explainable rules and physical limits: is the car about to exceed a safe following distance, cross into an occupied lane, exceed the grip available on this road, or ignore a sensor that just went blind? If the answer is yes, the supervisor doesn’t try to drive around the problem cleverly — it forces the car into a minimal risk manoeuvre: a controlled deceleration, a straightening of the wheel, a safe stop on the shoulder. This is also why safety-supervisor architectures lean so heavily on hardware and software redundancy, the same fail-operational principle behind the dual, physically separate steering and braking paths we covered in our article on steer-by-wire — a supervisor that shares a power source or a processor with the system it’s supposed to check on is not really independent at all.
None of this is unique to any one brand. It is the practical answer to a certification problem the whole industry shares: how do you prove a car is safe to sell when its main decision-making system is, by nature, unprovable in the traditional sense? You don’t try to prove the AI is perfect — you prove the watchdog is, and let the watchdog do the one thing it’s simple enough to guarantee: stop the car safely when it isn’t sure.
The EU research: from fail-operational hardware to certifiable AI
EU-funded research on this exact problem spans two generations, roughly a decade apart, because the underlying challenge changed as the primary AI systems themselves changed — from conventional embedded control software to deep learning.
- AutoDrive (2017-2020, €16.16M EU contribution, coordinated by Germany’s Infineon Technologies, under the ECSEL Joint Undertaking — the EU/national/industry partnership for electronic components and systems, not a CCAM or 2Zero project) laid the hardware and systems-architecture groundwork this whole approach rests on. It brought together Europe’s leading chipmakers, Tier-1 suppliers, carmakers and research institutes to design fail-aware, fail-safe and fail-operational electronic components and architectures — the redundant hardware and software layers, common across SAE Levels 3-5, that let a car keep enough control after a component fails to reach a minimal risk state rather than simply losing all function.
- SAFEXPLAIN (2022-2025, €3.89M, coordinated by the Barcelona Supercomputing Center under Horizon Europe’s Cluster 4 for Digital, Industry and Space — a general-purpose destination, not an automotive-specific partnership) tackles the newer half of the problem directly: how do you certify a deep-learning system against Functional Safety (FUSA) standards when its logic isn’t explicit code you can read, but weights learned from data? The project developed methods to make deep-learning software explainable and traceable enough to prove, to a certification body’s satisfaction, that it genuinely meets its safety requirements — with automotive as one of three industrial case studies alongside space and rail, worked alongside functional-safety specialist IKR and platform partner BSC itself.
Together, the two projects bracket the whole problem: AutoDrive built the redundant hardware and system architecture a supervisor layer needs to physically act on its own, independent of the primary AI; SAFEXPLAIN is building the methods to certify that the AI being supervised, and the logic doing the supervising, actually behave the way their designers claim. Neither project is about making the car’s main AI smarter — both are about proving, formally and to a regulator’s satisfaction, what happens when it isn’t.
Photo: © Car-Shooters