We have already explained how a self-driving car senses the world — the perception stage that builds a live model of everything around the vehicle. That model, however accurate, is useless on its own: it has to be turned into a decision, over and over, several times a second. This is the second stage of the sense, think, act pipeline every automated vehicle runs, and it’s arguably the hardest one, because “thinking” for a car means predicting what unpredictable humans are about to do, then choosing an action a court could later be asked to justify.
From objects to a story: prediction
Perception hands the “think” stage a list of objects — cars, cyclists, pedestrians — each with a position and a velocity. That’s a snapshot, not a plan, and a car can’t act on a snapshot: it needs to know where that cyclist is going, not just where they are right now. Behaviour prediction takes recent motion history and context — is that pedestrian looking at their phone, is that car’s indicator on, is this a school zone at 3pm — and projects several plausible next few seconds forward for every actor on the road simultaneously. Human drivers do a version of this instinctively and constantly; encoding it into a system that has to be right nearly all the time, and safe on the rare occasions it’s wrong, is most of what makes automated decision-making hard.
From prediction to a path: planning
Once the car has a predicted future for everyone else on the road, path planning generates the vehicle’s own trajectory through that predicted future: a smooth, physically drivable line that reaches the destination while keeping a safe margin against every other actor’s predicted path, obeying traffic rules, and staying comfortable for whoever’s inside. Planning has to happen continuously, because every one of those predictions about other road users gets revised as new sensor data comes in, which means the car’s own planned path is being quietly re-computed constantly, not decided once per manoeuvre.
The black box problem
Modern planning and prediction systems increasingly rely on deep learning models trained on enormous driving datasets, and they are, in narrow measurable terms, extremely good at it — but a neural network’s internal reasoning is notoriously difficult for even its own engineers to fully explain. That’s a serious problem the moment a self-driving car is involved in an incident: a regulator, an insurer, or a court will eventually ask why the car braked, swerved, or didn’t, and “the model weighted these features this way” is rarely an acceptable answer on its own. This has pushed a whole research field, Explainable AI (XAI), specifically into autonomous driving: the goal isn’t just an accurate decision, but one whose reasoning can be reconstructed and defended after the fact.
Where the state of the art is heading: EU research on trustworthy decision-making
Three genuinely different pieces of this problem are each the subject of dedicated EU-funded research. AIthena (2022-2025, €6.00M, coordinated by Vicomtech in Spain, Horizon Europe CCAM Partnership) is building Explainable AI specifically for CCAM development and testing, researching data, model and output-level interpretability so that a self-driving car’s AI can be trusted and audited rather than treated as an unexplainable black box producing “better-than-human” results nobody can account for. interACT (2017-2020, €5.53M, coordinated by DLR, the German Aerospace Center) went after a different gap: automated vehicles are good at detecting obstacles, but historically hopeless at communicating their own intentions back to the humans around them, so interACT researched how an AV should interpret the implicit signals other road users give off — and how it should sign its own next move back to them — specifically for the mixed traffic where automated and human-driven vehicles share the road. The purely algorithmic side of “think” has its own dedicated research too: EMP (2019-2021, €2.49M, developed by Swiss motion-planning specialist Embotech), funded through the EIC SME Instrument, brought a deterministic, physics-based motion-planning engine already proven in aerospace and industrial robotics into automotive use, generating certifiable, real-time trajectories — for manoeuvres from valet parking to highway and urban driving pilots — that run on standard automotive-grade ECUs rather than the specialised hardware such planning software had previously required. It is the layer that turns a predicted scene and a chosen manoeuvre into an actual, drivable path.
Why it matters beyond the spec sheet
An autonomous vehicle that senses perfectly but decides poorly is not meaningfully safer than one with worse sensors and better judgement — which is why this stage, not raw sensor specifications, is where most of the remaining gap to genuinely trustworthy Level 4 driving actually sits. Once a decision is made, though, it still has to become an actual steering, braking or acceleration command the car’s hardware can carry out safely — and that’s the final piece of the pipeline, coming next.
Photo: © Car-Shooters