Safeguarding Multi-Agent Reinforcement Learning for Connected and Automated Vehicles with Recursively Feasible Control Barrier Functions

Jianye Xu1, Yousef Sayed2, Bassam Alrifaee2

1Department of Computer Science, RWTH Aachen University, Germany

2Department of Aerospace Engineering, University of the Bundeswehr Munich, Germany

Paper (coming soon) Video Code

Video

Abstract

Multi-Agent Reinforcement Learning (MARL) shows promise in coordinating connected and automated vehicles but offers no safety guarantee. Although Control Barrier Functions (CBFs) can add this guarantee, their optimization problem can become infeasible in practice due to control bounds. Moreover, they often overapproximate vehicle and road geometry, which makes them conservative. This paper proposes the Backup-Horizon CBF (BH-CBF), which evaluates safety along a horizon until all vehicles stop. BH-CBF is recursively feasible under control bounds, and it considers the exact vehicle and road geometry. Based on BH-CBF, we derive a fully decentralized safety filter that only needs a single broadcast of vehicle states and routes per step and stays real-time in dense traffic. We use BH-CBF to safeguard a MARL policy over its life cycle: as a reward during pre-training, as a run-time filter at deployment, and as a filter in the training loop that keeps fine-tuning on unseen maps collision-free. In simulations on real-world maps of a roundabout and an intersection, BH-CBF avoids all collisions of unsafe MARL policies and keeps its safety guarantee at every step. Existing filters either collide, lose the guarantee, or fall back to full braking far more often. In recorded roundabout traffic with 1264 human-driven vehicles over 115 km, our safeguarded policies shorten the median travel time by 14% compared with the human drivers, without any collision.

Key Idea

BH-CBF evaluates the safety margins along a backup policy that brakes every vehicle to a standstill along its route. Its filter keeps this braking maneuver safe at every step, which makes it recursively feasible under control bounds.

An unsafe MARL policy safeguarded by our BH-CBF filter on a roundabout (left) and an intersection (right). Light trails: predictions of braking to a standstill along the route. Real time.

Results Along the Life Cycle of a MARL Policy

We evaluate BH-CBF on real-world maps of a roundabout and an intersection in four stages. The clips show example rollouts; the paper reports the statistics over all runs.

Stage 1Pre-training with safety rewards

We pre-train policies with three safety rewards and deploy them without a filter. Our BH-CBF-informed reward trains the safest policies, which are nearly collision-free on their own map.

Policies pre-trained with the Time-To-Collision (TTC) reward, the one-step CBF reward, and our BH-CBF-informed reward on the roundabout, from the same initial state. Red stars: collisions. 2× speed.

Stage 2Zero-shot deployment with run-time filters

We deploy unsafe pre-trained policies with run-time filters, also on unseen maps. One-step CBF filters can become infeasible and collide, whereas our BH-CBF filter avoids all collisions.

The same policy from the same initial state without a filter, with a one-step CBF filter, and with our BH-CBF filter. 2× speed.
An unsafe policy on the roundabout (top) and the intersection (bottom). Alone, it collides (paused at the collision); with our BH-CBF filter, it continues from the same joint state without collision. Blue and green: the conflicting vehicles. Real time.

Stage 3Fine-tuning with the filter in the loop

With our BH-CBF filter in the training loop, fine-tuning on an unseen map is collision-free. The filter's corrections become a learning signal that makes the policy itself safe.

A policy pre-trained on the roundabout, before and after fine-tuning on the intersection with our BH-CBF filter in the loop. Both are deployed without a filter. 2× speed.

Stage 4Comparison with human drivers

We replace every human-driven vehicle recorded on a roundabout of the rounD dataset by a controlled vehicle that enters at the recorded time and state. Over 1264 vehicles and 115 km, our BH-CBF-filtered policies shorten the median travel time by 14% compared with the human drivers, without any collision.

Recorded human drivers (left) and our fine-tuned policy with the BH-CBF filter (right), from the same recorded entries. Real time.

Code

The code that reproduces our evaluation results will be released in SigmaRL, our open-source framework for MARL of connected and automated vehicles.