Adaptive Vehicle Control Based on Pedestrian Behavior
Contributed to a pedestrian-aware ROS 2 control stack on a physical Polaris GEM, combining LiDAR/RGB-D fusion, motion-based time-to-collision estimates, and Stanley/PID control. Supervised tests reported 53 successes across 58 mixed scenario and controller trials.
Technical work
Four-person team · Supervised tests on a physical Polaris GEM
- Implemented ROS 2 sensor fusion for Ouster LiDAR and YOLOv11/OAK-D detections using approximate synchronization, a 2 m association gate, and separate range/bearing weights.
- Built TTC-driven cruise, caution, and yield decisions, issuing STOP_YIELD when perception data was older than 0.5 seconds and applying a 2-second recovery buffer before resuming cruise.
- Implemented Stanley lateral control and the safety-control layer for speed selection and braking, connecting pedestrian risk estimates to the Polaris GEM’s vehicle controllers.
- The four-person team reported 53 successes in 58 supervised scenario and controller tests, including 9/10 in each crossing condition and 8/10 for pedestrians walking along the road.
Overview
The team integrated pedestrian perception, approximate sensor synchronization, time-to-collision estimates, and risk-dependent driving behavior on a physical Polaris GEM at UIUC.
Stanley steering and PID speed control execute the behavior commands. This project uses heuristic motion prediction; the subsequent ADAPT project explores diffusion forecasts and MPPI planning. Tests were supervised by a safety driver.
Problem Statement
A vehicle needs to connect pedestrian detections to driving decisions while accounting for uncertainty, stale measurements, and crossing geometry. The project explores graduated speed responses and stopping behavior based on observed pedestrian motion.
Key Features
- Multi-Sensor Perception: LiDAR and RGB-D camera fusion for robust pedestrian detection
- Pedestrian Behavior Prediction: Trajectory prediction, motion forecasting, and Time-to-Collision (TTC) calculation
- Intelligent State Machine: Multi-phase decision system with CRUISE, STOP_YIELD, and SLOW_CAUTION states
- Real-time Adaptation: Speed adaptation based on pedestrian motion, with Stanley control following a preplanned path
- Safety Controller: Emergency braking and velocity control with PID feedback
- Stanley Controller: Precise lateral control for path following
- Sensor Fusion: Weighted fusion of LiDAR (0.8 distance, 0.3 direction) and Camera (0.2 distance, 0.7 direction) data
Technologies Used
Technical Architecture
Perception Stack
- LiDAR Processing: Voxelization, ground filtering, outlier removal, DBSCAN clustering, tracking with EMA smoothing, geometric and motion-based human detection
- RGB-D Processing: YOLOv11 object detection, depth extraction, pedestrian pose transformation to ego frame
- Sensor Fusion: Time synchronization, data association with Euclidean distance matching (2.0m threshold), weighted averaging
Prediction Module
- Pedestrian trajectory buffering and smoothing
- Motion prediction using historical trajectory data
- Ego vehicle trajectory prediction
- Time-to-Collision (TTC) calculation
Planning & Control
- High-Level Decision: Critical checks, context-dependent behavior, and recovery; perception data older than 0.5 seconds triggers a STOP_YIELD command, with a 2-second recovery buffer
- Safety Controller: Speed mapping (CRUISE → 5 m/s, SLOW → 2.5 m/s) and emergency braking
- Stanley Controller: Minimize heading and cross-track error for lateral control
- Velocity PID: Smooth acceleration/deceleration for longitudinal control
Results & Performance
| Experiment Type | Experiments | Success Rate |
|---|---|---|
| Cruise Mode | 5 | 100% (5/5) |
| No Pedestrian w/ Sign | 10 | 100% (10/10) |
| Crossing Pedestrian w/ Sign | 10 | 90% (9/10) |
| Stationary Pedestrian | 5 | 100% (5/5) |
| Crossing Pedestrian | 10 | 90% (9/10) |
| Pedestrian Walking Along Road | 10 | 80% (8/10) |
| Vehicle Stanley Control | 8 | 87.5% (7/8) |
Reported aggregate: 53/58 (91.4%) across the listed mixed tests, including eight Stanley-controller trials. Tests used a safety driver.
The slides and report retain scenario-level results. No controlled reduction in emergency braking relative to a reactive baseline is established.
Challenges & Solutions
- Challenge: Human movement is inherently uncertain and unpredictable
Solution: Used heuristic motion prediction with motion smoothing and TTC-based early warning - Challenge: Sensor fusion with different modalities (LiDAR vs Camera)
Solution: Developed weighted fusion approach leveraging LiDAR's distance accuracy and Camera's directional precision - Challenge: Real-time decision making with safety constraints
Solution: Designed hierarchical state machine with critical safety checks, context-aware behavior, and recovery mechanisms
Team
Het Patel • Sunny Deshpande • Ansh Bhansali • Keisuke Ogawa
University of Illinois Urbana-Champaign — December 2025
The architecture slide assigns Het sensor fusion, high-level decision logic, the safety controller, and Stanley control; Sunny RGB-D perception and pedestrian behavior prediction; and Keisuke and Ansh LiDAR processing. Results are team-level supervised tests.