AutoShield: Multimodal Pedestrian Motion Prediction and TTC-Gated Autonomous Control
Implemented sensor fusion, high-level decisions, safety-control commands, and Stanley steering in a four-person ROS 2 project on a physical Polaris GEM. The team reported 53/58 passes across supervised mixed scenario and controller trials; a safety driver retained stopping authority.
Technical work
Four-person team · Supervised tests on a physical Polaris GEM
- Engineered a ROS 2 multimodal perception pipeline integrating voxel-filtered LiDAR DBSCAN clustering, EMA-based tracking, and YOLOv11 RGB-D localization; implemented 100-ms temporal synchronization, Euclidean data association, weighted sensor fusion, and discrete-time Time-to-Collision (TTC) estimation using predicted pedestrian and ego-vehicle trajectories.
- Architected a hierarchical, risk-aware control framework integrating TTC thresholds, pedestrian motion classification, sensor-staleness detection, and regulatory-sign context; coupled Stanley lateral control with PID-based longitudinal regulation, state-dependent velocity mapping, and PACMod2 emergency-braking overrides for predictive stop/yield decisions.
- Developed an integrated perception-to-actuation autonomy stack on the UIUC GEM vehicle, addressing the fundamental limitations of reactive pedestrian avoidance through short-horizon motion prediction and anticipatory collision-risk assessment, enabling context-aware speed adaptation, conservative handling of uncertain observations, and supervised vehicle operation.
Overview
The team integrated pedestrian perception, approximate sensor synchronization, time-to-collision estimates, and risk-dependent driving behavior on a physical Polaris GEM at UIUC.
Stanley steering and PID speed control execute the behavior commands. This project uses heuristic motion prediction; the subsequent ADAPT project explores diffusion forecasts and MPPI planning. Tests were supervised by a safety driver.
Problem Statement
A vehicle needs to connect pedestrian detections to driving decisions while accounting for uncertainty, stale measurements, and crossing geometry. The project explores graduated speed responses and stopping behavior based on observed pedestrian motion.
Key Features
- Multi-Sensor Perception: LiDAR and RGB-D camera fusion for robust pedestrian detection
- Pedestrian Behavior Prediction: Trajectory prediction, motion forecasting, and Time-to-Collision (TTC) calculation
- Intelligent State Machine: Multi-phase decision system with CRUISE, STOP_YIELD, and SLOW_CAUTION states
- Real-time Adaptation: Speed adaptation based on pedestrian motion, with Stanley control following a preplanned path
- Safety Controller: Emergency braking and velocity control with PID feedback
- Stanley Controller: Precise lateral control for path following
- Sensor Fusion: Weighted fusion of LiDAR (0.8 distance, 0.3 direction) and Camera (0.2 distance, 0.7 direction) data
Technologies Used
Technical Architecture
Perception Stack
- LiDAR Processing: Voxelization, ground filtering, outlier removal, DBSCAN clustering, tracking with EMA smoothing, geometric and motion-based human detection
- RGB-D Processing: YOLOv11 object detection, depth extraction, pedestrian pose transformation to ego frame
- Sensor Fusion: Time synchronization, data association with Euclidean distance matching (2.0m threshold), weighted averaging
Prediction Module
- Pedestrian trajectory buffering and smoothing
- Motion prediction using historical trajectory data
- Ego vehicle trajectory prediction
- Time-to-Collision (TTC) calculation
Planning & Control
- High-Level Decision: TTC-dependent behavior and stale-input checks; more than 0.5 seconds without updates triggers a STOP_YIELD command. This is a command-level response, not a verified braking guarantee
- Safety Controller: Speed mapping (CRUISE → 5 m/s, SLOW → 2.5 m/s) and emergency braking
- Stanley Controller: Minimize heading and cross-track error for lateral control
- Velocity PID: Smooth acceleration/deceleration for longitudinal control
Results & Performance
| Experiment Type | Experiments | Success Rate |
|---|---|---|
| Cruise Mode | 5 | 100% (5/5) |
| No Pedestrian w/ Sign | 10 | 100% (10/10) |
| Crossing Pedestrian w/ Sign | 10 | 90% (9/10) |
| Stationary Pedestrian | 5 | 100% (5/5) |
| Crossing Pedestrian | 10 | 90% (9/10) |
| Pedestrian Walking Along Road | 10 | 80% (8/10) |
| Vehicle Stanley Control | 8 | 87.5% (7/8) |
Reported aggregate: 53/58 (91.4%) across the listed mixed tests, including eight Stanley-controller trials. Tests used a safety driver.
The slides and report retain scenario-level results. No controlled reduction in emergency braking relative to a reactive baseline is established.
Challenges & Solutions
- Challenge: Human movement is inherently uncertain and unpredictable
Solution: Used heuristic motion prediction with motion smoothing and TTC-based early warning - Challenge: Sensor fusion with different modalities (LiDAR vs Camera)
Solution: Developed weighted fusion approach leveraging LiDAR's distance accuracy and Camera's directional precision - Challenge: Real-time decision making with safety constraints
Solution: Designed hierarchical state machine with critical safety checks, context-aware behavior, and recovery mechanisms
Team
Het Patel • Sunny Deshpande • Ansh Bhansali • Keisuke Ogawa
University of Illinois Urbana-Champaign — December 2025
The architecture slide assigns Het sensor fusion, high-level decision logic, the safety controller, and Stanley control; Sunny RGB-D perception and pedestrian behavior prediction; and Keisuke and Ansh LiDAR processing. Results are team-level supervised tests.