C.A.R.E.: ROS 2 Humanoid Teleoperation Through AR Spectacles
Built a hackathon prototype connecting Snap AR Spectacles with a Booster K1 humanoid through ROS 2, live camera streaming, and head-orientation control. My work covered robot control, AR integration, and VLM functionality.
Technical work
Cal Hacks 12.0 · Working assistive-robot prototype
- Developed ROS 2 control software for the Booster K1, translating AR button and rotation events into Booster SDK locomotion commands; implemented a bridge with velocity limits and a command timeout to stop motion when commands become stale.
- Integrated robot camera feedback with the AR communication layer, converting ROS 2 NV12 images through OpenCV, preserving timestamps when republishing frames, and streaming base64-encoded PNG images over a dedicated WebSocket endpoint separate from control traffic.
- Assistive robotics prototype combining Snap Spectacles, live camera feedback, and humanoid control to explore assistance for people with limited mobility; contributed Gemini VLM experiments while collaborating with teammates responsible for SLAM/navigation and the AR communication interface.
Overview
C.A.R.E. bridges augmented reality and robotics to create an assistive robotic companion for individuals with mobility challenges. The system integrates Snap AR Spectacles with the Booster K1 robot, enabling intuitive human-robot interaction through an AR control interface.
Built at Cal Hacks 12.0 in October 2025. Assistance for older people and people with limited mobility motivated the prototype; it was not evaluated as a deployed care system.
What It Does
- AR Control Interface: Users view live robot camera feeds through AR glasses with joystick and HUD overlay for intuitive control
- Dual Operation Modes: Supports both autonomous patrol mode and manual control via head movements
- AI-Powered Detection: Gemini and OpenCV support human detection and interaction in the prototype
- Communication Architecture: ROS2, WebSockets, and ngrok tunneling maintain responsive control with dual channels — low-bandwidth control and high-bandwidth video
- Head Tracking Navigation: User head orientation maps directly to robot movement commands for natural, hands-free control
Target Applications
Proposed uses for further development and validation:
- Elderly and mobility-limited assistance — helping individuals navigate and interact with their environment
- Security and patrol operations — autonomous monitoring with human oversight
- Search and rescue missions — remote exploration in hazardous environments
- Construction and infrastructure inspection — safe inspection of dangerous or hard-to-reach areas
Technologies Used
Technical Challenges & Solutions
- Latency Reduction: Implemented dual WebSocket channels — one for low-bandwidth control signals and another for high-bandwidth video streaming to minimize control lag while maintaining video quality
- Perception Integration: Connected vision-language functionality with the robot interface; the project does not report end-to-end latency or human-following accuracy
- AR Interface Design: Created minimal, intuitive HUD elements that provide essential information without overwhelming the user's field of view
- Sensor Calibration: Developed mapping system to accurately translate AR camera rotation data to robot motor angle commands for precise head-tracking control
Team
Built at Cal Hacks 12.0 by:
- Het Patel — Booster Robot Control, Snap AR Integration, Vision Language Models
- Sunny Deshpande — SLAM and Navigation
- Atharv Mungale — Snap AR Software and Communication
- Vetrivel Balaji — Snap AR Software and Communication