Towards real-time adaptive robotic behavior: using multimodal feedback and reinforcement learning for human-robot collaboration, by and for humans and robots

Project Focus

ABSTRACT

This project addresses key challenges in human–robot collaboration (HRC) in prefabrication and construction: uncertainty, error propagation, and communication breakdowns that limit efficiency, safety, and user trust. State-of-the-art HRC systems often rely on static task assignments and one-directional interaction models, tasking users with error detection and recovery. In dynamic construction environments, characterized by material variation, interruptions, and shifting human roles, such approaches increase cognitive load, frustration, and reduce collaboration quality.

Resilient HRC requires co-adaptive processes where humans and robots jointly detect errors, negotiate corrective strategies, and communicate actions and intentions. While advances in reinforcement learning (RL), sensing, and multimodal interfaces show promise, significant gaps remain: existing error taxonomies insufficiently address uncertainty-rich workflows; adaptive robotic behaviors are rarely tested in construction; and multimodal communication is underexplored under site conditions, like noise and movement.

Building on prior studies in collaborative timber assembly, we aim to develop real-time adaptive robotic behaviors and multimodal communication methods for effective feedback loops in HRC. Previous work revealed persistent error propagation, mismatched collaboration speeds, limited coordination channels, and high cognitive demands, motivating a shift toward systems that adapt to both environmental conditions and human collaborators.

The project pursues three objectives: (1) studying how diverse users recognize and resolve errors and extending error taxonomies; (2) translating human strategies into adaptive robotic behaviors using sensor-supported reinforcement learning; and (3) designing and evaluating multimodal communication methods—visual, auditory, and haptic—that provide clear, low-cognitive-load feedback. Outcomes include validated HRC strategies, design guidelines, and open-source tools supporting resilient and inclusive collaboration in construction.

Project Members

Principal Investigators

• Prof. Achim Menges | Institute for Computational Design and Construction (Fac. 01)
• Prof. Thomas Wortmann | Institute for Computational Design and Construction (Fac. 01)
• Prof. Michael Sedlmair | Institute for Visualization and Interactive Systems (Fac. 05)

Researchers

• Gili Ron | Institute for Computational Design and Construction (Fac. 01)
• Tanja Blascheck | Institute for Visualization and Interactive Systems (Fac. 05)
• Regine Lendway | Institute for Visualization and Interactive Systems (Fac. 05)
• Jan Krieglstein | Institute for Visualization and Interactive Systems (Fac. 05)
• Jonas Vogelsang | Institute for Visualization and Interactive Systems (Fac. 05)