Towards self-reliant robots : skill learning, failure recovery, and real-time adaptation: integrating behavior trees, reinforcement learning, and vision-language models for robust robotic autonomy
Author
Summary, in English
Robots operating in real-world settings must manage task variability, environmental uncertainty, and failures during execution. This thesis presents a unified framework for building self-reliant robotic systems by integrating symbolic planning, reinforcement learning, behavior trees (BTs), and vision-language models (VLMs).
At the core of the approach is an interpretable policy representation based on behavior trees and motion generators (BTMGs), supporting both manual design and automated parameter tuning. Multi-objective Bayesian optimization enables learning skill parameters that balance performance metrics such as safety, speed, and task success. Policies are trained in simulation and successfully transferred to real robots for contact-rich manipulation tasks.
To support generalization, the framework models task variations using gaussian processes, enabling interpolation of BTMG parameters across unseen scenarios. This allows adaptive behavior without retraining for each new task instance.
Failure recovery is addressed through a hierarchical scheme. BTs are extended with a reactive planner that dynamically updates execution policies based on runtime observations. Vision-language models assist in detecting and identifying failures, and in generating symbolic corrections when tasks are predicted to fail.
The thesis concludes with a discussion of future work, including (1) using vision-language-action (VLA) models or diffusion policies to generate new skills on the fly from multimodal inputs, and (2) extending the reactive planner with proactive failure prediction to anticipate and prevent execution errors before they occur. Together, these directions aim to advance robotic systems that are more robust, adaptable, and autonomous.
At the core of the approach is an interpretable policy representation based on behavior trees and motion generators (BTMGs), supporting both manual design and automated parameter tuning. Multi-objective Bayesian optimization enables learning skill parameters that balance performance metrics such as safety, speed, and task success. Policies are trained in simulation and successfully transferred to real robots for contact-rich manipulation tasks.
To support generalization, the framework models task variations using gaussian processes, enabling interpolation of BTMG parameters across unseen scenarios. This allows adaptive behavior without retraining for each new task instance.
Failure recovery is addressed through a hierarchical scheme. BTs are extended with a reactive planner that dynamically updates execution policies based on runtime observations. Vision-language models assist in detecting and identifying failures, and in generating symbolic corrections when tasks are predicted to fail.
The thesis concludes with a discussion of future work, including (1) using vision-language-action (VLA) models or diffusion policies to generate new skills on the fly from multimodal inputs, and (2) extending the reactive planner with proactive failure prediction to anticipate and prevent execution errors before they occur. Together, these directions aim to advance robotic systems that are more robust, adaptable, and autonomous.
Department/s
Publishing year
2025-09-15
Language
English
Full text
- Available as PDF - 22 MB
- Download statistics
Document type
Dissertation
Publisher
Computer Science, Lund University
Topic
- Robotics and automation
Keywords
- Autonomous Robotics
- Behavior Trees
- Reinforcement learning
- Vision-Language Models
- Failure Recovery
Status
Published
Supervisor
ISBN/ISSN/Other
- ISBN: 978-91-8104-681-6
- ISBN: 978-91-8104-682-3
Defence date
10 October 2025
Defence time
13:00
Defence place
Lecture Hall E:1406, building E, Klas Anshelms väg 10, Faculty of Engineering LTH, Lund University, Lund. The dissertation will be live streamed, but part of the premises is to be excluded from the live stream.
Opponent
- Lazaros Nalpantidis (Prof.)