Diffusion Augmented Agents Framework

In this video, the presenter introduces the Diffusion Augmented Agents (DAAG) framework, which integrates large language models (LLMs), vision language models (VLMs), and diffusion models to enhance sample collection and transfer learning in reinforcement learning for embodied agents. The method, called Hindsight Experience Augmentation (HEA), utilizes diffusion models to relabel past experiences and transform videos consistently over time and space. This allows the autonomous process to be guided by an LLM without human intervention, making it ideal for lifelong learning scenarios. The DAAG framework reduces reliance on reward-labeled data for fine-tuning VLMs and training reinforcement learning agents on new tasks, demonstrating improved efficiency in robotic manipulation and navigation. The integration of components such as an LLM for orchestration, a fine-tuned VLM for reward detection, and a diffusion model for synthetic data generation enables agents to learn from limited past experiences. Empirical evaluations highlight DAAG’s ability to enhance task learning and transfer efficiency, showcasing its potential in overcoming data scarcity challenges in reinforcement learning. This advancement paves the way for more capable and adaptable lifelong learning agents.