Agent Frameworks vs Data Pipelines: What to Use

In this video, Dave Ebbelaar discusses his perspective on agent frameworks and why he believes they will fail for most business automation problems. He argues that agent frameworks, which use large language models (LLMs) to reason and determine the next steps in a workflow, are often too complex and not robust enough for practical applications. Instead, Dave advocates for a simpler approach using data pipelines, which have been a staple in computing for years and offer solid design principles. He explains that most business processes can be broken down into sequential steps, making them more reliable and easier to manage. Dave demonstrates his approach by showing a Python project template that uses a data pipeline to automate email classification and response generation. He emphasizes the importance of designing workflows as directed acyclic graphs (DAGs) to ensure data flows in one direction, enhancing reliability. The video includes a detailed walkthrough of the code, highlighting how to set up pipelines, process tasks, and manage intermediate steps. Dave concludes by encouraging viewers to build applications from the ground up using first principles, rather than relying on complex frameworks, to fully understand and control their systems.