Part 2/10:

Historically, data processing revolved around ETL (Extract, Transform, Load) and ELT workflows, with data stored in data warehouses or data lakes. These systems facilitated structured and unstructured data management, enabling analytics and reporting. However, as data volumes exploded and cloud platforms such as AWS, Azure, and GCP became dominant, a new paradigm emerged—allowing data to reside directly on cloud storage buckets, accessible for analysis with any query engine.

The speaker reflects on the typical frustrations faced by data engineers: pipeline failures, mismatched data, sleepless nights, and the complexities of managing stale metadata. The evolution now aims to address these pain points with more flexible, scalable, and unified approaches.