How to set up config driven data pipelines | Microsoft Learn
It encapsulates complex Spark clusters and job run-times and provides a simplified interface to users, who can focus on the business logic. The framework is based on PySpark and Delta Lake and managed by develo