Data pipeline

A data pipeline is an automated sequence of steps that moves data from sources to a target environment and processes it as needed. It can include extraction, transformation, validation, storage and handing data over to analytics or reporting tools.

How a data pipeline works

A pipeline can load data from databases, applications, APIs or files. It then adjusts and checks the data and stores it, for example, in a data warehouse or data lake. The process can run at scheduled intervals or respond to events. The individual steps build on each other, and their outcome affects data availability.

What matters in the design

Besides performance, it is necessary to address monitoring, retries of failed runs, error handling, data quality, security and documentation. Without these elements, a pipeline may move data but will not be reliable to operate.