Components

Components are the elements of Data Flow that enable you to query, process, store data, and apply custom logic. Arrange them according to your preference to build your custom ETL pipeline.

Source Components

Source components get data from the data source so you can use this data later in your data flow. There are two types of source components: Source and File Source. Choose one depending on where you are getting data from.

Component	Description
Source	Gets data from a database or cloud application and adds it to the data flow scope.
File Source	Gets data from a CSV, Avro, or Parquet file and adds it to the data flow scope. You can use a local file or one from a file storage service.

Target Components

Target components are used to modify data in a data source and create logs if needed. There are two types of target components: Target and File Target. To modify data in data source target components use actions. There are custom actions that are specific to certain connections and four actions available for each connection by default: Insert, Update, Delete, and Execute Command.

Component	Description
Target	Uses connection to choose a data source and modifies its data based on selected actions.
File Target	Inserts data to a CSV, JSON, Avro, or Parquet file that can be stored locally or on the storage service.

Transformation Components

Transformation components are in charge of data transformation and custom data flow logic. With their help you can create custom data pipelines and apply data transformations before uploading to data target destination.

Component	Description
Split	Divides data flow into multiple branches and allow actions to be executed in parallel.
Conditional Split	Enables you to route records based on conditions.
Bufferizer	Sets a specific size of the batch.
Row Count	Counts the number of input rows and writes the result to a variable.
Lookup	Matches input records with records from another data source and adds columns of the matched records to the scope.
Extend	Receives data from the output of the previous component and modifies it. You can use it to add calculated fields, change the field's type, modify nested objects, and more.
Unwind	Deconstructs nested type properties.