Components are the elements of Data Flow that enable you to query, process, store data, and apply custom logic. Arrange them according to your preference to build your custom ETL pipeline.
Source components get data from the data source so you can use this data later in your data flow. There two types of source components: Source and CSV Source. Choose one depending on where you are getting data from.
|Source||Gets data from a database or cloud application and adds it to the data flow scope.|
|CSV Source||Gets data from the CSV file and adds it to the data flow scope. You can use a local CSV file or the one from the file storage service.|
Target components are used to modify data in a data source and create logs if needed. There three types of target components: Target, CSV Target, and JSON Target. To modify data in data source target components use actions. There are custom actions that are specific to certain connections and four actions available for each connection by default: Insert, Update, Delete, and Execute Command.
|Target||Uses connection to choose a data source and modifies its data based on selected actions.|
|CSV Target||Inserts data to the CSV file that can be stored locally or on the storage service.|
|JSON Target||Inserts data to the JSON file that can be stored locally or on the storage service.|
Transformation components are in charge of data transformation and custom data flow logic. With their help you can create custom data pipelines and apply data transformations before uploading to data target destination.
|Split||Divides data flow into multiple branches and allow actions to be executed in parallel.|
|Conditional Split||Enables you to route records based on conditions.|
|Bufferizer||Sets a specific size of the batch.|
|Row Count||Counts the number of input rows and writes the result to a variable.|
|Lookup||Matches input records with records from another data source and adds columns of the matched records to the scope.|
|Extend||Receives data from the output of the previous component and modifies it. You can use it to add calculated fields, change the field's type, modify nested objects, and more.|
|Unwind||Deconstructs nested type properties.|