March 2021

Data Flow Beta — New Data Integration Type in Skyvia

This month, we are super excited to announce the release of Data Flow. It is a new kind of integration packages in Skyvia. So far data flow is available only in beta.

Data flow allows you to integrate multiple data sources in one package and enables advanced data transformations on the way from source to final destinations. Despite being intended for complex integration scenarios, data flow is quite an easy-to-use tool, which requires no coding and can be used both by IT and non-IT professionals.

You can use data flow in the scenarios when data replication or import is not enough. These are the scenarios when you need to:

• load data into multiple sources at once;
• use complex, multistage transformations like lookup by an expression;
• obtain data from one data source, enrich them with data from another data source and finally load them into the third one.

Easy-to-Use Designer

For users’ comfort and better usability, Skyvia team has developed a diagram editor tool, a data flow working area, which allows arranging the entire process of data movement and any layout of components on the diagram visually. It allows building data flows of any configuration, size and complexity and easily managing them.

When you start building a data flow, you drag components onto the diagram and configure their settings in the details sidebar, which automatically opens on the right as soon as you drop the component onto the diagram. You connect components with links, which are displayed as arrows on the diagram, to show the direction of data movement between components from source to target, and, if necessary, you apply data transformations inbetween.

With our convenient diagram navigation, you can move the entire data flow, minimize or maximize it, move the separate components around, change their location, etc.

Data Flow Components

Data flow components are divided into 3 categories: sources, targets and transformation components.

Using source, you extract data from a cloud app or database and bring them to the data flow. For various data modifications, you use various transformation components among which are Extend, Lookup, Conditional Split, Row Count, Split, etc.

Transformations help you add new columns to the input records, delete old columns or change variable values, split data according to specified conditions and send modified data to multiple destinations, i.e. targets (databases or cloud apps). We do not limit you in the number of targets you can create in the data flow. On the screenshot, among other package kinds you can see a data flow package with multiple targets.

Scheduling the Data Flow Package

Skyvia allows scheduling your data flow packages for automatic execution as any other type of data integration packages. All the advanced scheduling features are preserved in the new data integration type. Scheduling might be very useful if you want to configure a data flow package to run periodically on certain days and at a particular time or if you want to delay a data flow execution to a later time.

Viewing the Data Flow Runs

Skyvia displays the data flow runs in the same way as for other data integration packages. On the Monitor tab, you can view the current data flow status and 5 most recent runs, and, on the Log tab, you can view older runs as well as all runs for a specified period.

In data flows, you determine the number of success and error records yourself according to the result settings. You can also create and write logs on your own. When you click a certain package run, the History Details window pops up. If you have created logs, the data flow generates them, and you can download their results from this window.

Our Data Flow documentation will help you get familiar with all the advanced feautures the data flow offers.

New Connectors

We have also released four new connectors in Skyvia this month. Among them are Snowflake, Google Ads, Mailjet and Harvest. Note that they are supported only in the new data integration runtime. So far, Skyvia does not support backup for these connectors.

Snowflake — a scalable cloud-based data platform, which enables data storage, processing and analytic solutions that are faster, more flexible and easier to use. The Snowflake data platform is not built on any existing database technology or “big data” one, instead it combines a completely new SQL query engine with an innovative architecture natively designed for the cloud.

Google Ads — an online advertising platform developed by Google. Using Google Ads, you can create online ads to reach people exactly when they are interested in the products and services that you offer.

Mailjet — a secure and reliable cloud-based email delivery and tracking platform, which allows users to send marketing and transactional emails.

Harvest — a convenient cloud-based time tracking and invoicing tool, which has been designed for businesses of all sizes.