April 2020
New Data Integration Engine for Replication and Export
Our team keeps working hard updating Skyvia and making it better. Recently we have released a new, more convenient UI for our users, and now it’s time for changes under the hood. Today we announce a new data integration engine that extends Skyvia’s capabilities in integrating data between various cloud applications and databases.
Unlike our current data integration engine, the new engine is designed and written from scratch. The current engine somewhat limits us in extending data integration features, but it is no longer the case for the new engine. We designed it with extendability in mind, considering all the user-suggested features and requests that we could not implement previously, and we laid solid foundation for future data integration improvements.
Availability
Currently, the new engine is only available for replication and export only. It is not applied by default. In order to use the new engine, you need to select the Use new runtime check box in your package details. This provides you with a choice of whether to use the new engine and get new features or stay with the tried current engine.
SQL engine for cloud data allows much more flexible data access and data manipulation than native cloud API and allow you work with your cloud data just like with database.
New Engine Features
Except the mentioned check box, there are no noticeable changes to the user interface. All of the changes are under the hood, and you may only notice how your integration works better.
Better Package Execution and Error Control
The new engine provides better package execution and error control. For replication, in particular, this means that the new replication package won’t fail anymore if it finds that some tables for replication already exist in the target database. Or vice versa, if an existing replication package finds that some tables are deleted. Now the package is able to check whether the target table has a suitable structure and use it, or, in case of a missing table, re-create it and reload all the data automatically. This can reduce the number of errors and manual work for setting up replication greatly.
Agent Connection Support
Another noticeable and long-awaited feature is support for agent connections in the new data integration engine. Agent connections greatly simplify connecting to on-premise databases in local networks, using a secure locally installable Agent application as a tunnel for loading data.
A lot of our Connect and Query users already appreciate this feature, and now it finally becomes available for our replication and export users. You can easily copy your cloud data to your on-premise databases without allowing direct access to them from the Internet and making firewall exceptions. Additionally, you can now easily export their data to CSV files and upload these CSV automatically to file storage services or FTP servers.
New Expression Engine
The new data integration engine also includes new engine for expressions used for data transformation. It provides much more features and functions than SSIS expressions. However, since expressions are not used in export and replication packages, this feature is not yet available for our users. You will be able to assess it when we add support of import and synchronization packages to the new engine.