May 2016

Greatly Improved Data Import

We are glad to present the new version of Skyvia — our online service for cloud data integration, backup, and management.

The new Skyvia version offers lots of new features for Import — a member of our Data Integration toolkit for cloud and relational data. The new Import features include UPSERT support for all data sources, greatly improved Lookup functionality, data filtering support, importing data from several related source objects to one target object, etc.

UPSERT Operation Support

Now Skyvia Import can perform the UPSERT operation not just for Salesforce, but for all supported data sources. If a Null value is specified for the ID or primary key, UPSERT operation inserts the record, and if a non-null value is specified, UPSERT operation tries to update the record with the specified ID or primary key. This allows you to avoid inserting duplicate data.

The primary key value can be specified in various ways: as a source column value or via lookup or external ID, etc., so you can perform UPSERT even without knowing the ID values.

Powerful Lookup Mapping

Lookup mapping now offers powerful functionality to get values for target fields both from target and source objects. Its new features are available for both Import and Synchronization packages.

Source Lookup

Now Skyvia Import supports lookup not only on the objects from target connection, but also on objects from the source connection too.

Composite Lookup Key Support

When one lookup key column is not enough to uniquely identify the necessary record from the lookup object, you can use a lookup key that consists of multiple columns.

Lookup Error Processing

Now you can specify what to do when no such record was found by Lookup - whether to throw an error or to assign the Null value to the mapped target column. Additionally you can control what to do when multiple records are found - to throw an error or to take the first found record.

Case-insensitive Lookup

New lookup can perform both case-sensitive and case-insensitive comparisons.

Lookup Cache

Lookup can optionally select necessary fields all the rows from the lookup object to cache once, and then search for the necessary records in the cache. If you import a large number of rows with lookup, it can be more efficient to select all the data from a lookup object once than select necessary records from it for each imported rows. For cloud data sources with API call limits, it can decrease the number of API calls used.

Two-level Lookup

For advanced mapping cases you can use the second-level lookup to specify values for the lookup key of the first-level lookup. For example, suppose you perform an import between two CRMs or from two instances of the same CRM. Let it be an import from one Salesforce organization to another. You have already imported accounts and now want to import contacts so that they belong to the same accounts in target that they belong in source.

In the target Salesforce organization, imported accounts already have different IDs, so you cannot just import AccountID values of source contacts. We will use Target Lookup mapping to find the IDs of the imported accounts by account names. However, the source Contact table does not contain the names of the referenced Accounts, it contains only their IDs. So we will use the Source Lookup to supply the account name values to the first-level lookup from the source accounts by their IDs.

Another use case of the two-level lookup can be when you need to get the values from a table, which is a parent table of the parent table of the source table. You can take the foreign key value of the first parent table on the second-level lookup, and then take the necessary value from the grandparent table on the top-level lookup.

Data Filtering

Skyvia 4.1 provides data filtering for the source data. It can filter data in two ways. First, it provides advanced filter settings that allow you to configure custom filters as complex as you may ever need. Additionally it allows importing only the recently changed or inserted data.

In a single click Skyvia allows configuring an import task to load only the records that was inserted or updated since the last successful package run (or since package creation if it was never run). This allows you to schedule import package for automatic execution, and this package will load only the data changes each time instead of loading all the source data.

Additionally, this feature can be used for creating a trigger-action like integration. You can configure an import operation that will perform some data change in a target when a record is created or updated in a source and schedule it to execute every few minutes.

Import Joined Tables

Another new Import feature is the ability to import data not just from a single object or table, but to join several related tables and use their columns in mapping directly. For example, when importing data from Salesforce contacts, you can join the fields from their accounts, etc.

File Mask Support

For importing CSV files from FTP/SFTP and Dropbox, the new Skyvia Import allows specifying not only file names but also file masks, that include a date variable. When such Import package is scheduled for automatic execution, it will search the file to import by substituting the current date to the mask.

Support for Self-Referencing Relations

When using the Relation mapping, Skyvia now supports relations when object references itself. An example of such relation can be a relation of Salesforce Account object that references itself via the ParentId field. Now Skyvia can correctly build a self-referencing relation in target, based on such a relation in source.