ProviderConfiguration

The ProviderConfiguration section contains main provider settings that determine the connector authentication and behavior.

Most of the behavior settings can be overridden for specific connector objects and stored procedures in the Metadata section.

ProviderConfiguration section contains the following configuration sections and settings:

ProviderConfiguration section example:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
"ProviderConfiguration": {
    "BaseUrl": "https://api.stripe.com/v1",
    "AuthenticationType": "Basic",
    "TimeStampStoreMode": "SecondsSinceEpoch",
    "FieldsParameterSupported": true,
    "FieldsParameterName": "opt_fields",
    "ResultPath": "data",
    "Headers": [{
        "Key": "X-API-Version",
        "Value": 2
    }],
    "PagingStrategy": {
        "Type": "LimitOffset",
        "LimitParameterName": "limit",
        "OffsetParameterName": "offset",
        "PageSize": 100
    },
    "ErrorHandling": {
        "ErrorMarkJPath": "ok",
        "ErrorMarkConditionValue": true,
        "APILimitMessage": "request over limit",
        "ErrorMessageJPath": "error_description"
  },
}

Authentication Settings

The following settings define the web requests authentication to access the source API.

AuthenticationType

The setting which defines how the connector will authenticate the web requests.

Valid values are the following:

For example, "AuthenticationType": "Basic".

AuthenticationToken

If you use the AuthenticationToken or OAuth2 type, specify how the authentication token is passed.

TokenName

Name of the parameter which accepts the token value.

TokenType

This setting defines the way to pass the token.

For example:

1
2
3
4
5
"AuthenticationToken": {
    "TokenType": "Header",
    "TokenName": "Bearer",
    "HeaderName": "Authorization"
}

Valid values are the following:

HeaderName

Name of the header for the Header token type. The default value is Authorization.

General Settings

BaseUrl

The base URL is the initial part of the API URL. All API endpoints are relative to the base URL. For example, https://acuityscheduling.com/api/v1 or https://app.asana.com/api/1.0

ResultPath

The path to the field that contains the result collection of JSON objects in the response.

The field with the result collection of objects may be located on the root level of the data source response or nested within another object. For example,

Root Field ResultPath:

If the field is located on the root level of the response,

1
2
3
{
    "data": []
} 

Its ResultPath value in the connector will look like this

1
2
3
"ProviderConfiguration" : {
    "ResultPath": "data"
}

Nested Field ResultPath:

If the field is nested within another object,

1
2
3
4
5
{
    response: {
        data: []
    }
} 

You must specify it in the ResultPath, for example

1
2
3
"ProviderConfiguration" : {
    "ResultPath": "response.data"
}

In the Metadata section, you can also define or redefine this setting for a particular object.

BodyPattern

Data sources APIs require passing the web request body in a specific format.
Usually, the data sources accept the web request body in JSON object format.

For example,

1
2
3
4
{
    "firstName":"John",
    "lastName":"Doe" 
}

By default, the connector passes the web request body as a JSON object. Thus, you can skip this setting if the data source API requires passing the web requests body as a JSON object.

However, if the data source requires the unusual web request body format, you can define this format in the BodyPattern setting.

When the request body is an array:

1
2
3
4
5
6
  [
    {
    "firstName":"John",
    "lastName":"Doe" 
    }
  ]

Its BodyPattern will have the following format:

1
  "BodyPattern": "[ <$body$> ]"

When the request body is an array inside the data property :

1
  { "data": [ { "name": "rndwork.com" } ] }

Its BodyPattern will have the following format:

1
  "BodyPattern": "{ \"data\": [ <$body$> ] }"

When the request body is a root object inside the data property:

1
  { "data": { "name": "rndwork.com" } }

Its BodyPattern will have the following format:

1
  "BodyPattern": "{ \"data\":  <$body$>  }"

In the Metadata section, you can define or redefine this setting for a particular object.

Headers

Constant HTTP request headers that are added to every request to the connector. Specify Headers as a JSON array of objects with the Key and Value properties. For example,

1
2
3
4
5
6
"ProviderConfiguration": {
    "Headers": [{
        "Key": "X-API-Version",
        "Value": 2
    }]
}

Headers indicated in the ProviderConfiguration section apply to all REST requests in the connector.

If you want to apply a specific header to a particular object, you can define this setting directly in the Objects configuration section.

FieldsParameterSupported

This settings determines if the connector supports querying only a part of the object fields. The list of fields is passed as an URL parameter in this case. This functionality helps to improve the API performance when reading objects. The default value is false.

FieldsParameterName

This setting determines the name of the web request URL parameter, which contains the list of fields to query. Use this setting if the FieldsParameterSupported is set to true.

The default value is field. The name field is typical for most APIs.

You can define or redefine this setting for the particular object in the Metadata section.

Date and Time Settings

The following settings define the format of date and time data passed from and to the source API.

TimeStampStoreMode

Format in which timestamp data is processed in the connector.

1
2
3
"ProviderConfiguration": {
    "TimeStampStoreMode": "SecondsSinceEpoch"
}

Possible values are:

You can also define or redefine this setting for the particular field in the Metadata section.

DateTimeFormat

This setting defines how to operate with the DateTime data. Use it when the TimeStampStoreMode = Text.

The default value is yyyy-MM-ddTHH:mm:ssZ.

To set the DateTimeFormat to MM-dd-YYYY HH:mm in all objects and all fields specify the following statement.

1
2
3
"ProviderConfiguration": {
    "DateTimeFormat": "MM-dd-YYYY HH:mm"
}

DateFormat

This setting defines how to operate with the Date data. Use it when the TimeStampStoreMode = Text. The default value is yyyy-MM-dd.

TimeFormat

This setting defines how to operate with the Time data. The default value is HH:mm:ssZ.

You can define and redefine these settings for the specific field in the Metadata section.

RateLimitThrottling

RateLimitThrottling is an allowed number of API requests per unit of time. It helps to avoid the “429 Too Many Requests” error.

If the number of API calls reaches the limit per the specified time interval, the connector waits until this time ends and sends web requests afterward.

RateLimitThrottling works on a single connection and can’t prevent the 429 error if there are several parallel connections to one source.

Example of allowing five web requests per 1 second:

1
2
3
4
"RateLimitThrottling": {
    "RequestsLimit": 5,
    "TimeInterval": 1000
}

The RateLimitThrottling setting contains two nested settings.

RequestsLimit

Maximum allowed number of web requests during the time span defined in the TimeInterval setting.

TimeInterval

Time span in milliseconds during which the maximal number of web requests can be sent.

PagingStrategy

Pagination is a technique that allows the splitting of the retrieved data into pages according to the defined strategy.

You must define the PagingStrategy if the source uses pagination. Otherwise, the source returns only the first page.

If the pagination differs from object to object, you can define or redefine the PagingStrategy in the Metadata section for particular objects.

Common Paging Strategy Settings

InputType

This setting defines the way for passing paging strategy settings.

Valid values are:

Type

Determines the strategy type. Most PagingStrategy settings depend on the Type specified. The list of possible values is:

See the detailed description of paging strategy types and settings required for them below

WithoutPaging

With this type selected, all the records return at once (when the source doesn’t support pagination).

1
2
3
"PagingStrategy": {
    "Type": "WithoutPaging"
}

LimitOffset

Use this type when records are requested in pages using limit and offset parameters.

For example:

1
2
3
4
5
"PagingStrategy": {
    "Type": "LimitOffset",
    "LimitParameterName": "limit",
    "OffsetParameterName": "offset",
    "PageSize": 100

LimitParameterName

Name of the Limit parameter. The default value is limit.

OffsetParameterName

Offset parameter name. If you skip it, the default value is offset.

PageSize

Defines the number of records selected in one call. For example, PageSize = 100 means that the first API call selects the first 100 records, and the subsequent API call selects another 100 from 101 to 200.

StartIndex

This setting defines the offset counting start (first page index). The default value is 0, but some sources require to set it to 1.

PageNo

Use this type when a web request must include the required page number. It is passed as the URL parameter of the request. The number of the next page is specified in every subsequent request.

This paging strategy supports different ways how the data source can indicate whether the next page is available. The data source response may contain either the indicator of the next page’s existence or the total number of pages or the total number of records. Alternatively, if the data source does not return any such information, Skyvia can stop querying the next pages if the number of records in the received page is less than the specified page size. If page size is also unknown, Skyvia continues to query next pages till the current page still contains data.

Different settings are used for different ways of detecting whether the next page exists.

For example, it may look like this:

1
2
3
4
5
6
"PagingStrategy": {
    "Type": "PageNo",
    "PageNoParameterName": "page",
    "PageSizeParameterName": "per_page",
    "PageSize": 60
}

All the possible settings which may be defined for the PageNo type are the following:

PageNoParameterName

Name of the page number parameter in the request URL.

HasMoreJPath

This setting is used if the response contains an indicator that more pages exist. It specifies the JSON path to the response property indicating if the next page with data exists.

For example, the source response contains the following information:

1
2
3
4
5
6
7
8
9
{
  "data": {
        "pages": {
            "hasMoreRecords": true,
            "page": 3,
            "totalPages": 10
        }
    }
}

In this case, the HasMoreJPath setting for the connector’s paging strategy looks like this:

1
"HasMoreJPath": "data.pages.hasMoreRecords"

TotalPageCountJPath

This setting is used if the response contains the total number of pages. It specifies a JSON path to the response property with the overall number of pages. If the response doesn’t contain the indicator of the next page’s existence, reading continues until the page number is less than this value.

For example, ChartMogul source API returns the following pagination information in the object response:

1
2
3
4
5
6
{
    "per_page": 50,
    "page": 1,
    "current_page": 1,
    "total_pages": 4
}

Thus, the PagingStrategy for ChartMogul conector, will look like this:

1
2
3
4
5
6
7
  "PagingStrategy": {
    "Type": "PageNo",
    "PageNoParameterName": "page",
    "TotalPageCountJPath": "total_pages",
    "PageSizeParameterName": "per_page",
    "PageSize": 50
}

TotalRowsJPath

JSON path to search the general number of records on all pages. If there is no HasMoreJPath value in the response, the reading continues till the whole number of records is less than this value.

PageSizeParameterName

Name of the request parameter which defines the page size (Optional).

If the default page size is known, but there is no corresponding name parameter, you can leave the PageSizeParameterName empty but specify the default number of records on one page as the PageSize value. In this case the connector can determine that the next page does not exist

PageSize

Number of records loaded per page. If it needs to be set in the request, specify it together with the corresponding PageSizeParameterName.

You can also use it to determine the end of reading if you don’t specify HasMoreJPath, TotalPageCountJPath, or TotalRowsJPath. If you know page size used by the data source, you can specify it without the PageSizeParameterName. In this case connector stop requesting next page after it receives a page with less records than the specified page size.

The PageSize setting also has a special value -1, which means that the size of the page is unknown. In this case we do not pass the page size in requests. If you set the PageSize value to -1 and HasMoreJPath, TotalPageCountJPath, and TotalRowsJPath are not specified, the reading will continue until the number of records in the response exceeds 0.

StartIndex

The index of the first page. Default value is 1.

NextPageToken

Use this paging type when the data source returns the token of the subsequent page in the response body or header, and this token must be passed when requesting the next page.

1
2
3
4
5
6
7
"PagingStrategy": {
    "Type": "NextPageToken",
    "TokenJPath": "status.next_page",
    "NextPageParameterName": "next_page_token",
    "PageSizeParameterName": "limit",
    "PageSize": 500
}

NextPageParameterName

Name for the parameter used for passing the next page token in the request.

TokenJPath

JSON path to search the next page token in the response body. Required if the TokenHeaderName is not specified.

TokenHeaderName

The name of the response header, which contains the next page token. Required if the TokenJPath is not specified.

PageSizeParameterName

Name of the parameter which determines the page size. If you omit it, the request does not contain the page size.

PageSize

Number of records loaded in response to one call. It is desirable to determine the end of reading more accurately.

If it is not declared and PageSizeParameterName is not set, the reading continues until the request stops returning records.

If the PageSizeParameterName is set, the default PageSize value is 100.

NextPageUrl

Use this paging type when data source returns URL for reading the next page. It is returned in the response body or the header and is used as a URL for the next call.

1
2
3
4
5
6
"PagingStrategy": {
    "Type": "NextPageUrl",
    "UrlHeaderName": "Link",
    "PageSizeParameterName": "per_page",
    "PageSize": 100
}

UrlJPath

JSON path to search the next page URL in the response body. It is required if the UrlHeaderName is not set.

UrlHeaderName

Name of the header where the next page URL is located. It is required if the UrlJPath is not specified. The header value looks like URL;rel=”next”. The response may return several values, but you need only values containing rel=”next”. If the response doesn’t include the header with the rel=”next” attribute, there are no more pages to read.

PageSizeParameterName

Name of the parameter which defines the page size. If not specified, the page size is not contained in the call.

PageSize

Number of records loaded per page. It is desirable to determine the end of reading more accurately.

If the page size is not defined and the PageSizeParameterName is not specified, reading continues till the call stops returning records or the URL for the next call.

If PageSizeParameterName is set, the default PageSize value is 100.

Prefix

Prefix that is added to the URL before reading the next page. For example, if the server returns the relative URL, you need to add the base URL of the connector.

Suffix

Suffix that is added to the URL before reading the next page. It may be some additional request parameters.

ErrorHandling

If the API request returns an error instead of a response, you can process it and determine the connector behavior in case of errors.

For example,

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
"ErrorHandling": {
    "ErrorMarkJPath": "ok",
    "ErrorMarkConditionValue": false,
    "ErrorMessageJPath": "error_description",
    "ObjectNotFoundErrors": [
        "may not be available",
        "not found"
    ],
    "HtmlResponseError": "Connection settings are invalid. Please check your Domain, User, and/or Password values.",
    "FatalErrors": [
        "The API call for this organization has exceeded the maximum call rate limit"
    ],
    "Failover": {
        "FailoverErrors": [
            "Rate limited due to excessive requests",
            "Too Many Requests",
            "429"
        ],
        "MaxRetries": 5,
        "MinDelay": 500
    }
}

ErrorMessageJPath

JSON path to search the message in the error response. You can specify several search paths separated by semicolon.

For example,

1
2
3
"ErrorHandling" : {
        "ErrorMessageJPath": "meta.message"
        }

Use the $full_error_response$ value to display the whole error response as it is. For example,

1
2
3
"ErrorHandling": {
    "ErrorMessageJPath": "$full_error_response$"
}, 

If the value for this setting is not set, or it can’t find the error message, then the search tries to find the error tags in the response. It starts with errors or Errors and then checks tags error, Error, message, Message, statusmessage, title, Title, Exception.

ObjectNotFoundErrors

When Skyvia queries records with a filter by an ID value, it expects either a record or an empty result. However, some data sources return an error instead of an empty response when you request a non-existing record by an ID. In such cases we need to ignore this error and return an empty result.

By default, Skyvia ignores the messages with the Not Found, No Result found, Gone occurrences. Some data sources may return atypical errors. For such cases, you can specify a JSON array of error message strings to ignore in the ObjectNotFoundErrors setting, for example, "ObjectNotFoundErrors" : [ "may not be available", "not found" ]. The error is ignored if the error message contains at least one of the specified strings.

HtmlResponseError

Use this setting for the cases when the source returns the HTML page instead of an error message.

It allows replacing the HTML page with a readable error message. For example, Connection settings are invalid. Please check your Domain, User, and/or Password values.

In this case, all the error information sent from the source server will be lost.

ErrorMarkJPath

This setting defines the JSON path in the response, where you can find the error.

Use it if the source does not generate an exception in case of failure but immediately sends an error message in a field of the JSON response.

ErrorConditionJPath and ErrorConditionValue

Use these settings if the source returns a field with a sign that the request ends with an error instead of returning it.

In this case, you can specify the JSON path to search whether the execution failed or succeeded in the ErrorConditionJPath. And you can set the value which indicates that the request failed.

For instance, "ErrorConditionJPath" : "ok", "ErrorConditionValue" : false.

FatalErrors

Use this setting to define the array of the keywords inside the error messages which indicate fatal error. In case of such an error, the execution immediately stops without additional attempts to succeed.

Failover

Use this setting if you want the connector to retry the API request if it failed with certain errors.

For example, many sources limit the number of API calls per unit of time. If this limit is exceeded (e.g., during the integration execution in Skyvia), the source returns an error, and the execution fails. Failover makes the execution run again automatically after such failure.

FailoverErrors

The array of string message parts of the server error messages that indicate that the execution must try again.

MaxRetries

The maximum number of attempts to retry the request. If all the attempts failed, an error is returned.

The default value is 10.

MinDelay

MinDelay is a minimal delay in milliseconds between attempts to retry a request.

By default, the minimal delay is 300 milliseconds. The delay is not constant and increases with the number of attempts using the exponential backoff algorithm. MinDelay determines only its minimal value.

ReturningStrategy

This section determines how to process the data returned from the data source for INSERT or UPDATE operations.

ReturningStrategy section example:

1
2
3
4
5
6
7
8
    "ProviderConfiguration": {

        "ReturningStrategy": {
            "ResultPath": "data",
            "Type": "Row"
        },
        
    }

If you need to override the ReturningStrategy settings for a particular object in the Objects section, you need to specify the separate InsertReturningStrategy and UpdateReturningStrategy settings for INSERT and UPDATE operations respectively.

Type

This setting determines what data is returned from the data source after INSERT or UPDATE. Valid values are the following.

ResultPath

Path to the returned data, if needed. For example, you need to specify it if the returned data is not on the root level of the response body, but is nested.

KeyHeaderName

This setting defines the name of the response header, which contains the ID of the inserted object. Use it when the record ID returns not in the response itself but in its headers. This setting is used only with the Key type.

ConstantParameters

You can set constant parameters sent with all requests for all the objects. This setting is needed when you have to pass the same parameters for every object, for example, authentication.

For example:

1
2
3
4
5
6
7
8
9
10
    "ProviderConfiguration": {      
        "ConstantParameters": [
            {
                "ParameterName": "club_secret",
                "ParameterType": "UrlParameter",
                "DbType": "String",
                "Value": "CS-19195-ACCESS-WO0qRGel5fKZMnMSvDNKofmq4"
            }
        ]     
    }

ConstantParameters is an array of JSON objects with the following settings.

ParameterName

Constant parameter name.

ParameterType

This setting determines how the values are passed in the web request. Valid values are:

DbType

Defines parameter data type. The list of possible data types is the following:

Value

Defines the constant parameter value.

Disable

The condition under which the parameter isn’t used (for example, if the associated setting is NULL). It is usually used in conjunction with macros in Skyvia.

Example: ConstantParameters when Authentication Parameters Required for Every Object

For example, if a connector requires passing authentication parameters for every object, you can either add them to the Url setting of every object in the Objects section:

"Url":"/clubmembers?club_secret=CS-19195-ACCESS-WO0qRGel5fKZMnMSvDNKofmq4&club_key=HWGUG4xmhI7EWL7UOQ2Ew2V0vbmyjBUi2yfSOYoXCk"

Or you can configure ConstantParameters in the ProviderConfiguration section like this:

1
2
3
4
5
6
7
8
9
10
11
12
13
"ConstantParameters": [{
        "ParameterName": "club_secret",
        "ParameterType": "UrlParameter",
        "DbType": "String",
        "Value": "CS-19195-ACCESS-WO0qRGel5fKZMnMSvDNKofmq4"
    },
    {
        "ParameterName": "club_key",
        "ParameterType": "UrlParameter",
        "DbType": "String",
        "Value": "HWGUG4xmhI7EWL7UOQ2Ew2V0vbmyjBUi2yfSOYoXCk"
    }
]

As a result, the URL setting for objects in the Metadata section will be simplified and will look like the following: "Url":"/clubmembers".