Elasticsearch

Elasticsearch is a powerful search and analytics engine for all types of structured and unstructured data.

Data integration: Skyvia supports importing data to and from Elasticsearch, exporting Elasticsearch data to CSV files, replicating Elasticsearch data to relational databases and replicating other cloud apps’ data to Elasticsearch.

Backup: Skyvia Backup does not support Elasticsearch.

Query: Skyvia Query supports Elasticsearch.

Establishing Connection

You can create a connection to both cloud and on-premises Elasticsearch servers using API Key or Basic authentication. Use API Key authentication for cloud Elasticsearch servers and on-premise servers with Kibana. If there is no Kibana on your on-premise server, connect using basic authentication.

Make sure your server is available for external connections. Skyvia will access your server from the following IPs: 40.118.246.204, 13.86.253.112, and 52.190.252.0.

Getting Credentials

Domain

The Elasticsearch domain is the base address of the server where your Elasticsearch is deployed.

For the on-premises servers, domain means a server domain name or IP address.

To get Domain value for the cloud Elasticsearch server, do the following:

  1. Log in to Elasticsearch Cloud.
  2. Select the needed Deployment from the Deployments list.
  3. Click Copy endpoint near the Elasticsearch label.

domain

API Key

To obtain the API Key, perform the following actions:

  1. For the cloud server, log in to the Elasticsearch Cloud server and select the needed Deployment. For the on-premise server, log in to your Kibana.
  2. Click on the menu icon on the left and select Stack Management in the Management drop-down list.

  3. In the Security block select API Keys.

    API Keys list

  4. Click + Create API Key on the right.

  5. Name the API Key and click Create API Key.

    API Key name

  6. Copy the created API Key.

The API key is visible only once when you create it. Keep it somewhere safe to be able to access it later.

API Keys copy

Creating Connection

Connection Using API Key

To create a connection with Elasticsearch using API Key authentication, perform the following steps:

API Keys auth

  1. Insert your Elasticsearch server address into the Domain box.
  2. Select API Key authentication type.
  3. Insert API Key

Connection Using Basic Authentication

To create a connection with Elasticsearch using a basic authentication type, perform the following steps:

Basic auth

  1. Insert your Elasticsearch server address into the Domain box.
  2. Select Basic authentication type.
  3. Enter your Elasticsearch Username and Password.

Additional Connection Parameters

Port

By default, the port value for the HTTP connection is 9200, and 9243 for the HTTPS connection.

Specify your custom port value if it differs from the default value.

SearchKeepAliveTimeout

Number of seconds during which the source object state is “frozen” so that query results stay consistent if someone adds/deletes data in the source object. When the value equals ‘0’, possible changes made to the source data may be included in the search results. By default, the value is set to ‘0’.

ExtraArrayFields

Enable this parameter when your data contain values with nested arrays and objects. This allows reading fields of basic data types and complex array fields.

When this checkbox is selected, Skyvia creates additional JSON array fields with the -array suffix in the name for each existing field.

When this checkbox is not selected, Skyvia reads all the array fields the same way as basic data type fields. If you query Elasticsearch data containing arrays with ExtraArrayFields disabled, integration or query fails with an error.

By default, this checkbox value is set to true.

For example, we have a field f_long, which data type is long. When we set the ExtraArrayFields parameter as true, two fields are created in the connector: f_long(long) and f_long_array(JsonArray).

Case 1: The f_long field of a record contains a single value:

ExtraArrayFields = true: the f_long field returns the single long value, and the f_long_array field returns an array with single value.
ExtraArrayFields = false: the f_long field returns its long value.

Case 2: The field f_long contains an array of two or more values.

ExtraArrayFields = true: the f_long field returns null, and the f_long_array returns the array. ExtraArrayFields = false: the f_long field returns an error because it cannot convert an array to a number.

Connector Specifics

Replication Target

You can replicate cloud apps data to Elasticsearch.

If you try to replicate objects with the composite primary key, the integration will return an error.

The primary key field in a target object always has the name “Id” after replication, even if it had another name in the source object (for example, Account Id or Internal Id).

Object Peculiarities

All Elasticsearch objects are custom.

DML Operations Support

All Elasticsearch objects support INSERT, UPDATE and DELETE operations.

Incremental Replication and Synchronization

Skyvia does not support Incremental Replication (as a Source) and Synchronization because none of the Elasticsearch objects has creation and modification timestamp fields.

Skyvia supports Incremental Replication with Elasticsearch as a Target.

Supported Actions

Skyvia supports all the common actions for Elasticsearch.

On This Page