Product Updates in 2022 Q3

Product Updates Archive for 2022 Q3 period.

2022-09-22 - v22.38

Credentials Page: Search by component name

In our quest to improve every aspect of our User Interface, we added a search function to the Credentials page. When you navigate to Integrate - Credentials you can now Search by component name to find the credential card you need instead of scrolling to find it.

WebHook Step: Send sample request

When you configure a webhook component step in your integration flow, you can either Send sample request or Add sample manually. When you select Send sample request a WebHook address is provided which you can use to send your example sample.

We also highlight that this address to be used only one-time to send a representative sample which you can use to configure subsequent steps in your integration flow.

REST-API-V2 component POST and PUT body type

To improve the user experience and prevent an accidental JSON data structure pasting into the Body field while in an Integrator mode, we changed the User Interface behaviour to default into JSONata mode when an application/json content type is selected for the POST and PUT HTTP calls.

This change avoids possible errors when users forget to change the input field settings deom Integrator to JSONata mode.

Run-time environment variables

Components exchange information with various platform services using run-time environment variables during execution of each step of an integration flow.

We are constantly working to standardise and add new environment variables to streamline the information exchange. The following environment variables were added to the run-time:

• WEBHOOK_URI - This environment variable will be available to each step in the flow which exchanges data with the webhook service, eg. WebHook or Flow-linking. This environment variable from the Tenant attributes webhooks_domain or from the HELM3 environment variable WEBHOOKS_BASE_URI if the Tenant record is missing.

• ELASTICIO_CONTRACT_ID - To help with troubleshooting of errors in multi-contract cases this environment variable will be available during the run-time of every execution including:

  • Verify credentials
  • Select model
  • Dynamic metadata
  • Generated samples and flow execution

Updates

Node.js Sailor version 2.7.0

A new version of Node.JS Sailor library 2.7.0 is released. This update is preparatory groundwork for the message persistence feature.

Java Sailor version 3.4.0

A new version of Java Sailor library 3.4.0 is released. This update is preparatory groundwork for the message persistence feature.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Iron-bank service Scalability

Iron-bank service collects statistics from the running integration flows. The service is now running as Kubernetes DeamonSet together with the cadvisor. This reduces the load on the system significantly and increases the scalability of Iron Bank.

Externally Build Docker Components

Information in this section is intended for our customers who intend to extend the automatic component delivery to include custom developed components.

API changes

Added new endpoint to PATCH /v2/teams to add docker registry to a team. Check the API documentation for the details.

Components Pusher changes

HELM3 configuration changes

To extend the automatic component delivery to include custom developed components, we modified the componentsConfiguration section of HELM3 charts in the following way:

componentsConfiguration:
  developmentTeam: teamName
  components:
    componentName:
      # Image tag for pulling into the internal docker registry. Required, can be "latest"
      targetTag: latest
      # Image name by which to pull repo from registry. Optional, `team_name/repo_name` by default
      dockerRepoName: "orgName/repoName"
      # Team where create RepoBuild. Optional. Will override root `developmentTeam` on the `components` level
      developmentTeam: teamName
      # Access for the component in the platform context. Optional, `global` by default
      access: team

Please Note: These are breaking changes and you should carefully revise your setup if you intend to use this feature.

• developmentTeam - unchanged
• componentName - unchanged
• registry, url, repo and tag got changed into dockerRepoName, developmentTeam, targetTag and access. Please review these changes carefully.

Along with these changes the following new required variables were introduced:

• DOCKER_REGISTRIES_ENCRYPTION_KEY used to encrypt credentials for docker registries
• COMPONENTS_PUSHER_AUTH_USERNAME used to authenticate to components-pusher server
• COMPONENTS_PUSHER_AUTH_PASSWORD used to authenticate to components-pusher server

The components-pusher expects to find these variables to deploy the component docker images into your default team.

The new approach expects to also find a new optional TEAM_DOCKER_REGISTRY environment variable in case of custom developed components. It is defined in the following way:

TEAM_DOCKER_REGISTRY:
   uri: "https://index.docker.io/v1/"
   credentials:
       username: ""
       password: ""

For the uri system expects string containg the address of docker registry. The credentials object can have the following properties:

• authFile - path of the authentication file
• registryToken - bearer token for accessing the source registry
• username and password

If none of the above are used - components-pusher will try to access the registry anonymously.

Behavior Changes

The components-pusher is not longer a cronjob but a fully integrated service of the platform. It runs in Kubernetes cluster in a single instance (replicas non-configurable).

The service runs following way:

1. Regularly checks for new builds for repositories with dockerTargetTag fields in the repo documents.
2. When new builds of repository are detected they are copied into internal docker registry.
3. The frequency of checks is configurable with the .Values.componentsPusher.processLoopInterval parameter of the component-pusher service. The frequency of checks is set in millisecond(ms). The default is 86400000 ms (1 day).
4. The component-pusher uses the newly added internal endpoint (/copy/:component_id?version=:version) to pull of the specified component and version from docker registry. Uses basic authentication with COMPONENTS_PUSHER_AUTH_USERNAME and COMPONENTS_PUSHER_AUTH_PASSWORD values defined in the HELM3 charts.

Gendry changes

• createTeam task now creates/updates default team with docker registry data from the TEAM_DOCKER_REGISTRY environment variable
• New createDefaultRepos task creates/updates components listed in componentsConfiguration. Teams, in which these repositories are created, require to have docker registry data associated with them. If default team used - docker registry data will be taken from TEAM_DOCKER_REGISTRY variable. If any other team used - it needs to be patched with correct docker registry data using API endpoint PATCH /v2/teams.

Fixed bugs

• Fixed the User Interface bug where the execution logs could not be copied.
• Fixed unexpected behavior of the Time Picker when filtering in the Threads and Execution pages.
• Addressed the limitation when credentials fields (oauth and additional fields, baseUrl, etc ) could not be retrieved in the OAuthFieldView. Now all fields can be accessed directly from the cfg object.

Components

Microsoft Dynamics CRM V2 component 1.0.0

Introducing new component for Microsoft Dynamics CRM.

• ADDED Make Raw Request action

Marketo component 2.2.5

• FIXED the bug when non required field would show as required and block configuration of the step.

Shopware 6 component 1.1.0

• ADDED new authentication method using CLIENT_ID and CLIENT_SECRET credentials
• ADDED Lookup Object (at most one) action
• ADDED Lookup Objects (plural) action
• ADDED Upsert Object action
• ADDED Delete Object By ID action
• ADDED Get New and Updated Objects Polling trigger

HJSON component 1.0.3

• UPDATED the Sailor library version to 2.6.29
• UPDATED the component-commons library version to 3.0.2
• FIXED the vulnerabilities in dependencies

Zendesk component 1.0.2

• UPDATED the Sailor library version to 2.6.29
• UPDATED the component-commons library version to 3.0.2
• FIXED the vulnerabilities in dependencies

VTEX component 1.4.3

• FIXED output metadata for the Lookup Object (at Most 1) action for Object Type - Orders
• FIXED incorrect behavior of the Allow ID to be omitted option in the Lookup Object (at Most 1) action
• UPDATED the Sailor library version to 2.6.29

Shopify component 1.6.5

• ADDED retry on 429 error (respecting the Retry-After header)

Hubspot component 1.5.3

• FIXED the Verify credentials

2022-09-08 - v22.36

Credentials: New design & improved responsiveness

We continue to update our User Interface to improve ease of use and reduce UI lag. This time it is the UI surrounding Credentials that is updated. When you navigate to Integrate > Credentials you will notice larger icons and a more coherent design.

When you select a component from the Component Overview page, you will see the redesigned Component Credential view. The Salesforce component is shown here as an example:

In this view you can select the kebab menu (3 vertical dots) at the right of each credential to Verify, Edit, or Delete the Credentials. Alternatively, you can select the + icon to the right of the search field if you want to create a new Credential.

Here is how each credential page looks when you Edit it:

Retrieve new sample button

While developing new integration flows it is often necessary to retrieve new sample data after each flow step is created. This was previously possible by navigating Back. In this release we have added a new button Retrieve new sample from component that allows you to create new sample data without navigating away from your work.

Long lines in Code component

We improved the user interface of the Node.js Code component configuration field. Now you can use long lines of code witout spaces and the interface will wrap it for you.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

OEM Related

Information in this section is intended for our customers who use the OEM version of the elastic.io platform.

Registration page improvement

If your tenant is using the registration page and the email_verification parameter is enabled (set to true) then only the email address field will be displayed.

When an User enters their E-mail address and presses Register, an Email verification link is sent to the mail address. Through this link Users can then enter their password:

Fixed bugs

• We have made our platform more robust by ensuring that User metadata which defines views of each workspace is backed up and recreated even if the data has been deleted from its primary location (RabbitMQ).

Components

Maester component 1.0.5

• UPDATED the maester-client library version to 4.0.3

Batch component 2.0.4 and 2.0.5

• UPDATED the batching library version to 2.0.3

Request-reply component 1.2.6

• UPDATED the maester-client library version to 4.0.3
• UPDATED the component-commons library version to 3.0.2

SFTP component 1.5.2

• UPDATED the component-commons library version to 3.0.2

Email component 1.2.1 and 1.2.2

• UPDATED the Sailor library version to 2.6.29
• UPDATED the component-commons library version to 3.0.2
• FIXED the vulnerabilities in dependencies
• REPUBLISHED the component using components-build-helper version 2.0.0

CSV component 3.1.5

• UPDATED the Sailor library version to 2.6.29
• UPDATED the component-commons library version to 3.0.2
• FIXED the vulnerabilities in dependencies

rest-api-ntlm-auth component 1.0.0

• ADDED Make Raw Request action

Salesforce component v2 2.4.0

• ADDED new Get New and Updated Objects Polling trigger, old one set to deprecated
• UPDATED the Sailor library version to 2.6.29
• UPDATED the component-commons library version to 3.0.2
• FIXED the vulnerabilities in dependencies

IPaas Core component 1.5.0

• ADDED the Calculate Flow Dependencies action
• ADDED the Validate Deployability action
• ADDED ability to fetch more that 100 objects (in total) for the Lookup Objects (plural) action
• ADDED checkbox Retrieve only base fields for Lookup Objects (plural) action
• UPDATED the Sailor library version to 2.6.29

2022-08-25 - v22.34

Informative error messaging for invalid SSH key entries

When we introduced support for newest SSH versions for the user SSH keys used to deploy components, we deprecated the old ssh-dss types. If users enter an unsupported or incorrect SSH the user interface now displays an informative error message instead of simply disabling the ‘Save’ button.

Deployment of Externally Build Dockers

This capability will provide an alternative to Git push deployment of user developed components and is of interest to enterprise clients wishing to condense entire integration flows into single custom components and maintain control at the docker level.

At this earliest stages we lay our groundwork by updating the component commons and api to support components pull from external docker registries. The following new attributes were added:

• docker_registry object into the HTTP POST call to /v2/teams/
  • attributes.docker_registry.uri - Docker Registry URI
  • attributes.docker_registry.credentials - Docker Registry credentials
• docker_repo_name and docker_target_tag in the POST HTTP call to /v2/components (can be set if its team has docker_registry object)
  • attributes.docker_repo_name - String representing component’s name in docker registry. Available only if team supports docker registry
  • attributes.docker_target_tag - String representing component’s tag in docker registry. Available only if team supports docker registry

Please note, as of this stage these attributes have no functionality attached to them.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Dashboard Runlog retention period

The flow execution statistics presented on the Dashboard Runlog are stored in the platform Mongo Database. This is in contrast to the flow execution threads, shown on the Executions pages, which are stored separately in Clickhouse.

When you access the Dashboard Runlog, the platform queries the MongoDB to get these records. These database calls can delay display of the runlog for users with many active flows. To enable users to manage such circumstances we have added an environment variable which defines the retention period of runlogs in the database.

Previously you could only control the retention period by enforcing the expireAfterSeconds option in the Database index holding the execution records. In this release we introduce a new TASK_STAT_START_INDEX_TTL environment variable to the HELM3 charts, which you can use to set retention suitable for your environment. It is mandatory that a value is set for TASK_STAT_START_INDEX_TTL and the default value is set at 432000 seconds (5 days). Which means the platform will store the Dashboard Runlog records for 5 days in the Database.

Breaking Changes

Setting the TASK_STAT_START_INDEX_TTL environment variable to control the retention period of the Dashboard Runlog is recommended for the new installations of the platform.

If your platform installation already has the retention period set in the Database prior to your installation of the platform version then your environment variable will not be applied. If you need to change the value you must change the Database index setup prior to the deployment of the current release.

If you need to change the retention period of Dashboard runlogs in the future, remove the expireAfterSeconds option of the Database index containing the Dashboard Runlog records, set the value of TASK_STAT_START_INDEX_TTL environment variable and then deploy the platform version again.

Kubernetes 1.22

Our platform is now compatible with the Kubernetes version 1.22. Check out the new features enabled here.

MongoDB Transport Layer Security verification is enforceable

You can now enforce MongoDB TLS verification in all platform services by using newly introduced HELM3 chart configurations:

• .Values.global.secrets.mongodbTlsCertificateKey - secret name with the TLS certificate and key. If specified, will be mounted to the services and specified in the tlsCertificateKeyFile connection option.
• .Values.global.secrets.mongodbTlsCA - secret name with CA certificate to validate MongoDB server certificate on the client side. If specified, will be mounted to the services and specified in the tlsCAFile connection option.

OEM Related

Information in this section is intended for our customers who use the OEM version of the elastic.io platform.

Embedded credential management for OEM End Users

Here we present further improvements to the recently introduced White-label credential management feature. Now you can use HTML inline frame or iFrame to embed the credentials.

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <title>Embedded Credentials Demo</title>
  <style>html, body, iframe { height: 85%; width: 90%; margin: 0;}</style>
</head>
<body>
  <iframe src="https://[eio_platform_domain]/embedded-credentials/[repoId]?workspaceId=[workspaceId]&ssoProviderType=[ssoProviderType]&ssoProviderId=[ssoProviderId]" />
</body>
</html>

The embed URL construction scheme is the same. Refer to the 22.20 release for details.

Pleass Note: End Users must enable pop-ups in their browser settings in order to use the embedded credential management.

Fixed bugs

• Fixed the bug when an HTTP POST to /v2/workspaces/:id/secrets/:id/refresh endpoint would return outdated access_token instead of the newly refreshed value.

Components

Qualtrics component 1.0.0

• ADDED Get New and Updated Objects polling trigger
• ADDED Make Raw Request action
• ADDED Upsert Object action
• ADDED Generate Custom Distribution Link action
• ADDED Getting Survey Responses action
• ADDED Lookup Object (at most one) action

Google Spreadsheets V2 component 1.0.1

• IMPROVED the component behavior in case of 429 status code. Now it will retry the messages with an exponential backoff.
• IMPROVED the component behavior in case of 5xx errors. The messages will be retried with an exponential backoff as well.

SFTP component 1.5.1

• UPDATED component the component-commons-library to the version 3.0.1

SugarCRM component 1.1.6

• ADDED email1 and email2 fields for metadata of Contacts module in the Upsert Action
• UPDATED elasticio-sailor-nodejs to v2.6.29

Shopware V6 component 1.0.0

• ADDED Make Raw Request Action

Request-Reply component 1.2.5

• UPDATED the maester-client library version to 4.0.2
• UPDATED the component-commons-library to the version 3.0.1

Zoho Subscription component 2.0.0

• UPGRADED authentication mechanism to OAuth 2.0 and migrated to the Secrets service
• UPDATED the sailor library to version 2.6.29
• UPDATED build time dependencies

2022-08-11 - v22.32

Continued improvements

As part of ongoing program of improving our code base and in-line with our ISO27001 processes we have recently completed a number of improvements to both address existing vulnerabilities and prevent potential vulnerabilities being released. Instrumental to this approach is the use of regular scans of our code base during back ups.

An high-level overview of our InfoSec standards and processes are available here.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Scalability of services

elastic.io platform runs many services under the hood, such as webhooks, api, frontend, etc.

Most of these services can be replicated, which allows the services to be scaled horizontally to address the required load and volume. However, some platform services are inherently not scalable. In this update we removed the possibility to define scaling within HELM chart configurations for bloodygate and handmaiden services.

OEM Related

Information in this section is intended for our customers who use the OEM version of the elastic.io platform.

Contract level custom metadata

We introduce a new possibility to add a custom metadata to the contracts. A new section called custom_data can be used to store non-sensitive string data up-to 1MB in size. To add, edit and read contract custom metadata you need to have contracts.contract.edit_custom_data permission (tenantAdmin has this permission by default) to perform POST, PATCH and GET calls to the /v2/contract API endpoint.

This new feature enables Tenant Owners to better differentiate different types of Contracts they supply to their Clients and Users. For instance, a selection of different Contract types can be defined and automatically created, deployed, and maintained using this metadata as basis. Each contract type can have a wide range of specified features, such as GB RAM quota limits, numbers of users or workspaces, and even connector whitelists.

Fixed bugs

• We corrected the error type for the PATCH call to /v2/flows/{FLOW_ID} when an invalid team name or an invalid trigger/action name are defined in the command parameter. Now you will get 4xx type error instead of 5xx which better reflects the situation.

Components

Shopify component 1.6.4

• UPDATED error messages in actions
• UPDATED Upsert Object action to match on criteria other than Shopify ID

Snowflake component 1.3.0

• ADDED ability to read from views
• UPDATED Polling Trigger to allow to Emit Individual Records

Updates to multiple components

Dependencies updated for the following Node.js components where updated:

• SFTP component 1.5.0
• Batch component 2.0.2
• Request-reply component 1.2.4
• Maester component 1.0.3

2022-07-28 - v22.30

Step designer: Credentials part

We have improved the credentials part to provide more unified user experience during the flow design. The look and feel is now more in line with our new user interface approach.

In this new view we moved the Verify, Edit and Delete functions for each credential to a contextual menu which you can open by selecting three dots in the end of each credential record.

Node.js Sailor version 2.6.29

A new version of the Node.JS Sailor library 2.6.29 is released. For Users who develop their own components using our SDKs this update enables keep-alive functionality for the https outgoing connections as well. This functionality reduces the number of https reconnects in favour of using already open connections.

Component build helper 1.1.0

A new version of the component-build-helper library 1.1.0 is released. This library is used by the platform to build components when a new version of an integration component is deployed. For more information check the component build configuration requirements document for supported Java versions and limitations.

Fixed bugs

• Fixed a user interface bug in the recipe activation page when the password was not obfuscated while creating a Basic Authentication credential.
• Fixed a user interface a bug in the recipe activation page when the Additional parameters field in the Rest-API-component credential would not render properly.

Components

Slack component 1.0.0

• ADDED Make Raw Request action
• ADDED Lookup Object (at most one) action
• ADDED Post Message action

Xero component 1.2.0

• ADDED Upsert Object action
• FIXED the start time in the Get Updated Objects Polling trigger

Google Spreadsheets component 3.0.0

• UPDATED authentication mechanism. Component will use Secrets system service to authenticate and refresh tokens.
• ADDED Create/Upsert/Update Spreadsheet Row action
• ADDED Read Spreadsheet action

2022-07-14 - v22.28

Code base optimisation

As part of ongoing codebase optimisation several redundant parameters have been removed from our API - these changes are reflected in our API docs.

PLease visit our API documentation for more information.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Add Ability to Set Per-Service Custom Ingress Annotations

To increase the flexibility of in ingress configurations for different hosts we made a breaking change. We have changed the structure of the model properties tenant.ingressConfig.annotations and helm.Values.global.services.handmaiden.annotations. Please be aware that upgrading to elastic.io Tenants to this latest version requires the small changes described below:

• tenant.ingress_config.annotations format changed. This parameter must be reset via API call before updating to the latest version. Please use this API call:

  {
     "data": {
         "type": "tenant",
         "attributes": {
             "ingress_config": {}
             }
  }
}

• global.services.handmaiden.annotations HELM variable format changed, please set it to the new format during upgrade

Here you can see the corresponding code for PATCH v2/tenants endpoint:

Click to expand for more details

 {
     "data": {
         "type": "tenant",
         "attributes": {
             "ingress_config": {
                 "annotations": {
                     "app": {
                         "nginx/123": "123",
                         "nginx/456": "456"
                     },
                     "api": {
                         "nginx/123": "123",
                         "nginx/456": "456"
                     },
                     "webhooks": {
                         "nginx/mtls": "true",
                         "nginx/456": "456"
                     },
                     "apidocs": {
                         "nginx/mtls": "true",
                         "nginx/456": "456"
                     }
                 }
             }
         }
     }
 }

Please visit our API documentation for more information.

Add “–push-retry” option to kaniko.

To improve push behaviour we added --push-retry and made this configurable only via gitreceiver (deafult: 3). Please setup using .Values.global.gitReceiver.dockerImageBuildJobPushRetry.

OEM Related

Information in this section is intended for our customers who use the OEM version of the elastic.io platform.

Support new component builder for java components

We are introducing a new сomponent-build-helper version 0.1.0. It also means that the old appbuilder is totally deprecated and is no longer supported. We have added custom dockerfile support for Java components.

Please Note: For now our new component builder supports Java 8 based components only.

Check for vulnerabilities after build using grype

Component vulnerabilities check in the build docker image is done by сomponent-build-helper during component build process. Components-build-helper has build_component_docker command which checks docker image vulnerabilities using grype tool.

For more information please visit our article on Components build configuration requirements.

Fixed bugs

• The status page will now be displayed during maintenance on React pages. The property “location” (“head” or “body”, default: “head”) has been added to tenant.attributes.customScripts.

PLease visit our API documentation for more information.

Components

Stripe Component 1.0.0

• ADDED Make Raw Request Action

Shopify admin component 1.6.3

• UPDATED Lookup Objects action, now the method search is used instead of list for the customer object type

XERO component 1.1.0

• ADDED Lookup Objects (plural) Action
• ADDED Lookup Object (at most one) Action
• ADDED Void approved invoices or bills Action
• ADDED Get Updated Objects Polling Trigger