Product Updates in 2022 Q2

Product Updates Archive for 2022 Q2 period.

2022-06-30 - v22.26

Node.js Sailor version 2.6.28

A new version of the Node.JS Sailor library 2.6.28 is released. For Users who develop their own components using our SDKs this update provides improved handling of any errors generated by the Maester service during upload of lightweight messages.

Edit Snapshot button always present

With this release it is now possible to to set and edit snapshots while the flow is in Draft status - flows no longer need to have the status Published or have been executed.

Edit Snapshot always available

Information in this section is intended for our customers who use OEM version of the elastic.io platform.

Email domain blacklist (continuation)

This is further improvement of email domain blacklist feature from 22.22 release. Now you can use a wildcard character (*) with a domain name to exclude all variants, for example gmail.*, yahoo.*, etc.

Fixed bugs

  • Fixed the issue which enabled data sample editing in read-only mode of the flow designer. To edit data samples you must press edit first to make changes.

2022-06-21 - v22.24

To extend vertical space of the flow designer screen we moved the Footer to the bottom of side-menu. You can now see the platform version, Privacy Policy and ToS links by opening the menu.

New footer design

Re-authentication for sensitive actions (final part)

This is the 3rd and final part of our initiative to prevent accidental or malicious changes to sensitive data and follows on from releases 22.20 and 22.22.

In this final part we extend re-authentication for sensitive actions to support cases when a user logs in using OIDC/SAML. The user will be redirected to OIDC/SAML provider for authentication.

Again, please note: You will be asked to re-authenticate once every 6 hours.

This feature will work on all modern web browsers versions from:

  • Chrome 66+
  • Firefox 63+

Component development and custom Dockerfile

To improve component deployment and build process we modernised the appbuilder and apprunner docker images to address potential vulnerabilities in the Node.js packages. All platform services, as well as the Node.js based integration components are built with these images.

The modernisation brought some important changes which might affect you when you deploy the new version of your component code.

  • You can use a custom Dockerfile to define the build process in the Node.js components. It must be in the root directory of your component code.
  • If you have previously used a custom Dockerfile you might have noticed that it was not used in the build process. Now the build process will check for the existence of it and use it. Which means if you have not updated your Dockerfile recently, chances are your component build might fail or the newly built component might not function as expected. Check your Dockerfile before deploying or remove it from the component code if custom build parameters are not necessary.
  • We now support Node.js version 18.
  • IMPORTANT: custom Dockerfile is not yet supported for Java components.

Fixed bugs

  • Fixed the bug in the object and attachment handling service. Now the service will retry streams on failure and will handle errors properly.
  • Fixed the user interface bug when the required field symbol (*) would not show after the metadata reload in case the mandatory fields were controlled by a checkbox.

Components

Salesforce Component v2 2.2.4 and 2.3.0

  • ADDED new Type Of Search - External IDs to Upsert Object action
  • ADDED caching for metadata in the Upsert Object action (metadata needs to find fields that contain attachment)
  • ADDED timeout for Upsert Object action

REST API Component v2 2.0.13

  • IMPROVED secrets only load once per container lifetime (per token expiration time in case of OAUTH2)

Vtex Component 1.4.2

  • ADDED Last interaction option to Time stamp field to poll on in Get New And Updated Objects Polling trigger
  • IMPROVED Get New And Updated Objects Polling trigger to use scroll instead of search to collect all records

HJSON Component 1.0.2

  • FIXED url in attachment

Delta Detection Component 2.1.0

  • ADDED Read Entire Bucket Contents action
  • ADDED Write Entire Bucket Contents action

Utility Component 1.5.0

  • ADDED Network Diagnostics action

IPass Core Component 1.4.0

  • IMPROVED flow with pubSubTopicName at actions Lookup Object By Unique Criteria and Lookup Objects (plural)
  • ADDED logic to handle matching Pub-Sub topics at actions Upsert Object By Unique Criteria
  • IMPROVED flow with componentSemanticVersion at actions Lookup Object By Unique Criteria and Lookup Objects (plural)
  • ADDED logic to handle matching on Semantic Versions at actions Upsert Object By Unique Criteria

Ukraine Alerts Component 1.0.0 and 1.0.1

  • ADDED retries to every API call
  • ADDED Get Ukraine Alerts Polling Trigger
  • ADDED Webhook Trigger

Zoho CRM Component 1.3.3 and 1.3.4

  • ADDED version field to the component.json file
  • FIXED attachment processing
  • IMPROVED set node engine to 16.x (fix error with “Verify Credentials”)
  • ADDED ENV MAX_FILE_SIZE to control attachment size limit
  • IMPROVED upgrade component-commons-library to v2.1.0

Magento2 Component 1.6.7

  • ADDED support for Magento2 versions 2.4.4
  • FIXED integration tests

Shopify Component 1.6.1 and 1.6.2

  • ADDED information about required permissions for the credential verification
  • IMPROVED more info in logs if got errors

ID Linking Component 1.1.0

  • ADDED Write Entire Bucket Contents Action
  • ADDED Read Entire Bucket Contents Action
  • FIXED label names for Upsert Object input metadata

SFTP Component 1.4.9

  • ADDED file filter to the Poll Files trigger

Xero Component 1.0.0

  • ADDED Make Raw Request action

Email Component 1.2.0

  • ADDED support for attachments from metadata. The metadata now is shown in the body.
  • FIXED how the component handles incoming Arrays. Now it stringifies the JSON bodies by default preserving the strings, numbers, and booleans.
  • NEW main library for sending emails (old one is now deprecated).
  • IMPROVED the code by migration to TypeScript.

2022-06-02 - v22.22

Announcements

ISO 27001 certification

We are ISO 27001 certified! Check our dedicated page for more details and to download the copy of our certificate.

Temporary Duplication of Execution Numbers

As part of ongoing improvements we have refined our system architecture around executions to decrease latency when loading the executions page. As part of these changes there is a temporary duplication of the number of executions recorded in your executions page. No data is lost, there is no influence on quota consumption, and these duplications will disappear as we end the transition phase in approximately 30 days. If you have any questions please contact our support directly.

Attachment/object processing and storing

We are changing policies for our retention services. As platform traffic grows the resource burden of retaining files becomes significant.

  • The attachment and object (a.k.a. lightweight objects) retention period is reduced to 3 days (72 hours)
  • The maximum attachment/object size is capped at 100 MB.

Features

Flow designer header

To provide more vertical space during the flow design process we changed the flow designer header behaviour. Now when the left-hand-side menu is closed the action buttons and statuses merge into the same line with flow title and description.

Flow designer header

When your flow title or description is larger than the available space the visible text is cropped. By hovering over these fields you can reveal the full text.

Hover to see the description

Improvements and Updates

Copy code blocks from published flows

Reusing mappings or code snippets from published flows can accelerate flow development. You can now select and copy such text from your published flows without first making a draft of the flow. This feature includes:

  • Any mapping (including JSONata transform views)
  • The selected data sample
  • Any code from the code component
  • Data from retrieved sample

Example shows copied code from the Node.js code component configuration view:

Copy code blocks

Credential ID permanently shown

Now you can see the IDs of any credentials while browsing through the list of credentials in the Integrate > Credentials > Component Name view:

Credential ID in Credentials page

Or while configuring a step in your integration flow:

Credential ID in Flow designer

REST-API component header configuration

We now obfuscate authentication header values for the REST-API-V2 component. You will notice this in the Configuration view when basic or api-key authentication is used.

REST-API-V2 credential

Re-authentication for sensitive actions (continuation)

This is part 2 of our initiative to prevent accidental or malicious changes to sensitive data and follows on from release 22.20.

This time we extend the list of actions for which you will be asked to enter your password to authenticate:

  • Change user roles in Contracts and Workspaces.
  • Add or remove SSH Keys.
  • Remove member from Contract.

Again, please note: You will be asked to re-authenticate once every 6 hours.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Enforcing Maester object size

We introduce a new variable (MAESTER_MAX_SIZE_PER_OBJECT in bytes) you can use to control the maximum object size accepted by the Maester service. The default maximum value is set 1GB. The recommended range is from 100MB to 1GB.

Information in this section is intended for our customers who use OEM version of the elastic.io platform.

Email domain blacklist

If your tenant has an an open registration page then you can block registration from any domain by adding it into the domain blacklist. For this to work in your tenant, make an API call to update your tenant records using tenantAdmin privileges:

  • Enable the registration page - set the attributes.hide_register parameter to false.
  • Add records to the blacklist - create the attributes.email_domains_blacklist record to contain comma separated domain names in an array object like ["example.com", "foo.bar"].

Email verification with regular signup page

During the 20.20 platform release we introduced email verification process which worked only when the signin_v_2 were used.

Now you can use the email verification process even if you are using a regular signup page in your tenant. To enable this feature in your tenant the following steps must be done using an API call to update your tenant records using tenantAdmin privileges:

  • Enable the registration page - set the attributes.hide_register parameter to false.
  • Enable email_verification feature flag - set attributes.feature_flags.email_verification to true.

Fixed bugs

  • Fixed the user interface bug connected with slow loading of the executions page for workspaces containing executions belonging to a deleted user’s flows. Now you will see [Deleted User] instead.
  • Fixed the bug when "Cannot perform 'get' on a proxy that has been revoked" error was shown after evaluation of the JSONata mapping expression.
  • Fixed the Dashboard bug connected with the overlapping axis lines and values.
  • Fixed the bug when Flow Page would not load in case the aggregated flow versions document would exceed 16MB limit in MongoDB.

Components

REST API V2 component

  • IMPROVED the use of platform secrets

HJson component 1.0.2

  • FIXED the attachments URL

Delta-detection Component 2.1.0

  • ADDED Write Entire Bucket Contents action
  • ADDED Read Entire Bucket Contents action

utility component 1.4.2

  • ADDED Network Diagnostics action

Ipaas Core component 1.4.0

  • UPDATED Lookup Flow Hydrates Component Version Info- making this view dynamic
  • UPDATED Upsert Flow Handles Matching on Semantic Versions - making this view dynamic
  • UPDATED Lookup Flow Hydrates Pub-Sub Topic Info - making this view dynamic

Zoho CRM component 1.3.4

  • FIXED the attachment processing
  • FIXED set the node engine to 16.x (addresses error with Verify Credentials)
  • ADDED environment variable MAX_FILE_SIZE to control attachment size limit
  • UPDATED component dependencies

Magento2 component 1.6.7

  • ADDED support for Magento2 versions 2.4.4
  • FIXED the integration tests

Ukraine-alert component 1.0.0

  • NEW component to receive raid alerts notifications in Ukraine
  • ADDED new Get Ukraine Alerts Polling trigger
  • ADDED new Webhook trigger

Google spreadsheet component 2.0.3

  • IMPROVED configuration of the timing of calls is now available in the Credentials step
  • ADDED configuration of retry attempts to the Credentials step
  • ADDED retry logic that helps prevent Quota exceeded errors

Docuware component 1.1.1

  • FIXED bug with data shown in “Search dialog to use” field
  • FIXED attachments Url

JDBC component 2.5.5

  • UPDATED connection parameter connectionTimeZone now defaults to SERVER for MySQL connections

Paypal component 1.3.1

  • ADDED duplicate check of incoming message for Receive IPN message trigger

SFTP component 1.4.8

  • FIXED memory leak for Download Files/File by name actions and Read Files trigger
  • UPDATED component dependencies

Salesforce component 2.2.4

  • ADDED timeout mechanism which allows to retry never-ending requests

2022-05-19 - v22.20

Re-authentication for sensitive actions

While working on the platform interface it is possible that users can accidentally change or delete important or sensitive information.

For such cases we introduce re-authentication for sensitive actions. After for the following actions you will be asked to enter your password to authenticate:

  • Reading, copying and regenerating your API Keys.
  • Modifying your user details.
  • Disabling 2-Factor Authentication (2FA).
  • Inviting members to contracts.
  • Inviting and removing members from workspaces.

Please Note: You will be asked to re-authenticate once every 6 hours.

Sensitive action re-authentication

Email verification during registration

To prevent erroneous and bot registrations we introduce email verification step to our trial registration process. When you register first time with our platform, an email will be sent to your given email address. Click on the link in your received email to finish your registration. There are certain rules and restrictions to this process as well:

  • If you register using Single-Sign-on (SSO) Provider (OIDC/SAML) then the platform will not ask to verify your email address since you have verified it with your provider.
  • You must wait for 30 minutes before you can use the same email address to register.
  • The invitation to confirm your email is valid only for 1 day.

Please Note: If you are invited to the platform directly, no email verification will be required.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements included in this release.

Separate MongoDB for Maester

When your integration flow needs to transfer big objects or attachments, one of the platform microservices called Maester temporarily stores them in MongoDB while passing only the object IDs through the integration flow steps.

Starting from , you can use a dedicated database for storing Maester objects and the run-time attachments. To configure, add MAESTER_MONGO_URI environment variable to HELM3 secrets charts and set it to target this new database.

Note: You will need to migrate the grids to new MongoDB as well to ensure the Maester object availability after implementation.

Organisations who do not need a separate MongoDB instance to store their Maester objects should set the MAESTER_MONGO_URI variable to the same value as the MONGO_URI variable. This will save objects in your main database.

Setting authentication token lifetime

To configure the re-authentication for sensitive actions:

  • Add the SENSITIVE_ACTION_AUTH_LIFETIME environment variable to HELM3 secrets chart and set it to specify how long user authentications are valid. The system expect the value of this parameter in milliseconds. The default is set to 21600000 (6 hours).
  • Enable the tenant.featureFlags.sensitiveActionsReauth feature flag (set it to true).

Information in this section is intended for our customers who use OEM version of the elastic.io platform.

White-labeled credential management

Developing integrations requires a certain level of access to business systems. In cases where the integrators are not responsible for these systems it is critical that only secure access is allowed and that system admins can control credentials used To solve the dilemma we introduce White-labeled credential management feature.

How does the White-labeled credential management work?

As an integrator you provide a unique URL to the credential holder to enter their access credentials without registering or logging-in to the system. The access holder is presented an interface to enter their credentials, verify them and save.

OEM Create Credential

In addition they can modify the existing credentials as well.

OEM Modify Credentials

For both cases they are presented with a simple screen to work with only the specified credentials.

How to generate unique URL?

There are certain preconditions before you can enable this feature:

  • The end customer is already in the DB. This means you have to create a user for your credential holder, then add him/her to the contract and workspace where the integration is to be developed. All these actions you can do with API calls using administrative access.
  • The newly created user must have only access to the credentials entity.
  • Your tenant has SSO provider (OIDC/SAML) with autoCreateUsers set to false.

Once the above conditions are met you can take the following steps to generate the URL to request credentials from their owner:

The initial URL has a form /embedded-credentials/repoId where the repoId would be the ID of component for which you need to set the credential. Then you would need to add the following query parameters to this URL:

  • workspaceId (required) - this is the ID of the workspace where integration is set.
  • ssoProviderType (optional) - values can be SAMLProvider or OIDCProvider.
  • ssoProviderId (optional) - this is the ID of your SSO provider.

Let us construct the URL: TENANT_URL/embedded-credentials/[repoId]?workspaceId=[workspaceId]&ssoProviderType=[ssoProviderType]&ssoProviderId=[ssoProviderId]

If you do not specify ssoProviderType and ssoProviderId then they will be automatically taken from the tenant.loginRedirectSsoProvider property but only if this is defined.

Setup Email verification

To set the email verification process in your tenant the following must be done using an API call to update your tenant records using tenantAdmin privileges:

  • Enable the registration page - set the attributes.hide_register parameter to false.
  • Use signin_v_2 type - add the attributes.signin_v_2 attribute containing the URL of the provider logo using the attributes.signin_v_2.logo_url tenant parameter.
  • Enable email_verification feature flag (NEW) - set attributes.feature_flags.email_verification to true.

Fixed bugs

  • Fixed the error Cannot add property token, object is not extensible appears in the platform user interface when trying to evaluate JSONata expressions with "$count()".

Components

PayPal Component 1.3.0

  • ADDED Receive Instant Payment Notification trigger
  • ADDED Receive Payment Data Transfer trigger
  • ADDED Webhook trigger
  • ADDED Create Object action
  • ADDED Update Object action
  • IMPROVED input metadata in the Make a Payment action

Microsoft One Drive Component 1.0.5

  • UPDATED the component-commons-library to read and upload attachments through the Maester
  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities

Shopify Admin Component 1.6.0

  • ADDED validation HMAC signature for the Webhook subscription trigger

Petstore Component Java 1.0.6

  • IMPROVED the security by adding an automated vulnerability check run in CI/CD

JDBC Component 2.5.4

  • IMPROVED the security by adding an automated vulnerability check run in CI/CD

Vtex Component 1.4.1

  • ADDED Place Order action
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.

CSV Component v3 3.1.4

  • FIXED memory leak on Emit Batch behavior for the Read CSV attachment action

Updates to multiple components

As a part of our on-going improvements of integration components, we

  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.

Here is the list of components for release.

2022-05-05 - v22.18.1

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements done in this round.

Configure login attempts

You can now configure the number of failed login attempts allowed before users are locked out of the system. We added a new variable MAX_LOGIN_ATTEMPTS to the HELM3 secrets chart. You may set this variable according to your own requirements. The default value is set to five (5). Here is how we calculate this number:

  • In case you don’t have 2-factor Authentication (2FA) enabled, this is the number of failed login attempts.
  • In case you have 2FA enabled, this is combined number of failed 2FA code and login attempts. This means if your user has 2FA enabled and succeeded with login but failed with 2FA more than the value of MAX_LOGIN_ATTEMPTS then your user will be locked out of the system. To unlock such users the tenant administration must first disable 2FA for this user and instruct users to navigate /forgot address of the tenant to reset their password. This will reset the counter and user can login again.

Information in this section is intended for our customers who use OEM version of the elastic.io platform.

Removing docker registry builds on component deletion

The platform builds a component docker image every time you push a new version of your component. When you remove the component or a particular version of it from the platform, an associated docker build will remain until you would remove it at later stage directly from the docker registry.

With this update we improved the situation in the following way:

  • When you remove a version of your component, the platform removes the associated docker build.
  • When you remove your component entirely, the platform removes all builds of your component.
  • We created a special tool (remove-deleted-repos-from-docker) to help cleanup the remnant builds. Contact us to get a copy of this tool.
  • As an OEM customer, you must configure your own docker registry to enable delete images option.
  • As an OEM customer, please be aware that the deletion from docker registry does not free the memory automatically. To free the memory you would need to run the garbage collector job.

Components

CSV Component 3.1.3

  • FIXED Emit Batch behavior

Woocommerce Component 2.0.1

  • ADDED Raw Request action
  • ADDED Upsert Object action
  • ADDED Webhook trigger
  • REMOVED Create Object and Update Object actions removed in favor the new Upsert Object action.
  • ADDED metadata information to all actions.

Paypal component 1.3.0

  • ADDED Webhook trigger
  • ADDED Create Object action
  • ADDED Update Object action
  • IMPROVED the input metadata in Make a Payment action

Vtex component 1.4.0

  • ADDED Place Order action

MailChimp component 1.0.3

  • ADDED an ability to set Ansprache field for Add new Subscriber action

Updates to multiple components

As a part of our on-going improvements of integration components, we

  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.

Here is the list of components for release.

2022-04-21 - v22.16

Not enough quota to start task

If you are one of our many power users you might have noticed from time to time a pop-up warning with a message Not enough quota to start task. These messages were not related to your RAM quota limit. Instead these messages are triggered by real-time flows with more than 25 steps. Because the platform initiates each step sequentially, these errors are triggered when all flow steps cannot be initiated within the allowed time.

We have resolved this issue by ensuring that all steps (Kubernetes pods) are initiated in parallel, thus significantly shortening the time required to start the flow. We also changed the warning message to The flow FLOW NAME (FLOW ID) is taking longer than expected to transition.

Example of error

News in this section are for our customers who use OEM version of the elastic.io platform.

Enforce 2FA use

Tenant administration can now enforce the 2FA during the login. We introduce a tenant configuration flag called forced_totp to switch this feature on.

  • To enable forced_totp you must enable the enabled_totp as well.
  • If you remove the enabled_totp feature flag, system will remove forced_totp as well.

You can set these parameters either during the tenant creation or you can update the parameters of an existing tenant via an API call. An example of such an API call is shown below.

{
  "data": {
    "type": "tenant",
    "attributes":{
      "feature_flags": {
        "enabled_totp" : true,
        "forced_totp" : true
      }
    }
  }
}

From version, the platform checks status of the forced_totp parameter during your navigation of every page in the UI. If tenant administration enforced the 2FA and you have not enabled the 2FA for your account then you will be redirected to your profile page and presented a pop-up form to enable the 2FA.

Enable 2FA

Fixed bugs

  • Changed the platform behaviour when webhook would give an error when an integration flow would have adding a step after the request-reply component in the WebHook -> Request-reply flows.

Components

Hubspot component 1.5.2

  • FIXED bug connecting with the uploading attachments

Salesforce component 2.2.3

  • UPDATED dependencies and addressed the packaged vulnerabilities

Sftp component 1.4.7

  • FIXED uploading the attachments

Zendesk component 1.0.1

  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.
  • UPDATED component-commons-library version to 2.0.2
  • FIXED the attachments processing

Email component 1.0.14

  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.

CSV component 3.1.2

  • UPDATED the component-commons-library to read and upload attachments through the Maester service
  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities

Request-reply component 1.2.3

  • UPDATED Sailor version to 2.6.27
  • UPDATED dependencies and addressed the packaged vulnerabilities
  • ADDED the component pusher job to Circle.ci configuration.

2022-04-11 - v22.14

Step designer: Sample data part

We have improved the sample data retrieval to provide a more unified user experience during flow design. The look and feel is now more in line with our new approach.

Step designer: Sample Data part

The above images show the new Sample data retrieval view. This release adds new functionality, allowing users to skip sample generation by clicking on the Skip Sample button. When this function is used the platform will add an empty JSON {} instead of the Sample data.

When you retrieve the Sample the platform UI will show it only in JSON format. In we removed the Integrator view of Data Sample.

Step designer: Sample data retrieved

Here you can view the Sample, collapse different sections of the JSON data structure and edit it.

Quota usage page improvements

To show the RAM quota overuse we improved the quota Usage History histogram. If you go over the set RAM quota value the overused part will show in red.

Quota overuse

SSH Keys and SSH 8.8

To support the newest SSH versions we deprecated the old ssh-dss signature and we added ssh-ed25519 instead. Now when you upload your SSH key to our platform you will see the information about the supported types of signatures.

SSH Key upload screen

Please Note the old ssh-rsa keys are still supported. If you want to use them you have two options:

  1. Use an OpenSSH 8.8+ client and generate your SSH Key using the -oPubkeyAcceptedKeyTypes=+ssh-rsa parameter. It will generate an RSA+SHA-2 type SSH key which provides a high level of security and works with the OpenSSH 8.8+ clients.
  2. Use an OpenSSH <=8.7 client to generate your SSH Key.

Flow deletion times

The platform includes new capability to hard-delete the flows after an expiration time set by the platform management. Every time you try to delete the flow a warning message will show with the following text:

Are you sure you want to delete “FLOW NAME” flow permanently? Deleting this flow is an irreversible process, we cannot undo a deletion if you complete the process by accident. Please note that the credentials linked with the flow will not be deleted.

HELM3

We constantly improve different aspects of HELM3 deployments in collaboration with our partners. This section lists updates and improvements done in this round.

Changes to cadvisor and fluentd demons

To enable better coexistence of platform microservices within the multi-service Kubernetes clusters, we modified the logic used to run cadvisor and fluentd services. The platform will run these services when the node has uniquely defined selectors.

To achieve this we introduced a new HELM3 configuration for the node labels for the Kubernetes pod allocation.

global:
  nodeSelectors:
    platform: "platform"
    longRunningTask: "longrunning"
    ordinaryTask: "ordinary"

For these changes to take affect we deprecate the following HELM3 configurations:

  • KUBERNETES_ORDINARY_LABEL_VALUE
  • KUBERNETES_LONG_RUNNING_LABEL_VALUE

These variables ware removed from the secrets HELM3 chart and are no longer used by the admiral service starting from release. Please use Values.global.nodeSelectors platform HELM3 chart configuration instead.

  • The cadvisor service will run on the nodes with longRunningTask and ordinaryTask labels.
  • The fluentd service will run on the nodes with platform, longRunningTask and ordinaryTask labels.

Richer Ingress Management

To extend the Ingress setup capability we made changes in the Handmaiden microservice to provide 2 new ways to provide annotations to the Ingress records. With this new capability you can set unique Ingress values per each tenant in your cluster.

  1. Set in the HELM3 chart.
  2. Set per tenant using an API call.

Set HELM3 configuration

You can set the Ingress values directly in your HELM3 chart like:

global:
  services:
    handmaiden:
      annotations:
        kubernetes.io/ingress.class: 'nginx'
        testAnnotation: 'testValue'

If you set the above configuration the platform passes it to the new environment variable INGRESS_OPERATOR_ANNOTATIONS used by the Handmaiden service. This adds Ingress annotations to all tenants in your multi-tenant cluster.

Setting via API call

You can set unique Ingress annotations for each tenant using the new configuration while creating the tenant via an API call. A typical call body could look like:

{
  "data": {
    "type": "tenant",
    "attributes": {
      "ingress_config" : {
        "annotations": {
          "annotationName": "value"
        }
      }
    }
  }
}

Annotation Priorities

Our platform will serves these annotations based on priority of annotation the configurations (from low to high):

  • Default Ingress annotations,
  • Annotations provided in the INGRESS_OPERATOR_ANNOTATIONS environment variable,
  • Annotations, provided for the tenant using API call,
  • Static, hard-coded annotations.

Flow deletion job

Introducing a new cron job called clear-deleted-flows which permanently deletes flows from the MongoDB with the DELETED status. Every user trying to delete a flow will see a warning message informing that this is a permanent and irreversible action.

HELM3 optional variables

We introduce new environment variables which you can use to control the clear-deleted-flows job:

  • WIPER_CLEAR_DELETED_FLOWS_AGE_SECONDS - Time in seconds the job must wait before deleting the flow permanently after it is marked as DELETED in MongoDB. We set the default value to 86400 seconds (1 day).
  • WIPER_CLEAR_DELETED_FLOWS_LIMIT - Maximum number of flows the clear-deleted-flows job will permanently delete each time it runs.

Release process

This release process applies if you already have a cluster running previous versions of the platform code. The release will start an automatic migration.

  • Backup the MongoDB data base before the deployment to have an opportunity to reverse the process.
  • Set WIPER_CLEAR_DELETED_FLOWS_LIMIT environment variable to an applicable number for your case to perform this operation faster of you have large number of flows in your DB.

After the migration the marathonevents, requestbins and taskstartlogs deprecated collection will be removed.

Iron Bank table creation on all Clickhouse instances

New Environment Variables:

  • IRON_BANK_CLICKHOUSE_NODES (required) - array of objects with host (required), port, user and password values
  • IRON_BANK_CLICKHOUSE_DATABASE (optional) - ClickHouse database name (default iron_bank)
IRON_BANK_CLICKHOUSE_NODES:
   - host: "10.0.0.1"
     port: "8123"
     user: "default"
     password: "123"
   - host: "10.0.0.2"
     port: "8123"
     user: "user"
     password: "1234"
IRON_BANK_CLICKHOUSE_DATABASE: "iron_bank"

New Iron Bank provision logic

  • If all nodes in IRON_BANK_CLICKHOUSE_NODES are not available, an error is thrown and iron bank shuts down
  • If one of the nodes is not available, a warning is logged and provision to all other available nodes is executed
  • The main connection (to which all querying is executed) is set to the first available node from IRON_BANK_CLICKHOUSE_NODES

Removed Environment Variables:

IRON_BANK_CLICKHOUSE_URI

Fixed bugs

  • Addressed issue when the platform token refresher service would return 500 status code in case when the /refresh/[id] Oauth server would return an error. Now the service reports 4xx indicating the third party problem.
  • Fixed the bug connected with the Profile Page usage example returning 401 in case of API-Key regeneration.

Components

Hubspot Component 1.5.1

  • FIXED the bug in Get New and Updated Objects Polling preventing it from extracting more than 10K records.
  • UPDATED sailor library to 2.6.27 version.
  • ADDED added the component pusher job to Circle.ci configuration.

Salesforce Component 2.2.2

  • ADDED reconnect logic on errors.
  • UPDATED sailor library to 2.6.27 version.
  • ADDED added the component pusher job to Circle.ci configuration.

Vtex component 1.3.0

  • ADDED a new Lookup Object (at Most One) action

Paypal component 1.1.0 and 1.2.0

  • ADDED a new Get New and Updated Objects Polling trigger,
  • ADDED a new Lookup Object (at Most One) action,
  • ADDED a new Make a Payment action.

SOAP component 1.2.9

  • IMPROVED Metadata generation,
  • ADDED a possibility to handle WSDL files where the “message” doesn’t contain an element,
  • UPDATED the Java Sailor libraries to 3.3.9 version.

SFTP component 1.4.6

  • IMPROVED large files handling. Now downloading file sizes bigger 10MB would not truncate the file.

JDBC component 2.5.2

  • UPDATED the Java Sailor libraries to 3.3.9 version.

Petstore component Java 1.0.4

  • UPDATED the Java Sailor libraries to 3.3.9 version.