Product Updates in 2019 Q2

Product Updates Archive for 2019 Q2 period.

2019-05-30 - v3.16.0

Date Details
May 30th, 2019 Platform release v3.16.0

New Features

Performance Boosts

We have made quite a few performance-related improvements. Apart from general optimization of the platform this means that now Flows and Designer pages will work much faster. For once, we will not spam you with boring technical details on this one. You will see the difference.

Frontend

For your convenience we have set alphabetical ordering for the following:

  • Contracts switcher
  • Components list on Credentials page
  • Credentials list of a certain Component
  • Components list in Designer
  • Credentials of a certain Component in Designer
  • Component’s functions list in Designer
  • Workspace members list
  • Contract members list
  • Workspaces list on the Contract details page
  • Developer Teams list on Contract details page
  • Repositories list on Developer Team page

Additionally, you can now open links in new tabs. You know, those you couldn’t before, like: Workspace link in the left navigation bar, Flow link in the list of flows, Next and Previous links in execution log.

Also, some descriptions and messages are now more clear: mapping description for array mapping, limited Workspace text, etc. Some details here, if you require.

Content-Based Router component now shows branch setup directly after component selection on the first step.

Other

You can now filter data samples by ID in API endpoint GET /v2/data-samples?filter[id]=id1,id2,..,idN An example is here.

If you’re using Content-Based Router component, you’d be glad to know that it can now access passthrough. NOTE: The new version of Content-Based Router component needs to be applied to your installation. Details here.

In the following gendry migration 2019-05-16-set-task-last-start-time.js add flow.lastStartTime where it’s not set. This is essential for the new functionality.

Fixed Bugs

  • Flows older than 1.5 years don’t block Contract deletion any more, provided you run gendry migration 2019-05-02-set-task-currentStatus-for-deleted-status.js

  • Uppercase letters in Service Account username do not break login process anymore.

Technical Stuff

WARNING! Boring technical stuff ahead. Read only if you have to.

Frontend stuff

“Please, configure all steps in order to save the flow” message replaced with alert. It shows in case of non-configured components in a flow.

Other Stuff

FILTERING EXAMPLE:

curl https://api.elastic.io/v2/data-samples?filter[id]={DATASAMPLE_ID1},{DATASAMPLE_ID2} \
  -u {EMAIL}:{APIKEY} \
  -H 'Accept: application/json'

EXAMPLE FOR CBR PASSTHROUGH:

Let’s say that the first step of flow returns the following data in the sample:

{
  "fireTime": "2634-02-27T12:50:29.603Z",
  "lastPoll": "2840-04-29T18:20:58.174Z"
}

If you want these values to be evaluated please use the following expressions: elasticio."step_1".body.fireTime or elasticio."step_1".body.lastPoll

2019-04-25 - v3.15.0

Date Details
April 25th, 2019 Platform release v3.15.0

New Features

Full and Limited Workspaces

Now there are two types of workspaces: full and limited. Full workspaces are well-known to you, so limited type is the new thing. Basically, in limited workspaces the workflows are set to stop automatically after a certain time period.

Limited workspaces are mainly meant for trial periods. Check out more info about them here. By default, your workspaces will be full. Here are some more technical details for those interested.

Contracts and Workspaces

There will be no more “stray” contracts, meaning that any contract will always have an Owner. The first member of a contract will automatically be assigned the role of Owner. Also, you cannot just leave the contract without “administration”. If you’re the last member, you can either delete the contract or terminate your account from the platform. As a general rule, if the last member is removed from the contract, the contract is deleted automatically. Check out the technical details.

You can now rename workspaces in the UI. That is, if you have the right permission, of course.

As a casual improvement, you can now enjoy optimized internal resource management in contracts.

Other

All in all, we’ve made a few minor additions to improve performance. For example, we removed an entire tier of overhead by using pods instead of jobs for flows. While performance gain is not huge, all actions (retrieve sample, start/stop flow, etc.) are now faster, and this effect is cumulative.

Also, now user registration process is optimized. It works via API calls instead of directly via Database, which is generally better and more secure.

You can find details here.

Fixed Bugs

  • We updated JSONata versions used in the platform to 1.6.3, so all associated processes will go smoothly. Don’t forget to update mapper, router and JSONata-transform.

Technical Stuff

WARNING! Boring technical stuff ahead. Read only if you have to.

Full and Limited Workspaces Stuff

To define full type as default, set tenant.defaultWorkspaceType to full.

There is a new Wiper script that handles limited workspaces. It is a job stop-limited-flows that periodically stops flows in limited workspaces. Also, we implemented a new API endpoint for Wiper to get flows that have to be stopped in limited workspaces. The response consists of an array of IDs of flows that require stopping. You can modify the TTL (Time to Live) of the limited flows by using the environment variable LIMITED_WORKSPACE_FLOW_TTL_IN_MINUTES. The default value is 10 minutes.

The Wiper job suspend-tasks-with-failing-containers will stop failed flows in limited workspaces, instead of suspending them, as it does in the full workspaces.

Contracts and Workspaces Stuff

A user without owner role will not be added to empty contract via POST /v2/:contract_id/invites or POST /v2/:contract_id/members by default. POST /v2/contracts/:contract_id/members will return 409 if contract has no members and payload.role doesn’t contain owner. POST /v2/contracts/:contract_id/invites will return 409 if contract has no members and payload.role doesn’t contain owner.

To disable this effect you need to set environment variable DISABLE_CONTRACT_MUST_HAVE_OWNER to "true" into secret. After that it is possible to add member without owner role to empty contract.

EXAMPLE:

{
    "DISABLE_CONTRACT_MUST_HAVE_OWNER": "true"
}

Note: because of environment variables processing, the value false is being converted into a string ("false") and considered as truthy value. In order to set the behavior back to defaults, it is better to remove this environment variable from the secret at all.

To not create the default workspace

The default Workspace won’t be created in the following cases:

1. POST new member into the Contract.

2. Redeeming an invite.

3. New user registration.

The page with Create New Workspace button will appear instead.

Other Stuff

Added/changed environment variables that control CPU and memory requests/limits. Make sure the following environment variables are set:

COMPONENT_MEM_DEFAULT=256
COMPONENT_MEM_DEFAULT_LIMIT=256
COMPONENT_MEM_JAVA=512
COMPONENT_MEM_JAVA_LIMIT=512
COMPONENT_CPU=0.1
COMPONENT_CPU_LIMIT=1

All new pods in k8s are started with ImagePullPolicy: IfNotPresent.

Reset snapshot function is disabled while the flow is running. You can reset snapshot only when the flow is stopped.

2019-04-04 - v3.14.0

Date Details
April 4th, 2019 Platform release v3.14.0

New Features

OAuth

  • Introduced API endpoints for OAuth clients. Oauth client entity contains client ID and client secret for OAuth authorization.

  • Oauth client is defined in tenant per component. There may be several OAuth clients in different tenants for same component. Note: since OAuth client belongs to tenant, tenant removal means OAuth clients removal for this tenant.

  • In case a component was removed or hidden due to access change, OAuth client remains. However, no information about component is attached in responses.

  • Accessibility is calculated on tenant basis. So global components are recognized as accessible. If component tenant matches OAuth client tenant, components with tenant and team accesses are also recognized as accessible.

  • Blocked the ability to delete an OAuth client, which is used in the flow (or draft, or version).

  • Global OAuth components don’t appear in GET /v2/components response until a Tenant Admin creates an OAuth Client for them.

  • Set OAuth client related environment variables on each step execution.

  • Reworked OAuth code of Frontend to get tenant-specific OAuth parameters from OAuth clients API. New service account is introduced. It’s used by frontend to communicate with API. Login + pass pair is defined with the new environment variables:

    FRONTEND_SERVICE_ACCOUNT_USERNAME

    FRONTEND_SERVICE_ACCOUNT_PASSWORD.

    You should generate it any way you like (e.g. pwgen -n 15 -y) At this moment only tenants.oauth_client.get permission is granted to frontend service account.

UI

  • Number of workspaces shown on the left menu is now limited by browser height.

  • Added tooltips to the Plus button starting from the second step of the flow. “Add the initial trigger” and “Add a new action” hints.

  • Added tooltip explaining why the user cannot publish a draft. “Please complete all steps first” message is displayed next to the Publish Draft icon.

  • Changed tooltip for Create Draft button to “Edit Flow”. Note: changes will not affect the flow until you publish edited draft message is now appeared.

Other

  • All numeric log object properties are converted to strings to avoid the following Graylog parsing errors: {"type":"mapper_parsing_exception","reason":"failed to parse [err_code]","caused_by":{"type":"number_format_exception","reason":"For input string: \"string\""}} for platform apps logs.

  • To avoid MapperParsingException[failed to parse [err_code]]; nested: NumberFormatException[For input string: "INVALID_PARAM"]. Parsing JSON of components’ log messages removed from Graylog input and is done in-place on frontend. Deployment instructions: Remove all extractors for Graylog GEL input named "Input for flow steps and one-time execs". This should be done after instructions for Route platform app logs to own GELF input -> stream -> index.

  • Removed contract.membership.edit permission from contract.admin role. So that he/she could not change roles of Contract members.

  • New input field type RegExp (Regular Expression). Field object in component.json should have the next property in order to validate it as a regular expression field "type": "regexp".

  • All wiper Cron job descriptors were changed - “startingDeadlineSeconds”: 200 added.

  • Tenant admin can now include the custom .js and .css files in order to be able to change the behavior/appearance of the platform. Tenant model needs to accept the following new properties:

"customStylesheets": [
  {"href": "http://path-to-1.css"},
  {"href": "http://path-to-2.css"},
  ...
],
"customScripts": [
  {"src": "http://path-to-1.js"},
  {"src": "http://path-to-2.js"},
  ...
]

  • Workspace owner can rename workspace via API request.

  • Implemented statistics collector for one-time-executions.

  • CSS improved for custom tenant styles.

  • Unused samples will be deleted after choosing the definite sample.

  • Route platform app logs to own GELF input -> stream -> index. Added ability to send log messages of flow steps and one-time-execs (verify credentials, dynamic metadata, dynamic select model) to separate GELF input in Graylog. This allows applying own extractors configuration and logs records retention policies.

Deployment instructions:

BEFORE DEPLOYMENT:

• Save following .json as file and import as Graylog content pack

{"name":"{name} v3.14.0 update","description":"{name} v3.14.0
update","category":"{name}","inputs":[{"id":"5c641c5346bf4004e61835b5","title":"Input
for flow steps and one-time
execs","configuration":{"override_source":"eio-exec","recv_buffer_size":262144,"bind_address":"0.0.0.0","port":12202,"decompress_size_limit":8388608},"static_fields":{},"type":"org.graylog2.inputs.gelf.udp.GELFUDPInput","global":true,"extractors":[{"title":"message_cut","type":"REGEX_REPLACE","cursor_strategy":"COPY","target_field":"message","source_field":"message","configuration":{"replacement":"$1","regex":"^(.{0,524288}).*$"},"converters":[],"condition_type":"NONE","condition_value":"","order":0},{"title":"JSON","type":"JSON","cursor_strategy":"COPY","target_field":"message","source_field":"message","configuration":{"flatten":false,"key_separator":".","list_separator":",
","kv_separator":"="},"converters":[],"condition_type":"STRING","condition_value":"level","order":1},{"title":"K8S
Fluentd","type":"JSON","cursor_strategy":"COPY","target_field":"","source_field":"log","configuration":{"flatten":true,"list_separator":",
","kv_separator":"=","key_prefix":"","key_separator":"_","replace_key_whitespace":false,"key_whitespace_replacement":"_"},"converters":[],"condition_type":"NONE","condition_value":"","order":2}]}],"streams":[{"id":"5c641bc646bf4004e61830e1","title":"Flow
steps and one-time execs stream","description":"Logs from flow steps and
one-time execs
input","disabled":false,"matching_type":"AND","stream_rules":[{"type":"EXACT","field":"source","value":"eio-exec","inverted":false,"description":""},{"type":"EXACT","field":"source","value":"eio-exec","inverted":false,"description":""}],"outputs":[],"default_stream":false}],"outputs":[],"dashboards":[],"grok_patterns":[{"name":"BASE10NUM","pattern":"(?<![0-9.+-])(?>[+-]?(?:(?:[0-9]+(?:\\.[0-9]+)?)|(?:\\.[0-9]+)))"}],"lookup_tables":[],"lookup_caches":[],"lookup_data_adapters":[]}

• Apply content from imported {name} v3.14.0 update content pack named “Elasticio execs”

• Start Stream “Flow steps and one-time execs stream”

• Make sure udp port 12202 is not blocked by firewall on graylog

• Set GELF_PORT_EIO_EXEC environment variable to 12202 for eio-fluentd Daemon Set

End of Instructions

Fixed Bugs

  • FIXED excessively long text in Configure input section is overlaid table border in retrieved sample.

  • FIXED broken API docs start page when there is no slash at the end of the URL. /docs/v2/ is the new start page now.