Crosser Node 2.6.0

Crosser Node 2.6.0

Release Note

Release date: 2023-06-29


Note: A critical bug was found in 2.6.0 and is now deprecated, use 2.6.1 instead.

New features

Retry on all modules

Each module now has settings for Retry (Common Settings). When the  Max Number Of Retries setting has a value larger than 0 the module will retry sending a message if an attempt fails. When the max limit is reached the message will be dropped and the output will have crosser.success set to false, unless persistence is enabled (see below).

With this feature the Memory Buffer module is no longer needed for standard use cases. It may still be useful if more advanced retry logic is needed.

Where To Use This

This feature is only useful on modules where another try makes sense when a module fails to process a message. For example modules that communicate with external systems can benefit from this, while it does not make sense to use this on most analytic modules.

Persistent Messages

Each module now has a setting for persistence. You will find this setting on the Common settings tab.

Note: The persistent REST API is not available for remote session flows, only deployed flows.

How It Works

Let's say that we have these settings:

  1. Persistent Messages Enabled = true
  2. Max Number Of Retries = 3
  3. Retry Delay In Milliseconds = 1000


This would mean that:

  • The module will make sure that the message is saved before trying to process

  • If the module fails to process the message (an exception occurs or crosser.success is false) it will try again, but maximum 3 tries will be made

  • There will be a delay of 1 second before each retry

What Happens On Success?

If/When the message is successfully processed the persisted message will be deleted from the database.

What Happens When We Run Out Of Retries?

If a module fails to process the message and all retries are used the message will still not be lost. When we reach this state the failing message will be moved to a DeadLetter storage and stay there until:

  • A: The TTL setting for DeadLetters timeout (default 24 hours) and it will be deleted.

  • B: Someone decides to Restore the DeadLetter. It will then be added back to the end of the queue and be processed once again.

The Node offers a new REST API for handling Persistent Messages and DeadLetters

The setting for TTL can be configured with in appsettings.json

  1. "persistenceSettings": { "deadLetterHoursToLive": 24 }


What If The Flow Stops?

If the flow stops no more messages will be added to the persistent storage until the flow starts again. However, all messages persisted before the flow stopped will remain in the storage and when the flow starts each module will load all its messages into the queue again.

Important!

  • Each flow has its own database. This database will be available between restarts of the flow and the node. However, when changing flow-version any data in the database will be lost.

  • Using persistence will have a huge impact on throughput.

MQTT Notifications

The Node will publish messages when things occur that someone might want to know. The root topic for all the messages will be $crosser/. So for example $crosser/flows/{flowDefinitionId}/started will be published when a flow starts.

The $crosser topic will be forbidden by clients to publish to, only subscriptions are allowed. If a client tries to publish with a topic that starts with $crosser/ the client will be disconnected.

Note that MQTT Notifications are disabled for remote sessions. Only deployed flows will trigger MQTT Notifications.

Flow Topics

Topic: $crosser/flows/{flowDefinitionId}/state [RETAINED]
When a flow changes state (started, stopped) this topic will be triggered with the information.

  1. {
  2.     'flowId': <guid>,
  3.     'flowName': 'Demo Flow',
  4.     'version': '3',
  5.     'running': true,
  6.     'status': 'Ok', // or error if it was an error
  7.     'reason': 'Flow xyz was stopped',
  8.     'origin': 'Cloud' // or local
  9.     'timestamp': '2022-02-22 18:34:23'
  10. }

Topic: $crosser/flows/{flowDefinitionId}/status [RETAINED]
When a flow change status to Ok, Warning or Error

  1. {
  2.     'flowId': <guid>,
  3.     'flowName': 'Demo Flow',
  4.     'version': '3',
  5.     'status': 'Ok', // or error/warning
  6.     'reason': 'Information about the reason for the status',
  7.     'origin': 'Local'
  8.     'timestamp': '2022-02-22 18:34:23'
  9. }

Module Topics

Topic: $crosser/modules/{moduleId}/status [RETAINED]
When a module changes status (Ok, Warning, Error)

  1. {
  2.     "moduleId":"133efbb7-cfc6-400c-bcff-8312f02d6596",
  3.     "moduleName":"CSharp",
  4.     "version":"2.2.2",
  5.     "status":"Error",
  6.     "flowId":"64ad5939-2f68-47ba-a0d5-8bc5d8dc5b27",
  7.     "flowDefinitionId":"1234939-2f68-47ba-a0d5-8bc5d8dc5b27",
  8.     "flowName":"MQTT Notifications",
  9.     "reason":"Module CSharp, 2.2.2, Index: 2 changed to status Error: One or more errors occurred. (Boom)",
  10.     "queue":{
  11.         "mode":"DropWrite",
  12.         "size":1000,
  13.         "persistence": true,
  14.         "retries":5,,
  15.         "retryDelay":1,
  16.         "delay": 0
  17.     },
  18.     "timestamp":"2022-03-04T16:13:16.960Z"
  19. }

Topic: $crosser/modules/{moduleId}/messages/dropped
When the module queue is full and the strategy is not wait.

  1. {
  2.     "moduleId":"133efbb7-cfc6-400c-bcff-8312f02d6596",
  3.     "moduleName":"CSharp",
  4.     "version":"2.2.2",
  5.     "status":"Warning",
  6.     "flowId":"64ad5939-2f68-47ba-a0d5-8bc5d8dc5b27",
  7.     "flowDefinitionId":"1234939-2f68-47ba-a0d5-8bc5d8dc5b27",
  8.     "flowName":"MQTT Notifications",
  9.     "reason":"Queue full",
  10.     "queue":{
  11.         "mode":"DropWrite",
  12.         "size":1000,
  13.         "persistence": true,
  14.         "retries":5,,
  15.         "retryDelay":1,
  16.         "delay": 0
  17.     },
  18.     "timestamp":"2022-03-04T16:13:16.960Z"
  19. }

Topic: $crosser/modules/{moduleId}/messages/deadletters
When persistence is on and a message has failed maximum number of times it will be moved to the deadletter queue. I do not see any need for event when a messages fails since we will have retry until success or deadletter

  1. {
  2.     "moduleId":"133efbb7-cfc6-400c-bcff-8312f02d6596",
  3.     "moduleName":"CSharp",
  4.     "version":"2.2.2",
  5.     "status":"Warning",
  6.     "flowId":"64ad5939-2f68-47ba-a0d5-8bc5d8dc5b27",
  7.     "flowDefinitionId":"1234939-2f68-47ba-a0d5-8bc5d8dc5b27",
  8.     "flowName":"MQTT Notifications",
  9.     "reason":"Retries maxed out",
  10.     "queue":{
  11.         "mode":"DropWrite",
  12.         "size":1000,
  13.         "persistence": true,
  14.         "retries":5,,
  15.         "retryDelay":1,
  16.         "delay": 0
  17.     },
  18.     "timestamp":"2022-03-04T16:13:16.960Z"
  19. }

Subscribing

  • By subscribing to $crosser/# you would get all messages

  • Subscribing to $crosser/flows/# would get you all flow messages

  • Subscribing to $crosser/modules/# would get you all module messages

Of course you can also use other combinations and use + as a single level wildcard

  • Subscribing to $crosser/flows/+/stopped would get you only stopped flow messages

  • Subscribing to $crosser/modules/+/status would get you module status messages for all modules

Get the current status for all deployed flows

When a MQTT client creates a subscription for the topic $crosser/flows/# or $crosser/flows/+/status the broker will send out the current status for all deployed flows.

Settings To Enable/Disable Endpoints For HTTP/MQTT/API

Up until now the Node will always start the endpoints for HTTP, MQTT and REST API. There might be a number of reasons why you do not want to start all (or any) of them.

All of the endpoints will be enabled by default, but you can now disable them in the appsettings.json file.

  1. "httpServersEnabled": true,
  2. "mqttBrokersEnabled": true,
  3. "apiServerEnabled": true


If you prefer to use ENVIRONMENT VARIABLES.

Example for HttpServers

  1. NodeConfiguration__HttpServersEnabled: true


You can also use arguments if you need that...

Example for HttpServers

  1. --NodeConfiguration:HttpServersEnabled, true


The priority order highest to lowest for configurations are

  • Arguments

  • Environment Variables

  • Settings in json-files

When Disabling The HTTP Server

  • You will not be able to use HTTP to send data to Flows

  • You will not be able to use WebSockets to send data to Flows

When Disabling The MQTT Broker

  • You will not be able to use external MQTT clients to send data to Flows. You can still use MQTT Client modules to get data from external brokers.

  • You will not be able to use the Python Bridge module in Flows if the module version is less than 4.0.0

  • You will not be able to use the MQTT notifications feature

When Disabling The API Server

  • You will not be able to use the local UI (http://localhost:9191)

  • You will not be able to use the REST API for seeing Flows/Metrics/Logs etc

Handle Date & Time in Serialization

The Default JsonSerializer now supports the new C# types DateOnly and TimeOnly.


Changes

Halt On Error

If a flow process is stopped (crashes) with an error code and Halt On Error is set to false the Host will restart the flow.

Log Message For Modules

Previously we logged events with the name of the module in combination with a calculated index. This was confusing when you have complex flows with several branches. As of 2.6.0 we log events using the name of the module which can be changed by the user when building the flow.

MQTT Deprecated as RemoteSession Transport

In previous versions of the Node you could use MQTT as transport for Remote Sessions. This option has now been removed and instead the options are WebSockets (preferred) or HTTP.

Add Support For Rolling Files

Prior to 2.6.0 no more logs was written when the file size limit was reached (10MB). As of 2.6.0 the logger will role the file when the limit is reached.

Sample output

log20220119.log

log20220119_001.log

log20220119_002.log

log20220119_003.log

Proxy Configuration Changes

In previous versions of the Node you could set username and password for proxies as a separate setting. This is now changed and the username and password will be set directly on the ENVIRONMENT variable:


  1. // Non-authenticated HTTP server:
  2. HTTP_PROXY=http://10.10.1.10:1180

  4. // Authenticated HTTP server:
  5. HTTP_PROXY=http://username:password@10.10.1.10:1180

  7. // Non-authenticated HTTPS server:
  8. HTTPS_PROXY=http://10.10.1.10:1180

  10. // Authenticated HTTPS server:
  11. HTTPS_PROXY=http://username:password@10.10.1.10:1180


DateTime Serialization

Previously always to UTC, now always local time with information about timezone in the format yyyy-MM-ddTHH:mm:ss.fffzzz.


Bugfixes

Metric For MQTT Connections

When having a very instable network the MQTT connections metric would show a negative number of connections. This was due to the fact that the incrementation of connections was done after reading the MQTT connect message. In cases where the connect message was never received only decrementation was done and this caused negative connection metrics.

The incrementation of MQTT connections is now done before the MQTT connect message is complete.

Timeout When Starting Many Flows At Once

In previous versions of the Node the flow processes were all started at the same time. This could cause timeouts if the machine does not have sufficient performance to be able to handle the load.

As of 2.6.0 the flows will be started in sequence to make sure that machines with low performance and many flows deployed still can handle a safe startup.

Windows Service Log-Size To Small

As of 2.5.2 the maximum log-file-size was increased to 10MB, the Windows Service did not get this change and still had 1MB in 2.5.2. This is now changed so that the Windows Service also has a limit at 10MB.

Increased Timeout For C# Module

The default initialize timeout for modules is 30 seconds. The C# module might require more time to compile the code on machines with poor performance, therefor the initialize timeout for the C# module was increased to 2 minutes as of 2.6.0.

Copying FlowMessages With byte[] Was Slow

When a Flow message had large byte arrays as properties, cloning the Flow message was unnecessary slow & also allocated memory in an insufficient way. This is now fixed.

WebSocket Did Not Work Behind Proxies

In previous Nodes the Remote Session could not use WebSockets as transport behind a HTTP proxy. This has now been fixed. There are still HTTP proxies that does not allow WebSocket traffic, but chances are that you now can use the standard transport for Remote Sessions

HTTP Request Timeout Missing

If a HTTP client start sending data to the Node and not send all the data the Node would wait infinite for the data to be received. As of 2.6.0 the Node will terminate the HTTP connection if no data is received in 30 seconds.

Numeric Overflow When Reporting Metrics

An invalid cast to Int32 was used preventing metrics to have numbers over 2147483647. This could cause negative values for network traffic and number of messages in Crosser Cloud.




    • Related Articles

    • Crosser Security Overview

      Crosser Security Overview November 2023 About this document This document describes security aspects related to the Crosser Streaming Analytics solution. The Crosser Streaming Analytics solution The Crosser Streaming Analytics solution has two main ...

    • Crosser Node 2.5.3

      Crosser Node 2.5.3 May 2022 Bugfix - Prevent Crash When Getting ChannelClosedExceptions When running remote sessions with unstable network connections the node host process could crash due to not handling a ChannelClosed exception in a proper way.

    • Monitoring the Crosser Node

      Introduction Once you have your first flows deployed, you might think about how to integrate the Crosser Node and Flows into your existing monitoring solution. In this article we describe what options you have and how to utilize provided interfaces ...

    • Crosser Node 3.1.0

      Release Note Release date: 2023-12-20 Changes Sending metrics for Host and Runtimes Now that Crosser Control Center can handle metrics both for the Node and the deployed flows (runtimes), the Node will report metrics for the host as well as all ...

    • Crosser Node 2.5.4

      Crosser Node 2.5.4 Note: Version 2.5.4 of the node has been pulled back due to critical issues that were identified after the release. Please upgrade to a later version. September 2022 Bugfix - Prevent Invalid Http Requests To Cause 100% CPU This bug ...