App Builder Guide

Introduction

Modern Real-Time Business Applications often have, at their core, a collection of disparate data streams that contain inter-related information. Value can only be extracted from this data after it has been massaged, filtered, and combined with other data flowing through separate parts of the business. This data-munging is an inevitable part of any business and is a prerequisite to kicking off higher level business processes that can be modeled with a tool like the Collaboration Builder. Enter the App Builder: a visual tool for describing and automating the data ingestion and processing pipeline that drives business decisions.

The App Builder allows users to capture, transform, and make decisions on streams of events occurring within the Vantiq system without writing much, if any, code. The app builder goes beyond traditional ETL pipelines by not just transforming data, but also identifying important business situations within the event streams which can automatically drive collaborations. Vantiq’s mission is to help businesses Sense, Analyze, and Act on real-time data, and the App Builder helps complete that mission by dramatically reducing the amount of code (and development time) that goes into sensing and analyzing the data.

Apps are built graphically using the App Builder, which allows users to layout the general flow of data through their system, from initial acquisition through to the identification of a high-level business situation which can drive a collaboration. Each box in the diagram represents a task that operates on a stream of inbound events produced by its parent task, and produces outbound events that can be consumed by each child task. The root tasks at the top of an app identify the initial streams of data that will be combined by the app, and then each child activity operates on some transformed subset of the initial event stream.

Some of the high level capabilities of the App Builder include:

Tutorial

To walkthrough building your first app, check out the tutorial.

App Builder Overview

An app builder diagram consists of boxes, representing tasks, and lines, representing streams of events flowing between tasks. Tasks perform an operation on inbound events and then emit outbound events for the next task to consume. Each task in an app consumes one or more inbound streams of events, and produces an outbound stream of events. All apps flow downwards, starting with the root Event Stream tasks at the top of the diagram. Certain tasks can only appear at the end of an app diagram, for instance the Detect Situation task creates a situation which can kick off one or more collaborations, at which point any further logic would be diagramed in the Collaboration Builder.

Apps are, under the covers, completely implemented in VAIL. When an app is created or updated, a collection of rules and procedures are generated to implement each task. The app builder is a convenient tool for jumpstarting your development, but it’s important to remember that the app builder models a subset of what VAIL is capable of. When the app builder isn’t expressive enough to implement a particular piece of application logic, it’s almost always possible to implement the logic directly in a VAIL rule or procedure instead.

Apps are assembled from a collection of predefined tasks. The current set includes:

Creating an App

When creating a new App, the initial diagram consists of a single, unconfigured Event Stream task. From there, more event streams and downstream tasks can be added by right clicking on an existing task and selecting one of the options from the contextual menu. Each task added to the app diagram needs to be properly configured in order for the app to compile. Apps are compiled whenever they are saved, and if no errors are found during compilation, the app will generate a collection of hidden rules that implement each of the tasks. If errors are detected during compilation of the app, it will still be saved, however none of the generated code will be saved. When this happens the task that the first error was detected in will be highlighted in red.

The app builder is designed to facilitate an iterative development process. Start with an initial event stream and a log stream task to see what raw events are being received. Next add a transformation task that processes the outbound events from the raw event stream and then log the output of the transformation. Then add a filter to the output of the transformation and log the filtered event stream. Once you’ve tuned the filter and transformation activities to produce the output you expect, you can add a detect situation task to the output of the filter task, and you’ve built an entire app.

Collaborations and Apps

Apps are intended to simplify a piece of the development process, however at some point the app will identify a situation in which a solution cannot be fully automated. At this point human users need to be involved: enter collaborations. Once a detect situation activity in an app has been configured, right click on the task and it should contain an option to “Add new Collaboration Type”. Clicking this will start a new collaboration builder window in the same project with a preconfigured triggering condition that will automatically trigger whenever the detect situation activity in the app is reached by an event.

App Activity Patterns

Tasks in an App are configurations of predefined Activity Patterns. Each activity pattern represents a generalizable process that can be expressed with just a few configuration properties. Given an Activity Pattern and a configuration of the pattern, we can produce VAIL code that implements the specifics of the task in the App Builder.

There are some configuration properties that are generic to (almost) all activity patterns. These include:

Below are brief descriptions of the existing activity patterns and the configuration properties that each contains in addition to the common ones described above:

Detect Situation

When an event reaches a Detect Situation task, a situation instance is automatically created that includes the event data that caused it to be created. The contextual menus that pop up when right-clicking on a task will include an option to create a new collaboration type from a detect situation task. This will automatically configure the initiate activity in the collaboration to trigger when the Detect Situation task creates a situation.

No additional configuration parameters need to be specified for a Detect Situation task.

Dwell

Dwell can be used to identify a state that hasn’t for a duration of time across sequential events. For instance, a stream of events might contain temperature readings from a sensor on a machine, and it’s dangerous for the machine to have a temperature greater than 100 degrees for more than a minute. If the temperature jumps over 100 degrees temporarily, that’s not necessarily dangerous. This is an ideal use case for dwell, which can be configured to only emit events when the condition event.temperature > 100 is true for 60 seconds. The desired state is expressed as a VAIL boolean expression, and dwells are detected by sequential events satisfying the condition.

The Dwell activity pattern contains the following configuration properties:

Event Stream

The root tasks in any app are Event Streams. Event Streams identify a resource on which events occur, and an optional constraint that restricts which events to include in the stream. The currently supported resources are types, sources, and topics, so any insert on a type, message arriving on a source, or publish to a topic can be the root of an App. Apps can have multiple event streams which identify completely different pieces of data, and these can be joined downstream to detect more complex situations in parallel streams.

The Event Stream activity pattern contains the following configuration properties:

Enrich

The Enrich task is useful for attaching persistent data to a stream of events. For instance, an event stream may produce temperature readings for a motor that look like:

{
    "engineId": "XYZ",
    "temp": 165
}

And there could be an existing Engine Type that contains data associated with each engine that looks like:

{
    "engineId": "XYZ",
    "manufacturer": "Ford",
    "modelYear": 2010,
    "optimalTemp": 155,
    ...
}

The enrich task would automatically join the Engine data to the event to produce outbound events that look like:

{
    "engineId": "XYZ",
    "temp": 165,
    "Engine": {
        "engineId": "XYZ",
        "manufacturer": "Ford",
        "modelYear": 2010,
        "optimalTemp": 155,
        ...
    }
}

The Enrich activity pattern requires 2 configuration properties:

Filter

A simple task that restricts which events flow through to downstream tasks. The condition is specified as a VAIL conditional expression and can operate on any of the properties in the inbound event.

The Filter activity pattern contains only one non-standard configuration property:

Join

The Join activity pattern allows users to identify conditions that occur across multiple event streams and merge these into singular events for the consumption of downstream tasks.

For an example, consider the engine monitoring demo application, which detects when the engine enters a bad state. There are two independent streams in the demo, one containing engine speed readings and another with the engine temperature. Each reading is associated with a specific engineId, so to identify an engine in a bad state we need to be able to identify readings on the engine speed stream which have a speed over 1000 and readings on the engine temperature stream over 140, with the same engineId, where both events occur within 30 seconds of each other.

This “Bad Engine” situation can only be captured by identifying conditions on two parallel streams that occur within a constrained time window.

The Join Activity Pattern includes the following configuration properties:

Log Stream

The Log Stream task is useful for debugging purposes. It takes no configuration properties, and simply logs each event that reaches the log stream task at the INFO level. When developing an app, it’s often useful to attach a log stream to each task as you experiment with its configuration in order to check the output of a task. Then, once the task is properly configured, remove the log stream, begin adding a new task and attach a new log stream to the new task.

Merge

Sometimes it is possible to end up in a situation where multiple streams are producing identical data, and in these cases it is often helpful to merge them into a single stream that can be consumed by downstream tasks. The Merge task does this for any number of streams. For best results, the inbound data on each of the independent parent streams should adhere to the same schema so that downstream tasks can all process the output of the merge in a consistent way.

The Merge task takes no configuration parameters.

Missing

The Missing task detects the absence of an event for a fixed time duration. This can be useful for monitoring the heartbeat or pulse of a system (or person). The Missing task takes inbound events and produces no output as long as events occur within the specified window; but if no event is detected for the specified duration, then an output event is produced indicating no inbound event was received. As long as no new inbound event is received, an output event will be produced for every interval that passes without an inbound event.

The Missing activity pattern accepts 2 additional configuration properties:

The output of the missing task looks like:

{
    "missingAt": "2017-10-09T21:45:28.231Z", 
    "task": "MissingMyTopic",
    "lastEvent": {
        ... // Copy of last event received
    }
}

Threshold

Threshold can be used to detect sequential events crossing a threshold. The threshold is expressed as a condition, and crossing the threshold is determined by the condition evaluating to different values for the sequential events. For instance, if we have an inbound stream of temperature readings from sensors, and we want to detect when the temperature goes over 100 degrees, the threshold condition would be event.temp > 100. If one event came in with a temp value of 90, then the next came in with a value of 101, the threshold would trigger. Likewise it would also trigger if initially the temperature was first 101, then a new reading came in at 90 degrees.

The threshold activity pattern contains the following configuration properties:

Transformation

The Transformation task allows users to inject their own procedure into an app. The transformation procedure is expected to take a single object (the inbound event) as input, and return the mutated outbound event. In between the procedure can do whatever the user wants, including produce events that trigger other event streams in the app (for instance by publishing to a topic). Returning null from the transformation procedure indicates the inbound event should be filtered out, in which case the transformation task will produce no outbound event.

The Transformation activity pattern contains only one non-standard configuration property:

Unwind

When an inbound event contains an array property, it is sometimes helpful to process the entries within the array independently. The Unwind task allows users to do just that. For example, the inbound event:

{
    "name": "joe",
    "heartrates": [
        {"ts": "2017-09-07T23:45:13.573Z", "value": 60}, 
        {"ts": "2017-09-07T23:46:14.512Z", "value": 65}, 
        {"ts": "2017-09-07T23:51:11.513Z", "value": 90}
    ]
}

contains an array of heartrates that were recorded at different times. Using the unwind activity, it would be possible to turn this one event into 3 sequential events like:

{
    "name": "joe",
    "rate": {"ts": "2017-09-07T23:45:13.573Z", "value": 60}
},
{
    "name": "joe",
    "rate": {"ts": "2017-09-07T23:46:14.512Z", "value": 65}
},
{
    "name": "joe",
    "rate": {"ts": "2017-09-07T23:51:11.513Z", "value": 90}
}

each of which would be processed by downstream tasks independently.

The Unwind Activity Pattern contains the following configuration properties:

We’re always looking for new patterns in the applications users develop to turn into new tasks, so if you have any ideas please send them to the Vantiq Product Team.