External Source Integration Guide
Vantiq supports the integration of external data sources into the automation service. A source may supply data or be a recipient of notifications produced by Vantiq or both supply data and be a recipient of notifications. At the present time Vantiq supports following source types:
- Email servers
- REST services
- SMS services
- Kafka topics
A source implements one or more of the following capabilities:
- Stream data This provides a continuous stream of objects from the source. Examples of this include an MQTT topic with new messages delivered asynchronously to the server or a web site that is read periodically to produce a stream of data.
- Query the source for data This requests a specific result be retrieved from the source. For example, you can send a request to a REST API in order to interact with some external service.
- Publish data to the source This sends data from the Vantiq server to the specified source. Examples include sending a message to an AMQP topic or sending a push notification via SMS.
Sources are defined by creating a source resource that represents the definition of the source and submitting the definition to Vantiq for registration. Once registered, the source will immediately begin reading and processing messages if a stream is configured and will also immediately be available for query and publish operations.
The specific details of the ArsSource properties required are described in the separate documentation for each source class:
The general representation of a source resource is defined in Resource Reference Guide: Source. In summary, a source contains the following properties:
- name - the name asigned to the source.
- type - the type of source. Currently must be one of thes string values:
- direction - indicates whether the source sends (SINK), receives (SOURCE) or both sends and receives (BOTH) data and must be one of the string values:
- config - contains the source type specific parameters associated with the source. The detailed contents of config is defined in the source type specific documentation referenced above.
As described above sources can support up to 3 operations, all of which operate on data in the form of “messages”. The operations and message formats supported are specific to each source implementation and described in the source specific documentation. Here we describe the general capabilities.
Vantiq supports 3 distinct content types for data sent and received by sources:
- XML – XML data is sent and received using the standard XML textual notation. At runtime XML objects are manipulated using the GPath notation.
- Text – text data is unformatted and uninterpreted by the system. It will be transmitted and received using the UTF-8 character set.
Each source implementation is responsible for transforming data sent and received between the external form supported by the source and one of these 3 content types supported by the Vantiq system. For example, an Email source may accept a JSON document which describes the elements of an Email message and transform that document into the actual SMTP message.
A stream of information is read from the external system using either an asynchronous model or a synchronous (polling) model depending on the capabilities of the external system. The process stream method must run continuously and may not terminate until the source is shut down. For example, the MQTT source type asynchronously reads messages when they are published to the corresponding MQTT topic while the REMOTE source periodically polls the specified REST endpoint for new data.
When a message arrives on a streaming source an event is generated which will be delivered to any subscribed rules for processing. The event may also be delivered to any web socket clients which have subscribed to the event’s id.
The ultimate result of processing is usually, but not always, a change to the state of the automation model that may trigger additional rules.
If load balancing is required on the MQTT or AMQP stream it is best if the processing rule is stateless and not use internal state. Stateful data can be placed in the automation model so that the change can dispatch rules across the server cluster. It is also practical to partition the work across servers by assigning messages to different servers.
A query can be presented to sources that support queries to obtain data from the external system. A query is executed by invoking the query function from the body of a rule. The query method is provided with the source type specific object that defines the query to execute. For example, the REMOTE source type that supports REST endpoints is invoked with an object that defines the URL, headers and body for the REST request that obtains the data from the remote endpoint.
A source can send data or notifications to the external system or, perhaps more appropriately, via the external system since the ultimate receiver of the notification may be a person. Examples of notify method uses:
- MQTT source - posts the message to the specified MQTT topic. Ultimately, the message will be delivered to all MQTT clients subscribed to the MQTT topic.
- EMAIL source - the message is emailed to the set of recipients attached to the message.
- REMOTE source - the message is POSTed to the URL defined on the remote source.
The notify function is invoked from a rule to send data or notifications via a source.