Go Reference Go code stats Go Report Card Slack Widget

Simple Iot is a platform that enables you to add remote sensor data, telemetry, configuration, and device management to your project or product.

Implementing IoT systems is hard. Most projects take way longer and cost more than they should. The fundamental problem is getting data from remote locations (edge) to a place where users can access it (cloud). We also need to update data and configuration at the edge in real time from any location. Simple IoT is an attempt to solve these problems by embracing the fact that IoT systems are inherently distributed and building on simple concepts that scale.

Simple IoT provides:

  • a single application with no dependencies that can be run in both cloud and edge instances
  • efficient synchronization data in both directions
  • a powerful UI to view configuration and current values
  • a rules engine that runs on all instances that can trigger notifications or set data
  • extensive support for Modbus -- both server and client
  • support for the Linux 1-wire subsystem.
  • flexible graph organization of instances, users, groups, rules, and configuration
  • integration with other services like InfluxDB and Twilio
  • a system that is easy to extend in any language using NATS
  • a number of useful Go packages to use in your custom application

See vision, architecture, and integration for addition discussion on these points.

See detailed documentation for installation, usage, and development information.

Motivation

This project was developed while building real-world IoT applications and has been driven by the following requirements:

  • Data (state or configuration) can be changed anywhere — at edge devices or in the cloud and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify data.
  • Data bandwidth is limited in some IoT systems — especially those connected with Cat-M modems (< 100kb/s). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.

Core ideas

The process of developing Simple IoT has been a path of reducing what started as a fairly complex IoT system to simpler ideas. This is what we discovered along the way:

  1. treat configuration and state data the same for purposes of storage and synchronization.
  2. represent this data using simple types (Nodes and Points).
  3. organize this data in a graph.
  4. all data flows through a message bus.
  5. run the same application in the cloud and at the edge.
  6. automatically sync common data between instances.

Design is the beauty of turning constraints into advantages. -- Ava Raskin

These constraints have resulted in Simple IoT becoming a flexible distributed graph database optimized for IoT datasets. We'll explore these ideas more in the documentation.

Support, Community, Contributing, etc.

Pull requests are welcome -- see development for more thoughts on architecture, tooling, etc. Issues are labelled with "help wanted" and "good first issue" if you would like to contribute to this project.

For support or to discuss this project, use one of the following options:

License

Apache Version 2.0

Contributors

Thanks to contributors:

Made with contrib.rocks.

Installation

Simple IoT will run on the following systems:

  • ARM/x86/RiscV Linux
  • MacOS
  • Windows

The computer you are currently using is a good platform to start with as well as any common embedded Linux platform like the Raspberry PI or Beaglebone Black.

If you needed an industrial class device, consider something from embeddedTS like the TS-7553-V2.

The Simple IoT application is a self contained binary with no dependencies. Download the latest release for your platform and run the executable. Once running, you can log into the user interface by opening http://localhost:8080 in a browser. The default login is:

  • user: admin@admin.com
  • pass: admin

Cloud/Server deployments

When on the public Internet, Simple IoT should be proxied by a web server like Caddy to provide TLS/HTTPS security. Caddy by default obtains free TLS certificates from Let's Encrypt and ZeroSSL with automatic fallback if one provider fails.

There are Ansible recipes available to deploy Simple IoT, Caddy, Influxdb, and Grafana that work on most Linux servers.

Yocto Linux

Yocto Linux is a popular edge Linux solution. There is a Bitbake recipe for including Simple IoT in Yocto builds.

Use Cases

Simple IoT is platform that can be used to build IoT systems where you want to synchronize data between a number of distributed devices to a common central point (typically in the cloud). A common use case is connected devices where users want to remotely monitor and control these devices.

use

Some examples systems include:

  • irrigation monitoring
  • alarm/building control
  • industrial vehicle monitoring (commercial mowers, agricultural equipment, etc)
  • factory automation

SIOT is optimized for systems where you run Embedded Linux at the edge and have fairly complex config/state that needs synchronized between the edge and the cloud.

Changes can be made anywhere

Changes to config/state can be made locally or remotely in a SIOT system.

edit anywhere

Integration

There are many ways to integrate Simple IoT with other applications.

integration

There are cases where some tasks like machine learning are easier to do in languages like C++, then you can connect these applications to SIOT via NATS to access config/state. See the Integration reference guide for more detailed information.

Multiple upstreams

Because we run the same SIOT application everywhere, we can add upstream instances at multiple levels.

multiple upstream

This flexibility allows us to run rules and other logic at any level (cloud, local server, or edge gateway) -- wherever it makes sense.

User Interface

Contents

Basic Navigation

After Simple IoT is started, a web application is available on port :8080 (typically http://localhost:8080). After logging in (default user/pass is admin@admin.com/admin), you will be presented with a tree of nodes.

nodes

The Node is the base unit of configuration. Each node contains Points which describe various attributes of a node. When you expand a node, the information you see is a rendering of the point data in the node.

You can expand/collapse child nodes by clicking on the arrow arrow to the left of a node.

You can expand/edit node details by clicking on the dot dot to the left of a node.

node edit

Adding nodes

Child nodes can be added to a node by clicking on the dot to expand the node, then clicking on the plus icon. A list of available nodes to add will then be displayed:

node add

Some nodes are populated automatically if a new device is discovered, or a downstream device starts sending data.

Deleting, Moving, Mirroring, and Duplicating nodes

Simple IoT provides the ability to re-arrange and organize your node structure.

To delete a node, expand it, and then press the delete icon delete icon.

To move or copy a node, expand it and press the copy copy icon icon. Then expand the destination node and press the paste paste icon icon. You will then be presented with the following options:

paste options

  • move - moves a node to new location
  • mirror - is useful if you want a user or device to be a member of multiple groups. If you change a node, all of the mirror copies of the node update as well.
  • duplicate - recursively duplicates the copied node plus all its descendants. This is useful for scenarios where you have a device or site configuration (perhaps a complex Modbus setup) that you want to duplicate at a new site.

Graphing and advanced dashboards

If you need graphs and more advanced dashboards, consider coupling Simple IoT with Grafana. Someday we hope to have dashboard capabilities built in.

Custom UIs

See the frontend reference documentation.

Users/Groups

Users and Groups can be configured at any place in the node tree. The way permissions work is users have access to the parent node and the parent nodes children. In the below example, Joe has access to the SBC device because both Joe and SBC are members of the Site 1 group. Joe does have access to the root node.

group user

If Joe logs in, the following view will be presented:

joe nodes

Upstream connections

Simple IoT provides for simple upstream connections via NATS or NATS over Websocket.

upstream

To create an upstream, add an upstream node to the root node on the downstream instance. If your upstream server has a name of myserver.com, then you can use the following connections URIs:

  • nats://myserver.com:4222 (4222 is the default nats port)
  • ws://myserver.com (websocket unencrypted connection)
  • wss://myserver.com (websocket encrypted connection)

IP addresses can also be used for the server name.

Auth token is optional and needs to be configured in an environment variable for the upstream server. If your upstream is on the public internet, you should use an auth token. If both devices are on an internal network, then you may not need an auth token.

Typically, wss are simplest for servers that are fronted by a web server like Caddy that has TLS certs. For internal connections, nats or ws connections are typically used.

Occasionally, you might also have edge devices on networks where nats outgoing connections on port 4222 are blocked. In this case, its handy to be able to use the wss connection, which just uses standard HTTP(S) ports.

upstream

There are also several videos that demostrate upstream connections:

Rules

Contents

The Simple IoT application has the ability to run rules. That are composed of one or more conditions and actions. All conditions must be true for the rule to be active.

Node point changes cause rules of any parent node in the tree to be run. This allows general rules to be written higher in the tree that are common for all device nodes (for instance device offline).

In the below configuration, a change in the SBC propagates up the node tree, thus both the D5 on rule or the Device offline rule are eligible to be run.

rules

Node linking

Both conditions and actions can be linked to a node ID. If you copy a node, its ID is stored in a virtual clipboard and displayed at the top of the screen. You can then paste this node ID into the Node ID field in a condition or action.

rule-linking

Conditions

Each condition may optionally specify a minimum active duration before the condition is considered met. This allows timing to be encoded in the rules.

Node state

A point value condition looks at the point value of a node to determine if a condition is met. Qualifiers that filter points the condition is interested in may be set including:

  • node ID (if left blank, any node that is a descendent of the rule parent)
  • point type ("value" is probably the most common type)
  • point Key (used to index into point arrays and objects)

If the provided qualification is met, then the condition may check the point value/text fields for a number of conditions including:

  • number: >, <, =, !=
  • text: =, !=, contains
  • boolean: on, off

Schedule

TODO:

Actions

Every action has an optional repeat interval. This allows rate limiting of actions like notifications.

Notifications

Notifications are the simplest rule action and are sent out when:

  • all conditions are met
  • time since last notification is greater than the notify action repeat interval.

Every time a notification is sent out by a rule, a point is created/updated in the rule with the following fields:

  • id: node of point that triggered the rule
  • type: "lastNotificationSent"
  • time: time the notification was sent

Before sending a notification we scan the points of the rule looking for when the last notification was sent to decide if its time to send it.

Set node point

Rules can also set points in other nodes. For simplicity, the node ID must be currently specified along with point parameters and a number/bool/text value.

Typically a rule action is only used to set one value. In the case of on/off actions, one rule is used to turn a value on, and another rule is used to turn the same value off. This allows for hysteresis and more complex logic than in one rule handled both the on and off states. This also allows the rules logic to be stateful.

Notifications

Notifications are sent to users when a rule goes from inactive to active and contains a notification action. This notification travels up the node graph. At each parent node, users potentially listen for notifications. If a user is found, then a message is generated. This message likewise travels up the node graph. At each parent node, messaging service nodes potentially listen for messages and then process the message. Each node in Simple IoT that generates information is not concerned with the recipient of the information or how the information is used. This decoupling is the essence of a messaging based system (we use NATS) and is very flexible and powerful. Because nodes can be aliased (mirrored) to different places, this gives us a lot of flexibility in how points are processed. The node tree also gives us a very visual view of how things are connected as well as an easy way to expand or narrow scope based on high in the hierarchy a node is placed.

Example

There is hierarchy of nodes in this example system:

  • Company XYZ
    • Twilio SMS
    • Plant A
      • Joe
      • Motor overload Rule
      • Line #1
        • Motor Overload
    • Plant B

The node hierarchy is used to manage scope and permissions. The general rule is that a node has access to (or applies to) its parent nodes, and all of its parents dependents. So in this example, Joe has access to everything in Plant A, and likewise gets any Plant A notifications. The Motor overload rule also applies to anything in Plant A. This allows us to write one rule that could apply to multiple lines. The Twilio SMS node processes any messages generated in Company XYZ including those generated in Plant A, Line #1, Plant B, etc. and can be considered a company wide resource.

The process for generating a SMS notification to a user is as follows:

  1. Line #1 contains a Motor Overload sensor. When this value changes, a point (blue) gets sent to its parent Line #1 and then to Plant A. Although it is not shown below, the point also gets sent to the Company XYZ and root nodes. Points always are rebroadcast on every parent node back to the root.
  2. Plant A contains a rule (Motor Overload) that is then run on the point, which generates a notification (purple) that gets sent back up to its parent (Plant A).
  3. Plant A contains a user Joe so a notification + user generates a message (green), which gets sent back upstream to Plant A and then to Company XYZ.
  4. Company XYZ contains a messaging service (Twilio SMS), so the message gets processed by this service a SMS message gets sent to Joe.

The Motor Overload sensor node only generates what it senses. The Motor Overload rule listens for points in Plant A (its parent) and processes those points. The Joe user node listens for points at the Plant A node (its parent) and processes any points that are relevant. The Twilio SMS node listens for point changes at the Company XYZ node and processes those points. Information only travels upstream (or up the node hierarchy).

message process

In this example, the admin user does not receive notifications from the Twilio SMS messaging service. The reason is that the Twilio SMS node only listens for messages on its parent node. It does not have visibility into messages sent to the root node. With the node hierarchy, we can easily partition who gets notified. Additional group layers can be added if needed. No explicit binding is required between any of the nodes -- the location in the graph manages all that. The higher up you go, the more visibility and access a node has.

Twilio SMS Messaging

Simple IoT supports sending SMS messages using Twilio's SMS service. Add a Messaging Service node and then configure.

twilio

Email Messaging

will be added soon ...

I/O Devices

Simple IoT is a framework that allows for the management of various Input/Output (IO) devices. Each devices is represented by a node and optionally subnodes.

Modbus

Modbus is popular data communications protocol used for connecting industrial devices. The specification is open and available at the Modbus website.

Simple IoT can function as both a Modbus client or server and supports both RTU and TCP transports. Modbus client/server is used as follows:

  • client: typically a PLC or Gateway -- the device reading sensors and initiating Modbus transactions. This is the mode to use if you want to read sensor data and then process it or send to an upstream instance.
  • server: typically a sensor, actuator, or other device responding to Modbus requests. Functioning as a server allows SIOT to simulate Modbus devices or to provide data to another client device like a PLC.

Modbus is a prompt response protocol. With Modbus RTU (RS485), you can only have one client (gateway) on the bus and multiple servers (sensors). With Modbus TCP, you can have multiple clients and servers.

Modbus is configured by adding a Modbus node to the root node, and then adding IO nodes to the Modbus node.

modbus

Modbus IOs can be configured to support most common IO types and data formats:

modbus io config

1-Wire

1-Wire is a device communication bus that provides low-speed data over a single conductor. It is also possible to power some devices over the data signal as well, but often a third wire is run for power.

Simple IoT supports 1-wire buses controlled by the 1-wire (w1) subsystem in the Linux kernel. Simple IoT will automatically create nodes for 1-wire buses and devices it discovers.

1-wire nodes

Bus Controllers

Raspberry PI GPIO

There are a number of bus controllers available but one of the simplest is a GPIO on a Raspberry PI. To enable, add the following to the /boot/config.txt file:

dtoverlay=w1-gpio

This enables a 1-wire bus on GPIO 4.

To add a bus to a different pin:

dtoverlay=w1-gpio,gpiopin=x

A 4.7kΩ pull-up resistor is needed between the 1-wire signal and 3.3V. This can be wired to a 0.1" connector as shown in the following schematic:

1-wire schematic

See this page for more information.

1-Wire devices

DS18B20 Temperature sensors

Simple IoT currently supports 1-wire temperature sensors such as the DS18B20. This is a very popular and practical digital temperature sensor. Each sensor has a unique address so you can address a number of them using a single 1-wire port. These devices are readily available at low cost from a number of places including eBay -- search for DS18B20, and look for an image like the below:

DS18B20

MCU Devices

Status: Specification

Microcontroller (MCU) devices can be connected to Simple IoT systems via various serial transports (RS232, RS485, CAN, and USB Serial). The Arduino platform is one example of a MCU platform that is easy to use and program. Simple IoT provides a serial interface module that can be used to interface with these systems. The combination of a laptop or a Raspberry PI makes a useful lab device for monitoring analog and digital signals. Data can be logged to InfluxDB and viewed in the InfluxDB Web UI or Grafana. This concept can be scaled into products where you might have a Linux MPU handling data/connectivity and a MCU doing real-time control.

mcu

TODO: add instructions for setting up an Arduino system.

Debug Levels

You can set the following debug levels to log information.

  • 0: no debug information
  • 1: log ASCII strings (must be COBS wrapped)
  • 2: log raw data (must be COBS wrapped)

USB

Graphing

Simple IoT is designed to work with several other applications for storing time series data and viewing this data in graphs.

InfluxDB

InfluxDB is currently the recommended way to store historical data. This database is efficient and can run on embedded platforms like the Raspberry PI as well as desktop and server machines. To connect SIOT to InfluxDB, simply add an InfluxDB node in your setup, and fill in the parameters.

Grafana

Grafana is a very powerful graphing solution that works well with InfluxDB. Although InfluxDB has its own web interface and graphing capability, generally we find Grafana to be more full featured and easier to use.

Configuration

Environment variables are used to control various aspects of the application. The following are currently defined:

  • General
    • SIOT_HTTP_PORT: http network port the SIOT server attaches to (default is 8080)
    • SIOT_DATA: directory where any data is stored
    • SIOT_AUTH_TOKEN: auth token used for NATS and HTTP device API, default is blank (no auth)
    • OS_VERSION_FIELD: the field in /etc/os-release used to extract the OS version information. Default is VERSION, which is common in most distros. The Yoe Distribution populates VERSION_ID with the update version, which is probably more appropriate for embedded systems built with Yoe. See ref/version.
  • NATS configuration
    • SIOT_NATS_PORT: Port to run NATS on (default is 4222 if not set)
    • SIOT_NATS_HTTP_PORT: Port to run NATS monitoring interface (default is 8222)
    • SIOT_NATS_SERVER: defaults to nats://localhost:4222
    • SIOT_NATS_TLS_CERT: points to TLS certificate file. If not set, TLS is not used.
    • SIOT_NATS_TLS_KEY: points to TLS certificate key
    • SIOT_NATS_TLS_TIMEOUT: Configure the TLS upgrade timeout. NATS defaults to a 0.5s timeout for TLS upgrade, but that is too short for some embedded systems that run on low end CPUs connected over cellular modems (we've see this process take as long as 4s). See NATS documentation for more information.
    • SIOT_NATS_WS_PORT: Port to run NATS websocket (default is 9222, set to 0 to disable)
  • Particle.io
    • SIOT_PARTICLE_API_KEY: key used to fetch data from Particle.io devices running Simple IoT firmware

Status

The Simple IoT project is still in a heavy development phase. Most of the core concepts are stable, but APIs, packet formats, and implementation will continue to change for some time yet. SIOT has been used in several production systems to date with good success, but be prepared to work with us (report issues, help fix bugs, etc.) if you want to use it now.

Database

We are currently using a key/value store based on bbolt as the data store. This will likely be changed out for SQLite in the near future. That ability to run SQLite in a pure Go application only recently because available. This should not affect developers much as all data access goes through NATS, but data for existing instances will need to be manually migrated from the old to the new disk formats. This is done by exporting the data using the old version, and then importing the data using the new version. SQLite has a very stable disk format so we don't anticipate any migration issues in the future after we switch to SQLite.

Handling of high rate sensor data

Currently each point change requires quite a bit computation to update the HASH values in upstream graph nodes. For repetitive data, this is not necessary as new values are continually coming in, so we will at some point make an option to specify points values as repetitive. This will allow SIOT to scale to more devices and higher rate data.

User Interface

The web UI is currently polling the SIOT backend every 4 seconds via HTTP. This works OK for small data sets, but uses more data than necessary and has a latency of up to 4s. Long term we will run a NATS client in the frontend over a websocket so the UI response is real-time and new data gets pushed to the browser.

Security

Currently, and device that has access to the system can write or write to any data in the system. This may be adequate for small or closed systems, but for larger systems, we need per-device authn/authz. See issue #268, PR #283, and our security document for more information.

Errata

Any issues we find during testing we log in Github issues, so if you encounter something unexpected, please search issues first. Feel free to add your observations and let us know if an issues is impacting you. Several issues to be aware of:

  • we don't handle loops in the graph tree yet. This will render the instance unusable and you'll have to clean the database and start over.
  • for now, create a different email for the admin user on each instance. See #366.
  • there are still several corner cases with upstream connections that need improved. (#367, #366, #339)

Frequently Asked Questions

Q: How is SIOT different than Home Assistant, OpenHAB, Domoticz, etc.?

Although there may be some overlap and Simple IoT may eventually support a number of off the shelf consumer IoT devices, the genesis and intent of the project is for developing IoT products and the infrastructure required to support them.

Q: How is SIOT different than Particle.io, etc.?

Particle.io provides excellent infrastructure to support their devices and solve many of the hard problems such as remote FW update, getting data securely from device to cloud, efficient data bandwidth usage, etc. But they don't provide a way to provide a user facing portal for a product that customers can use to see data and interact with the device.

Q: How is SIOT different than AWS/Azure/GCP/... IoT?

SIOT is designed to be simple to develop and deploy without a lot of moving parts. We've reduced an IoT system to a few basic concepts that are exactly the same in the cloud and on edge devices. This symmetry is powerful and allows us to easily implement and move functionality wherever it is needed. If you need Google Scale, SIOT may not be the right choice; however, for smaller systems where you want a system that is easier to develop, deploy, and maintain, consider SIOT.

Q: Can't NATS Jetstream do everything SIOT does?

This is a good question and I'm not sure yet. NATS has some very interesting features like Jetstream which can queue data and and store data in a key-value store and data can be syncronized between instances. NATS also has a concept of leaf-nodes, which conceptually makes sense for edge/gateway connections. Jetstream is optimized for data flowing in one direction (ex: orders through fullfillment). SIOT is optimized for data flowing in any direction and data is merged using data structures with CRDT (conflic-free replicated data types) properties. SIOT also stores data in a DAG (directed acyclic graph) which allows a node to be a child of multiple nodes, which is difficult to do in a hiearchical namespace. Additionally, each node is defined by an array of points and modifications to the system are communicated by transferring points. SIOT is a batteries included complete solution for IoT solutions, including a web framework, clients for various types of IO (ex: Modbus) and cloud services (ex: Twilio). We will continue to explore using more of NATS core functionality as we move forward.

Documentation

Good documentation is critical for any project and to get good documentation, the process to create it must be as frictionless as possible. With this in mind, we've structured SIOT documentation as follows:

  • Markdown is the primary source format.
  • documentation lives in the same repo as the source code. When you update the code, update the documentation at the same time.
  • documentation is easily viewable in Github, or our generated docs site. This allows any snapshot of SIOT to contain a viewable snapshot of the documentation for that revision.
  • mdbook is used to generate the documentation site.
  • all diagrams are stored in a single draw.io file. This allows you to easily see what diagrams are available and easily copy pieces from existing diagrams to make new ones. Then generate a PNG for the diagram in the images/ directory in the relevant documentation directory.

Vision

This document attempts to outlines the project philosophy and core values. The basics are covered in the readme. As the name suggests, a core value of the project is simplicity. Thus any changes should be made with this in mind. Although this project has already proven useful on several real-world project, it is a work in progress and will continue to improve. As we continue to explore and refine the project, many things are getting simpler and more flexible. This process takes time and effort.

“When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions.” -- Steve Jobs

Guiding principles

  1. Simple concepts are flexible and scale well.
  2. IoT systems are inheriently distributed, and distrbuted systems are hard.
  3. There are more problems to solve than people to solve them, thus it makes sense to collaborate on the common technology pieces.
  4. There are a lot of IoT applications that are not Google scale (10-1000 device range).
  5. There is significant opportunity in the long tail of IoT, which is our focus.
  6. There is value in custom solutions (programming vs drag-n-drop).
  7. There is value in running/owning our own platform.
  8. A single engineer should be able to build and deploy a custom IoT system.
  9. We don't need to spend excessive amounts of time on operations. For smaller deployments, we deploy one binary to a cloud server and we are done with operations. We don't need 20 microservices when one monolith will work just fine.
  10. For many applications, a couple hours of down time is not the end of the world. Thus a single server that can be quickly rebuilt as needed is adequate and in many cases more reliable than complex systems with many moving parts.

Technology choices

Choices for the technology stack emphasize simplicity, not only in the language, but just as important, in the deployment and tooling.

  • Backend
    • Go
      • simple language and deployment model
      • nice balance of safety + productivity
      • excellent tooling and build system
      • see this thread for more discussion/information
  • Frontend
    • Single Page Application (SPA) architecture
      • fits well with real-time applications where data is changing all the time
      • easier to transition to Progressive Web Apps (PWA)
    • Elm
      • nice balance of safety + productivity
      • excellent compiler messages
      • reduces possibility for run time exceptions in browser
      • does not require a huge/complicated/fragile build system typical in Javascript frontends.
      • excellent choice for SPAs
    • elm-ui
      • What if you never had to write CSS again?
      • a fun, yet powerful way to lay out a user interface and allows you to efficiently make changes and get the layout you want.
  • Database
    • Eventually support multiple databased backends depending on scaling/admin needs
    • Embedded db using Genji
      • no external services to configure/admin
  • Cloud Hosting
    • Any machine that provides ability run long-lived Go applications
    • Any MAC/Linux/Windows/rPI/Beaglebone/Odroid/etc computer on your local network.
    • Cloud VMs: Digital Ocean, Linode, GCP compute engine, AWS ec2, etc. Can easily host on a $5/mo instance.
  • Edge Devices
    • any device that runs Linux (rPI, Beaglebone-black, industrial SBCs, your custom hardware ...)

In our experience, simplicity and good tooling matter. It is easy to add features to a language, but creating a useful language/tooling that is simple is hard. Since we are using Elm on the frontend, it might seem appropriate to select a functional language like Elixir, Scala, Clojure, Haskell, etc. for the backend. These environments are likely excellent for many projects, but are also considerably more complex to work in. The programming style (procedural, functional, etc.) is important, but other factors such as simplicity/tooling/deployment are also important, especially for small teams who don't have separate staff for backend/frontend/operations. Learning two simple languages (Go and Elm) is a small task compared to dealing with huge languages, fussy build tools, and complex deployment environments.

This is just a snapshot in time -- there will likely be other better technology choices in the future. The backend and frontend are independent. If either needs to be swapped out for a better technology in the future, that is possible.

Architecture

This document describes how the Simple IoT project fulfills the basic requirements as described in the top level README.

There are two levels of architecture to consider:

  • System: how multiple SIOT instances and other applications interact to form a system.
  • Application: how the SIOT application is structured.
  • Clients: all about SIOT clients where most functionality is implemented.

High Level Overview

Simple IoT functions as a collection of connected, distributed instances that communicate via NATS. Data in the system is represented by nodes which contain an array of points. Data changes are communicated by sending points within an instance or between instances. Points in a node are merged such that newer points replace older points. This allows granular modification of a node's properties. Nodes are organized in a DAG (directed acyclic graph). This graph structure defines many properties of the system such as what data users have access to, the scope of rules and notifications, and which nodes external services apply to. Most functionality in the system is implemented in clients, which subscribe and publish point changes for nodes they are interested in.

application architecture

System Architecture

Contents

IoT Systems are distributed systems

IoT systems are inherently distributed where data needs to be synchronized between a number of different systems including:

  1. Cloud (one to several instances depending on the level of reliability desired)
  2. Edge devices (many instances)
  3. User Interface (phone, browser)

IoT Distributed System

Typically, the cloud instance stores all the system data, and the edge, browser, and mobile devices access a subset of the system data.

Extensible architecture

Any siot app can function as a standalone, client, server or both. As an example, siot can function both as an edge (client) and cloud apps (server).

  • full client: full siot node that initiates and maintains connection with another siot instance on a server. Can be behind a firewall, NAT, etc.
  • server: needs to be on a network that is accessible by clients

We also need the concept of a lean client where an effort is made to minimize the application size to facilitate updates over IoT cellular networks where data is expensive.

Device communication and messaging

In an IoT system, data from sensors is continually streaming, so we need some type of messaging system to transfer the data between various instances in the system. This project uses NATS.io for messaging. Some reasons:

  • allows us to push realtime data to an edge device behind a NAT, on cellular network, etc -- no public IP address, VPN, etc required.
  • is more efficient than HTTP as it shares one persistent TCP connection for all messages. The overhead and architecture is similar to MQTT, which is proven to be a good IoT solution. It may also use less resources than something like observing resources in CoAP systems, where each observation requires a separate persistent connection.
  • can scale out with multiple servers to provide redundancy or more capacity.
  • is written in Go, so possible to embed the server to make deployments simpler for small systems. Also, Go services are easy to manage as there are no dependencies.
  • focus on simplicity -- values fit this project.
  • good security model.

For systems that only need to send one value several times a day, CoAP is probably a better solution than NATS. Initially we are focusing on systems that send more data -- perhaps 5-30MB/month. There is no reason we can't support CoAP as well in the future.

Data modification

Where possible, modifying data (especially nodes) should be initiated over nats vs direct db calls. This ensures anything in the system can have visibility into data changes. Eventually we may want to hide db operations that do writes to force them to be initiated through a NATS message.

data flow

Simple, Flexible data structures

As we work on IoT systems, data structures (types) tend to emerge. Common data structures allow us to develop common algorithms and mechanism to process data. Instead of defining a new data type for each type of sensor, define one type that will work with all sensors. Then the storage (both static and time-series), synchronization, charting, and rule logic can stay the same and adding functionality to the system typically only involves changing the edge application and the frontend UI. Everything between these two end points can stay the same. This is a very powerful and flexible model as it is trivial to support new sensors and applications.

Constant vs Varying parts of System

See Data for more information.

Node Tree

The same Simple IoT application can run in both the cloud and device instances. The node tree in a device would then become a subset of the nodes in the cloud instance. Changes can be made to nodes in either the cloud or device and data is sycnronized in both directions.

cloud device node tree

The following diagram illustrates how nodes might be arranged in a typical system.

node diagram

A few notes this structure of data:

  • A user has access to its child nodes, parent nodes, and parent node descendants (parents, children, siblings, nieces/nephews).
  • Likewise, a rule node processes points from nodes using the same relationships described above.
  • A user can be added to any node. This allows permissions to be granted at any level in the system.
  • A user can be added to multiple nodes.
  • A node admin user can configure nodes under it. This allows a service provider to configure the system for their own customers.
  • If a point changes, it triggers rules of upstream nodes to run (perhaps paced to some reasonable interval)
  • The Edge Dev Offline rule will fire if any of the Edge devices go offline. This allows us to only write this rule once to cover many devices.
  • When a rule triggers a notification, the rule node and any upstream nodes can optionally notify its users.

The distributed parts of the system include the following instances:

  • Cloud (could be multiple for redundancy). The cloud instances would typically store and synchronize the root node and everything under it.
  • Edge Devices (typically many instances (1000's) connected via low bandwidth cellular data). Edge instances would would store and synchronize the edge node instance and descendants (ex Edge Device 1)
  • Web UI (potentially dozens of instances connected via higher bandwidth browser connection).

As this is a distributed system where nodes may be created on any number of connected systems, node IDs need to be unique. A unique serial number or UUID is recommended.

Application Architecture

Contents

The Simple IoT Go application is a single binary with embedded assets. The database and NATS server are also embedded by default for easy deployment. There are five main parts to a Simple IoT application:

  1. NATS Message Bus: all data goes through this making it very easy to observe the system.
  2. Store: persists the data for the system, merges incoming data, maintains node hash values for synchronization, rules engine, etc. (the rules engine may eventually move to a client)
  3. Clients: interact with other devices/systems such as Modbus, 1-wire, etc. This is where most of the functionality in a SIOT system lives, and where you add your custom functionality. Clients can exist inside the Simple IoT application or as external processes written in any language that connect via NATS. Clients are represented by a node (and optionally child nodes) in the SIOT store. When a node is updated, its respective clients are updated with the new information. Likewise, when a client has new information, it sends that out to be stored and used by other nodes/instances as needed.
  4. HTTP API: provides a way for HTTP clients to interact with the system.
  5. Web UI: Provides a user interface for users to interact with the system. Currently it uses the HTTP API, but will eventually connect directly to NATS.

The simplicity of this architecture makes it easy to extend with new functionality by writing a new client. Following the constraints of storing data as nodes and points ensures all data is visible and readable by other clients, as well as being automatically synchronized to upstream instances.

application architecture

Application Lifecycle

Simple IoT uses the Start()/Stop() pattern for any long running processes. With any long running process, it is important to not only Start it, but also to be able to cleanly Stop it. This is important for testing, but is also good practice. Nothing runs forever so we should never operate under this illusion. The oklog/run packaged is used to start and shutdown these processes concurrently. Dependencies between processes should be minimized where possible through retries. If there are hard dependencies, these can be managed with WaitStart()/WaitStop() functions. See server.go for an example.

NATS lends itself very well to a decoupled application architecture because the NATS clients will buffer messages for some time until the server is available. Thus we can start all the processes that use a NATS client without waiting for the server to be available first.

Long term, a NATS API that indicates the status of various parts (rules engine, etc.) of the system would be beneficial. If there are dependencies between processes, this can be managed inside the process instead of in the code that starts/stops the processes.

NATS Integration

The NATS API details the NATS subjects used by the system.

Echo concerns

Any time you potentially have two sources modifying the same resource (a node), you need to be concerned with echo'd messages. This is a common occurance in Simple IoT. Because another resource may modify a node, typically a client needs to subscribe to the node messages as well. This means when it sends a message, it will typically be echo'd back. See the client documentation for ideas on how to handle the echo problem.

The server.NewServer function returns a nats connection. This connection is used throughout the application and does not have the NoEcho option set.

User Interface

Currently, the User Interface is implemented using a Single Page Architecture (SPA) Web Application. This keeps the backend and frontend implementations mostly independent. See User Interface and Frontend for more information.

There are many web architectures to chose from and web technology is advancing at a rapid pace. SPAs are not in vogue right now and more complex architectures are promoted such as Next.js, SveltKit, Deno Fresh, etc. Concerns with SPAs include large initial load and stability (if frontend code crashes, everything quits working). These concerns are valid if using Javascript, but with Elm these concerns are minimal as Elm compiles to very small bundles, and run time exceptions are extremely rare. This allows us to use a simple web architecture with minimal coupling to the backend and minimal build complexity. And it will be a long time until we write enough Elm code that bundle size matters.

A decoupled SPA UI architecture is also very natural in Simple IoT as IoT systems are inherently distributed. The frontend is just another client, much the same as a separate machine learning process, a downstream instance, a scripting process, etc.

Simple IoT Clients

Contents

Most functionality in Simple IoT is implemented in Clients.

app-arch

Each client can be configured by one or more nodes in the SIOT store graph. These nodes may be created by a user, a process that detects new plug and play hardware, or other clients.

A client interacts with the system by listening for new points it is interested in and sending out points as it acquires new data.

Creating new clients

Simple IoT provides utilities that assist in creating new clients. See the Go package documentation for more information. A client manager is created for each client type. This manager instantiates new client instances when new nodes are detected and then sends point updates to the client. Two levels of nodes are currently supported for client configuration. An example of this would be a Rule node that has Condtion and Action child nodes.

Client lifecycle

It is important the clients cleanly implement the Start()/Stop() pattern and shut down cleanly when Stop() is called releasing all resources. If nodes are added or removed, clients are started/stopped. Additionally if a child node of a client config is added or removed, the entire client is stopped and then restarted. This relieves the burden on the client from managing the addition/removal of client functionality. Thus it is very important that clients stop cleanly and release resources in case they are restarted.

Message echo

Clients need to be aware of the "echo" problem as they typically subscribe as well as publish to the points subject for the nodes they manage. When they publish to these subjects, these messages will be echoed back to them. There are several solutions:

  1. create a new NATS connection for the client with the NoEcho option set. For this to work, each client will need to establish its own connection to the server. This may not work in cases where subjects are aliased into authenticated subject namespaces.
  2. inspect the Point Origin field -- if is blank, then it was generated by the node that owns the point and does not need to be processed by the client generating the data for that node. If is not blank, then the Point was generated by a user, rule, or something other than the owning node and must be processed. This may not always work -- example: user is connected to a downstream instance and modifies a point that then propagates upstream -- it may get echo'd back to an authenticated client.
  3. (investigation stage) A NATS messages header can be populated with the ID of the client that sent the message. If it is an authenticated client, then the message will not be echo'd on the authenticated client subject namespace of the same ID. This information is not stored, so cannot be used for auditing purposes.

The SIOT client manager filters out all points received with Origin set to a blank string. Thus, if you want a point to get to a client, you must set the Origin field. This helps ensure that the Origin field is used consistently as otherwise stuff won't work.

See also tracking who made changes.

Development

Go Package Documentation

The Simple IoT source code is available on Github.

Simple IoT is written in Go. Go package documentation is available.

Code Organization

Currently, there are a lot of subdirectories. One reason for this is to limit the size of application binaries when building edge/embedded Linux binaries. In some use cases, we want to deploy app updates over cellular networks, therefore we want to keep packages as small as possible. For instance, if we put the natsserver stuff in the nats package, then app binaries grow a couple MB, even if you don't start a NATS server. It is not clear yet what Go does for dead code elimination, but at this point, it seems referencing a package increases the binary size, even if you don't use anything in it. (Clarification welcome!)

For edge applications on Embedded Linux, we'd eventually like to get rid of net/http, since we can do all network communications over NATS. We're not there yet, but be careful about pulling in dependencies that require net/http into the NATS package, and other low level packages intended for use on devices.

Directories

See Go docs directory descriptions

Coding Standards

Please run siot_test from envsetup.sh before submitting pull requests. All code should be formatted and linted before committing.

Please configure your editor to run code formatters:

  • Go: goimports
  • Elm: elm-format
  • Markdown: prettier (note, there is a .prettierrc in this project that configures prettier to wrap markdown to 80 characters. Whether to wrap markdown or not is debatable, as wrapping can make diffs harder to read, but Markdown is much more pleasant to read in an editor if it is wrapped. Since more people will be reading documentation than reviewing, lets optimize for the reading in all scenarios -- editor, Github, and generated docs)

Pure Go

We plan to keep the main Simple IoT application a pure Go binary if possible. Statically linked pure Go has huge advantages:

  1. you can easily cross compile to any target from any build machine.
  2. blazing fast compile times
  3. deployment is dead simple – zero dependencies. Docker is not needed.
  4. you are not vulnerable to security issues in the host systems SSL/TLS libs. What you deploy is pretty much what you get.
  5. although there is high quality code written in C/C++, it is much easier to write safe, reliable programs in Go, so I think long term there is much less risk using a Go implementation of about anything – especially if it is widely used.
  6. Go’s network programming model is much simpler than about anything else. Simplicity == less bugs.

Once you link to C libs in your Go program, you forgo many of the benefits of Go. The Go authors made a brilliant choice when they chose to build Go from the ground up. Yes, you loose the ability to easily use some of the popular C libraries, but what you gain is many times more valuable.

Building Simple IoT

Requirements:

  • Go
  • Node/NPM

Simple IoT build has currently been testing on Linux and MacOS systems. See envsetup.sh for scripts used in building.

To build:

  • source envsetup.sh
  • siot_setup
  • siot_build

Running unit tests

There are not a lot of unit tests in the project yet, but below are some examples of running tests:

  • test everything: go test -race ./...
  • test only client directory: go test -race ./client
  • run only a specific: `go test -race ./client -run BackoffTest (run takes a RegEx)
  • siot_test runs tests as well as vet/lint, frontend tests, etc.

The leading ./ is important, otherwise Go things you are giving it a package name, not a directory. The ... tells Go to recursively test all subdirs.

Document and test during development

It is much more pleasant to write documentation and tests as you develop, rather than after the fact. These efforts add value to your development if done concurrently. Quality needs to be designed-in, and leading with documentation will result in better thinking and a better product.

If you develop a feature, please update/create any needed documentation and write any tests (especially end-to-end) to verify the feature works and continues to work.

Data

Contents

Data Structures

As a client developer, there are two main primary structures: NodeEdge and Point. A Node can be considered a collection of Points.

These data structures describe most data that is stored and transferred in a Simple IoT system.

The core data structures are currently defined in the data directory for Go code, and frontend/src/Api directory for Elm code.

A Point can represent a sensor value, or a configuration parameter for the node. With sensor values and configuration represented as Points, it becomes easy to use both sensor data and configuration in rule or equations because the mechanism to use both is the same. Additionally, if all Point changes are recorded in a time series database (for instance Influxdb), you automatically have a record of all configuration and sensor changes for a node.

Treating most data as Points also has another benefit in that we can easily simulate a device -- simply provide a UI or write a program to modify any point and we can shift from working on real data to simulating scenarios we want to test.

Edges are used to describe the relationships between nodes as a directed acyclic graph.

dag

Nodes can have parents or children and thus be represented in a hierarchy. To add structure to the system, you simply add nested Nodes. The Node hierarchy can represent the physical structure of the system, or it could also contain virtual Nodes. These virtual nodes could contain logic to process data from sensors. Several examples of virtual nodes:

  • a pump Node that converts motor current readings into pump events.
  • implement moving averages, scaling, etc on sensor data.
  • combine data from multiple sensors
  • implement custom logic for a particular application
  • a component in an edge device such as a cellular modem

Like Nodes, Edges also contain a Point array that further describes the relationship between Nodes. Some examples:

  • role the user plays in the node (viewer, admin, etc)
  • order of notifications when sequencing notifications through a node's users
  • node is enabled/disabled -- for instance we may want to disable a Modbus IO node that is not currently functioning.

Being able to arranged nodes in an arbitrary hierarchy also opens up some interesting possibilities such as creating virtual nodes that have a number of children that are collecting data. The parent virtual nodes could have rules or logic that operate off data from child nodes. In this case, the virtual parent nodes might be a town or city, service provider, etc., and the child nodes are physical edge nodes collecting data, users, etc.

Synchronization

See research for information on techniques that may be applicable to this problem.

Typically, configuration is modified through a user interface either in the cloud, or with a local UI (ex touchscreen LCD) at an edge device. Rules may also eventually change values that need to be synchronized. As mentioned above, the configuration of a Node will be stored as Points. Typically the UI for a node will present fields for the needed configuration based on the Node Type, whether it be a user, rule, group, edge device, etc.

In the system, the Node configuration will be relatively static, but the points in a node may be changing often as sensor values changes, thus we need to optimize for efficient synchronization of points. We can't afford the bandwidth to send the entire node data structure any time something changes.

As IoT systems are fundamentally distributed systems, the question of synchronization needs to be considered. Both client (edge), server (cloud), and UI (frontend) can be considered independent systems and can make changes to the same node.

  • An edge device with a LCD/Keypad may make configuration changes.
  • Configuration changes may be made in the Web UI.
  • Sensor values will be sent by an edge device.
  • Rules running in the cloud may update nodes with calculated values.

Although multiple systems may be updating a node at the same time, it is very rare that multiple systems will update the same node point at the same time. The reason for this is that a point typically only has one source. A sensor point will only be updated by an edge device that has the sensor. A configuration parameter will only be updated by a user, and there are relatively few admin users, and so on. Because of this, we can assume there will rarely be collisions in individual point changes, and thus this issue can be ignored. The point with the latest timestamp is the version to use.

Real-time Point synchronization

Point changes are handled by sending points to a NATS topic for a node any time it changes. There are three primary instance types:

  1. Cloud: will subscribe to point changes on all nodes (wildcard)
  2. Edge: will subscribe to point changes only for the nodes that exist on the instance -- typically a handful of nodes.
  3. WebUI: will subscribe to point changes for nodes currently being viewed -- again, typically a small number.

With Point Synchronization, each instance is responsible for updating the node data in its local store.

Catch-up/non real-time synchronization

Sending points over NATS will handle 99% of data synchronization needs, but there are a few cases this does not cover:

  1. One system is offline for some period of time
  2. Data is lost during transmission
  3. Other errors or unforeseen situations

There are two types of data:

  1. periodic sensor readings (we'll call sample data) that is being continuously updated
  2. configuration data that is infrequently updated

Any node that produces sample data should send values every 10m, even if the value is not changing. There are several reasons for this:

  • indicates the data source is still alive
  • makes graphing easier if there is always data to plot
  • covers the synchronization problem for sample data. A new value will be coming soon, so don't really need catch-up synchronization for sample data.

Config data is not sent periodically. To manage synchronization of config data, each edge will have a Hash field.

The edge Hash field is a hash of:

  • edge and node point timestamps except for sample points. Sample points are excluded from the hash as discussed above.
  • child edge Hash fields

We store the hash in the edge structures because nodes (such as users) can exist in multiple places in the tree.

The points are sorted by timestamp and child nodes are sorted by hash so that the order is consistent when the hash is computed.

This is essentially a Merkle Tree -- see research.

Comparing the node Hash field allows us to detect node differences. We then compare the node points and child nodes to determine the actual differences.

Any time a node point (except for sample date) is modified, the node's Hash field is updated, and the Hash field in parents, grand-parents, etc are also computed and updated. This may seem like a lot of overhead, but if the database is local, and the graph is reasonably constructed, then each update might require reading a dozen or so nodes and perhaps writing 3-5 nodes. Additionally, non sample-data changes are relatively infrequent.

Initially synchronization between edge and cloud nodes is supported. The edge device will contain an "upstream" node that defines a connection to another instance's NATS server -- typically in the cloud. The edge node is responsible for synchronizing of all state using the following algorithm:

  1. occasionally the edge device fetches the edge device root node hash from the cloud.
  2. if the hash does not match, the edge device fetches the entire node and compares/updates points. If local points need updated, this process can happen all on the edge device. If upstream points need updated, these are simply transmitted over NATS.
  3. if node hash still does not match, a recursive operation is started to fetch child node hashes and the same process is repeated.

The md5 algorithm is used to compute the hash fields because it is relatively efficient to compute and reasonably small. While sha246 might be more secure, the application of this hash is not security, but rather verifiability. The failure mode is that two different trees will generate the same hash. Because the root hash is always changing, this is not really a problem as the next change to the tree will likely trigger a new hash -- usually within a short amount of time.

Node Topology changes

Nodes can exist in multiple locations in the tree. This allows us to do things like include a user in multiple groups.

Add

Node additions are detected in real-time by sending the points for the new node as well as points for the edge node that adds the node to the tree.

Copy

Node copies are are similar to add, but only the edge points are sent.

Delete

Node deletions are recorded by setting a tombstone point in the edge above the node to true. If a node is deleted, this information needs to be recorded, otherwise the synchronization process will simply re-create the deleted node if it exists on another instance.

Move

Move is just a combination of Copy and Delete.

If the any real-time data is lost in any of the above operations, the catch up synchronization will propagate any node changes.

Tracking who made changes

The Point type has an Origin field that is used to track who generated this point. If the node that owned the point generated the point, then Origin can be left blank -- this saves data bandwidth -- especially for sensor data which is generated by the client managing the node. There are several reasons for the Origin field:

  • track who made changes for auditing and debugging purposes. If a rule or some process other than the owning node modifies a point, the Origin should always be populated. Tests that generate points should generally set the origin to "test".
  • eliminate echos where a client may be subscribed to a subject as well as publish to the same subject. With the Origin field, the client can determine if it was the author of a point it receives, and if so simply drop it. See client documentation for more discussion of the echo topic.

Converting Nodes to other data structures

Nodes and Points are convenient for storage and synchronization, but cumbersome to work with in application code that uses the data, so we typically convert them to another data structure. data.Decode and data.Encode can be used to convert Node data structures to your own custom struct, much like the Go json package.

Simple IoT Store

Locking

Currently, the store is made of up:

  • data stored to disk in a database (Genji)
  • a node cache
  • an edge cache

The caches are used to speed up read to nodes as loading them from the database is an expensive operation. We need to manage locking for concurrent access to the cache. Managing locking can be tricky as you need to figure out at what level to do the locking:

  1. lock entire functions at a high level
  2. only lock bits of code that access the cache data structures

It seems locking bits of code that access the data structures is more maintainable as then you can nest functions without worrying about deadlocks where the entire function locks. However, we likely have issues with in that the store does not do transactions when updating a point -- need to investigate this more.

Reliability

Reliability is an important consideration in any IoT system as these systems are often used to monitor and control critical systems and processes. Performance is a key aspect of reliability because if the system is not performing well, then it can't keep up and do its job.

Point Metrics

The fundamental operation of SimpleIoT is that it process points, which are changes to nodes. If the system can't process points at the rate they are coming in, then we have a problem as data will start to back up and the system will not be responsive.

Points and other data flow through the NATS messaging system, therefore it is perhaps the first place to look. We track several metrics that are written to the root device node to help track how the system is performing.

The NATS client buffers messages that are received for each subscription and then messages are dispatched serially one message at a time. If the application can't keep up with processing messages, then the number of buffered messages increases. This number is occasionally read and then min/max/avg writen to the metricNatsPending* points in the root device node.

The time required to process points is tracked in the metricNatsCycle* points in the root device node. The cycle time is in milliseconds.

We also track point throughput (messages/sec) for various NATS subjects in the metricNatsThroughput* points.

These metrics should be graphed and notifications sent when they are out of the normal range. Rules that trigger on the point type can be installed high in the tree above a group of devices so you don't have to write rules for every device.

Database interactions

Database operations greatly affect system performance. When Points come into the system, we need to store this data in the primary (ex Genji) and time series stores (ex InfluxDB). The time it takes to read and write data greatly impacts how much data we can handle.

IO failures

All errors reading/writing IO devices should be tracked at both the device and bus level. These can be observed over time and abnormal rates can trigger notifications. Error counts should be reported at a low rate to avoid using bandwidth and resources -- especially if multiple counts are incremented on an error (IO and bus).

Logging

Many errors are currently reported as log messages. Eventually some effort should be made to turn these into error counts and possibly store them in the time series store for later analysis.

API

Contents

The Simple IoT server currently provides both Http and NATS.io APIs. We've tried to keep the two APIs a similar as possible so it is easy to switch from one to the other. The Http API currently accepts JSON, and the NATS API uses protobuf.

NOTE, the Simple IoT API is not final and will continue to be refined in the coming months.

NATS

NATS.io allows more complex and efficient interactions between various system components (device, cloud, and web UI). These three parts of the system make IoT systems inherently distributed. NATS focuses on simplicity and is written in Go which ensures the Go client is a 1st class citizen and also allows for interesting possibilities such as embedding in the NATS server in various parts of the system. This allows us to keep our one-binary deployment model.

The siot binary embeds the NATS server, so there is no need to deploy and run a separate NATS server.

For the NATS transport, protobuf encoding is used for all transfers and are defined here.

  • Nodes
    • node.<id>
      • returns an array of data.EdgeNode structs that meets the specified id and parent.
      • If id = "root", then the root node is fetched.
      • the parent can can optionally by specified by setting the message payload to one of the following:
        • the ID of the parent node
        • "all" to find all instances of the node. If "all" is specified, tombstoned nodes are not returned.
    • node.<id>.children
      • can be used to request the immediate children of a node
      • parameters can be specified as points in payload
        • tombstone with value field set to 1 will include deleted points
        • nodeType with text field set to node type will limit returned nodes to this type
    • node.<id>.points
      • used to listen for or publish node point changes.
    • node.<id>.<parent>.points
      • used to publish/subscribe node edge points. The tombstone point type is used to track if a node has been deleted or not.
    • up.<upstreamId>.<nodeId>.points
      • node points are rebroadcast at every upstream ID so that we can listen for point changes at any level. The sending node is also included in this.
    • up.<upstreamId>.<nodeId>.<parentId>.points
      • edge points rebroadcast at every upstream node ID.
  • Legacy APIs that are being deprecated
    • node.<id>.not
      • used when a node sends a notification (typically a rule, or a message sent directly from a node)
    • node.<id>.msg
      • used when a node sends a message (SMS, email, phone call, etc). This is typically initiated by a notification.
    • node.<id>.file (not currently implemented)
      • is used to transfer files to a node in chunks, which is optimized for unreliable networks like cellular and is handy for transfering software update files.
  • Auth
    • auth.user
      • used to authenticate a user. Send a request with email/password points, and the system will respond with the User nodes if valid. There may be multiple user nodes if the user is instantiated in multiple places in the node graph. A JWT node will also be returned with a token point. This JWT should be used to authenticate future requests. The frontend can then fetch the parent node for each user node.
  • System
    • error
      • any errors that occur are sent to this subject

HTTP

For details on data payloads, it is simplest to just refer to the Go types which have JSON tags.

Most APIs that do not return specific data (update/delete) return a StandardResponse

  • Nodes
    • data structure
    • /v1/nodes
      • GET: return a list of all nodes
      • POST: insert a new node
    • /v1/nodes/:id
      • GET: return info about a specific node. Body can optionally include the id of parent node to include edge point information.
      • DELETE: delete a node
    • /v1/nodes/:id/parents
      • POST: move node to new parent
      • PUT: mirror/duplicate node
      • body is JSON api/nodes.go:NodeMove or NodeCopy structs
    • /v1/nodes/:id/points
      • POST: post points for a node
    • /v1/nodes/:id/cmd
      • GET: gets a command for a node and clears it from the queue. Also clears the CmdPending flag in the Device state.
      • POST: posts a cmd for the node and sets the node CmdPending flag.
    • /v1/nodes/:id/not
      • POST: send a notification to all node users and upstream users
  • Auth
    • /v1/auth
      • POST: accepts email and password as form values, and returns a JWT Auth token

HTTP Examples

You can post a point using the HTTP API without authorization using curl:

curl -i -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '[{"type":"value", "value":100}]' http://localhost:8080/v1/nodes/be183c80-6bac-41bc-845b-45fa0b1c7766/points

If you want HTTP authorization, set the SIOT_AUTH_TOKEN environment variable before starting Simple IoT and then pass the token in the authorization header:

curl -i -H "Authorization: f3084462-3fd3-4587-a82b-f73b859c03f9" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '[{"type":"value", "value":100}]' http://localhost:8080/v1/nodes/be183c80-6bac-41bc-845b-45fa0b1c7766/points

Frontend

Elm Reference Implementation

The reference Simple IoT frontend is implemented in Elm as a Single Page Application (SPA) and is located in the frontend/ directory.

Code Structure

The frontend is based on elm-spa, and is split into the following directories:

  • Api: contains core data structures and API code to communicate with backend (currently REST).
  • Pages: the various pages of the application
  • Components: each node type has a separate module that is used to render it. NodeOptions.elm contains a struct that is used to pass options into the component views.
  • UI: Various UI pieces we used
  • Utils: Code that does not fit anywhere else (time, etc)

We'd like to keep the UI optimistic if possible.

Creating Custom Icons

SIOT icons are 24x24px pixels (based on feather icon format). One way to create them is to:

  • create a 24x24px drawing in InkScape, scale=1.0
  • draw your icon
  • if you use text
    • convert text to path: select text, and then menu Path -> Object to Path
    • make sure fill is set for path
  • save as plain SVG
  • set up a new Icon in frontend/src/UI/Icon.elm and use an existing custom icon like variable as a template.
  • copy the SVG path strings from the SVG file into the new Icon
  • you'll likely need to adjust the scaling transform numbers to get the icon to the right size

(I've tried using: https://levelteams.com/svg-to-elm, but this has not been real useful, so I usually end up just copying the path strings into an elm template and hand edit the rest)

SIOT JavaScript library using NATS over WebSockets

This is a JavaScript library avaiable in the frontend/lib directory that can be used to interface a frontend with the SIOT backend.

Usage:

import { connect } from "./lib/nats";

async function connectAndGetNodes() {
  const conn = await connect();
  const [root] = await conn.getNode("root");
  const children = await conn.getNodeChildren(root.id, { recursive: "flat" });
  return [root].concat(children);
}

This library is also published on NPM (in the near future).

(see #357)

(Note, we are not currently using this yet in the SIOT frontend -- we still poll the backend over REST and fetch the entire node tree, but we are building out infrastructure so we don't have to do this.)

Custom UIs

The current SIOT UI is more an engineering type view than something that might be used by end users. For a custom/company product IoT portal where you want a custom web UI optimized for your products, there are several options:

  1. modify the existing SIOT frontend.
  2. write a new frontend, mobile app, desktop app, etc. The SIOT backend and frontend are decoupled so that this is possible.

Database

Currently, Simple IoT supports bbolt as a data store. This is an embedded keyvalue store that is used similar to how NoSQL databases are used. Genji is used to provide some convenience for storing and querying Go types on top of Bolt.

The current database schema is several MongoDb schema design posts (1, 2, 3).

As described in the architecture document, nodes and edges are the primary data structures stored in database.

We currently use an external InfluxDB 2.x database for storing timeseries data, but eventually would like to have an embedded timeseries option -- perhaps built on bolt.

Evolvability

One important consideration in data design is the can the system be easily changed. With a distributed system, you may have different versions of the software running at the same time using the same data. One version may use/store additional information that the other does not. In this case, it is very important that the other version does not delete this data, as could easily happen if you decode data into a type, and then re-encode and store it.

With the Node/Point system, we don't have to worry about this issue because Nodes are only updated by sending Points. It is not possible to delete a Node Point. So it one version writes a Point the other is not using, it will be transferred, stored, syncronized, etc and simply ignored by version that don't use this point. This is another case where SIOT solves a hard problem that typically requires quite a bit of care and effort.

Rules

Rules are defined by nodes and are composed of additional child nodes for conditions and actions. See the node/point schema for more details.

All points should be sent out periodically, even if values are not changing to indicate a node is still alive and eliminate the need to periodically run rules. Even things like system state should be sent out to trigger device/node offline notifications.

Notifications

(see notification user documentation

Notifications are messages that are sent to users. There several concerns when processing a notification:

  1. The message itself and how it is generated.
  2. Who receives the messages.
  3. Mechanism for sending the message (Twilio SMS, SMTP, etc)
  4. State of the notification
    1. sequencing through a list of users
    2. tracking if it was acknowledged and by who
  5. Distributed concerns (more than one SIOT instance processing notifications)
    1. Synchronization of notification state between instances.
    2. Which instance is processing the notification.

For now, we assume all notifications will be handled on a single SIOT instance (typically in the cloud) -- distributed aspects of notifications will implemented later.

Notifications can be initiated by:

  1. rules
  2. users sending notifications through the web UI

Notification Data Structures

Elsewhere in the system, configuration and sensor data are represented as Points. The Point data structure is optimized for synchronization and algorithms where simplicity and consistency allows us to easily process points with common code. But a Point does not have enough fields to represent a message or notification. We could encode the message as JSON in the Point text field, but it would be nice to have something a little more descriptive. Additionally, all notifications and messages should be stored in the time series database so there is a history of everything that was sent.

Time series databases like InfluxDB store records with the following attributes:

  • timestamp
  • measurement (similar to collection, bucket, or table in other databases)
  • keys (string only, indexed)
  • values (can be a variety of data types: float, integer, string, boolean)

Notifications will be handled by two data structures:

  • Notification
    • typically generated by a rule or a node that is directly sending a message
    • stored in main database as they may contain state that needs to be processed over time
  • Message
    • an individual message to a user (SMS, Email, voice call)
    • stored in time series database as they are transient

Notifications are initiated (as is all write data in the system) by sending a message through NATS. The typical flow is as follows:

rule -> notification -> msg

Integration

This page discusses ways you can integration Simple IoT into your system. At its core, SIOT is a distributed graph database optimized for storing, viewing, and synchronizing state/config in IoT systems. This makes it very useful for any system where you need distributed state/config.

With SIOT, you run the same application in both the cloud and edge devices, so you can use any of the available integration points at either place.

integration

The SIOT API

This primary way to interact with Simple IoT is through a NATS API. You can add additional processes written in any language that has a NATS client. Additionally, the NATS wire protocol is fairly simple so could be implemented from scratch if needed. If your most of your system is written in C++, but you needed a distributed config/state store, then run SIOT along side your existing processes and add a NATs connection to SIOT. If you want easy scripting in your system, consider writing a Python application that can read/modify the SIOT store over NATS.

SIOT Data Structures

The SIOT data structures are very general (nodes and points) arranged in a graph, so you can easily add your own data to the SIOT store by defining new node and point types as needed. This makes SIOT very flexible and adaptable to about any purpose. You can use points in a node to represent maps and arrays. If you data needs more structure, then nested nodes can accomplish that. It is important with SIOT data to retain CRDT properties. These concepts are discussed more in ADR-1.

The requirement to only use nodes and points may seem restrictive at first, but can be viewed as a serialization format with CRDT properties that are convenient for synchronization. Any distributed database requires meta data around your data to assist with synchronization. With SIOT, we have chosen to make this metadata simple and accessible to the user. It is typical to convert this data to more convenient data structures in your application -- much the same way you would deserialize JSON.

The architecture page discusses data structures in more detail.

Time series data and Graphing

If you need history and graphs, you can add InfluxDB and Grafana. This instantly gives you history and graphs of all state and configuration changes that happened in the system.

Embedded Linux Systems

Simple IoT was designed with Embedded Linux systems in mind, so it is very efficient -- a single, statically linked Go binary with all assets embedded that is ~20MB in size and uses ~20MB of memory. There are no other dependencies required such as a runtime, other libraries, etc. This makes SIOT extremely easy to deploy and update. An Embedded Linux system deployed at the edge can be synchronized with a cloud instance using an upstream connection.

Integration with MCU (Microcontroller) systems

MCUs are processors designed for embedded control and are typically 32-bit CPUs that run bare-metal code or a small OS like FreeRTOS or Zephyr and don't have as much memory as MPUs. MCUs cannot run the full SIOT application or easily implement a full data-centric data store. However, you can still leverage the SIOT system by using the node/point data structures to describe configuration and state and interacting with a Simple IoT like any other NATS client. nanopb can be used on MCUs to encode and decode protobuf messages, which is the default encoding for SIOT messages.

If your MCU supports MQTT, then it may make sense to use that to interact with Simple IoT as MQTT is very similar to NATS, and NATS includes a built-in MQTT server. The NATS wire protocol is also fairly simple and can also be implemented on top of any TCP/IP stack.

If your MCU interfaces with a local SIOT system using USB, serial, or CAN, then you can use the SIOT serial adapter.

Serial Devices

Status: Specification

Contents

It is common in embedded systems architectures for a MPU (Linux based running SIOT) to be connected via a serial link (RS232, RS485, CAN, USB serial) to a MCU.

mcu connection

See this article for a discuss on the differences between a MPU and MCU. These devices are not connected via a network interface, so can't use the SIOT NATS API directly, thus we need to define a proxy between the serial interface and NATS for the MCU to interact with the SIOT system.

State/config data in both the MCU and MPU systems is represented as nodes and points. An example of nodes and points are shown below. These can be arranged in any structure that makes sense and is convenient. Simple devices may only have a single node with a handful of points.

nodes/points

SIOT does not differentiate between state (ex: sensor values) and config (ex: pump turn on delay) -- it is all points. This simplifies the transport and allows changes to be made in multiple places. It also allows for the granular transmit and synchronization of data -- we don't need to send the entire state/config anytime something changes.

SIOT has the ability to log points to InfluxDB, so this mechanism can also be used to log messages, events, state changes, whatever -- simply use an existing point type or define a new one, and send it upstream.

Protocol

The SIOT serial protocol mirrors the NATS PUB message with a few assumptions:

  • we don't have mirrored nodes inside the MCU device
  • the number of nodes and points in a MCU is relatively small
  • the payload is always an array of points
  • only the following SIOT NATS APIs are supported:
    • node.<id>.points (used to send node points)
    • node.<id>.<parent>.points (used to send edge points)
  • we don't support NATS subscriptions or requests -- on startup, we send the entire dataset for the MCU device in both directions (see On connection section), merge the contents, and then assume any changes will get sent and received after that.

The serial protobuf type is used to transfer these messages:

message Serial {
 string subject = 1;
 repeated Point points = 2;
}

subject can be left blank when sending/receiving points for the MCU root node. This saves quite a bit of data in the serial messages.

The point type nodeType is used to create new nodes and to send the node type on connection.

Encoding

All packets between the SIOT and serial MCU systems are structured as follows:

  • sequence (1 byte, rolls over)
  • Serial protobuf
  • crc (2 bytes) (Currently using CRC-16/KERMIT)

All packets are ack'd by an empty packet with the same sequenced number. If an ack is not received in X amount of time, the packet is retried up to 3 times, and then the other device is considered "offline".

Protobuf is used to encode the data on the wire. Find protobuf files here. nanopb can be used to generate C based protobuf bindings that are suitable for use in most MCU environments.

On connection

On initial connection between a serial device and SIOT, the following steps are done:

  • the MCU sends the SIOT system an empty packet with its root node ID
  • the SIOT systems sends the current time to the MCU (point type currentTime)
  • the MCU updates any "offline" points with the current time (see offline section).
  • the SIOT acks the current time packet.
  • all of the node and edge points are sent from the SIOT system to the MCU, and from the MCU to the SIOT system. Each system compares point time stamps and updates any points that are newer. Relationships between nodes are defined by edge points (point type tombstone).

Timestamps

Simple IoT currently uses the default protobuf timestamp format, which includes the following fields:

Field nameTypeDescription
secondsint64Represents seconds of UTC time since Unix epoch 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to 9999-12-31T23:59:59Z inclusive.
nanosint32Non-negative fractions of a second at nanosecond resolution. Negative second values with fractions must still have non-negative nanos values that count forward in time. Must be from 0 to 999,999,999 inclusive.

Fault handling

Any communication medium has the potential to be disrupted (unplugged/damaged wires, one side off, etc). Devices should continue to operate and when re-connected, do the right thing.

If a MCU has a valid time (RTC, sync from SIOT, etc), it will continue operating, and when reconnected, it will send all its points to re-sync.

If a MCU powers up and has no time, it will set the time to 1970 and start operating. When it receives a valid time from the SIOT system, it will compute the time offset from the SIOT time and its own 1970 based time, index through all points, add the offset to any points with time less than 2020, and then send all points to SIOT.

When the MCU syncs time with SIOT, if the MCU time is ahead of the SIOT system, then it set its time, and look for any points with a time after present, and reset these timestamps to the present.

Packet Framing

Protocols like RS232 and USB serial do not have any inherent framing; therefore, this needs to be done at the application level. SIOT uses COBS (Consistent Overhead Byte Stuffing) to encode data for these transports.

RS485

Status: Idea

RS485 is a half duplex, prompt response transport. SIOT periodically prompts MCU devices for new data at some configurable rate. Data is still COBS encoded so that is simple to tell where packets start/stop without needing to rely on dead space on the line.

Simple IoT also supports modbus, but the native SIOT protocol is more capable -- especially for structured data.

Addressing: TODO

CAN

Status: Idea

CAN messages are limited to 8 bytes. The J1939 Transport Protocol can be used to assemble multiple messages into a larger packet for transferring up to 1785 bytes.

Implementation notes

Both the SIOT and MCU side need to store the common set of nodes and points between the systems. This is critical as the point merge algorithm only uses an incoming point if the incoming point is newer than the one currently stored on the device. For SIOT NATS clients, we use the NodeEdge data structure:

type NodeEdge struct {
        ID         string
        Type       string
        Parent     string
        Points     Points
        EdgePoints Points
	Origin     string
}

Something similar could be done on the MCU.

If new nodes are created on the MCU, the ID must be a UUID, so that it does not conflict with any of the node IDs in the upstream SIOT system(s).

On the SIOT side, we keep a list of Nodes on the MCU and periodically check if any new Nodes have been created. If so, we send the new Nodes to the MCU. Subscriptions are set up for points and edges of all nodes, and any new points are sent to the MCU. Any points received from the MCU simply forwarded to the SIOT NATS bus.

DFU

Status: Idea

For devices that support USB Device Firmware Upgrade (DFU), SIOT provides a mechanism to do these updates. A node that specifies USB ID and file configures the process.

Version

The Simple IoT app stores and uses three different version values:

  • App Version
  • OS Version
  • HW Version

The App version is compiled into the application Go binary by the build (see the envsetup.sh file). This version is based on the latest Git tag plus hash if there have been any changes since the tag.

On Linux, the OS version is extracted from the VERSION field in /etc/os-release. The field can be changed using the OS_VERSION_FIELD environment variable.

The versions are displayed in the root node as shown below:

versions display

Security

Users and downstream devices will need access to a Simple IoT instance. Simple IoT currently provides access via HTTP and NATS.

Server

For cloud/server deployments, we recommend installing a web server like Caddy in front of Simple IoT. See the Installation page for more information.

Edge

Simple IoT Edge instances initiate all connections to upstream instances; therefore, no incoming connections are required on edge instances and all incoming ports can be firewalled.

HTTP

The Web UI uses JWT (JSON web tokens).

Devices can also communicate via HTTP and use a simple auth token. Eventually may want to switch to JWT or something similar to what NATS uses.

NOTE, it is important to set an auth token -- otherwise there is no restriction on accessing the device API.

NATS

Currently devices communicating via NATS use a common auth token. It would be nice to move to something where each device has its own authentication (TODO, explore NATS advanced auth options).

Long term we plan to leverage the NATS security model for user and device authn/authz.:

Research

This document contains information that has been researched during the course of creating Simple IoT.

Synchronization

An IoT system is inherently distributed. At a minimum, there are three components:

  1. device (Go, C, etc.)
  2. cloud (Go)
  3. multiple browsers (Elm, Js)

Data can be changed in any of the above locations and must be seamlessly synchronized to other locations. Failing to consider this simple requirement early in building the system can make for brittle and overly complex systems.

Resgate

resgate.io is an interesting project that solves the problem of creating a real-time API gateway where web clients are synchronized seamlessly. This project uses NATS.io for a backbone, which makes it interesting as NATS is core to this project.

The Resgate system is primarily concerned with synchronizing browser contents.

Couch/pouchdb

Has some interesting ideas.

Merkle Trees

  • https://medium.com/@rkkautsar/synchronizing-your-hierarchical-data-with-merkle-tree-dbfe37db3ab7
  • https://en.wikipedia.org/wiki/Merkle_tree
  • https://jack-vanlightly.com/blog/2016/10/24/exploring-the-use-of-hash-trees-for-data-synchronization-part-1
  • https://www.codementor.io/blog/merkle-trees-5h9arzd3n8
    • Version Control Systems Version control systems like Git and Mercurial use specialized merkle trees to manage versions of files and even directories. One advantage of using merkle trees in version control systems is we can simply compare hashes of files and directories between two commits to know if they've been modified or not, which is quite fast.
    • No-SQL distributed database systems like Apache Cassandra and Amazon DynamoDB use merkle trees to detect inconsistencies between data replicas. This process of repairing the data by comparing all replicas and updating each one of them to the newest version is also called anti-entropy repair. The process is also described in Cassandra's documentation.

Scaling Merkel trees

One limitation of Merkel trees is the difficulty of updating the tree concurrently. Some information on this:

Distributed key/value databases

  • etcd

Distributed Hash Tables

  • https://en.wikipedia.org/wiki/Distributed_hash_table

CRDT (Conflict-free replicated data type)

  • https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type
  • Yjs
    • https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/

Timestamps

Other IoT Systems

AWS IoT

  • https://www.thingrex.com/aws_iot_thing_attributes_intro/
    • Thing properties include the following, which are analogous to SIOT node fields.
      • Name (Desription)
      • Type (Type)
      • Attributes (Points)
      • Groups (Described by tree structure)
      • Billing Group (Can also be described by tree structure)
  • https://www.thingrex.com/aws_iot_thing_type/
    • each type has a specified attributes -- kind of a neat idea

Architecture Decision Records

This directory is used to capture Simple IoT architecture and design decisions.

For background on ADRs see Documenting Architecture Decisions by Michael Nygard. Also see an example of them being used in the NATS project. The Go proposal process is also a good reference.

Process

When thinking about architectural changes, we should lead with documentation. This means we should start a branch, draft a ADR, and then open a PR. An associated issue may also be created.

ADRs should used primarily when a number of approaches need to be considered, thought through, and we need a record of how and why the decision was made. If the task is a fairly straightforward implementation, write documentation in the existing User and Reference Guide sections.

When an ADR is accepted and implemented, a summary should typically be added to the Reference Guide documentation.

An ADR can have the following sections as needed. The highlighted sections should probably be in every ADR.

  • Header
    • Author: AUTHOR_NAME Last updated: 2021-11-04
    • Issue:
    • PR/Discussion:
    • Status: [Discussion|Proposed|Accepted|Deprecated|Superseded]
  • Problem -- what problem are we trying to solve?
  • Context -- background, facts surrounding this discussion.
  • Design -- discussion on implementation -- may present several different options.
  • Decision -- what was decided.
    • Objections/concerns
  • Consequences -- what is the impact, both negative and positive.
  • Additional Notes/Reference -- links to reference material that may be relevant.

ADRs

IndexDescription
ADR-1Consider changing/expanding point data type
ADR-2Authorization considerations.
ADR-3Node lifecycle

Point Data Type Changes

  • Author: Cliff Brake Last updated: 2021-11-04
  • Issue at: https://github.com/simpleiot/simpleiot/issues/254
  • PR/Discussion: https://github.com/simpleiot/simpleiot/pull/279
  • Status: Accepted

Contents

Problem

The current point data type is fairly simple and has proven useful and flexible to date, but we may benefit from additional or changed fields to support more scenarios. It seems in any data store, we need at the node level to be able to easily represent:

  1. arrays
  2. maps

IoT systems are distributed systems that evolve over time. If can't easily handle schema changes and synchronize data between systems, we don't have anything.

Context/Discussion

Should we consider making the point struct more flexible?

The reason for this is that it is sometimes hard to describe a sensor/configuration value with just a few fields.

Requirements

  • IoT systems are often connected by unreliable networks (cellular, etc). All devices/instances in a SIOT should be able to functional autonomously (run rules, etc) and then synchronize again when connected.
  • all systems must converge to the same configuration state. We can probably tolerate some lost time series data, but configuration and current state must converge. When someone is remotely looking at a device state, we want to make sure they are seeing the same things a local operator is seeing.

evolvability

From Martin Kleppmann's book:

In a database, the process that writes to the database encodes the data, and the process that reads from the database decodes it. There may just be a single process accessing the database, in which case the reader is simply a later version of the same process—in that case you can think of storing something in the database as sending a message to your future self.

Backward compatibility is clearly necessary here; otherwise your future self won’t be able to decode what you previously wrote.

In general, it’s common for several different processes to be accessing a database at the same time. Those processes might be several different applications or services, or they may simply be several instances of the same service (running in parallel for scalability or fault tolerance). Either way, in an environment where the application is changing, it is likely that some processes accessing the database will be running newer code and some will be running older code—for example because a new version is currently being deployed in a rolling upgrade, so some instances have been updated while others haven’t yet.

This means that a value in the database may be written by a newer version of the code, and subsequently read by an older version of the code that is still running. Thus, forward compatibility is also often required for databases.

However, there is an additional snag. Say you add a field to a record schema, and the newer code writes a value for that new field to the database. Subsequently, an older version of the code (which doesn’t yet know about the new field) reads the record, updates it, and writes it back. In this situation, the desirable behavior is usually for the old code to keep the new field intact, even though it couldn’t be interpreted.

The encoding formats discussed previously support such preservation of unknown fields, but sometimes you need to take care at an application level, as illustrated in Figure 4-7. For example, if you decode a database value into model objects in the application, and later re-encode those model objects, the unknown field might be lost in that translation process. Solving this is not a hard problem; you just need to be aware of it.

Some discussion of this book: https://community.tmpdir.org/t/book-review-designing-data-intensive-applications/288/6

CRDTs

Some good talks/discussions:

I also agree CRDTs are the future, but not for any reason as specific as the ones in the article. Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively. And certainly to build reliable systems. The abstraction of a single, global, logical truth is so nice and tidy and appealing, but it becomes so leaky that I think all successful systems for distributed state will abandon it beyond a certain scale. -- Peter Bourgon

CRDTs, the hard parts by Martin Kleppmann

Infinite Parallel Universes: State at the Edge

Wikipedia article

Properties of CRDTs:

  • Associative (order in which operations are performed does matter)
  • Commutative (changing order of operands does not change result)
  • Idempotent (operation can be applied multiple times without changing the result, tolerate over-merging)

The existing SIOT Node/Point data structures were created before I know what a CRDT was, but they happen to already give a node many of the properties of a CRDT -- IE, they can be modified independently, and then later merged with a reasonable level of conflict resolution.

For reliable data synchronization in distributed systems, there has to be some metadata around data that facilitates synchronization. This can be done in two ways:

  1. add meta data in parallel to the data (turn JSON into a CRDT, example automerge or yjs)
  2. express all data using simple primitives that facilitate synchronization

Either way, you have to accept constraints in your data storage and transmission formats.

To date, we have chosen to follow the 2nd path (simple data primitives).

Operational transforms

There are two fundamental schools of thought regarding data synchronization:

  1. Operation transforms. In this method, a central server arbitrates all conflicts and hands the result back to other instances. This is an older technique and is used in applications like Google docs.
  2. CRDTs -- this is a newer technique that works with multiple network connections and does not require a central server. Each instance is capable of resolving conflicts themselves and converging to the same point.

While a classical OT arrangement could probably work in a traditional SIOT system (where all devices talk to one cloud server), it would be nice if we are not constrained to this architecture. This would allow us to support peer synchronization in the future.

Other Standards

Some reference/discussion on other standards:

Sparkplug

https://github.com/eclipse/tahu/blob/master/sparkplug_b/sparkplug_b.proto

The sparkplug data type is huge and could be used to describe very complex data. This standard came out of the industry 4.0 movement where a factory revolves around a common MQTT messaging server. The assumption is that everything is always connected to the MQTT server. However, with complex types, there is no provision for intelligent synchronization if one system is disconnected for some amount of time -- its all or nothing, thus it does not seem like a good fit for SIOT.

SenML

https://datatracker.ietf.org/doc/html/draft-ietf-core-senml-08#page-9

tstorage

The tstorage Go package has an interesting data storage type:

type Row struct {
	// The unique name of metric.
	// This field must be set.
	Metric string
	// An optional key-value properties to further detailed identification.
	Labels []Label
	// This field must be set.
	DataPoint
}

type DataPoint struct {
	// The actual value. This field must be set.
	Value float64
	// Unix timestamp.
	Timestamp int64
}

type Label struct {
	Name  string
	Value string

In this case there is one value and an array of labels, which are essentially key/value strings.

InfluxDB

InfluxDB's line protocol contains the following:

type Metric interface {
	Time() time.Time
	Name() string
	TagList() []*Tag
	FieldList() []*Field
}

type Tag struct {
	Key   string
	Value string
}

type Field struct {
	Key   string
	Value interface{}
}

where the Field.Value must contain one of the InfluxDB supported types (bool, uint, int, float, time, duration, string, or bytes).

time-series storage considerations

Is it necessary to have all values in one point, so they can be grouped as one entry in a time series data base like influxdb? Influx has a concept of tags and fields, and you can have as many as you want for each sample. Tags must be strings and are indexed and should be low cardinality. Fields can be any datatype influxdb supports. This is a very simple, efficient, and flexible data structure.

Example: location data

One system we are working with has extensive location information (City/State/Facility/Floor/Room/Isle) with each point. This is all stored in influx so we can easily query information for any location in the past. With SIOT, we could not currently store this information with each value point, but would rather store location information with the node as separate points. One concern is if the device would change location. However, if location is stored in points, then we will have a history of all location changes of the device. To query values for a location, we could run a two pass algorithm:

  1. query history and find time windows when devices are in a particular location.
  2. query these time ranges and devices for values

This has the advantage that we don't need to store location data with every point, but we still have a clear history of what data come from where.

Example: file system metrics

When adding metrics, we end up with data like the following for disks partitions:

Filesystem     Size Used Avail Use% Mounted on
tmpfs          16806068224 0 16806068224   0% /dev
tmpfs          16813735936 1519616 16812216320   0% /run
ext2/ext3      2953064402944 1948218814464 854814945280  70% /
tmpfs          16813735936 175980544 16637755392   1% /dev/shm
tmpfs          16813740032 3108966400 13704773632  18% /tmp
ext2/ext3      368837799936 156350181376 193680359424  45% /old3
msdos          313942016 60329984 253612032  19% /boot
ext2/ext3      3561716731904 2638277668864 742441906176  78% /scratch
tmpfs          3362746368 118784 3362627584   0% /run/user/1000
ext2/ext3      1968874332160 418203766784 1450633895936  22% /run/media/cbrake/59b35dd4-954b-4568-9fa8-9e7df9c450fc
fuseblk        3561716731904 2638277668864 742441906176  78% /media/fileserver
ext2/ext3      984372027392 339508314112 594836836352  36% /run/media/cbrake/backup2

It would be handy if we could store filesystem as a tag, size/used/avail/% as fields, and mount point as text field.

We already have an array of points in a node -- can we just make one array work? The size/used/avail/% could easily be stored as different points. The text field would store the mount point, which would tie all the stats for one partition together. Then the question is how to represent the filesystem? With the added Key field in proposal #2, we can now store the mount point as the key.

TypeKeyTextValue
filesystemSize/home1243234
filesystemUsed/home234222
filesystemType/homeext4
filesystemSize/home1000000
filesystemUsed/date10000
filesystemType/homebtrfs

Representing arrays

With the index field, we can already represent arrays as a group of points, where index defines the position in the array. One example where we do this is for selecting days of the week in schedule rule conditions. The index field is used to select the weekday. So we can have a series of points to represent Weekdays. In the below, Sunday is the 1st point set to 0, and Monday is the 2nd point, set to 1. https://community.tmpdir.org/t/the-tstorage-time-series-package-for-go/331

[]data.Point{
  {
    Type: "weekday",
    Index: 0,
    Value: 0,
  },
  {
    Type: "weekday",
    Index: 1,
    Value: 0,
  },
}

In this case, the condition node has a series of weekday points with indexes 0-6 to represent the days of the week.

If we change value to map (propsal #1) of key/values, then weekday values could be represented in the field map:

  • "0":0
  • "1":1
  • "2":0
  • "3":0

or

  • "Sun":0
  • "Mon":1
  • "Tues:0
  • "Wed":0

A single point could then represent weekdays instead of requiring multiple points.

In practice, I've found presenting weekdays by numbers is easier to deal with in programs.

However, an embedded map does not have strong CRDT properties, so lets return to proposal #2, where an array is represented as a set of points.

Concurrent modifications to node arrays

With SIOT, concurrent modifications to a point typically work out pretty well, as most settings can be described in a single point. However, with arrays represented as a group of points, there is more likelihood of conflict, because order in an array is sometimes important.

Much of the research with ordered set (sequence) CRDTs deals with a text editor scenario where interleaving letters and words is a problem. In our use case, interleaving is not an concern. We simply need to deal with add, remove, and index changes (moves) reliably.

Example: weekdays in rule condition stored as a point array

In this example, we will have a list of 7, where a value of 1 indicates that weekday is enabled. In this case, the size and order of the array is fixed, so the current mechanism will work just fine.

Example: various array add/move operations.

The following are all points in a node with a common Type value.

t1: A creates initial array:

IndexKeyTimeTextTombstone
0UUID11a0
1UUID21b0

t2: C adds item to array:

IndexKeyTimeTextTombstone
0UUID11a0
1UUID21b0
2UUID32c0

t3: B adds a' between a and b.

IndexKeyTimeTextTombstone
0UUID11a0
0.5UUID43a'0
1UUID21b0
2UUID32c0

t4: move a' between b and c:

IndexKeyTimeTextTombstone
0UUID11a0
1UUID21b0
1.5UUID44a'0
2UUID32c0

t5: instance A adds a'' between a' and b, and instance B adds b' between a' and b.

IndexKeyTimeTextTombstone
0UUID11a0
0.5UUID55a''0
0.5UUID65b'0
1UUID21b0
1.5UUID44a'0
2UUID32c0

t6: move a' back to between a and a'':

IndexKeyTimeTextTombstone
0UUID11a0
0.25UUID46a'0
0.5UUID55a''0
0.5UUID65b'0
1UUID21b0
2UUID32c0

Note, for this to all work, points are matched only by Type and Key, not Index, as the index may change for a point. Any time an Index is used, the Key must be populated with a globally unique value such that this point can always be identified later.

Point deletions

To date, we've had no need to delete points, but it may be useful in the future.

Consider the following sequence of point changes:

  1. t1: we have a point
  2. t2: A deletes the point
  3. t3: B concurrently change the point value

The below table shows the point values over time with the current point merge algorithm:

TimeIndexKeyValueTombstone
t10UUID1100
t20UUID1101
t30UUID1200

In this case, the point becomes undeleted because the last write wins (LWW). Is this a problem? What is the desired behavior? A likely scenario is that a device will be continually sending value updates and a user will make a configuration change in the portal that deletes a point. Thus it seems delete changes should always have precedence. However, with the last write wins (LWW) merge algorithm, the tombstone value could get lost. It may make sense to:

  • make the tombstone value an int
  • only increment it
  • when merging points, the highest tombstone value wins
  • odd value of tombstone value means point is deleted

Thus the tombstone value is merged independently of the timestamp and thus is always preserved, even if there concurrent modifications.

The following table shows the values with the modified point merge algorithm.

TimeIndexKeyValueTombstone
t10UUID1100
t20UUID1101
t30UUID1201

Duration, Min, Max

The current Point data type has Duration, Min, and Max fields. This is used for when a sensor value is averaged over some period of time, and then reported. The Duration, Min, Max fields are useful for describing what time period the point was obtained, and what the min/max values during this period were.

Representing maps

In the file system metrics example below, we would like to store a file system type for a particular mount type. We have 3 pieces of information:

data.Point {
  Type: "fileSystem",
  Text: "/media/data/",
  ????: "ext4",
}

Perhaps we could add a key field:

data.Point {
  Type: "fileSystem",
  Key: "/media/data/",
  Text: "ext4",
}

The Key field could also be useful for storing the mount point for other size/used, etc points.

making use of common algorithms and visualization tools

A simple point type makes it very nice to write common algorithms that take in points, and can always assume the value is in the value field. If we store multiple values in a point, then the algorithm needs to know which point to use.

If an algorithm needs multiple values, it seems we could feed in multiple point types and discriminated by point type. For example, if an algorithm used to calculate % of a partition used could take in total size and used, store each, and the divide them to output %. The data does not necessarily need to live in the same point. Could this be used to get rid of the min/max fields in the point? Could these simply be separate points?

  • Having min/max/duration as separate points in influxdb should not be a problem for graphing -- you would simply qualify the point on a different type vs selecting a different field.
  • if there is a process that is doing advanced calculations (say taking the numerical integral of flow rate to get total flow), then this process could simply accumulate points and when it has all the points for a timestamp, then do the calculation.

Schema changes and distributed synchronization

A primary consideration of Simple IoT is easy and efficient data synchronization and easy schema changes.

One argument against embedded maps in a point is that adding these maps would likely increase the possibility of schema version conflicts between versions of software because points are overwritten. Adding maps now introduces a schema into the point that is not synchronized at the key level. There will also be a temptation to put more information into point maps instead of creating more points.

With the current point scheme, it is very easy to synchronize data, even if there are schema changes. All points are synchronized, so one version can write one set of points, and another version another, and all points will be sync'd to all instances.

There is also a concern that if two different versions of the software use different combinations of field/value keys, there could be information lost. The simplicity and ease of merging Points into nodes is no longer simple. As an example:

Point {
  Type: "motorPIDConfig",
  Values: {
    {"P": 23},
    {"I": 0.8},
    {"D": 200},
  },
}

If an instance with an older version writes a point that only has the "P" and "I" values, then the "D" value would get lost. We could merge all maps on writes to prevent losing information. However if we have a case where we have 3 systems:

Aold -> Bnew -> Cnew

If Aold writes an update to the above point, but only has P,I values, then this point is automatically forwarded to Bnew, and then Bnew forwards it to Cnew. However, Bnew may have had a copy with P,I,D values, but the D is lost when the point is forwarded from Aold -> Cnew. We could argue that Bnew has previously synchronized this point to Cnew, but what if Cnew was offline and Aold sent the point immediately after Cnew came online before Bnew synchronized its point.

The bottom line is there are edge cases where we don't know if the point map data is fully synchronized as the map data is not hashed. If we implement arrays and maps as collections of points, then we can be more sure everything is synchronized correctly because each point is a struct with fixed fields.

Is there any scenario where we need multiple tags/labels on a point?

If we don't add maps to points, the assumption is any metadata can be added as additional points to the containing node. Will this cover all cases?

Is there any scenario where we need multiple values in a point vs multiple points?

If we have points that need to be grouped together, they could all be sent with the same timestamp. Whatever process is using the points could extract them from a timeseries store and then re-associate them based on common timestamps.

Could duration/min/max be sent as separate points with the same timestamp instead of extra fields in the point?

The NATS APIs allow you to send multiple points with a message, so if there is ever a need to describe data with multiple values (say min/max/etc), these can simply be sent as multiple points in one message.

Is there any advantage to flat data structures?

Flat data structures where the fields consist only of simple types (no nested objects, arrays, maps, etc). This is essentially what tables in a relational database are. One advantage to keeping the point type flat is it would map better into a relational database. If we add arrays to the Point type, then it will not longer map into a single relational database table.

Design

Original Point Type

type Point struct {
	// ID of the sensor that provided the point
	ID string `json:"id,omitempty"`

	// Type of point (voltage, current, key, etc)
	Type string `json:"type,omitempty"`

	// Index is used to specify a position in an array such as
	// which pump, temp sensor, etc.
	Index int `json:"index,omitempty"`

	// Time the point was taken
	Time time.Time `json:"time,omitempty"`

	// Duration over which the point was taken. This is useful
	// for averaged values to know what time period the value applies
	// to.
	Duration time.Duration `json:"duration,omitempty"`

	// Average OR
	// Instantaneous analog or digital value of the point.
	// 0 and 1 are used to represent digital values
	Value float64 `json:"value,omitempty"`

	// Optional text value of the point for data that is best represented
	// as a string rather than a number.
	Text string `json:"text,omitempty"`

	// statistical values that may be calculated over the duration of the point
	Min float64 `json:"min,omitempty"`
	Max float64 `json:"max,omitempty"`
}

Proposal #1

This proposal would move all the data into maps.

type Point struct {
    ID string
    Time time.Time
    Type string
    Tags map[string]string
    Values map[string]float64
    TextValues map[string]string
}

The existing min/max would just become fields. This would map better into influxdb. There would be some redundancy between Type and Field keys.

Proposal #2

type Point struct {
	// The 1st three fields uniquely identify a point when receiving updates
	Type string
	Key string

	// The following fields are the values for a point
	Time time.Time
	Index float64
	Value float64
	Text string
	Data []byte

	// Metadata
	Tombstone int
}

Notable changes from the existing implementation:

  • removal of the ID field, as any ID information should be contained in the parent node. The ID field is a legacy from 1-wire setups where we represented each 1-wire sensor as a point. However, it seems now each 1-wire sensor should have its own node.
  • addition of the Key field. This allows us to represent maps in a node, as well as add extra identifying information for a point.
  • the Point is now identified in the merge algorithm using the Type and Key. Before, the ID, Type, and Index were used.
  • any Point that uses Index, should also set the Key to some globally unique value. This Key is used on subsequent updates to identify the point.
  • Index is changed from int to float64 so that we can move points in arrays between existing points. See example above.
  • the Data field is added to give us the flexibility to store/transmit data that does not fit in a Value or Text field. This should be used sparingly, but gives us some flexibility in the future for special cases. This came out of some comments in an Industry 4.0 community -- basically types/schemas are good in a communication standard, as long as you also have the capability to send a blob of data to handle all the special cases. This seems like good advice.
  • the Tombstone fields is added as an int and is always incremented. Odd values of Tombstone mean the point was deleted. When merging points, the highest tombstone value always wins.

Decision

Going with proposal #2 -- we can always revisit this later if needed. This has minimal impact on the existing code base.

Objections/concerns

(Some of these are general to the node/point concept in general)

  • Q: with the point datatype, we lose types
    • A: in a single application, this concern would perhaps be a high priority, but in a distributed system, data syncronization and schema migrations must be given priority. Typically these collections of points are translated to a type by the application code using the data, so any concerns can be handled there. At least we won't get JS undefined crashes as Go will fill in zero values.
  • Q: this will be inefficient converting points to types
    • A: this does take processing time, but this time is short compared to the network transfer times from distributed instances. Additionally, applications can cache nodes they care about so they don't have to translate the entire point array every time they use a node. Even a huge IoT system has a finite # of devices that can easily fit into memory of modern servers/machines.
  • Q: this seems crude not to have full featured protobuf types with all the fields explicitely defined in protobuf. Additionally, can't protobuf handle type changes elegantly?
    • A: protobuf can handle field additions and removal but we still have the edge cases where a point is sent from an old version of software that does not contain information written by a newer versions. Also, I'm not sure it is a good idea to have application specific type fields defined in protobuf, otherwise, you have a lot of work all along the communication chain to rebuild everything every time anything changes. With a generic types that rarely have to change, your core infrastructure can remain stable and any features only need to touch the edges of the system.
  • Q: with nodes and points, we can only represent a type with a single level of fields
    • A: this is not quite true, because with the key/index fields, we can now have array and map fields in a node. However, the point is taken that a node with its points cannot represent a deeply nested data structure. However, nodes can be nested to represent any data structure you like. This limitation is by design because otherwise syncronization would be very difficult. By limitting the complexity of the core data structures, we are making synchronziation and storage very simple. The tradeoff is a little more work to marshall/unmarshall node/point data structures into useful types in your application. However, marshalling code is easy compared to distributed systems, so we need to optmize the system for the hard parts. A little extra typing will not hurt anyone, and tooling could be developed if needed to assist in this.

Generic core data structures also opens up the possibility to dynamically extend the system at run time without type changes. For instance, the GUI could render new nodes it has never seen before by sending it configuration nodes with declarative instructions on how to display the node. If core types need to change to do this type of thing, we have no chance at this type of intelligent functionality.

Consequences

Removing the Min/Max/Duration fields should not have any consequences now as I don't think we are using these fields yet.

Quite a bit of code needs to change to remove ID and add Key to code using points.

Additional Notes/Reference

We also took a look at how to resolve loops in the node tree:

https://github.com/simpleiot/simpleiot/issues/294

This is part of the verification to confirm our basic types are robust and have adequate CRDT properties.

Authorization

  • Author: Blake Miner
  • Issue: https://github.com/simpleiot/simpleiot/issues/268
  • PR / Discussion: https://github.com/simpleiot/simpleiot/pull/283
  • Status: Brainstorming

Problem

SIOT currently does not prevent unauthorized NATS clients from connecting and publishing / subscribing. Presently, any NATS client with access to the NATS server connection can read and write any data over the NATS connection.

Discussion

This document describes a few mechanisms for how to implement authentication and authorization mechanisms within Simple IoT.

Current Authentication Mechanism

Currently, SIOT supports upstream connections through the use of upstream nodes. The connection to the upstream server can be authenticated using a simple NATS auth token; however, all NATS clients with knowledge of the auth token can read / write any data over the NATS connection. This will not work well for a multi-tenant application or applications where user access must be closely controlled.

Similarly, web browsers can access the NATS API using the WebSocket library, but since they act as another NATS client, no additional security is provided; browsers can read / write all data over the NATS connection.

Proposal

NATS supports decentralized user authentication and authorization using NKeys and JSON Web Tokens (JWTs). While robust, this authentication and authorization mechanism is rather complex and confusing; a detailed explanation follows nonetheless. The end goal is to dynamically add NATS accounts to the NATS server because publish / subscribe permissions of NATS subjects can be tied to an account.

Background

Each user node within SIOT will be linked to a dynamically created NATS account (on all upstream nodes); each account is generated when the user logs in. Only a single secret is stored in the root node of the SIOT tree.

NATS has a public-key signature system based on Ed25519. These keypairs are called NKeys. Put simply, NKeys allow one to cryptographically sign and verify JWTs. An NKey not only consists of a Ed25519 private key / seed, but it also contains information on the "role" of the key. In NATS, there are three primary roles: operators, accounts, and users. In SIOT, there is one operator for a given NATS server, and there is one account for each user node.

Start-up

When the SIOT server starts, an NKey for the operator role is loaded from a secret stored as a point in the root node of the tree. This point is always stripped away when clients request the root node, so it's never transmitted over a NATS connection. Once the NATS server is running, SIOT will start an internal NATS client and connect to the local NATS server. This internal client will authenticate to the NATS server with a superuser, whose account has full permissions to publish and subscribe to all subjects. Unauthenticated NATS clients only have permission to publish to authsubject and listen for a reply.

Authentication / Login

External NATS clients (including web browsers over WebSockets) must first log into the NATS server anonymously (using the auth token if needed) and send a request to the auth subject with the username and password of a valid user node. The default username is admin@admin.com, and the default password is admin. The internal NATS client will subscribe to requests on the auth subject, and if the username / password is correct, it will respond with a user NKey and user JWT Token, which are needed to login. The user JWT token will be issued and signed by the account NKey, and the account NKey will be issued and signed by the operator NKey. The NATS connection will then be re-established using the user JWT and signing a server nonce with the user's NKey.

JWT expiration should be a configurable SIOT option and default to 1 hour. Optionally, when the user JWT token is approaching its expiration, the NATS client can request re-authenticate using the auth subject and reconnect using the new user credentials.

Storing NKeys

As discussed above, in the root node, we store the seed needed to derive the operator NKey. For user nodes, account and user NKeys are computed as-needed from the node ID, the username, and the password.

Authorization

An authenticated user will have publish / subscribe access to the subject space of $nodeID.> where $nodeID is the node ID for the authenticated user. The normal SIOT NATS API will work the same as normal with two notable exceptions:

  • The API subjects are prepended with $nodeID.
  • The "root" node is remapped to the set of parents of the logged in user node

Examples

Example #1

Imagine the following a SIOT node tree:

  • Root (Device 82ad…28ae)
  • Power Users (Group b723…008d)
    • Temperature Sensor (Device 2820…abdc)
    • Humidity Sensor (Device a89f…eda9)
    • Blake (User ab12…ef22)
  • Admin (User 920d…ab21)

In this case, logging in as "Blake" would reveal the following tree with a single root node:

  • Power Users (Group b723…008d)
    • Temperature Sensor (Device 2820…abdc)
    • Humidity Sensor (Device a89f…eda9)
    • Blake (User ab12…ef22)

To get points of the humidity sensor, one would send a request to this subject: ab12...ef22.node.a89f...eda9.points.

Example #2

Imagine the following a SIOT node tree:

  • Root (Device 82ad…28ae)
    • Temperature Sensor (Device 2820…abdc)
      • Blake (User ab12…ef22)
    • Humidity Sensor (Device a89f…eda9)
      • Blake (User ab12…ef22)
    • Admin (User 920d…ab21)

In this case, logging in as "Blake" would reveal the following tree with two root nodes:

  • Temperature Sensor (Device 2820…abdc)
    • Blake (User ab12…ef22)
  • Humidity Sensor (Device a89f…eda9)
    • Blake (User ab12…ef22)

To get points of the humidity sensor, one would send a request to this subject: ab12...ef22.node.a89f...eda9.points.

Implementation Notes

// Note: JWT issuer and subject must match an NKey public key
// Note: JWT issuer and subject must match roles depending on the claim NKeys

import (
	"github.com/nats-io/jwt/v2"
	"github.com/nats-io/nkeys"
	"github.com/nats-io/nats-server/v2/server"
)

// Example code to start NATS server
func StartNatsServer(o Options) {
	op, err := nkeys.CreateOperator()
	if err != nil {
		log.Fatal("Error creating NATS server: ", err)
	}
	pubKey, err := op.PublicKey()
	if err != nil {
		log.Fatal("Error creating NATS server: ", err)
	}
	acctResolver := server.MemAccResolver{}
	opts := server.Options{
		Port:                     o.Port,
		HTTPPort:                 o.HTTPPort,
		Authorization:            o.Auth,
		// First we trust all operators
		// Note: DO NOT USE conflicting `TrustedKeys` option
		TrustedOperators:         []{jwt.NewOperatorClaims(pubKey)},
		AccountResolver:          acctResolver,
	}
}

// Create an Account
acct, err := nkeys.CreateAccount()
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}
pubKey, err := acct.PublicKey()
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}
claims := jwt.NewAccountClaims{pubKey}
claims.DefaultPermissions = Permissions{
	// Note: subject `_INBOX.>` allowed for all NATS clients
	// Note: subject publish on `auth` allowed for all NATS clients
	Pub: Permission{
		Allow: StringList([]string{userNodeID+".>"}),
	},
	Sub: Permission{
		Allow: StringList([]string{userNodeID+".>"}),
	},
}
claims.Issuer = opPubKey
claims.Name = userNodeID
// Sign the JWT with the operator NKey
jwt, err := claims.Encode(op)
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}

acctResolver.Store(userNodeID, jwt)

Node Lifecycle

  • Author: Cliff Brake, last updated: 2022-02-16
  • PR/Discussion:
  • Status: discussion

Context

In the process of implementing a feature to duplicate a node tree, several problems have surfaced related to the lifecycle of creating and updating nodes.

Node creation (current)

  • if a point was sent and node did not exist, SIOT created a "device" node as a child of the root node with this point. This was based on this initial use of SIOT with 1-wire devices.
  • there is also a feature where if we send a point to a Device node that does not have an upstream path to root, or that path is tombstoned, we create this path. This ensures that we don't have orphaned device nodes in an upstream if they are still active. This happens to the root node on clear startup.
  • by the user in the UI -- Http API, /v1/nodes POST, accepts a NodeEdge struct and then sends out node points and then edge points via NATs to create the node.
  • node is sent first, then the edge

The creation process for a node involves:

  1. sending all the points of a node including a meta point with the node type.
  2. sending the edge points of a node to describe the upstream connection

current problems

There are currently two problems:

  1. When creating a node, we send all the node points, then the edge points. However this can create an issue in that an upstream edge for a device node does not exist yet, so in a multi-level upstream configuration A->B->C, if B is syncing to C for the first time, multiple instances of A will be created on C.
  2. If a point is sent for a node that does not exist, a new device node will be created.

experiments

An attempt was made to switch the sending edge points of new nodes before node points, however this created other issues (TODO: detail these).

discussion

Sending node and edge points separately for new nodes creates an issue in that these don't happen in one communication transaction, so there is a period of time between the two where the node state is indeterminate. Consideration was given to adding a NATS endpoint to create nodes where everything could be sent at once. However, this is problematic in that now there is another NATS subject for everyone to listen to and process, rather than just listening for new points. It seems less than ideal to have multiple subjects that can create/modify node points.

It seems at this point we can probably deprecate the feature to create new devices nodes based on a single point. This will force new nodes to be explicitly created. This is probably OK as new nodes are created in several ways:

  1. by the user in the UI
  2. by the upstream sync mechanism -- if the hash does match or a node does not exist upstream, it is sent. This is continuously checked so if a message does not succeed, it will eventually get resent.
  3. plug-n-play discovery mechanisms that detect new devices and automatically populate new nodes. Again, it is not a big deal if a message gets lost as the discovery mechanism will continue to try to create the new device if it does not find it.

Sneding edge before parents can be problematic for things like the client manager that might be listening for tombstone points to detect node creation/deletion.

Decision

Consequences