Introduction

Simple IoT enables you to add remote sensor data, telemetry, configuration, and device management to your project or product.

Implementing IoT systems is hard. Most projects take way longer and cost more than they should. The fundamental problem is getting data from remote locations (edge) to a place where users can access it (cloud). We also need to update data and configuration at the edge in real time from any location. Simple IoT is an attempt to solve these problems by embracing the fact that IoT systems are inherently distributed and building on simple concepts that scale.

Simple IoT provides:

• A single application with no dependencies that can be run in both cloud and edge instances
• efficient synchronization of data in both directions
• Local queuing of data at the edge while a device is offline, and automatic delivery of that backlog when the connection returns
• A flexible UI to view configuration and current values
• A rules engine that runs on all instances that can trigger notifications or set data
• Extensive support for Modbus - both server and client
• Support for the Linux 1-wire subsystem.
• Flexible graph organization of instances, users, groups, rules, and configuration.
• Integration with other services like InfluxDB and Twilio
• A system that is easy to extend in any language using NATS.
• A number of useful Go packages to use in your custom application

See vision, architecture, and integration for addition discussion on these points.

See detailed documentation for installation, usage, and development information.

Motivation

This project was developed while building real-world IoT applications and has been driven by the following requirements:

• Data (state or configuration) can be changed anywhere — at edge devices or in the cloud and this data needs to be synchronized seamlessly between instances. Sensors, users, rules, etc. can all change data. Some edge systems have a local display where users can modify the configuration locally as well as in the cloud. Rules can also run in the cloud or on edge devices and modify data.
• Data bandwidth is limited in some IoT systems — especially those connected with Cat-M modems (< 100Kb/sec). Additionally, connectivity is not always reliable, and systems need to continue operating if not connected.
• Data collected while a device is offline still needs to reach the cloud. Every instance writes to a local store first, so sensor readings, configuration changes, and history are queued while the link is down and delivered in order once it comes back.

Core ideas

The process of developing Simple IoT has been a path of reducing what started as a fairly complex IoT system to simpler ideas. This is what we discovered along the way:

1. Treat configuration and state data the same for purposes of storage and synchronization.
2. Represent this data using simple types (Nodes and Points).
3. Organize this data in a graph.
4. All data flows through a message bus.
5. Run the same application in the cloud and at the edge.
6. Automatically sync common data between instances.

Design is the beauty of turning constraints into advantages.

• Ava Raskin

These constraints have resulted in Simple IoT becoming a flexible distributed graph database optimized for IoT datasets. We’ll explore these ideas more in the documentation.

Support, Community, Contributing, etc.

Pull requests are welcome - see development for more thoughts on architecture, tooling, etc. Issues are labeled with “help wanted” and “good first issue” if you would like to contribute to this project.

For support or to discuss this project, use one of the following options:

• Documentation
• Simple IoT community forum
• open a GitHub issue
• Simple IoT YouTube channel
• Subscribe to our email newsletter for project updates.

If you use this project, please let us know! It is really helpful to hear from users.

License

Apache Version 2.0

Contributors

Thanks to contributors:

Made with contrib.rocks.

Installation

Simple IoT will run on the following systems:

• ARM/x86/RISC-V Linux
• MacOS
• Windows

The computer you are currently using is a good platform to start with as well as any common embedded Linux platform like the Raspberry PI.

If you needed an industrial class device, consider something from embeddedTS like the TS-7553-V2.

The Simple IoT application is a self contained binary with no dependencies. Download the latest release for your platform and run the executable. On Linux and MacOS, the download needs to be marked executable first:

chmod +x simpleiot-vX.Y.Z-linux-x86_64
./simpleiot-vX.Y.Z-linux-x86_64

Renaming it to siot is convenient if you plan to keep it in your PATH.

Once running, you can log into the user interface by opening http://localhost:8118 in a browser. The default login is:

• user: admin
• pass: admin

Simple IoT self-install (Linux only)

Simple IoT self-installation does the following:

• creates a Systemd service file
• creates a data directory
• starts and enables the service

To install as user, copy the siot binary to some location like /usr/local/bin and then run:

siot install

To install as root:

sudo siot install

The default ports are used, so if you want something different, modify the generated siot.service file.

Updating

Simple IoT can update itself to the latest release:

siot update

This downloads the release for the platform it is running on, verifies it against the checksums published with the release, and replaces the binary in place. The new binary is written to the directory the current one lives in, so if Simple IoT is installed somewhere like /usr/local/bin, run sudo siot update. To see what is available without installing it, use:

siot update -check

The new version starts running the next time Simple IoT starts, so if it is installed as a service, restart the service:

systemctl restart siot

Updating replaces the executable and leaves the data directory alone, so configuration and historical data carry forward. The previous binary is removed once the new one is in place, so keep a copy if you want to be able to return to it. On Windows, the previous version is left alongside the new one as siot.exe.old.

Note that siot update updates the Simple IoT application itself. To update the operating system on an embedded device, see the update client.

Cloud/Server deployments

When on the public Internet, Simple IoT should be proxied by a web server like Caddy to provide TLS/HTTPS security. Caddy by default obtains free TLS certificates from Let’s Encrypt and ZeroSSL with automatic fallback if one provider fails.

There are Ansible recipes available to deploy Simple IoT, Caddy, InfluxDB, and Grafana that work on most Linux servers.

• Simple IoT
• Caddy, InfluxDB, Grafana, etc.

Video: Setting up a Simple IoT System in the cloud

Yocto Linux

Yocto Linux is a popular edge Linux solution. There is a BitBake recipe for including Simple IoT in Yocto builds.

Networking

By default, Simple IoT runs an embedded NATS server and the SIOT NATS client is configured to connect to nats://127.0.0.1:4222.

Use Cases

Simple IoT is platform that can be used to build IoT systems where you want to synchronize data between a number of distributed devices to a common central point (typically in the cloud). A common use case is connected devices where users want to remotely monitor and control these devices.

Some examples systems include:

• Irrigation monitoring
• Alarm/building control
• Industrial vehicle monitoring (commercial mowers, agricultural equipment, etc.)
• Factory automation

SIOT is optimized for systems where you run Embedded Linux at the edge and have fairly complex config/state that needs synchronized between the edge and the cloud.

Changes can be made anywhere

Changes to config/state can be made locally or remotely in a SIOT system.

Devices keep working when the connection drops

Edge devices are often on cellular or shared networks where outages are a normal part of operation. A SIOT instance does not depend on its upstream to run: it writes every point to its own local store first, then replicates that store upstream. Sensor readings, rule activity, and configuration changes made while the link is down are queued on disk and delivered in order when it returns. Replication picks up at the point it stopped, so only the missed data is sent, which matters on a metered or low bandwidth connection.

The same applies in the other direction. Configuration changed in the cloud for a device that is offline, or one that has not been deployed yet, waits until the device connects.

How long a device can be offline and still catch up in full depends on how much history the store keeps. The store retains a bounded number of points per value (5000 by default), and the limit is adjustable, so a device that samples slowly can be offline far longer than one writing every second. See Synchronization for setting up an upstream connection, Store for the retention setting, and the synchronization reference for the mechanics of queuing and catch-up.

Integration

There are many ways to integrate Simple IoT with other applications.

There are cases where some tasks like machine learning are easier to do in languages like C++, then you can connect these applications to SIOT via NATS to access config/state. See the Integration reference guide for more detailed information.

Multiple upstreams

Because we run the same SIOT application everywhere, we can add upstream instances at multiple levels.

This flexibility allows us to run rules and other logic at any level (cloud, local server, or edge gateway) - wherever it makes sense.

User Interface

Contents

Basic Navigation

After Simple IoT is started, a web application is available on port :8118 (typically http://localhost:8118). After logging in (default user/pass is admin/admin), you will be presented with a tree of nodes.

The Node is the base unit of configuration. Each node contains Points which describe various attributes of a node. When you expand a node, the information you see is a rendering of the point data in the node.

You can expand/collapse child nodes by clicking on the arrow to the left of a node.

You can expand/edit node details by clicking on the dot to the left of a node.

Adding nodes

Child nodes can be added to a node by clicking on the dot to expand the node, then clicking on the plus icon. A list of available nodes to add will then be displayed:

Some nodes are populated automatically if a new device is discovered, or a downstream device starts sending data.

Deleting, Moving, Mirroring, and Duplicating nodes

Simple IoT provides the ability to re-arrange and organize your node structure.

To delete a node, expand it, and then press the delete icon.

To move or copy a node, expand it and press the copy icon. Then expand the destination node and press the paste icon. You will then be presented with the following options:

• move - moves a node to new location
• mirror - is useful if you want a user or device to be a member of multiple groups. If you change a node, all the mirror copies of the node update as well.
• duplicate - recursively duplicates the copied node plus all its descendants. This is useful for scenarios where you have a device or site configuration (perhaps a complex Modbus setup) that you want to duplicate at a new site.

Raw Node View

If a node is expanded, a raw node button is available that allows you to view the raw type and points for any node in the tree. It is useful at times during development and debugging to be able to view the raw points for a node.

After the raw button is pressed, the type and points are displayed:

Unknown nodes will also be displayed as raw nodes.

Points can also be edited, added, or removed in raw mode.

A custom node type can also be added by specifying the node type when adding a node. This can be useful when developing new clients or external clients that run outside of the SImple IoT application.

Graphing and advanced dashboards

If you need graphs and more advanced dashboards, consider coupling Simple IoT with Grafana. Someday we hope to have dashboard capabilities built in.

Custom UIs

See the frontend reference documentation.

Users/Groups

Users and Groups can be configured at any place in the node tree. The way permissions work is users have access to the parent node and the parent nodes children. In the below example, Joe has access to the SBC device because both Joe and SBC are members of the Site 1 group. Joe does have access to the root node.

If Joe logs in, the following view will be presented:

Schema

The configuration of a group and a user in it:

nodes:
  - group:
      description: Site 1
  - user:
      parent: Site 1
      email: joe@example.com
      firstName: Joe
      lastName: Smith
      pass: his-password
      phone: "+12155551212"
      edgePoints:
        role: admin

A group carries a description and nothing else. Its place in the tree is what gives it meaning, and the users and devices below it are what it groups.

A user is the one node type with no description, so a file finds it by email, and by name when there is no email. phone is written as text so the leading + is kept.

role is admin or user and lives under edgePoints rather than with the points, because a role belongs to the connection between the user and the node above rather than to the user. The same user mirrored into two places can hold a different role in each.

An export carries pass as it was entered, so treat a file that contains user nodes the way you would treat the passwords in it.

Notifications

Notifications are sent to users when a rule goes from inactive to active and contains a notification action. This notification travels up the node graph. At each parent node, users potentially listen for notifications. If a user is found, then a message is generated. This message likewise travels up the node graph. At each parent node, messaging service nodes potentially listen for messages and then process the message. Each node in Simple IoT that generates information is not concerned with the recipient of the information or how the information is used. This decoupling is the essence of messaging based systems (we use NATS) and is very flexible and powerful. Because nodes can be aliased (mirrored) to different places, this gives us a lot of flexibility in how points are processed. The node tree also gives us a very visual view of how things are connected as well as an easy way to expand or narrow scope based on high in the hierarchy a node is placed.

Example

There is hierarchy of nodes in this example system:

• Company XYZ
  • Twilio SMS
  • Plant A
    • Joe
    • Motor overload Rule
    • Line #1
      • Motor Overload
  • Plant B

The node hierarchy is used to manage scope and permissions. The general rule is that a node has access to (or applies to) its parent nodes, and all of its parents dependents. So in this example, Joe has access to everything in Plant A, and likewise gets any Plant A notifications. The Motor overload rule also applies to anything in Plant A. This allows us to write one rule that could apply to multiple lines. The Twilio SMS node processes any messages generated in Company XYZ including those generated in Plant A, Line #1, Plant B, etc. and can be considered a company wide resource.

The process for generating an SMS notification to a user is as follows:

1. Line #1 contains a Motor Overload sensor. When this value changes, a point (blue) gets sent to its parent Line #1 and then to Plant A. Although it is not shown below, the point also gets sent to the Company XYZ and root nodes. Points always are rebroadcast on every parent node back to the root.
2. Plant A contains a rule (Motor Overload) that is then run on the point, which generates a notification (purple) that gets sent back up to its parent (Plant A).
3. Plant A contains a user Joe so a notification + user generates a message (green), which gets sent back upstream to Plant A and then to Company XYZ.
4. Company XYZ contains a messaging service (Twilio SMS), so the message gets processed by this service an SMS message gets sent to Joe.

The Motor Overload sensor node only generates what it senses. The Motor Overload rule listens for points in Plant A (its parent) and processes those points. The Joe user node listens for points at the Plant A node (its parent) and processes any points that are relevant. The Twilio SMS node listens for point changes at the Company XYZ node and processes those points. Information only travels upstream (or up the node hierarchy).

In this example, the admin user does not receive notifications from the Twilio SMS messaging service. The reason is that the Twilio SMS node only listens for messages on its parent node. It does not have visibility into messages sent to the root node. With the node hierarchy, we can easily partition who gets notified. Additional group layers can be added if needed. No explicit binding is required between any of the nodes - the location in the graph manages all that. The higher up you go, the more visibility and access a node has.

Clients

Simple IoT is a framework that allows for clients to be added to manage IO, run rules, process data, etc. See documentation for individual clients. If you would like to develop a custom client, see the client reference documentation.

CAN Bus Client

The CAN client allows loading a standard CAN database file, receiving CAN data, and translating the CAN data into points via the database.

Usage

The CAN client can be used as part of the SimpleIoT library or through the web UI. The first step in either case is to create a CAN database in .kbc format.

Create the CAN Database

Create a file in the folder with the Go code named “test.kcd” containing the following:

<NetworkDefinition xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://kayak.2codeornot2code.org/1.0" xsi:schemaLocation="Definition.xsd">
  <Document name="Some Document Name">some text</Document>
  <Bus name="sampledatabase">
    <Message id="0x123" name="HelloWorld" length="8">
      <Notes></Notes>
      <Signal name="Hello" offset="0" length="8"/>
      <Signal name="World" offset="8" length="8"/>
    </Message>
    <Message id="0x12345678" name="Food" length="8" format="extended">
      <Notes></Notes>
      <Signal name="State" offset="0" length="32"/>
      <Signal name="Type" offset="32" length="32"/>
    </Message>
  </Bus>
</NetworkDefinition>

You can create any CAN database you want by crafting it in Kvaser’s free DBC editor and then using the canmatrix tool to convert it to KCD format. Note that canmatrix does not support all features of the DBC and KCD formats.

Next, setup the virtual SocketCan interface.

Setup Virtual CAN Interface

Run this in the command line. Reference

sudo modprobe vcan
sudo ip link add dev vcan0 type vcan
sudo ip link set up vcan0

Option #1 - Use In Web UI

Follow the instructions to install SimpleIoT, run it, and navigate to the web UI.

Expand the root node and click the + symbol to add a sub node. Select “CAN Bus” and click “add”.

Configure the CAN Bus node with a File subnode and upload the .kcd file you created.

Once the file has been uploaded, you should see the following stats in the CAN bus node:

Messages in db: 2 Signals in db: 4

Test with Messages

In a separate terminal:

cansend vcan0 123#R{8}
cansend vcan0 12345678#DEADBEEF

Ensure that there are no errors logged in the terminal by the application.

In the Web UI you should see the "Db msgs received" field increase to 2.

Option #2 - Use As Library

Copy this code to a Go file on your Linux machine in a folder by itself.

package main

import (
	"log"

	"github.com/nats-io/nats.go"
	"github.com/simpleiot/simpleiot/client"
	"github.com/simpleiot/simpleiot/data"
	"github.com/simpleiot/simpleiot/server"
)

// exNode is decoded data from the client node
type exNode struct {
	ID          string `node:"id"`
	Parent      string `node:"parent"`
	Description string `point:"description"`
	Port        int    `point:"port"`
	Role        string `edgepoint:"role"`
}

// exNodeClient contains the logic for this client
type exNodeClient struct {
	nc            *nats.Conn
	config        client.SerialDev
	stop          chan struct{}
	stopped       chan struct{}
	newPoints     chan client.NewPoints
	newEdgePoints chan client.NewPoints
	chGetConfig   chan chan client.SerialDev
}

// newExNodeClient is passed to the NewManager() function call -- when
// a new node is detected, the Manager will call this function to construct
// a new client.
func newExNodeClient(nc *nats.Conn, config client.SerialDev) client.Client {
	return &exNodeClient{
		nc:            nc,
		config:        config,
		stop:          make(chan struct{}),
		newPoints:     make(chan client.NewPoints),
		newEdgePoints: make(chan client.NewPoints),
	}
}

// Start runs the main logic for this client and blocks until stopped
func (tnc *exNodeClient) Run() error {
	for {
		select {
		case <-tnc.stop:
			close(tnc.stopped)
			return nil
		case pts := <-tnc.newPoints:
			err := data.MergePoints(pts.ID, pts.Points, &tnc.config)
			if err != nil {
				log.Println("error merging new points:", err)
			}
			log.Printf("New config: %+v\n", tnc.config)
		case pts := <-tnc.newEdgePoints:
			err := data.MergeEdgePoints(pts.ID, pts.Parent, pts.Points, &tnc.config)
			if err != nil {
				log.Println("error merging new points:", err)
			}
		case ch := <-tnc.chGetConfig:
			ch <- tnc.config
		}
	}
}

// Stop sends a signal to the Run function to exit
func (tnc *exNodeClient) Stop(err error) {
	close(tnc.stop)
}

// Points is called by the Manager when new points for this
// node are received.
func (tnc *exNodeClient) Points(id string, points []data.Point) {
	tnc.newPoints <- client.NewPoints{id, "", points}
}

// EdgePoints is called by the Manager when new edge points for this
// node are received.
func (tnc *exNodeClient) EdgePoints(id, parent string, points []data.Point) {
	tnc.newEdgePoints <- client.NewPoints{id, parent, points}
}

func main() {
	nc, root, stop, err := server.TestServer()

	if err != nil {
		log.Println("Error starting test server:", err)
	}

	defer stop()

	canBusTest := client.CanBus{
		ID:          "ID-canBus",
		Parent:      root.ID,
		Description: "vcan0",
		Device:      "vcan0",
	}

	err = client.SendNodeType(nc, canBusTest, "test")
	if err != nil {
		log.Println("Error sending CAN node:", err)
	}

	// Create a new manager for nodes of type "testNode". The manager looks for new nodes under the
	// root and if it finds any, it instantiates a new client, and sends point updates to it
	m := client.NewManager(nc, newExNodeClient)
	m.Start()

	// Now any updates to the node will trigger Points/EdgePoints callbacks in the above client
}

Run the following commands:

• go mod init example.com/m
• go run <file>.go
• Run the go get commands suggested by go run
• go mod tidy
• go run <file>.go

Run it!

go run <file.go>

Follow instructions from the “Test with Messages” section above.

Schema

The configuration of a CAN bus node and the file node holding its database:

nodes:
  - canBus:
      bitRate: "250000"
      description: Vehicle bus
      device: can0
      children:
        - file:
            binary: 0
            data: |
              <NetworkDefinition xmlns="http://kayak.2codeornot2code.org/1.0">
                <Bus name="sampledatabase">
                  ...
                </Bus>
              </NetworkDefinition>
            description: Vehicle database
            name: vehicle.kcd

device is the SocketCAN interface name and bitRate is text, so it is quoted.

The database rides along in the data point of the child file node, which is what makes an export of a CAN node enough to recreate it elsewhere. See the File client for the rest of that node.

The message and signal counts and the received counts shown in the UI are points the client maintains, so an export of a running node carries them as well.

Future Work

• Scale and translate messages based on scale and offset parameters in database
• Auto connect to CAN bus in case it is brought up after SIOT client is started
• Attempt to bring up CAN bus within client, handle case where it is already up
• Support multiple CAN database files per node (be selective in which internal db is updated when a name or data point is received in the client)
• Support sending messages (concept of nodes and send/receive pulled from databases??)
• Support .dbc file format in addition to .kcd
• Add the concept of a device to the CAN message points

File

The file node can be used to store files that are then used by other nodes/clients. Some examples include the CAN and Serial clients.

The default max payload of NATS is 1MB, so that is currently the file size limit, but NATS can be configured for a payload size up to 64MB. 8MB is recommended.

See the Frontend documentation for more information how the file UI is implemented.

If the Binary option is selected, the data is base64 encoded before it is transmitted and stored.

Schema

The configuration of a file node:

nodes:
  - file:
      binary: 0
      data: |
        nodes:
          - group:
              description: Tank farm
      description: Tank farm groups
      name: 10-groups.yaml

description is what names the node in the tree and name is the file name, so the two are separate and either can change without the other. The contents live in the data point, written as a YAML block scalar when the file has several lines, and base64 encoded first when binary is set.

The client maintains three more points, so an export of a running node carries them as well and an imported file settles on the right values without them being given: hash is the MD5 of the contents, size is their length in bytes, and created is a Unix timestamp written once when the node comes into existence, which is what orders provisioning files uploaded through the UI.

Database Client

The main SIOT store is NATS JetStream, which retains a bounded history of points for each node. For long-term storage, dashboards, and ad-hoc queries, a Database client can forward points to an external time-series database.

VictoriaMetrics is the primary time-series store for SIOT. The Database client speaks the InfluxDB v2 write API, which VictoriaMetrics supports, so InfluxDB 2.x can also be used.

Reliable delivery with durable consumers

The Database client reads points from the store’s JetStream streams using durable consumers rather than subscribing to live message traffic. A durable consumer is a named position in a stream that the NATS server persists to disk alongside the stream data. The client acknowledges each message only after the database has accepted the points it carried, and the server advances the saved position only on acknowledgment.

This makes delivery resumable. If the Database client, the SIOT instance, or the connection to the database is down for a period of time, points continue to accumulate in the streams. When the client comes back, delivery resumes from the saved position and the missed points are written to the database. Each Database node keeps its own position, so multiple Database clients can consume the same streams independently.

Two limits apply:

• Stream retention bounds how far behind a client can fall. By default the store keeps the last 5000 points per subject (one subject is one point type and key on one node). If a client is down long enough that a signal exceeds this limit, the oldest points for that signal are dropped from the stream and will be missing from the database.
• High-rate points are not stored in streams. They are delivered live and are not recovered after downtime, including downtime of the database itself.

A newly added Database node starts recording from the present; it does not backfill history already in the streams. Streams that appear after the client starts, such as the replica stream for a newly adopted device, are consumed from their beginning so the device’s initial catch-up is captured.

When the database is unavailable

The same mechanism covers an outage in the time-series database itself. If VictoriaMetrics is stopped, restarted, upgraded, or simply unreachable across the network, points sent during that window are written once it comes back and the recorded history has no gap.

The stream that already holds the points serves as the buffer, and the client’s saved consumer position records how far it has gotten. There is no separate spool file or in-memory queue in the client to size or manage. The sequence is:

1. The client collects points into batches of up to 500 and writes at least once a second, so each batch travels as a single write request.
2. It acknowledges the stream messages behind a batch only after the database accepts the write. Until then the points remain in the stream.
3. A failed batch returns to the stream with a retry delay that starts at about a second and doubles up to a maximum of one minute. The client makes no further connection attempts until the delay expires, so an outage lasting hours costs about one attempt per minute.
4. Meanwhile points keep arriving and accumulate. Up to 5000 may be outstanding at once; beyond that, JetStream stops delivering to this client and the rest wait in the stream.
5. When the database answers again, the held points are redelivered and written. Each point carries its original timestamp, so the history fills in at the times the readings happened.

Restarting SIOT during an outage is safe: none of the affected points were acknowledged, so they are still in the stream and arrive again on the next run. Stopping the client returns anything it had taken from the stream but had not yet written.

Rejections work differently. Bad credentials or a line the database cannot parse would fail identically on every attempt, so the client logs those points and drops them instead of blocking everything behind them.

Three log messages describe this behavior, each prefixed with the Database node’s description:

Db client site db: database write failed, holding points in the stream until it recovers: ...
Db client site db: database write succeeded after 4 failed attempts
Db client site db: dropping 37 points the database rejected: ...

The first appears once when an outage starts, not on every attempt. The second confirms recovery and how many attempts it took, and the third reports points that were discarded.

The limits above still apply: an outage that outlasts stream retention for a fast-changing signal loses that signal’s oldest points, and high-rate points are not buffered at all.

Choosing a database type

Add a Database node and choose the database type: InfluxDB 2.x or Victoria Metrics. Both are written using the InfluxDB version 2 line protocol, so the connection settings are similar, but the two differ in what they store and in how you graph the result. Existing Database nodes have no database type set and continue to behave as InfluxDB.

Victoria Metrics

Set the database type to Victoria Metrics and set the URI to the write endpoint, typically http://myserver:8428 for a single-node instance. VictoriaMetrics has no concept of an organization or a bucket, so those fields are hidden when this type is selected.

A single-node VictoriaMetrics has no authentication on the write path, so the Auth Token field can be left blank. VictoriaMetrics expects authentication to be handled by vmauth or vmgateway in front of it. The client sends the token as an Authorization: Token <token> header, which is one of the formats vmauth accepts, so setting the Auth Token here works when you point the URI at vmauth.

VictoriaMetrics does not support storing strings; it converts any non-numeric field value to 0. The client therefore writes only the numeric value field and skips string points, which keeps a points_text series of zeros out of the database. If you want to filter or graph on a value, publish it as a number. The GPS client does this for its fix status points for exactly this reason.

Each point arrives in VictoriaMetrics as the metric points_value, with the point type and key and all of the node tags described below available as labels.

Query latency offset

New points typically reach VictoriaMetrics within a second, because the write client sends a batch every second (or sooner once 500 points accumulate). Queries, however, do not see them for another 30 seconds by default: VictoriaMetrics shifts the end of every query range back by -search.latencyOffset, which defaults to 30s so that slow Prometheus scrapes are still counted.

To see data as soon as it is written, start VictoriaMetrics (or vmselect in a cluster) with:

victoria-metrics -search.latencyOffset=0s

Use a small value such as 1s if the clocks on the writing devices and the database server may differ slightly. The offset can also be set per request with a latency_offset=0s URL parameter, which in Grafana can be added to the data source’s custom query parameters when changing the server flag is not an option.

Setting the flag under systemd

Most Linux packages run VictoriaMetrics from a systemd unit that reads extra flags from an environment file, so the flag belongs there rather than in the unit itself. Check which file your unit uses:

systemctl cat victoriametrics.service

A packaged unit typically contains lines such as:

EnvironmentFile=/etc/default/victoriametrics
ExecStart=/usr/bin/victoria-metrics -storageDataPath /var/lib/victoriametrics $ARGS

Add the flag to the variable that ExecStart expands, ARGS in this example, by editing /etc/default/victoriametrics:

ARGS="-search.latencyOffset=0s"

Separate additional flags with spaces inside the quotes. Then restart the service and confirm the setting, which appears in the list of flags that differ from their defaults:

sudo systemctl restart victoriametrics
curl -s localhost:8428/flags

Keeping the flag in the environment file means a package upgrade can replace the unit without discarding the setting. Distributions vary: some use /etc/sysconfig/victoriametrics or a different variable name, and a unit with no EnvironmentFile needs a drop-in override created with sudo systemctl edit victoriametrics.service that sets ExecStart to the full command line.

Graphing Victoria Metrics data

Use Grafana with a Victoria Metrics (Prometheus-compatible) data source and query points_value with MetricsQL. See the Graphing documentation for how the node tags map to graph labels.

InfluxDB 2.x

Point data can also be stored in an InfluxDB 2.x database by adding a Database node:

Tags

Tags are the labels you filter and group by when querying or graphing. Every point written to the database carries the point’s own type and key, plus three tags describing the node that emitted it:

• node.id, the node’s ID (typically a UUID)
• node.type, the node type, such as signalGenerator or modbusIo
• node.description, the node’s Description field

These are always present and need no configuration. Anything beyond them (which machine a reading came from, which site a machine sits at) is added by turning node points into tags, described next.

Adding custom tags

Custom tags come from points on the node, so adding one takes two steps: put the point on the node that should carry the label, then tell the Database node which point types become tags.

Step 1: add a tag point to the node. Most node types have a Tags field with an Add Tag button. Enter a name, which becomes the tag’s key, then fill in its value. Naming a tag machine and setting it to press-3 adds a tag point with key machine and text press-3 to that node. The example below adds a machine tag to the signal generator producing the data.

Step 2: list the point type on the Database node. The client turns a point into a tag only when its type appears in this list. Open the Database node, find Tag Point Types, press Add Point Type, and enter tag. This is the point type, not the tag name, so the single entry tag covers every tag added through the Tags field, however many there are.

Result. Points flowing through the client now carry the tag, named node.<point type>.<point key>. A tag named machine added through the Tags field is written as node.tag.machine, since the point type is tag and the point key is machine:

The naming rule also covers point types other than tag. If a node has a machine point and you add machine to Tag Point Types, its points are written as node.machine.<key>. Listing a type that a node does not have is harmless: it contributes no tag.

Two things to know when planning tags:

• Tags apply going forward. Adding or editing a tag starts a new series in the database from that moment, and a query spanning the change sees both the old and the new series. The same is true of node.description. Settle on tag names before collecting history you intend to keep.
• Adding tags is inexpensive. The database indexes tag values and stores each distinct string once, so a descriptive tag repeated across millions of samples costs far less than its length suggests.

See the Graphing documentation for how to map these tags to graph labels.

Tag inheritance

One tag point can cover a whole subtree, so step 1 rarely needs repeating on every node. Tag points are inherited from ancestor nodes, so a label can be set once on the node that represents the thing being described (a machine group, a device, a site), and every point emitted beneath it carries that tag. Set site on the device node instead of on each of its sensors. For example, with tag listed in Tag Point Types:

device        tag: site=plant-a, customer=acme
└── press-3   tag: machine=press-3, site=plant-b
    └── temp-1    tag: sensor=inlet

a point emitted by temp-1 is written with node.tag.sensor=inlet, node.tag.machine=press-3, node.tag.site=plant-b, and node.tag.customer=acme.

The resolution rules are:

• All tags resolve into the same flat node.<point type>.<point key> namespace, so queries do not depend on the depth at which a tag was set.
• When the same tag is defined at more than one level, the value closest to the emitting node wins, so a local tag overrides an inherited one (site above).
• Inheritance stops at the Database client’s parent node, whose own tags are included. Nodes above the Database client’s scope do not contribute tags.
• A node can have more than one parent. When two ancestors at the same depth define the same tag, the node with the lowest ID wins, and the client logs the ambiguity the first time it is seen.
• node.id, node.type, and node.description always describe the emitting node and are never inherited.

Schema

Below is an export of a Victoria Metrics node and an InfluxDB node:

nodes:
  - db:
      dbType: victoriaMetrics
      description: Victoria Metrics
      tagPointType: tag
      uri: http://localhost:8428
  - db:
      authToken: T0k3n
      bucket: siot
      dbType: influxdb
      description: InfluxDB
      org: bec
      tagPointType:
        - machine
        - tag
      uri: http://localhost:8086

dbType is victoriaMetrics or influxdb; a node with no dbType behaves as InfluxDB. org and bucket apply to InfluxDB alone, and Victoria Metrics nodes leave them out.

tagPointType is the Tag Point Types field described above. It is a list, so a single point type is written as one value and several are written as a sequence. Each entry is a point type, and the client adds it to every sample as node.<point type>.<point key>.

An export carries authToken as it was entered, so treat a file that contains database nodes the way you would treat the token itself.

GPS Client

The GPS client reads position data and publishes it as points on a gps node. It supports three sources:

• Serial reads NMEA sentences directly from a receiver on a serial port.
• gpsd subscribes to the gpsd daemon over TCP. This is a good choice on a Linux system where gpsd already manages the receiver, where several processes need the same position, or where gpsd’s device detection and driver support are useful.
• Simulated generates a plausible track without any hardware, which makes it easy to develop rules, dashboards, and graphs.

All three sources publish the same points, so anything consuming the data works the same way no matter which source is configured.

Configuration

Select the source first. The remaining fields change to match it.

Serial

The client reopens the port automatically when a receiver is unplugged and plugged back in, so a USB receiver can be moved without restarting SimpleIoT.

The client reads the GGA, GSA, RMC, and VTG sentences and ignores the rest. A receiver reports one position across several of these, so the client collects a full cycle of sentences and publishes them together. Every point from one position therefore carries the same timestamp, which is what makes the data plottable on a map.

gpsd

The client reconnects with a growing delay whenever the daemon becomes unreachable. It also reports itself disconnected if no position arrives for ten seconds, because gpsd keeps the connection open when a receiver goes quiet or is unplugged.

Simulated

The heading drifts randomly within the configured rate, so the track wanders naturally instead of running straight or jumping between positions. Set the heading drift to 0 for a straight track.

Positions follow a great circle, so tracks behave correctly at high latitudes and when crossing the antimeridian.

The simulator continues from the node’s last published position, so changing the speed, the update period, or restarting SimpleIoT picks the track up where it left off rather than returning to the start. Switching a node from a hardware source to the simulator likewise continues from the last real position.

Two things send the simulated receiver back to the configured start position: pressing Reset location, and editing Start latitude, Start longitude, or Start heading, which moves the receiver to the position just entered.

The simulator reports a normal GPS fix rather than marking its data as simulated, so rules and dashboards behave exactly as they would with real hardware. The node’s source setting is what identifies the data as synthetic.

Debug Levels

The Debug level field controls how much the client logs. Every source logs connection changes and configuration problems whatever the level is set to.

Level 2 is the one to reach for when a receiver is connected but no position appears, since it names the sentence or report that could not be used. Levels 3 and above 4 behave the same as 2 and 4 respectively.

A new level takes effect on the next message. The source keeps running, so raising the level while chasing a problem leaves the serial port open, the gpsd session connected, and the simulated track where it is.

Level 4 is verbose. A receiver at the default one second period sends several sentences per fix, so expect a handful of lines every second. The simulator has no raw input to show, so it logs the points it generated for each fix instead, which is a way to watch a track advance without querying the node.

Published Points

A source publishes only what it actually reports. A receiver that sends no GSA sentences, for example, leaves fixType unset; no value is guessed for it.

Fix Type and Fix Quality

The three sources describe a fix in three different vocabularies, so the client normalizes them into two points. fixType covers whether the fix is 2D or 3D, which determines whether the altitude can be trusted. fixQuality covers which augmentation produced the fix, which determines how accurate the position is.

Both are stored as numbers rather than as text, which lets them be graphed and keeps them intact in databases that store only numeric values. The web UI displays them as labels.

fixType follows the gpsd encoding:

fixQuality follows the NMEA GGA encoding:

Values 7 and 8 are available through gpsd. The NMEA library SimpleIoT uses validates fix quality against the range 0 to 6, so a serial receiver reporting 7 or 8 is treated as reporting no fix. Both values are rare enough from real receivers that this is unlikely to come up in practice.

A Note on Altitude

The serial source reports altitude above mean sea level, taken from the GGA sentence. The gpsd source prefers gpsd’s altMSL field, which is the same measurement, and falls back to altHAE when that is all the daemon provides. altHAE is measured from the WGS84 ellipsoid instead, and the two differ by the local geoid separation, which reaches tens of meters in some parts of the world. If altitude accuracy matters for your application, check which field your gpsd version reports.

Storing and Graphing

Add a database node to store GPS points in InfluxDB or Victoria Metrics. All the numeric points listed above are stored normally.

Plotting a Track on a Map

Grafana’s Geomap panel needs latitude and longitude as two numeric fields on the same row. The GPS client stamps every point from one position with the same timestamp, which is what allows the two to be brought back together.

With InfluxDB, a Flux pivot() puts them into one row directly:

from(bucket: "siot")
  |> range(start: v.timeRangeStart, stop: v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "points" and
    r._field == "value" and
    (r.type == "latitude" or r.type == "longitude"))
  |> filter(fn: (r) => r["node.description"] == "My GPS")
  |> pivot(rowKey: ["_time"], columnKey: ["type"], valueColumn: "_value")

Set the panel’s Map Layer to Coordinates, and the latitude and longitude fields to the pivoted columns.

With Victoria Metrics, add one query per field. Set each query’s Legend to the field name, which is how the Geomap panel finds the coordinates later.

Each query selects one point type from the GPS nodes:

max by(node.description) (points_value{node.type="gps", node.description="$node", type="latitude"})
max by(node.description) (points_value{node.type="gps", node.description="$node", type="longitude"})
max by(node.description) (points_value{node.type="gps", node.description="$node", type="speed"})

$node is a dashboard variable holding the GPS node description, so one dashboard can serve several receivers. max by(node.description) collapses the result to one series per node.

The tag names the database client writes contain dots, as in node.type and node.description. Victoria Metrics accepts these as label names and they can be used directly in a query, as above.

Set Step to the GPS update rate, 1s in the example. A larger step samples the track instead of drawing every position.

Then add two transformations:

1. Join by field in Outer (time series) mode on Time. This is where the shared timestamp matters: the separate series line up onto single rows only because every point in one fix carries the same time.
2. Organize fields by name, which sets the field order and confirms the names carried over from the query legends.

Finally, set the panel’s Map Layer to Coordinates. Grafana locates the latitude and longitude fields by name.

To display the fix codes as labels on a Grafana panel, add value mappings using the tables above.

Schema

Below is an export of a simulated GPS node:

nodes:
  - gps:
      altitude: 85.3
      connected: 1
      description: Test track
      fixQuality: 1
      fixType: 3
      gpsSource: sim
      hdop: 0.92
      heading: 92.4
      latitude: 40.03567
      longitude: -75.52006
      numSat: 11
      period: 1
      simHeading: 90
      simHeadingRate: 5
      simLatitude: 40.0354
      simLongitude: -75.5198
      simSpeed: 12
      speed: 12

Modbus

Modbus is popular data communications protocol used for connecting industrial devices. The specification is open and available at the Modbus website. See also this Modbus Overview.

Simple IoT can function as both a Modbus client or server and supports both RTU and TCP transports. Modbus client/server is used as follows:

• Client: typically a PLC or Gateway - the device reading sensors and initiating Modbus transactions. This is the mode to use if you want to read sensor data and then process it or send to an upstream instance.
• Server: typically a sensor, actuator, or other device responding to Modbus requests. Functioning as a server allows SIOT to simulate Modbus devices or to provide data to another client device like a PLC.

Modbus is a prompt response protocol. With Modbus RTU (RS485), you can only have one client (gateway) on the bus and multiple servers (sensors). With Modbus TCP, you can have multiple clients and servers.

Modbus is configured by adding a Modbus node to the root node or to any group below it, and then adding IO nodes to the Modbus node.

The Response timeout parameter determines how long the Modbus client will wait for a response from a device. The default is 100ms, which is adequate for most devices, but it can be increased if you are communicating with a slow device.

Modbus IOs can be configured to support most common IO types and data formats:

The Scale and Offset parameters convert between the raw register value and the value stored in the node: value = raw * scale + offset. A scale of zero is treated as one, so an IO with no scale entered still reads its register.

Adding or removing an IO restarts the bus, which reopens the port. A Modbus server drops the connections it holds when this happens. This applies when a person edits the configuration, not during normal polling.

Schema

The configuration of an RTU client bus with one IO, and of a TCP server:

nodes:
  - modbus:
      baud: "9600"
      clientServer: client
      debug: 0
      description: Sensor bus
      disabled: 0
      pollPeriod: 500
      port: /dev/ttyUSB0
      protocol: RTU
      timeout: 100
      children:
        - modbusIo:
            address: 3
            dataFormat: uint16
            description: Tank level
            disabled: 0
            id: 1
            modbusIoType: modbusHoldingRegister
            offset: 0
            readOnly: 1
            scale: 0.1
            units: cm
  - modbus:
      clientServer: server
      description: PLC facing
      id: 5
      port: "502"
      protocol: TCP
      timeout: 100

clientServer is client or server and protocol is RTU or TCP. Which of the remaining connection settings apply follows from those two: an RTU bus uses port and baud, a TCP server uses port as the port it listens on, and a TCP client uses uri, written as host:port. port and baud are text, so both are quoted, including a TCP port number.

id is the Modbus device address rather than a node ID, which is why it is spelled like any other point. A server carries it on the bus node, and a client carries it on each IO, so one client bus can address several devices.

pollPeriod applies to a client and timeout to both, and both are in milliseconds. A timeout of zero or less is replaced with 100.

modbusIoType is one of modbusDiscreteInput, modbusCoil, modbusInputRegister, or modbusHoldingRegister. dataFormat is uint16, int16, uint32, int32, or float32, and it applies to the register types along with scale, offset, and units.

The values read and written, the error counts, and the connection state are points the client maintains, so an export of a running bus carries them as well.

Videos

Simple IoT Integration with PLC Using Modbus

Simple IoT upstream synchronization support

Simple IoT Modbus Demo

1-Wire

(note, this client has been refactored, but not tested. Testing is welcome …)

1-Wire is a device communication bus that provides low-speed data over a single conductor. It is also possible to power some devices over the data signal as well, but often a third wire is run for power.

Simple IoT supports 1-wire buses controlled by the 1-wire (w1) subsystem in the Linux kernel.

To use a bus, add a 1-Wire node where you want it in the tree and set its Index to the number of the bus controller, which matches the w1_bus_master<index> directory the kernel creates in /sys/bus/w1/devices. The first controller is index 0. Simple IoT then detects the sensors on that bus and creates a node for each one.

Bus Controllers

Raspberry PI GPIO

There are a number of bus controllers available but one of the simplest is a GPIO on a Raspberry PI. To enable, add the following to the /boot/config.txt file:

dtoverlay=w1-gpio

This enables a 1-wire bus on GPIO 4.

To add a bus to a different pin:

dtoverlay=w1-gpio,gpiopin=x

A 4.7kΩ pull-up resistor is needed between the 1-wire signal and 3.3V. This can be wired to a 0.1 inch connector as shown in the following schematic:

See this page for more information.

1-Wire devices

DS18B20 Temperature sensors

Simple IoT currently supports 1-wire temperature sensors such as the DS18B20. This is a very popular and practical digital temperature sensor. Each sensor has a unique address so you can address a number of them using a single 1-wire port. These devices are readily available at low cost from a number of places including eBay - search for DS18B20, and look for an image like the below:

Readings are in degrees Celsius by default. Set Units on a device node to F to report degrees Fahrenheit instead.

Schema

The configuration of a 1-wire bus and one of its devices:

nodes:
  - oneWire:
      debug: 0
      description: Tank sensors
      disabled: 0
      index: 0
      pollPeriod: 3000
      children:
        - oneWireIO:
            description: Tank top
            disabled: 0
            id: 28-0000073b6f4d
            units: F

index is the number of the bus controller, matching the w1_bus_master<index> directory in /sys/bus/w1/devices. pollPeriod is in milliseconds and defaults to 3000 when it is zero or missing.

id on a device is its 1-wire address rather than a node ID, which is why it is spelled like any other point. Simple IoT creates a device node for each sensor it detects on the bus, so these usually arrive on their own; what a file adds is a lasting description and, where wanted, units.

Leaving units out reports degrees Celsius. The readings and error counts are points the client maintains, so an export of a running bus carries them as well.

Messaging Services

SIOT can support multiple messaging services.

Twilio SMS Messaging

Simple IoT supports sending SMS messages using Twilio’s SMS service. Add a Messaging Service node and then configure.

Email Messaging

will be added soon …

Schema

Below is an export of a Twilio messaging service node:

nodes:
  - msgService:
      authToken: your-twilio-auth-token
      description: Twilio SMS
      from: "+12155551212"
      service: twilio
      sid: ACxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

service selects the service; twilio is what is supported today and smtp is reserved for email. sid and authToken are the Twilio account SID and auth token, and from is the number messages are sent from, written as text so the leading + is kept.

Where the node sits in the tree decides which messages it processes, as described in the notifications documentation, so a service that serves a whole company belongs on the company group rather than on any one device.

An export carries authToken as it was entered, so treat a file that contains messaging service nodes the way you would treat the token itself.

MCU Devices

Microcontroller (MCU) devices can be connected to Simple IoT systems via various serial transports (RS232, RS485, CAN, and USB Serial). The Arduino platform is one example of an MCU platform that is easy to use and program. Simple IoT provides a serial interface module that can be used to interface with these systems. The combination of a laptop or a Raspberry PI makes a useful lab device for monitoring analog and digital signals. Data can be logged to InfluxDB and viewed in the InfluxDB Web UI or Grafana. This concept can be scaled into products where you might have a Linux MPU handling data/connectivity and a MCU doing real-time control.

See the Serial reference documentation for more technical details on this client.

File Download

Files (or larger chunks of data) can be downloaded to the MCU by adding a File node to the serial node. Any child File node will then show up as a download option.

​

Protocols

The serial client speaks two wire protocols, selected with the Protocol setting on the node.

Binary is the default and is what existing nodes use. Points are exchanged as COBS-framed packets with a sequence number and a CRC. It is compact, and it supports high-rate data and file transfer. See the Serial reference documentation for the packet format.

Zephyr shell exchanges points as lines of ASCII over an MCU’s console shell. Everything on the wire is readable, so the same link you use for debugging is the link Simple IoT uses for data. Choose this when your firmware already has a Zephyr shell, or when bringing up a new board where being able to see and type on the link matters more than efficiency.

An empty Protocol value means binary, so nodes created before shell mode existed keep working unchanged.

Shell protocol

The MCU emits each point as a line, and Simple IoT writes points back using the p command the Zephyr firmware already registers:

pt uptime 0 INT 3600                              MCU to Simple IoT
p description 0 STR "lab bench" 2026-07-31T12:00:00.000000000Z

Both directions use the same fields and differ only in the verb, so an emitted line becomes a replayable command by changing one character. The verbs differ deliberately: Simple IoT must never mistake an echoed command for a point report.

Anything on the console that is not a point line is tolerated. Zephyr log messages become log points, and the boot banner, shell prompt, and command output are ignored rather than counted as errors.

High-rate data, file transfer, and packet acknowledgement are not available in shell mode, and the UI hides those controls when it is selected. Nodes that need them should stay on the binary protocol.

Timeout is how many seconds the link may be silent before the node is marked not connected, defaulting to 60. An open serial port says nothing about whether anything is alive on the other end, particularly on a USB port that survives an MCU reset.

Log Console Output

Log console output mirrors every line the MCU prints to the Simple IoT server log, tagged with the node description. Shell protocol only.

Once Simple IoT holds the serial port nothing else can read it, so this is what keeps the board observable. It works headless, lands in the journal when Simple IoT runs under systemd, and lets you watch a board boot in the terminal you started the server in. With it on, a separate terminal program is only needed before a node is attached to the port at all.

It is deliberately not a debug level: watching a board boot and diagnosing why a point is not arriving are different questions, and a single verbosity dial would force one to imply the other. Expect it to be loud on a board with network logging enabled.

Debug Levels

You can set the following debug levels to log information.

Binary protocol:

• 0: no debug information
• 1: log ASCII strings (must be COBS wrapped) (typically used for debugging code on the MCU)
• 4: log points received or sent to the MCU
• 8: log cobs decoded data (must be COBS wrapped)
• 9: log raw serial data received (pre-COBS)

Shell protocol:

• 0: no debug information
• 2: log malformed point lines, oversize lines, and warnings about points the MCU will truncate
• 4: log points decoded and each p command written to the MCU
• 9: log raw serial data received, before line assembly

Level 1 has no shell-mode meaning; console output is the separate checkbox described above, so the two can be used independently.

Schema

The configuration of a serial node using the shell protocol, with a file node available for download to the MCU:

nodes:
  - serialDev:
      baud: "115200"
      debug: 0
      description: Lab bench
      disabled: 0
      logConsole: 1
      maxMessageLength: 1024
      port: /dev/ttyACM0
      protocol: shell
      syncParent: 0
      timeout: 60
      children:
        - file:
            binary: 1
            data: SGVsbG8sIE1DVQ==
            description: Calibration table
            name: cal.bin

port and baud are text, so both are quoted. protocol is binary or shell, and an empty value means binary, so nodes created before shell mode existed carry no protocol point at all.

timeout is in seconds and maxMessageLength in bytes. logConsole applies to the shell protocol alone.

The counts, the connection state, the uptime, and the log line shown in the UI are points the client maintains, so an export of a running node carries them as well.

A node sending high rate data also carries an hrDest point holding the ID of the destination node. Unlike a point of type nodeID, it is written as the ID rather than as a description, so it names a node in the instance it was exported from.

Zephyr Examples

The zephyr-siot repository contains examples of MCU firmware that can interface with Simple IoT over serial, USB, and Network connections. This is a work in progress and is not complete.

Arduino Examples (no longer maintained)

Several Arduino examples are available that can be used to demonstrate this functionality.

Metrics

An important part of maintaining healthy systems is to monitor metrics for the application and system. SIOT can collect metrics for:

• the system
• the SIOT application
• any named processes

For the named process, if there are multiple processes of the same name, then we add values for all processes found.

System Metrics

Thermal Metrics

On Linux systems, the system metrics also include the thermal state of the board:

• Temperature comes from the hwmon sensors and from the thermal zones in /sys/class/thermal. Both are read because many SoCs expose their board sensors through hwmon while reporting the CPU, SoC, and junction temperatures through the zones alone. On a Jetson AGX Orin, for example, tj-thermal is the junction reading that governs throttling.
• Fan RPM and PWM come from the hwmon fan and pwm attributes. PWM is the raw kernel value, which runs from 0 to 255.
• Cooling State is the current state of each entry in /sys/class/thermal/cooling_device*, keyed by device type. Any value above zero means the thermal governor is limiting the system: a cpufreq or devfreq device reports how far the clocks have been pulled back, and a fan reports how hard it has been asked to run. Temperature tells you how warm a board is, while the cooling state tells you whether that warmth is costing performance, so the two are worth reading together. Cooling State Max gives the scale each device is measured against and is collected once at startup.

Power and Clocks

Two more readings round out the picture of how hard a board is working:

• Voltage, Current, and Power come from the hwmon power monitors, such as the INA3221 devices on a Jetson, and are published in volts, amps, and watts. A channel is published when its driver labels it, which is how a board names the rail that channel measures, so the points arrive keyed by rail name: VDD_GPU_SOC, VIN_SYS_5V0, and so on. Monitors that do not report power themselves still report voltage and current, and the product stands in for the missing reading.
• CPU MHz is the current clock of each core, keyed by cpu0, cpu1, and so on. Cores that are offline are left out. This is the reading that completes the thermal story: temperature says how warm the board is, cooling state says the governor stepped in, and the clock says what that cost.

Every reading above is taken on its own, so one that is unavailable, which happens when a rail is powered down or a monitor channel is disabled, does not affect the rest. Sensor names are not guaranteed to be unique; repeated names are numbered, as in tmp451 and tmp451_2.

SIOT Application Metrics

Named Process Metrics

Schema

The configuration of a system metrics node and a named process node:

nodes:
  - metrics:
      description: System
      period: 10
      tag:
        machine: press-1
      type: system
  - metrics:
      description: NATS server
      name: nats-server
      period: 10
      type: process

type is system, app, or process, and period is how often readings are taken, in seconds. name is the process name to watch and applies to a process node; values for all processes of that name are added together.

tag is a set of keyed points, and each one becomes a label on the samples the database client writes when its point type is listed there.

The readings themselves are points on the same node, so an export of a running instance carries them alongside the settings above.

Particle.io

SIOT provides a client for pulling data from Particle.io. Particle provide modules to quickly implement cellular connected MCU based IoT systems. They take care of managing the device (cellular connection, firmware deployments, etc.), and you only need to write the application.

The Particle cloud event API is used to obtain the data. A connection is made from the SIOT instance to the Particle Cloud and then data is sent back to SIOT using Server Sent Events (SSE). The advantage of this mechanism is that complex webhooks are not needed on the SIOT side, which requires additional firewall/web server configuration.

A Particle API key is needed which can be generated using the particle token CLI command.

The above example shows data provided by the Particle based Simple IoT Particle Gateway and 1-wire temperature sensors, and SIOT firmware.

Data is published to Particle in the following format:

[
	{
		"id": "4B03089794485728",
		"type": "temp",
		"value": 15.25
	}
]

The SIOT Particle client populates the point key field with the 1-wire device ID.

Schema

The configuration of a Particle node:

nodes:
  - particle:
      authToken: your-particle-token
      description: Particle Cloud
      disabled: 0

authToken is the token generated by particle token. An export carries it as it was entered, so treat a file that contains Particle nodes the way you would treat the token itself.

The data from the cloud arrives as points on this same node, one point per reading with the device ID as the point key, so an export of a running node carries those as well.

Rules

Contents

The Simple IoT application has the ability to run rules - see the video below for a demo:

Rules are composed of one or more conditions and actions. All conditions must be true for the rule to be active.

Node point changes cause rules of any parent node in the tree to be run. This allows general rules to be written higher in the tree that are common for all device nodes (for instance device offline).

In the below configuration, a change in the SBC propagates up the node tree, thus both the D5 on rule or the Device offline rule are eligible to be run.

Node linking

Both conditions and actions can be linked to a node ID. If you copy a node, its ID is stored in a virtual clipboard and displayed at the top of the screen. You can then paste this node ID into the Node ID field in a condition or action.

Conditions

Each condition may optionally specify a minimum active duration before the condition is considered met. This allows timing to be encoded in the rules.

Node state

A point value condition looks at the point value of a node to determine if a condition is met. Qualifiers that filter points the condition is interested in can set including:

• Node ID (if left blank, any node that is a descendant of the rule parent)
• Point type (“value” is probably the most common type)
• Point Key (used to index into point arrays and objects)

If the provided qualification is met, then the condition may check the point value/text fields for a number of conditions including:

• number: >, <, =, !=
• text: =, !=, contains
• boolean: on, off

Schedule

Rule conditions can be driven by a schedule that is composed of:

• start/stop time
• weekdays
• dates

If no weekdays are selected, then all weekdays are included.

When the dates are used, then weekdays are disabled.

Conversely, when a weekday is enabled, dates are disabled.

As a time range can span two days, the start time is used to qualify weekdays and dates.

See also a video demo:

Actions

Every action has an optional repeat interval. This allows rate limiting of actions like notifications.

Notifications

Notifications are the simplest rule action and are sent out when:

• All conditions are met
• Time since last notification is greater than the notify action repeat interval.

Every time a notification is sent out by a rule, a point is created/updated in the rule with the following fields:

• id: node of point that triggered the rule
• type: “lastNotificationSent”
• time: time the notification was sent

Before sending a notification we scan the points of the rule looking for when the last notification was sent to decide if its time to send it.

Set node point

Rules can also set points in other nodes. For simplicity, the node ID must be currently specified along with point parameters and a number/bool/text value.

Typically a rule action is only used to set one value. In the case of on/off actions, one rule is used to turn a value on, and another rule is used to turn the same value off. This allows for hysteresis and more complex logic than in one rule handled both the on and off states. This also allows the rules logic to be stateful. If you don’t need hysteresis or complex state, the rule “inactive action” can be used, which allows the rule to take action when it goes both active and inactive.

Disable Rule/Condition/Action

Disable Rule

A rule can be disabled. If the rule is disabled while active, then the rule inactive actions are run so that things get cleaned up if necessary and the actions are not left active.

Disable Condition

If there are no conditions, or all conditions are disabled, the rule is inactive. Otherwise, disabled conditions are simply ignored. For example, if there is a disabled condition and a non-disabled active condition, the rule is active.

Disable Action

A disabled action is not run.

Schema

The configuration of a rule with a point value condition, a schedule condition, and an action for each direction:

nodes:
  - rule:
      description: Tank low
      disabled: 0
      children:
        - condition:
            conditionType: pointValue
            description: Level below 10
            disabled: 0
            minActive: 5
            nodeID: Tank level
            operator: <
            pointKey: ""
            pointType: value
            value: 10
            valueType: number
        - condition:
            conditionType: schedule
            description: Working hours
            end: "17:00"
            start: "08:00"
            weekday:
              - 0
              - 1
              - 1
              - 1
              - 1
              - 1
              - 0
        - action:
            action: notify
            description: Tell the operators
        - actionInactive:
            action: setValue
            description: Clear the alarm
            nodeID: Alarm relay
            pointType: switchSet
            value: 0
            valueType: onOff

Conditions and actions are children of the rule, and an inactive action is a child of type actionInactive, which is what lets one rule act in both directions.

nodeID names the node a condition watches or an action writes to, and it is written as that node’s description rather than as an ID, so a rule can be moved between instances. Leaving it out of a condition watches every node below the rule’s parent. See referring to another node for how the name is resolved.

conditionType is pointValue or schedule. A point value condition qualifies the points it is interested in with pointType and pointKey, and valueType decides how it compares them: a number condition compares value using operator, one of >, <, =, or !=; a text condition compares valueText using =, !=, or contains; and an onOff condition matches a value of 1 or 0 and needs no operator. minActive is how many minutes the condition has to hold before the rule goes active.

A schedule condition uses start and end, written as text so 08:00 keeps its leading zero, along with weekday and date. weekday is seven points, Sunday first, each 1 or 0. date is a list of dates, and a schedule carries dates or weekdays rather than both.

action is notify, setValue, or playAudio. A setValue action names what to write with nodeID, pointType, and pointKey, and what to write with valueType and value or valueText. A playAudio action names the WAV file to play with filePath, the ALSA device to play it on with device, and the channel with channel.

The rule’s active state, the time the last notification was sent, and any error are points the client maintains, so an export of a running rule carries them as well.

Shelly IoT

Shelly sells a number of reasonably priced open IoT devices for home automation and industrial control. Most support Wi-Fi network connections and some of the Industrial line also supports Ethernet. The API is open and the devices support a number of communication protocols including HTTP, MQTT, CoAP, etc. They also support mDNS so they can be discovered on the network.

• Main website
• Device documentation
• API

Simple IoT provides the following support:

• Automatic discovery of all Shelly devices on the network using mDNS
• Support for the following devices:
  • 1pm (not tested)
  • Bulb Duo (on/off only)
  • Plus 1
  • Plus 1PM (not tested)
  • Plus 2PM
  • Plus Plug (only US variant tested)
    • Measurements such as Current, Power, Temp, Voltage are collected.
  • Plus i4
• Currently status is polled via HTTP every 2 seconds

Setup

• Configure the Shelly devices to connect to your Wi-Fi network. There are several options:
  1. Use the Shelly phone app
  2. A new device will start up in access point mode. Attach a computer or phone to this AP, open http://192.168.33.1 (default address of a reset device), and then configure the Wi-Fi credentials using the built-in Web UI.
• Add the Shelly client in SIOT
• The Shelly client will then periodically scan for new devices and add them as child nodes.

Example

Plug Example

Schema

The configuration of a Shelly node and one of the devices it found:

nodes:
  - shelly:
      description: Shelly
      disabled: 0
      children:
        - shellyIo:
            controlled: 1
            description: Bench light
            deviceID: shellyplusplugus-b0b21c12ad58
            disabled: 0
            ip: 192.168.1.42
            type: PlugUS

The Shelly node itself only carries a description and whether it is disabled. Everything else follows from what it finds: the client scans the network and adds a child node for each device, filling in deviceID, ip, and type itself. type is the device model, such as PlugUS, Plus1, Plus2PM, or PlusI4.

What you configure on a device is its description, whether it is disabled, and, for a device that can be driven, controlled. With controlled set, the client drives the device to the switchSet and lightSet values whenever they differ from what the device reports, which is what lets a rule or the UI change its state.

The readings and states, along with whether the device is currently reachable, are points the client maintains, so an export of a running node carries them as well. A device with more than one channel carries one point per channel, keyed by channel number.

Signal Generator Client

The signal generator can be used to generate various signals including:

• Sine wave
• Square wave
• Triangle wave
• Random walk

Below is a screen-shot of the generated data displayed in Grafana.

Configuration

The signal generated can be configured with the following parameters:

Most of the parameters are self-explanatory. With a Random Walk, you typically need to enter a negative number for the minimum. Increment as shown above. This causes the negative number generated to be negative roughly half the time.

The rounding can also be used to generate binary signals. Imagine a signal generator with these settings:

• Max. value = 1
• Min. value = 0
• Initial value = 0
• Round to = 1
• Min. increment = -7
• Max. increment = 3
• Sample Rate = 20 milliseconds

Due to min/max/round to options, this is a binary value, either 0 or 1, biased toward 0 (due to min/max increment options). This could be useful for simulating binary switches or something like it. Effectively, this will hold the value for at least 20m and picks a random number between -7 and 3. Due to rounding, if value is currently 0, there’s a 25% chance it becomes 1. If 1, there’s a 65% chance it becomes 0. This means that the value will be 0 roughly 91.25% (= 75% + (1 - 75%) * 65%) of the time.

Schema

signalType is sine, square, triangle, or random walk. frequency applies to the three waveforms and minIncrement, maxIncrement, and roundTo apply to a random walk. sampleRate is in Hz and batchPeriod is in milliseconds.

A generator writes to itself unless it is told otherwise. A node configured to write elsewhere carries a destination mapping, whose keys are nodeID, parent, highRate, pointType, and pointKey. nodeID there is a key rather than a point type, so it is written as the ID of the node it names rather than as a description, which is worth knowing when moving a generator between instances.

Below is an export of several types of signal generator nodes:

nodes:
  - signalGenerator:
      batchPeriod: 1000
      description: Variable pulse width
      frequency: 1
      initialValue: "0"
      maxIncrement: 3
      maxValue: 1
      minIncrement: -7
      minValue: "0"
      roundTo: 1
      sampleRate: 5
      signalType: random walk
      units: Amps
      value: 1
  - signalGenerator:
      batchPeriod: 1000
      description: Triangle
      frequency: 1
      initialValue: "0"
      maxIncrement: 0.5
      maxValue: 10
      minIncrement: 0.1
      minValue: "0"
      sampleRate: 100
      signalType: triangle
      value: 6.465714272450723e-12
  - signalGenerator:
      batchPeriod: 1000
      description: Square
      frequency: 1
      initialValue: "0"
      maxValue: 10
      minValue: "0"
      sampleRate: 100
      signalType: square
      value: 10
  - signalGenerator:
      batchPeriod: 1000
      description: Sine
      frequency: 1
      initialValue: "0"
      maxValue: 10
      minValue: "0"
      sampleRate: 100
      signalType: sine
      value: 4.999999999989843
  - signalGenerator:
      batchPeriod: 1000
      description: Random Walk
      frequency: 1
      initialValue: "0"
      maxIncrement: 0.5
      maxValue: 10
      minIncrement: -0.5
      minValue: "0"
      roundTo: 0.1
      sampleRate: 10
      signalType: random walk
      units: Amps
      value: 9.1

Synchronization

Simple IoT provides for synchronized upstream connections via NATS or NATS over WebSocket.

To create an upstream sync, add a sync node to the root node on the downstream instance. If your upstream server has a name of myserver.com, then you can use the following connections URIs:

• nats://myserver.com:4222 (4222 is the default NATS port)
• ws://myserver.com (WebSocket unencrypted connection)
• wss://myserver.com (WebSocket encrypted connection)

IP addresses can also be used for the server name.

Auth token is optional and needs to be configured in an environment variable for the upstream server. If your upstream is on the public internet, you should use an auth token. If both devices are on an internal network, then you may not need an auth token.

Typically, wss are simplest for servers that are fronted by a web server like Caddy that has TLS certs. For internal connections, nats or ws connections are typically used.

Occasionally, you might also have edge devices on networks where NATS outgoing connections on port 4222 are blocked. In this case, it’s handy to be able to use the wss connection, which just uses standard HTTP(S) ports.

How synchronization behaves

Synchronization works by replicating the JetStream streams that store each instance’s data — see the synchronization reference for how this works. The behavior you will observe:

• First connect: the device announces itself and appears under the upstream root node; its full tree (structure, configuration, and history) then arrives through replication. Configuration written on the upstream for a device that has not connected yet is delivered on first connect.
• Offline changes catch up. Changes made on either side while the connection is down are delivered when it comes back — replication resumes exactly where it left off, and only missed data is sent. See Queuing while offline below.
• Both sides can edit. Configuration can be changed on either instance; the newest change wins everywhere.
• Deleting a device on the upstream detaches it. The device keeps running standalone and does not add itself back; undelete the device node on the upstream to resume synchronization.

Queuing while offline

An edge instance does not need its upstream to keep working. It writes every point to its own local store first, and the sync client replicates that store upstream. When the connection drops, the instance keeps collecting data, running rules, and accepting local configuration changes, all of which queue on disk.

On reconnect:

• The backlog is sent in order, with the original timestamps, so history upstream has no gap.
• Only the missed messages are sent. Replication resumes at the position it reached before the outage, which keeps the recovery cheap on a metered or low bandwidth link.
• Clients that act on current values (rules, protocol clients, the UI) see one update per changed value once the backlog drains rather than a replay of every intermediate reading, so a device coming back online does not re-trigger rules on stale data.
• History consumers still receive every point. A Db client feeding a time-series database reads the stream with its own durable consumer, so the backlog reaches the database as well.

Configuration written upstream while a device is offline, or before it has ever connected, waits and is delivered on the next connect.

How long a device can be offline and still catch up in full depends on how much history the store keeps. The default is 5000 points per value, which is adjustable per instance. See Store for the setting, and the synchronization reference for how the queuing works.

Schema

The configuration of a sync node:

nodes:
  - sync:
      authToken: your-auth-token
      description: Cloud
      disabled: 0
      uri: wss://myserver.com

uri is the upstream connection, written as one of the forms described above. authToken matches SIOT_AUTH_TOKEN on the upstream server and is left out when the upstream needs no token.

A sync node belongs on the root node of the downstream instance, so a file that carries one leaves parent out and it attaches to the device node this instance runs as.

An export carries authToken as it was entered, so treat a file that contains sync nodes the way you would treat the token itself.

The count of synchronizations is a point the client maintains, so an export of a running node carries it as well.

Videos

There are also several videos that demonstrate upstream connections:

Simple IoT upstream synchronization support

Simple IoT Integration with PLC Using Modbus

Update

The Simple IoT update client facilitates updating software. Currently, it is designed to download images for use by the Yoe Updater. The process can be executed manually, or there are options to automatically download and install new updates.

There are several options:

• Update server: HTTP server that contains the following files:
  • files.txt: contains a list of update files on the server
  • update files named: <prefix>_<version>.upd
    • version should follow Semantic Versioning: MAJOR.MINOR.PATCH
    • prefix must match what the updater on the target device is expecting typically host/machine name.
• prefix: described above - typically host/machine name. This is auto detected on first startup, but can be changed if necessary.
• Dest dir: Destination directory for downloaded updates. Defaults to /data.
• Chk interval: time interval at which the client checks for new updates.
• Auto download: option to periodically check the server for new updates and download the latest version.
• Auto reboot/install: option to auto install/reboot if a new version is detected and downloaded.

Schema

The configuration of an update node:

nodes:
  - update:
      autoDownload: 1
      autoReboot: 0
      description: Updates
      directory: /data
      pollPeriod: 60
      prefix: myboard
      uri: http://updates.example.com

pollPeriod is how often the server is checked, in minutes, and defaults to 30 when it is zero or missing. directory defaults to /data.

prefix is detected on first startup, so a file usually leaves it out and lets each unit fill in its own; give it only when every unit the file applies to expects the same one.

The versions found on the server, the version downloaded, and the current OS version are points the client maintains, so an export of a running node carries them as well.

USB

Browser

The browser client enables control and configuration of the Yoe Kiosk Browser as it is when installed as part of Yoe Distro. On changing the configuration, changes are saved to /etc/default/yoe-kiosk-browser for the browser and /etc/default/eglfs.json for EGLFS, and the yoe-kiosk-browser service is restarted automatically.

Schema

Below is an export of a browser node:

nodes:
  - browser:
      debugport: "9222"
      defaultdialogs: 0
      description: Kiosk
      dialogcolor: "#1c1c1c"
      disabled: 0
      disablesandbox: 1
      displaycard: /dev/dri/card0
      exceptionurl: http://localhost:8118/offline.html
      fullscreen: 1
      ignorecerterr: 0
      keyboardscale: 1
      retryinterval: 10
      rotate: 0
      screenresolution: 1920x1080
      touchquirk: 0
      url: http://localhost:8118

The point types are lower case throughout, which matches the settings written to /etc/default/yoe-kiosk-browser. Checkboxes are stored as 1 and 0. debugport is text, so it is quoted; rotate and retryinterval are numbers. displaycard and screenresolution are the two settings that land in /etc/default/eglfs.json.

Graphing Data

Simple IoT is designed to work with several other applications for storing time series data and viewing this data in graphs.

InfluxDB

InfluxDB is currently the recommended way to store historical data. This database is efficient and can run on embedded platforms like the Raspberry PI as well as desktop and server machines. To connect SIOT to InfluxDB, add a database node and fill in the parameters.

Grafana

Grafana is a very powerful graphing solution that works well with InfluxDB. Although InfluxDB has its own web interface and graphing capability, generally we find Grafana to be more full featured and easier to use.

Changing the Display name (labels) in Grafana

Often with an Influx query, we’ll get trace display names that look like the below:

Often, much of this data is irrelevant or redundant with the query. One way to change the label is with an Override:

This can be tedious to set up and maintain.

Often a better way is to add tags to the nodes generating the data and then display the node tags in the display name by using the Influx map function.

from(bucket: "siot")
  |> range(start: v.timeRangeStart, stop:v.timeRangeStop)
  |> filter(fn: (r) =>
    r._measurement == "points" and
    r._field == "value" and
    r.type == "value")
  |> filter(fn: (r) => r["node.type"] == "signalGenerator")
  |> map(fn: (r) => ({_value:r._value, _time:r._time, _field:r["node.tag.machine"] + ":" + r["node.description"]}))

In this case we are displaying the node machine tag and description. The result is very nice:

Configuration

Environment variables

Environment variables are used to control various aspects of the application. The following are currently defined:

• General
  • SIOT_HTTP_PORT: HTTP network port the SIOT server attaches to (default is 8118)
  • SIOT_DATA: directory where any data is stored
  • SIOT_AUTH_TOKEN: auth token used for NATS and HTTP device API, default is blank (no auth)
  • OS_VERSION_FIELD: the field in /etc/os-release used to extract the OS version information. Default is VERSION, which is common in most distros. The Yoe Distribution populates VERSION_ID with the update version, which is probably more appropriate for embedded systems built with Yoe. See ref/version.
• NATS configuration
  • SIOT_NATS_PORT: Port to run NATS on (default is 4222 if not set)
  • SIOT_NATS_HTTP_PORT: Port to run NATS monitoring interface (default is 8222)
  • SIOT_NATS_SERVER: defaults to nats://127.0.0.1:4222
  • SIOT_NATS_TLS_CERT: points to TLS certificate file. If not set, TLS is not used.
  • SIOT_NATS_TLS_KEY: points to TLS certificate key
  • SIOT_NATS_TLS_TIMEOUT: Configure the TLS upgrade timeout. NATS defaults to a 0.5 second timeout for TLS upgrade, but that is too short for some embedded systems that run on low end CPUs connected over cellular modems (we’ve see this process take as long as 4 seconds). See NATS documentation for more information.
  • SIOT_NATS_WS_PORT: Port to run NATS WebSocket (default is 9222, set to 0 to disable)
• Provisioning
  • SIOT_PROVISIONING_DIR: directory of YAML files applied at start-up and whenever they change. If it is not set, <SIOT_DATA>/provisioning is used when that directory exists, so an image can ship the directory and say nothing else.
  • SIOT_PROVISIONING_INTERVAL: how often to look for changes the directory watch and the tree subscription might have missed, written as a Go duration such as 60s. The default is one minute.
• Particle.io
  • SIOT_PARTICLE_API_KEY: key used to fetch data from Particle.io devices running Simple IoT firmware

The configuration file format

One format describes a tree of nodes, and siot export, siot import, and provisioning all use it. The node type is the key, and each point type is a key of its own:

apiVersion: 1
nodes:
  - group:
      description: Sensors
      children:
        - modbus:
            description: Modbus sensors
            port: /dev/ttyS1
            baud: 9600
            debug: 0

How a value is written decides what it becomes:

Quoting is what tells a text value from a numeric one, which matters when a value looks like a number: port: 502 is numeric and port: "502" is text. If a client expects text and the file gives it a number, the client reads an empty value, so quote anything that is really text.

A mapping under a point type is a set of keyed points, and a sequence is an array:

- metrics:
    metricSysCPUFreq:
      cpu0: 1400
      cpu1: 1600
    tag: [alpha, beta] # keys "0" and "1"

Three keys inside a node are reserved: parent, children, and edgePoints. Every other key is a point type, id included – Modbus and OneWire nodes configure a point named id, and it is written like any other point. A node’s own ID never appears in a file.

Edge points, such as a user’s role, are spelled the same way under their own key:

- user:
    firstName: Admin
    email: admin@example.com
    edgePoints:
      role: admin

Points that a file does not need to carry are left out of an export. The nodeType edge point is one of them, since the node type is the key each node is written under, and the system fills it in when a file is applied.

How nodes are found

A file describes what the tree should look like rather than naming the nodes it means by ID, so applying one twice does what applying it once did. A node in a file matches an existing node when the parent and the description agree:

• No match: the node is created.
• A match of the same type: only the points whose values differ are sent.
• A match of a different type: an error, since a file that says modbus where the tree holds a group is either a mistake or a rename.
• More than one match: an error, since nothing says which node was meant.

A user node has no description, so an email address identifies it, and a name if there is no email. An entry with no description at all matches the single node of its type, which is how a metrics or serial node is addressed.

A description is how a file finds a node. Renaming one in the UI detaches it from the file that describes it, and the next time that file is applied it creates a second node beside the renamed one. The same is true of renaming a node in a file. Renaming deliberately is a two step change: delete the old description in the same file that introduces the new one. Give nodes descriptions that are meant to last.

Where nodes attach

A top level entry with no parent is applied under this instance’s device node. A parent names a node anywhere in the tree by description, which is how a file adds to a subtree it did not create:

nodes:
  - group:
      description: Tank farm
  - variable:
      parent: Tank farm
      description: Tank level

Entries apply in the order they are written, so a parent naming a node the same file creates has to come after the entry that creates it.

Referring to another node

A point of type nodeID names the node it refers to by description, and is resolved the same way parent is:

nodes:
  - variable:
      description: Tank level
  - rule:
      description: Tank low
      children:
        - condition:
            description: Level below 10
            nodeID: Tank level
            operator: "<"
            value: 10

References resolve after the whole file has been read, so one may point at a node the file creates further down, or at a node another file created.

Removing nodes

Applying a file adds and updates; it never removes something for going unmentioned. A delete list removes nodes, matched the way nodes entries are:

delete:
  - modbus:
      parent: Tank farm
      description: Old sensors

Deleting what is already gone does nothing, so a file with a delete list is as safe to apply repeatedly as any other.

Configuration export

Nodes can be exported to a YAML file. This is useful to:

• Back up the current configuration
• Dump node data for debugging
• Transfer a configuration, or part of one, from one instance to another
• Build a configuration in the UI and then ship it as a provisioning file

To export the entire tree:

siot export > backup.yaml

A subset of the tree can be exported by specifying the node ID:

siot export -nodeID 9d7c1c03-0908-4f8b-86d7-8e79184d441d > export.yaml

An export describes configuration and nothing else, which is what makes it usable as a provisioning file:

• The root node is left out. It is this instance rather than configuration, and a file describing it would match nothing anywhere else. Exporting the tree exports what is under the root.
• Node IDs are left out, since a file finds its nodes by description. A nodeID point is written as the description of the node it points at.
• Points that carry no value are left out, as is the origin recording which client last wrote each point.

Two nodes that share a parent and a description cannot be told apart by a file, so siot export reports that rather than writing a file that would do the wrong thing when applied. Give those nodes distinct descriptions, which is worth doing anyway.

Configuration import

siot import applies a file to a running instance, reading it from STDIN:

siot import < config.yaml

Nodes are matched by description, as described above, so importing a file creates what is missing, updates what has drifted, and does nothing when the tree already agrees. Importing the same file twice does what importing it once did.

siot import -dryRun < config.yaml prints what the file would do without applying any of it.

If authentication or a different server is required, this can be specified through command line arguments or the following environment variables (see descriptions above):

• SIOT_NATS_SERVER
• SIOT_AUTH_TOKEN

siot import --help for more details.

Example YAML file:

nodes:
  - group:
      description: group 1
      children:
        - variable:
            description: var 1
            value: 10

Configuration provisioning

An instance can be configured from files rather than by hand. Provisioning applies the same files siot import does, and applies them at start-up and whenever they change, so a unit built from an image comes up configured with no import step and no operator involvement.

There are two places files come from:

• A directory on disk, given by -provisioningDir or SIOT_PROVISIONING_DIR, defaulting to <SIOT_DATA>/provisioning when that directory exists. Files are applied in lexical order, so the familiar 10-, 20- prefixes express which file goes first.
• Files uploaded through the UI, which are file nodes under the provisioning node. This is how a unit whose filesystem you cannot reach gets configured.

Files on disk are applied first and uploads layer on top, so an uploaded file can attach to a group a shipped file created. Uploads are applied oldest first, by when the file was added rather than when its contents were last replaced, so correcting a file does not change its place in the order.

A file node exists from the moment it is added and its contents arrive when you upload them, so provisioning waits for the upload and leaves an empty file node alone.

A file is applied when its contents change, which is what leaves a value edited in the UI alone until the file describing it changes. A file that fails to parse or apply records its error and leaves the other files alone.

Checking files

siot provision -dir ./provisioning prints what the files in a directory would do to a running instance without applying any of it.

siot provision -dir ./provisioning -check only parses them, which needs no running instance and is what a build can use to fail on a bad file.

Seeing what happened

A provisioning node under the root records what was applied. Each file on disk gets a provisioningFile child carrying its name, the checksum of what was applied, and the last error if it failed. An uploaded file records the same thing on the file node itself, so a file and its status are one node in the UI.

Removing a file from the directory removes its status. The nodes it created stay where they are: provisioning describes what should exist, and does not own what it made. Use a delete list to remove nodes.

Status

The Simple IoT project is still in a heavy development phase. Most of the core concepts are stable, but APIs, packet formats, and implementation will continue to change for some time yet. SIOT has been used in several production systems to date with good success, but be prepared to work with us (report issues, help fix bugs, etc.) if you want to use it now.

Handling of high rate sensor data

Currently each point change requires quite a bit computation to update the HASH values in upstream graph nodes. For repetitive data, this is not necessary as new values are continually coming in, so we will at some point make an option to specify points values as repetitive. This will allow SIOT to scale to more devices and higher rate data.

User Interface

The web UI is currently polling the SIOT backend every 4 seconds via HTTP. This works OK for small datasets, but uses more data than necessary and has a latency of up to 4 seconds. Long term we will run a NATS client in the frontend over a WebSocket so the UI response is real-time and new data gets pushed to the browser.

Security

Currently, and device that has access to the system can write or write to any data in the system. This may be adequate for small or closed systems, but for larger systems, we need per-device authn/authz. See issue #268, PR #283, and our security document for more information.

Errata

Any issues we find during testing we log in GitHub issues, so if you encounter something unexpected, please search issues first. Feel free to add your observations and let us know if an issues is impacting you. Several issues to be aware of:

• We don’t handle loops in the graph tree yet. This will render the instance unusable and you’ll have to clean the database and start over.

Frequently Asked Questions

Q: How is SIOT different than Home Assistant, OpenHAB, Domoticz, etc.?

Although there may be some overlap and Simple IoT may eventually support a number of off the shelf consumer IoT devices, the genesis, and intent of the project is for developing IoT products and the infrastructure required to support them.

Q: How is SIOT different than Particle.io, etc.?

Particle.io provides excellent infrastructure to support their devices and solve many of the hard problems such as remote firmware update, getting data securely from device to cloud, and efficient data bandwidth usage. But, they don’t provide a way to provide a user facing portal for a product that customers can use to see data and interact with the device.

Q: What happens to data collected while a device is offline?

It is queued on the device and delivered when the connection returns. Every instance writes to its own local store first and replicates that store upstream, so an outage stops the transfer and not the collection. On reconnect, only the missed data is sent, in order and with the original timestamps. Configuration changed in the cloud while the device is away is delivered at the same time. See Synchronization for details and for the retention limits that determine how long a device can be offline and still catch up in full.

Q: How is SIOT different than AWS/Azure/GCP/… IoT?

SIOT is designed to be simple to develop and deploy without a lot of moving parts. We’ve reduced an IoT system to a few basic concepts that are exactly the same in the cloud and on edge devices. This symmetry is powerful and allows us to easily implement and move functionality wherever it is needed. If you need Google Scale, SIOT may not be the right choice; however, for smaller systems where you want a system that is easier to develop, deploy, and maintain, consider SIOT.

Q: Can’t NATS JetStream do everything SIOT does?

This is a good question and I’m not sure yet. NATS has some very interesting features like JetStream which can queue data and store data in a key-value store and data can be synchronized between instances. NATS also has a concept of leaf-nodes, which conceptually makes sense for edge/gateway connections. JetStream is optimized for data flowing in one direction (ex: orders through fulfillment). SIOT is optimized for data flowing in any direction and data is merged using data structures with CRDT (conflict-free replicated data types) properties. SIOT also stores data in a DAG (directed acyclic graph) which allows a node to be a child of multiple nodes, which is difficult to do in a hierarchical namespace. Additionally, each node is defined by an array of points and modifications to the system are communicated by transferring points. SIOT is a batteries included complete solution for IoT solutions, including a web framework, clients for various types of IO (ex: Modbus) and cloud services (ex: Twilio). We will continue to explore using more of NATS core functionality as we move forward.

Documentation

Good documentation is critical for any project and to get good documentation, the process to create it must be as frictionless as possible. With this in mind, we’ve structured SIOT documentation as follows:

• Markdown is the primary source format.
• Documentation lives in the same repo as the source code. When you update the code, update the documentation at the same time.
• Documentation is easily viewable in GitHub, or our generated docs site. This allows any snapshot of SIOT to contain a viewable snapshot of the documentation for that revision.
• mdbook is used to generate the documentation site.
• All diagrams are stored in a single draw.io file. This allows you to easily see what diagrams are available and easily copy pieces from existing diagrams to make new ones. Then generate a PNG for the diagram in the images/ directory in the relevant documentation directory.

Vision

This document attempts to outlines the project philosophy and core values. The basics are covered in the readme. As the name suggests, a core value of the project is simplicity. Thus, any changes should be made with this in mind. Although this project has already proven useful on several real-world project, it is a work in progress and will continue to improve. As we continue to explore and refine the project, many things are getting simpler and more flexible. This process takes time and effort.

“When you first start off trying to solve a problem, the first solutions you come up with are very complex, and most people stop there. But if you keep going, and live with the problem and peel more layers of the onion off, you can often times arrive at some very elegant and simple solutions.” - Steve Jobs

Guiding principles

1. Simple concepts are flexible and scale well.
2. IoT systems are inherently distributed, and distributed systems are hard.
3. There are more problems to solve than people to solve them, thus it makes sense to collaborate on the common technology pieces.
4. There are a lot of IoT applications that are not Google scale (10-1000 device range).
5. There is significant opportunity in the long tail of IoT, which is our focus.
6. There is value in custom solutions (programming vs drag-n-drop).
7. There is value in running/owning our own platform.
8. A single engineer should be able to build and deploy a custom IoT system.
9. We don’t need to spend excessive amounts of time on operations. For smaller deployments, we deploy one binary to a cloud server and we are done with operations. We don’t need 20 microservices when one monolith will work just fine.
10. For many applications, a couple of hours of down time is not the end of the world. Thus, a single server that can be quickly rebuilt as needed is adequate and in many cases more reliable than complex systems with many moving parts.

Technology choices

Choices for the technology stack emphasize simplicity, not only in the language, but just as important, in the deployment and tooling.

• Backend
  • Go
    • Simple language and deployment model
    • Nice balance of safety + productivity
    • Excellent tooling and build system
    • See this thread for more discussion/information
• Frontend
  • Single Page Application (SPA) architecture
    • Fits well with real-time applications where data is changing all the time
    • Easier to transition to Progressive Web Apps (PWA)
  • Elm
    • Nice balance of safety + productivity
    • Excellent compiler messages
    • Reduces possibility for run time exceptions in browser
    • Does not require a huge/complicated/fragile build system typical in JavaScript frontends.
    • excellent choice for SPAs
  • elm-ui
    • What if you never had to write CSS again?
    • Fun, yet powerful way to lay out a user interface and allows you to efficiently make changes and get the layout you want.
• Database
  • SQLite
    • See Store
  • Eventually support multiple database backends depending on scaling/admin needs
• Cloud Hosting
  • Any machine that provides ability run long-lived Go applications
  • Any MAC/Linux/Windows/rPI/Beaglebone/Odroid/etc. computer on your local network.
  • Cloud VMs: Digital Ocean, Linode, GCP compute engine, AWS EC2, etc. Can easily host on a $5/mo instance.
• Edge Devices
  • Any device that runs Linux (rPI, Beaglebone-black, industrial SBCs, your custom hardware …)

In our experience, simplicity and good tooling matter. It is easy to add features to a language, but creating a useful language/tooling that is simple is hard. Since we are using Elm on the frontend, it might seem appropriate to select a functional language like Elixir, Scala, Clojure, Haskell, etc. for the backend. These environments are likely excellent for many projects, but are also considerably more complex to work in. The programming style (procedural, functional, etc.) is important, but other factors such as simplicity/tooling/deployment are also important, especially for small teams who don’t have separate staff for backend/frontend/operations. Learning two simple languages (Go and Elm) is a small task compared to dealing with huge languages, fussy build tools, and complex deployment environments.

This is just a snapshot in time - there will likely be other better technology choices in the future. The backend and frontend are independent. If either needs to be swapped out for a better technology in the future, that is possible.

Architecture

This document describes how the Simple IoT project fulfills the basic requirements as described in the top level README.

There are two levels of architecture to consider:

• System: how multiple SIOT instances and other applications interact to form a system.
• Application: how the SIOT application is structured.
• Clients: all about SIOT clients where most functionality is implemented.

High Level Overview

Simple IoT functions as a collection of connected, distributed instances that communicate via NATS. Data in the system is represented by nodes which contain an array of points. Data changes are communicated by sending points within an instance or between instances. Points in a node are merged such that newer points replace older points. This allows granular modification of a node’s properties. Nodes are organized in a DAG (directed acyclic graph). This graph structure defines many properties of the system such as what data users have access to, the scope of rules and notifications, and which nodes external services apply to. Most functionality in the system is implemented in clients, which subscribe and publish point changes for nodes they are interested in.

System Architecture

Contents

IoT Systems are distributed systems

IoT systems are inherently distributed where data needs to be synchronized between a number of different systems including:

1. Cloud (one to several instances depending on the level of reliability desired)
2. Edge devices (many instances)
3. User Interface (phone, browser)

Typically, the cloud instance stores all the system data, and the edge, browser, and mobile devices access a subset of the system data.

Extensible architecture

Any siot app can function as a standalone, client, server or both. As an example, siot can function both as an edge (client) and cloud apps (server).

• Full client: full SIOT node that initiates and maintains connection with another SIOT instance on a server. Can be behind a firewall, NAT, etc.
• Server: needs to be on a network that is accessible by clients

We also need the concept of a lean client where an effort is made to minimize the application size to facilitate updates over IoT cellular networks where data is expensive.

Device communication and messaging

In an IoT system, data from sensors is continually streaming, so we need some type of messaging system to transfer the data between various instances in the system. This project uses NATS.io for messaging. Some reasons:

• Allows us to push real-time data to an edge device behind a NAT, on cellular network, etc. - no public IP address, VPN, etc. required.
• Is more efficient than HTTP as it shares one persistent TCP connection for all messages. The overhead and architecture is similar to MQTT, which is proven to be a good IoT solution. It may also use less resources than something like observing resources in CoAP systems, where each observation requires a separate persistent connection.
• Can scale out with multiple servers to provide redundancy or more capacity.
• Is written in Go, so possible to embed the server to make deployments simpler for small systems. Also, Go services are easy to manage as there are no dependencies.
• Focus on simplicity - values fit this project.
• Good security model.

For systems that only need to send one value several times a day, CoAP is probably a better solution than NATS. Initially we are focusing on systems that send more data - perhaps 5-30MB/month. There is no reason we can’t support CoAP as well in the future.

Data modification

Where possible, modifying data (especially nodes) should be initiated over NATS vs direct db calls. This ensures anything in the system can have visibility into data changes. Eventually we may want to hide db operations that do writes to force them to be initiated through a NATS message.

Simple, Flexible data structures

As we work on IoT systems, data structures (types) tend to emerge. Common data structures allow us to develop common algorithms and mechanism to process data. Instead of defining a new datatype for each type of sensor, define one type that will work with all sensors. Then the storage (both static and time-series), synchronization, charting, and rule logic can stay the same and adding functionality to the system typically only involves changing the edge application and the frontend UI. Everything between these two end points can stay the same. This is a very powerful and flexible model as it is trivial to support new sensors and applications.

See Data for more information.

Node Tree

The same Simple IoT application can run in both the cloud and device instances. The node tree in a device would then become a subset of the nodes in the cloud instance. Changes can be made to nodes in either the cloud or device and data is synchronized in both directions.

The following diagram illustrates how nodes might be arranged in a typical system.

A few notes this structure of data:

• A user has access to its child nodes, parent nodes, and parent node descendants (parents, children, siblings, nieces/nephews).
• Likewise, a rule node processes points from nodes using the same relationships described above.
• A user can be added to any node. This allows permissions to be granted at any level in the system.
• A user can be added to multiple nodes.
• A node admin user can configure nodes under it. This allows a service provider to configure the system for their own customers.
• If a point changes, it triggers rules of upstream nodes to run (perhaps paced to some reasonable interval)
• The Edge Dev Offline rule will fire if any of the Edge devices go offline. This allows us to only write this rule once to cover many devices.
• When a rule triggers a notification, the rule node and any upstream nodes can optionally notify its users.

The distributed parts of the system include the following instances:

• Cloud (could be multiple for redundancy). The cloud instances would typically store and synchronize the root node and everything under it.
• Edge Devices (typically many instances (1000’s) connected via low bandwidth cellular data). Edge instances would store and synchronize the edge node instance and descendants (ex Edge Device 1)
• Web UI (potentially dozens of instances connected via higher bandwidth browser connection).

As this is a distributed system where nodes may be created on any number of connected systems, node IDs need to be unique. A unique serial number or UUID is recommended.

Application Architecture

Contents

The Simple IoT Go application is a single binary with embedded assets. The database and NATS server are also embedded by default for easy deployment. There are five main parts to a Simple IoT application:

1. NATS Message Bus: all data goes through this making it very easy to observe the system.
2. Store: persists the data for the system in JetStream streams, merges incoming data, and replicates streams for synchronization with other instances.
3. Clients: interact with other devices/systems such as Modbus, 1-wire, etc. This is where most of the functionality in a SIOT system lives, and where you add your custom functionality. Clients can exist inside the Simple IoT application or as external processes written in any language that connect via NATS. Clients are represented by a node (and optionally child nodes) in the SIOT store. When a node is updated, its respective clients are updated with the new information. Likewise, when a client has new information, it sends that out to be stored and used by other nodes/instances as needed.
4. HTTP API: provides a way for HTTP clients to interact with the system.
5. Web UI: Provides a user interface for users to interact with the system. Currently it uses the HTTP API, but will eventually connect directly to NATS.

The simplicity of this architecture makes it easy to extend with new functionality by writing a new client. Following the constraints of storing data as nodes and points ensures all data is visible and readable by other clients, as well as being automatically synchronized to upstream instances.

Application Lifecycle

Simple IoT uses the Run()/Stop() pattern for any long running processes. With any long running process, it is important to not only Start it, but also to be able to cleanly Stop it. This is important for testing, but is also good practice. Nothing runs forever so we should never operate under this illusion. The oklog/run packaged is used to start and shutdown these processes concurrently. Dependencies between processes should be minimized where possible through retries. If there are hard dependencies, these can be managed with WaitStart()/WaitStop() functions. See server.go for an example.

NATS lends itself very well to a decoupled application architecture because the NATS clients will buffer messages for some time until the server is available. Thus, we can start all the processes that use a NATS client without waiting for the server to be available first.

Long term, a NATS API that indicates the status of various parts (rules engine, etc.) of the system would be beneficial. If there are dependencies between processes, this can be managed inside the process instead of in the code that starts/stops the processes.

Provisioning

Almost everything in Simple IoT is configured by nodes in the tree, and is implemented as a client. Provisioning is not, because it has to work before there is any configuration in the tree to read. It is a server level concern, set by a command line flag and an environment variable and started in server.Run() alongside the store and the client manager. This is the same reasoning behind Grafana configuring provisioning in grafana.ini rather than in the database it populates.

Provisioning applies the same files siot import does, using the engine in client/apply.go, so a file works the same whichever way it is applied. See user/configuration.

NATS Integration

The NATS API details the NATS subjects used by the system.

Echo concerns

Any time you potentially have two sources modifying the same resource (a node), you need to be concerned with echoed messages. This is a common occurrence in Simple IoT. Because another resource may modify a node, typically a client needs to subscribe to the node messages as well. This means when it sends a message, it will typically be echoed back. See the client documentation for ideas on how to handle the echo problem.

The server.NewServer function returns a NATS connection. This connection is used throughout the application and does not have the NoEcho option set.

User Interface

Currently, the User Interface is implemented using a Single Page Architecture (SPA) Web Application. This keeps the backend and frontend implementations mostly independent. See User Interface and Frontend for more information.

There are many web architectures to chose from and web technology is advancing at a rapid pace. SPAs are not in vogue right now and more complex architectures are promoted such as Next.js, SveltKit, Deno Fresh, etc. Concerns with SPAs include large initial load and stability (if frontend code crashes, everything quits working). These concerns are valid if using JavaScript, but with Elm these concerns are minimal as Elm compiles to very small bundles, and run time exceptions are extremely rare. This allows us to use a simple web architecture with minimal coupling to the backend and minimal build complexity. And it will be a long time until we write enough Elm code that bundle size matters.

A decoupled SPA UI architecture is also very natural in Simple IoT, as IoT systems are inherently distributed. The frontend is just another client, much the same as a separate machine learning process, a downstream instance, a scripting process, etc.

Simple IoT Clients

Contents

Most functionality in Simple IoT is implemented in Clients.

Each client can be configured by one or more nodes in the SIOT store graph. These nodes may be created by a user, a process that detects new plug and play hardware, or other clients.

A client interacts with the system by listening for new points it is interested in and sending out points as it acquires new data.

Creating new clients

See Development for information on how to set up a development system.

Simple IoT provides utilities that assist in creating new clients. See the Go package documentation for more information. A client manager is created for each client type. This manager instantiates new client instances when new nodes are detected and then sends point updates to the client. Two levels of nodes are currently supported for client configuration. An example of this would be a Rule node that has Condition and Action child nodes.

A “disabled” option is useful and should be considered for every new client.

Creating a new client typically requires the following steps:

1. Add any new node and points types to schema.go, Node.elm, and Point.elm. Please try to reuse existing point types when possible.
2. Create a new client in client/ directory. A client is defined by a type that satisfies the Client interface. A constructor must also be defined that is passed to NewManager and a struct that represents the client data. The name of the struct must match the node type - for instance a node of type canBus needs to be defined by a struct named CanBus. Additionally, each field of the client struct must have point tags. This allows us to automatically create and modify client structs from arrays of node points.
3. Create a new manager for the client in client/client.go
4. Create an Elm UI for the client in frontend/src/Components/
5. Create plumbing for new NodeXYZ in frontend/src/Pages/Home_.elm. Note, this can likely be improved a lot.

It is easiest to copy one of the existing clients to start. The NTP client is relatively simple and may be a good example.

Client life-cycle

It is important the clients cleanly implement the Run()/Stop() pattern and shut down cleanly when Stop() is called releasing all resources. If nodes are added or removed, clients are started/stopped. Additionally, if a child node of a client config is added or removed, the entire client is stopped and then restarted. This relieves the burden on the client from managing the addition/removal of client functionality. Thus, it is very important that clients stop cleanly and release resources in case they are restarted.

Message echo

Clients need to be aware of the “echo” problem as they typically subscribe as well as publish to the points subject for the nodes they manage. When they publish to these subjects, these messages will be echoed back to them. There are several solutions:

1. Create a new NATS connection for the client with the NoEcho option set. For this to work, each client will need to establish its own connection to the server. This may not work in cases where subjects are aliased into authenticated subject namespaces.
2. Inspect the Point Origin field - if is blank, then it was generated by the node that owns the point and does not need to be processed by the client generating the data for that node. If is not blank, then the Point was generated by a user, rule, or something other than the client owning the node and must be processed. This may not always work - example: user is connected to a downstream instance and modifies a point that then propagates upstream
   • it may get echoed back to an authenticated client.
3. (investigation stage) A NATS messages header can be populated with the ID of the client that sent the message. If it is an authenticated client, then the message will not be echoed on the authenticated client subject namespace of the same ID. This information is not stored, so cannot be used for auditing purposes.

The SIOT client manager filters out points for the following two scenarios:

1. A point with the same ID as the client and Origin set to a blank string.
2. A point received for a client where Origin matches the client root node ID.

Thus, if you want to set a point in one client and get that point to another node client, you must set the Origin field. This helps ensure that the Origin field is used consistently as otherwise stuff won’t work.

This gets a little tricky for clients that manage a node and its children (for instance the rule client - it has condition and action child nodes). If we follow the following rule:

Clients must set the point Origin field for any point sent to anything other than its root node.

If we following the above rule, then things should work. We may eventually provide clients with a function to send points that handles this automatically, but for now it is manual.

See also tracking who made changes.

Development

Go Package Documentation

The Simple IoT source code is available on GitHub.

Simple IoT is written in Go. Go package documentation is available.

Building Simple IoT

Requirements:

• Go
• Node/NPM

Simple IoT build has currently been testing on Linux and MacOS systems. See envsetup.sh for scripts used in building.

To build:

• source envsetup.sh
• siot_setup
• siot_build

Developing Simple IoT

• npm install -g run-pty
• . envsetup.sh
• siot_setup
• siot_watch

The siot_watch command can be used when developing Simple IoT. This does the following:

• Starts elm-watch on the Elm code. elm-watch will automatically update the UI without losing state any time an Elm file changes.
• Runs the Go backend and rebuilds it anytime a Go module changes (only tested on Linux and MacOS, but should be easy to set up Windows as well)

Both of the above are run in a run-pty wrapper, which allows you to see the output of either process. The output of the Elm compile is displayed in the browser, so it is rarely necessary to view the elm-watch side.

Using Simple IoT as a library

Simple IoT can be used a library for your custom application. The SIOT main.go illustrates how to start the SIOT server, and add clients. You can do this from any Go application. With a few lines of code, this gives you a lot of functionality including a NATS server.

Developing a new SIOT client

Most SIOT functionality is implemented in clients. See the client documentation for more information.

Customizing the UI

Currently, there is no simple way to customize the SIOT UI when using SIOT as a library package. Forking and changing the SIOT Elm code is probably the simplest way if you want to make a small change now.

In the future, we plan to provide an API for passing in a custom UI to the SIOT Server. You can also implement a custom HTTP client that serves up a custom UI.

Code Organization

Currently, there are a lot of subdirectories. One reason for this is to limit the size of application binaries when building edge/embedded Linux binaries. In some use cases, we want to deploy app updates over cellular networks, therefore we want to keep packages as small as possible. For instance, if we put the natsserver stuff in the nats package, then app binaries grow a couple MB, even if you don’t start a NATS server. It is not clear yet what Go does for dead code elimination, but at this point, it seems referencing a package increases the binary size, even if you don’t use anything in it. (Clarification welcome!)

For edge applications on Embedded Linux, we’d eventually like to get rid of net/HTTP, since we can do all network communications over NATS. We’re not there yet, but be careful about pulling in dependencies that require net/HTTP into the NATS package, and other low level packages intended for use on devices.

Directories

See Go docs directory descriptions

Coding Standards

Please run siot_test from envsetup.sh before submitting pull requests. All code should be formatted and linted before committing.

Please configure your editor to run code formatters:

• Go: goimports
• Elm: elm-format
• Markdown: prettier (note, there is a .prettierrc in this project that configures prettier to wrap markdown to 80 characters. Whether to wrap markdown or not is debatable, as wrapping can make diffs harder to read, but Markdown is much more pleasant to read in an editor if it is wrapped. Since more people will be reading documentation than reviewing, let’s optimize for the reading in all scenarios - editor, GitHub, and generated docs)

Pure Go

We plan to keep the main Simple IoT application a pure Go binary if possible. Statically linked pure Go has huge advantages:

1. You can easily cross compile to any target from any build machine.
2. Blazing fast compile times
3. Deployment is dead simple – zero dependencies. Docker is not needed.
4. You are not vulnerable to security issues in the host systems SSL/TLS libraries. What you deploy is pretty much what you get.
5. Although there is high quality code written in C/C++, it is much easier to write safe, reliable programs in Go. Long term there is much less risk using a Go implementation of about anything – especially if it is widely used.
6. Go’s network programming model is much simpler than about anything else. Simplicity == less bugs.

Once you link to C libraries in your Go program, you forgo many of the benefits of Go. The Go authors made a brilliant choice when they chose to build Go from the ground up. Yes, you loose the ability to easily use some of the popular C libraries, but what you gain is many times more valuable.

Running unit tests

There are not a lot of unit tests in the project yet, but below are some examples of running tests:

• test everything: go test -race ./...
• test only client directory: go test -race ./client
• Run only a specific: go test -race ./client -run BackoffTest (run takes a RegEx)
• siot_test runs tests as well as vet/lint, frontend tests, etc.

The leading ./ is important, otherwise Go things you are giving it a package name, not a directory. The ... tells Go to recursively test all sub directories.

Document and test during development

It is much more pleasant to write documentation and tests as you develop, rather than after the fact. These efforts add value to your development if done concurrently. Quality needs to be designed-in, and leading with documentation will result in better thinking and a better product.

If you develop a feature, please update/create any needed documentation and write any tests (especially end-to-end) to verify the feature works and continues to work.

Data

Contents

See also:

• Data store
• Data synchronization

Data Structures

As a client developer, there are two main primary structures: NodeEdge and Point. A Node can be considered a collection of Points.

These data structures describe most data that is stored and transferred in a Simple IoT system.

The core data structures are currently defined in the data directory for Go code, and frontend/src/Api directory for Elm code.

A Point can represent a sensor value, or a configuration parameter for the node. With sensor values and configuration represented as Points, it becomes easy to use both sensor data and configuration in rule or equations because the mechanism to use both is the same. Additionally, if all Point changes are recorded in a time series database (for instance InfluxDB), you automatically have a record of all configuration and sensor changes for a node.

Treating most data as Points also has another benefit in that we can easily simulate a device. Provide an UI or write a program to modify any point and we can shift from working on real data to simulating scenarios we want to test.

Edges are used to describe the relationships between nodes as a directed acyclic graph.

Nodes can have parents or children and thus be represented in a hierarchy. To add structure to the system, you simply add nested Nodes. The Node hierarchy can represent the physical structure of the system, or it could also contain virtual Nodes. These virtual nodes could contain logic to process data from sensors. Several examples of virtual nodes:

• A pump Node that converts motor current readings into pump events.
• Implement moving averages, scaling, etc. on sensor data.
• Combine data from multiple sensors
• Implement custom logic for a particular application
• A component in an edge device such as a cellular modem

Like Nodes, Edges also contain a Point array that further describes the relationship between Nodes. Some examples:

• Role the user plays in the node (viewer, admin, etc.)
• Order of notifications when sequencing notifications through a node’s users
• Node is enabled/disabled for instance we may want to disable a Modbus IO node that is not currently functioning.

Being able to arranged nodes in an arbitrary hierarchy also opens up some interesting possibilities such as creating virtual nodes that have a number of children that are collecting data. The parent virtual nodes could have rules or logic that operate off data from child nodes. In this case, the virtual parent nodes might be a town or city, service provider, etc., and the child nodes are physical edge nodes collecting data, users, etc.

The Point Key field constraint

The Point data structure has a Key field that can be used to construct Array and Map data structures in a node. This is a flexible idea in that it is easy to transition from a scaler value to an array or map. However, it can also cause problems if one client is writing key values of "" and another client (say a rule action) is writing value of "0". One solution is to have fancy logic that equates "" to "0" on point updates, compares, etc. Another approach is to consider "" and invalid key value and set key to "0" for scaler values. This incurs a slight amount of overhead, but leads to more predictable operation and eliminates the possibility of having two points in a node that mean the same things.

The Simple IoT Store always sets the Key field to "0" on incoming points if the Key field is blank.

Clients should be written with this in mind.

The Point Type and Key character constraint

A point travels on a NATS subject that ends in its type and key, so both are subject tokens and may not contain a period, whitespace, or the NATS wildcards * and >. Listeners read the node ID and parent ID from fixed positions in a subject, so a period would add a token, shift everything after it, and deliver the point to the wrong handler.

The store rejects points it cannot publish rather than rewriting them, since a key is data the sender chose and often the name it writes back to. A rejected point is logged with its type and key, and an error point is set on the node so the sender is visible in the UI.

A client that builds keys from names it does not control – kernel device names, mount points, network interface names – should pass them through data.SubjectSafeToken, which replaces the offending characters with underscores. The metrics client does this for sensor and interface names, which is why a cooling device the kernel calls devfreq-17000000.gpu appears as devfreq-17000000_gpu.

Converting Nodes to other data structures

Nodes and Points are convenient for storage and synchronization, but cumbersome to work with in application code that uses the data, so we typically convert them to another data structure. data.Decode, data.Encode, and data.MergePoints can be used to convert Node data structures to your own custom struct, much like the Go json package.

Arrays and Maps

Points can be used to represent arrays and maps. For an array, the key field contains the index "0", "1", "2", etc. For maps, the key field contains the key of the map. An example:

The above would map to the following Go type:

type myNode struct {
    ID              string      `node:"id"`
    Parent          string      `node:"parent"`
    Description     string      `node:"description"`
    IpAddresses     []string    `point:"ipAddress"`
    Switches        []bool      `point:"switch"`
    DiscPercentUsed []float64   `point:"diskPercentUsed"`
}

The data.Decode() function can be used to decode an array of points into the above type. The data.Merge() function can be used to update an existing struct from a new point.

Best practices for working with arrays

To make changes to an array in UI/Client code when storing the array in a native structure, store a length field as well so you know how long the original array was. After modifying the array, check if the new length is less than the original - if it is, then add a tombstone points to the end so that the deleted points get removed.

Generally it is simplest to send the entire array as a single message any time any value in it has changed - especially if values are going to be added or removed. The data.Decode will then correctly handle the array resizing.

Technical details of how data.Decode works with slices

Some consideration is needed when using Decode and MergePoints to decode points into Go slices. Slices are never allocated / copied unless they are being expanded. Instead, deleted points are written to the slice as the zero value. However, for a given Decode call, if points are deleted from the end of the slice, Decode will re-slice it to remove those values from the slice. Thus, there is an important consideration for clients: if they wish to rely on slices being truncated when points are deleted, points must be batched in order such that Decode sees the trailing deleted points first. Put another way, Decode does not care about points deleted from prior calls to Decode, so “holes” of zero values may still appear at the end of a slice under certain circumstances. Consider points with integer values [0, 1, 2, 3, 4]. If tombstone is set on point with Key 3 followed by a point tombstone set on point with Key 4, the resulting slice will be [0, 1, 2] if these points are batched together. But, if they are sent separately (thus resulting in multiple Decode calls), the resulting slice will be [0, 1, 2, 0].

Node Topology changes

Nodes can exist in multiple locations in the tree. This allows us to do things like include a user in multiple groups.

Add

Node additions are detected in real-time by sending the points for the new node as well as points for the edge node that adds the node to the tree.

Copy

Node copies are similar to add, but only the edge points are sent.

Delete

Node deletions are recorded by setting a tombstone point in the edge above the node to true. If a node is deleted, this information needs to be recorded, otherwise the synchronization process will simply re-create the deleted node if it exists on another instance.

Move

Move is just a combination of Copy and Delete.

If the any real-time data is lost in any of the above operations, the catch up synchronization will propagate any node changes.

Tracking who made changes

The Point type has an Origin field that is used to track who generated this point. If the node that owned the point generated the point, then Origin can be left blank - this saves data bandwidth - especially for sensor data which is generated by the client managing the node. There are several reasons for the Origin field:

• Track who made changes for auditing and debugging purposes. If a rule or some process other than the owning node modifies a point, the Origin should always be populated. Tests that generate points should generally set the origin to “test”.
• Eliminate echos where a client may be subscribed to a subject as well as publish to the same subject. With the Origin field, the client can determine if it was the author of a point it receives, and if so simply drop it. See client documentation for more discussion of the echo topic.

Evolvability

One important consideration in data design is the can the system be easily changed. With a distributed system, you may have different versions of the software running at the same time using the same data. One version may use/store additional information that the other does not. In this case, it is very important that the other version does not delete this data, as could easily happen if you decode data into a type, and then re-encode and store it.

With the Node/Point system, we don’t have to worry about this issue because Nodes are only updated by sending Points. It is not possible to delete a Node Point. So it one version writes a Point the other is not using, it will be transferred, stored, synchronized, etc. and simply ignored by version that don’t use this point. This is another case where SIOT solves a hard problem that typically requires quite a bit of care and effort.

Simple IoT Store

Simple IoT stores all application data in NATS JetStream, using the NATS server that is already embedded in every SIOT instance. There is no separate database: the same technology that moves messages between components also persists them, retains their history, and (see Data Synchronization) replicates them between instances.

ADR-7 records the full analysis behind this design. Earlier versions of SIOT used SQLite; existing SQLite data can be migrated with siot export / siot import.

Why JetStream

A JetStream stream is an append-only, persistent log of messages with sequence numbers and per-subject indexing. That shape matches IoT data unusually well:

• Points are already messages. SIOT components communicate by publishing points over NATS. Persisting them is a matter of capturing the same messages in a stream, not translating them into a second data model.
• History is the natural byproduct. A stream retains every point written to a subject, so time-series history is stored in the same place as current state, rather than requiring a separate time-series database.
• Sequence numbers replace the hash tree. Streams are ordered and sequence-tracked, so another instance can replicate one and know exactly what it has and has not seen. This is the foundation of the synchronization design.
• Small and embedded. JetStream runs inside the NATS server SIOT already ships, on cloud instances and small edge devices alike.

Boundaries and streams

Streams are created per boundary, not per node. A boundary is a node that represents a SIOT instance:

• the local instance’s root node, and
• any device-type node, which corresponds to a (potentially synced) remote instance.

Every node is owned by the nearest boundary found walking up the tree through undeleted edges. A node reachable from no boundary, or from more than one, is owned by the instance root boundary. Boundaries align with the natural units of synchronization and authorization: a device’s subtree syncs as a unit, and permissions are typically granted at device or group level.

Each (boundary, origin instance) pair gets one stream:

inst_<boundaryID>_<originID>

where originID is the root node ID of the instance that writes the stream. The inst (instance) prefix keeps the word “node” reserved for nodes in the data tree — both stream tokens identify instances, since a boundary is a node that represents one. Only the origin instance ever appends to its stream — this single-writer property is what makes synchronization simple and echo-free. A standalone instance with root R has a single stream, inst_R_R, holding its entire tree.

A hub R with a synced device X sees three streams:

Subjects

Two subject spaces are in play. Wire subjects are how points move between components in real time — they are plain NATS, not stored:

Listeners tell the two fan-out subjects apart by counting tokens, so a point type or key may not contain a period. The store checks this on every point it accepts — see checkPoints in store/store.go and the data reference.

Storage subjects are what streams capture. They carry both routing tokens so stream subject spaces never overlap:

The stream inst_<b>_<o> captures inst.<b>.<o>.>. Edges are stored with the parent node’s boundary, so the edge attaching a device into a hub’s tree lives in the hub’s stream — the device never needs it.

Current state: merge of subject tips

The current value of a point is the tip (last message) of its storage subject. Because a boundary can have streams from several origins (the device’s own data plus hub-written configuration), current state is the merge of tips across all inst_<boundaryID>_* streams, under one rule:

1. The newest point timestamp wins (timestamps are embedded in the point, not taken from the stream).
2. Equal timestamps from different origins resolve to the lexically greater origin ID, so every instance converges on the same winner.
3. An identical (timestamp, origin) delivery is a no-op, which makes the merge idempotent when the same point arrives more than once.

The store holds this merged state in two in-memory caches — an edge cache (the tree) and a point cache (current points) — populated by reading every stream’s subject tips at startup. The caches are the read path; queries never touch JetStream. Writes check the cache tip first, append to the stream, then update the cache, with a load-on-miss backstop.

Writes, deletes, and moves

A local write routes to inst_<owningBoundary>_<self>. Deleting a node writes a tombstone point on its parent edge — history is preserved and the delete can be undone. Moving a node (or subtree) across boundaries republishes its subject tips into the new boundary’s stream, preserving original point timestamps, then purges the old subjects; ownership follows the tree.

Retention and durability

Streams keep the most recent 5000 messages per subject by default. Because the limit is per subject, current state — including rarely-written configuration points — is always preserved, which time- or size-based retention could not guarantee. The default is sized so that:

• data reported every 10 minutes keeps about a month of local history,
• configuration subjects, written a handful of times, are effectively unlimited, and
• disk use on unattended edge devices stays bounded (a 1-per-minute subject would otherwise grow by ~525k messages a year).

History is tiered by write rate: fast subjects wrap sooner locally, and long-term history for them belongs in an external time-series database fed by the Db client, which reads the streams gap-free.

--storeMaxMsgsPerSubject (or SIOT_STORE_MAX_MSGS_PER_SUBJECT) overrides the default; -1 means unlimited. Each instance applies its own policy to every stream on its own disk — including replica streams, which the sync pumps create bare and the store configures when it discovers them — so a hub and a device can retain different amounts of the same data.

The JetStream file store fsyncs on a 2-minute interval by default. --storeSyncInterval (or SIOT_STORE_SYNC_INTERVAL) accepts a Go duration to shorten that window, or always to fsync every write, for edge devices with unreliable power, at a write-throughput cost.

Instance metadata

A small META key/value bucket (also JetStream) holds the instance’s root node ID and JWT signing key.

Data Synchronization

Simple IoT synchronizes data between instances by replicating the store’s streams rather than by comparing and copying tree state. Each instance appends only to its own streams (see Store); other instances hold replicas of those streams and merge them at read time. This page explains how the pieces of NATS — core subjects, JetStream streams, durable consumers, and message headers — combine to do this.

ADR-7 records the design analysis. The previous implementation, which compared Merkle-style node hashes and pushed subtrees, is fully replaced; streams carry their own sequence numbers, so nothing needs to be compared to know what is missing.

The model in one paragraph

Every stream has exactly one writing instance (its origin). A device with root X writes everything to its stream inst_X_X; a hub with root R writes configuration for the device’s subtree to its own stream inst_X_R. Sync means each side keeps a copy of the other’s stream: the hub holds a replica of inst_X_X, the device holds a replica of inst_X_R. Current state on either side is the merge of the subject tips of both streams — newest timestamp wins, with a deterministic origin tie-break. Because no instance ever writes remote data into its own streams, points cannot echo back and forth between instances: there is no loop to suppress.

   device X                                hub R
  ┌───────────────────┐                  ┌───────────────────┐
  │ inst_X_X (owned)  │ ──── push ─────► │ inst_X_X (replica)│
  │ inst_X_R (replica)│ ◄─── pull ────── │ inst_X_R (owned)  │
  │                   │                  │ inst_R_R (owned)  │
  └───────────────────┘                  └───────────────────┘
     merge tips of both                     merge tips of both
     = current state                        = current state

How each NATS feature is used

Core NATS subjects (p.>, ep.>, up.>) carry points between components within an instance in real time, exactly as before — clients, rules, and the UI are unaware of synchronization. The store subscribes to these wire subjects, persists local writes to its origin streams, and fans points out to up.> subjects for listeners like rules and database clients.

JetStream streams are both the store and the unit of sync. Because storage subjects embed the boundary and origin (inst.<boundary>.<origin>.…), a replica stream on another instance can use the same name and subjects — a copied message needs no translation.

Durable consumers drive replication. The sync client (which runs on the downstream instance and connects to the upstream’s NATS server, using the URI and auth token on its Sync node) runs two pumps:

• push: a durable consumer on the local inst_X_X delivers each message, and the pump publishes it — same subject, same payload — to the upstream, where the replica stream captures it.
• pull: a durable consumer on the upstream’s inst_X_R does the same in the other direction.

A message is acknowledged only after the receiving side confirms the write. A durable consumer remembers its position across disconnects, so a reconnect delivers exactly the messages the other side missed — no rescan, no comparison. This is what replaces the hash tree: the stream sequence is the synchronization state.

The pumps move messages and nothing else: they create a missing replica stream but never change an existing stream’s configuration. Each instance’s store owns the configuration of the streams on its own disk and applies its retention policy to replica streams when it discovers them, so a hub can keep more (or less) history of a device’s data than the device keeps itself.

Message headers solve origin attribution. When a store consumes a replica stream, it merges each message into its caches and, when a tip changes, re-broadcasts it on the ordinary wire subjects so local clients react — tagged with a Siot-Origin header naming the writing instance. A store receiving a wire message tagged with a remote origin merges it and fans it out but never persists it; the replica stream is the persistent copy. This single rule keeps the single-writer property intact everywhere.

Life of a connection

1. The sync client connects to the upstream NATS server (plain NATS or NATS over WebSocket).
2. Adoption: if the upstream tree has no node with this instance’s root ID, the client announces itself with one edge message; the upstream persists a device node under its root. (This is an ordinary untagged write — from the upstream’s view it is its own edge, in its own boundary.)
3. The push pump ensures the replica stream exists upstream and starts copying; the device’s whole tree — structure, configuration, and history — arrives through it, from sequence 1 on first connect.
4. The pull pump discovers upstream-origin streams for this instance’s boundary and copies them down; the first hub-side configuration write creates inst_X_R, and the device picks it up on its next scan.
5. Each store’s replica consumers merge the arriving messages and re-broadcast changed tips locally.

Configuration written on the hub before the device ever connects (pre-provisioning) simply waits in inst_X_R and arrives on first connect.

Offline catch-up

While disconnected, both sides keep writing to their own streams. On reconnect, the durable consumers resume and deliver only the backlog. Two kinds of consumers see that backlog differently:

• State clients (rules, protocol clients, the UI) should not see a replay of stale intermediate values. The store therefore holds re-broadcasts while a replica consumer has a backlog and emits one message per changed subject — the final tip — once it drains.
• History needs every point. It is preserved automatically: the replica stream receives the full backlog in order with original embedded timestamps, so local history stays gap-free (up to each stream’s retention limit). The Db (InfluxDB) client works this way: it consumes the streams with its own durable consumers, so an external time-series database receives every point — including the backlog after a sync outage or the client’s own downtime — rather than only what happened to cross the wire while it was listening. External sinks can follow the same pattern.

Conflicts

Concurrent writes to the same point from two instances are rare in practice — a sensor value has one source, a setting is usually edited in one place. When they happen, every instance applies the same merge rule to the same streams: newest embedded timestamp wins, and equal timestamps resolve to the lexically greater origin ID, so all instances converge on the same value without coordination.

Deleting a device (detach)

The edge that attaches a device into the hub’s tree lives in the hub’s own boundary stream, which the device does not replicate. Tombstoning that edge on the hub therefore detaches the device: the hub stops showing it, while the device keeps operating standalone, unaware. The device does not force itself back into the tree; only the hub can restore the edge (undelete), after which replication resumes where it left off.

Current limitations and direction

• Only the instance’s root boundary replicates today; nested device boundaries (a device that itself syncs devices) are planned.
• Replication runs over the ordinary upstream client connection. JetStream sourcing across NATS leaf connections — where the NATS servers replicate the streams themselves — is verified to work (see store/leafnode_spike_test.go) and is the intended replacement once per-instance JetStream domain configuration is worked out.
• Authorization currently uses the shared auth token; per-stream permissions issued via NATS auth callout are the planned tightening, and the stream-per-boundary layout is what makes one-rule-per-device grants possible.

See the Stage 3 plan for the full status list.

Reliability

Reliability is an important consideration in any IoT system as these systems are often used to monitor and control critical systems and processes. Performance is a key aspect of reliability because if the system is not performing well, then it can’t keep up and do its job.

Point Metrics

The fundamental operation of SimpleIoT is that it process points, which are changes to nodes. If the system can’t process points at the rate they are coming in, then we have a problem as data will start to back up and the system will not be responsive.

Points and other data flow through the NATS messaging system, therefore it is perhaps the first place to look. We track several metrics that are written to the root device node to help track how the system is performing.

The NATS client buffers messages that are received for each subscription and then messages are dispatched serially one message at a time. If the application can’t keep up with processing messages, then the number of buffered messages increases. This number is occasionally read and then min/max/avg written to the metricNatsPending* points in the root device node.

The time required to process points is tracked in the metricNatsCycle* points in the root device node. The cycle time is in milliseconds.

We also track point throughput (messages/sec) for various NATS subjects in the metricNatsThroughput* points.

These metrics should be graphed and notifications sent when they are out of the normal range. Rules that trigger on the point type can be installed high in the tree above a group of devices so you don’t have to write rules for every device.

Database interactions

Database operations greatly affect system performance. When Points come into the system, we need to store this data in the primary and time series stores (ex InfluxDB). The time it takes to read and write data greatly impacts how much data we can handle.

IO failures

All errors reading/writing IO devices should be tracked at both the device and bus level. These can be observed over time and abnormal rates can trigger notifications. Error counts should be reported at a low rate to avoid using bandwidth and resources - especially if multiple counts are incremented on an error (IO and bus).

Logging

Many errors are currently reported as log messages. Eventually some effort should be made to turn these into error counts and possibly store them in the time series store for later analysis.

API

Contents

The Simple IoT server currently provides both HTTP and NATS.io APIs. We’ve tried to keep the two APIs a similar as possible so it is easy to switch from one to the other. The Http API currently accepts JSON, and the NATS API uses a binary encoding for points and protobuf for node requests.

NOTE, the Simple IoT API is not final and will continue to be refined in the coming months.

NATS

NATS.io allows more complex and efficient interactions between various system components (device, cloud, and web UI). These three parts of the system make IoT systems inherently distributed. NATS focuses on simplicity and is written in Go which ensures the Go client is a 1st class citizen and allows for interesting possibilities such as embedding in the NATS server in various parts of the system. This allows us to keep our one-binary deployment model.

The siot binary embeds the NATS server, so there is no need to deploy and run a separate NATS server.

Point data uses a compact binary encoding (see data/point.go). Node requests still use protobuf, defined here. Each node point is sent as a single NATS message with type and key encoded in the subject.

Because the type and key become subject tokens, they may not contain a period, whitespace, or the NATS wildcards * and >. Listeners read the node ID and parent ID from fixed positions in a subject, so a type or key carrying a period would add a token and shift everything after it, delivering the point to the wrong handler. The store checks every point on the way in and rejects any that cannot be published, logging the offending type and key and setting an error point on the node so the sender can be found. A client that generates keys from names it does not control – sysfs device names, mount points, network interface names – should pass them through data.SubjectSafeToken first.

• Nodes
  • nodes.<parentId>.<nodeId>.<type>.<key>
    • Request/response – returns an array of data.EdgeNode structs.
    • parent="all", then all instances of the node are returned.
    • parent is set and id="all", then all child nodes of the parent are returned.
    • parent="root" and id="all" to fetch the root node(s).
    • The following combinations are invalid:
      • parent="all" && id="all"
    • Parameters can be specified as points in payload
      • tombstone with value field set to 1 will include deleted points
      • nodeType with text field set to node type will limit returned nodes to this type
  • p.<nodeId>.<type>.<key>
    • used to listen for or publish node point changes.
  • ep.<nodeId>.<parentId>.<type>.<key>
    • used to publish/subscribe node edge points. The tombstone point type is used to track if a node has been deleted or not.
  • phr.<nodeId> (not currently used)
    • high rate point data
  • phrup.<upstreamId>.<nodeId>
    • High rate point data is rebroadcast upstream. upstreamId is the parent of the node that is interested in HR data (currently the db node). nodeId is the node that is providing the HR data. In the case of a custom HR Dest Node (serial client), the serial client may not be a child of the upstream node.
  • up.<upstreamId>.<nodeId>.<type>.<key>
    • node points are rebroadcast at every upstream ID so that we can listen for point changes at any level. The sending node is also included in this. The store is responsible for posting to up subjects. Individual clients should not do this.
  • up.<upstreamId>.<nodeId>.<parentId>.<type>.<key>
    • edge points rebroadcast at every upstream node ID.
• Legacy APIs that are being deprecated
  • node.<id>.not
    • used when a node sends a notification (typically a rule, or a message sent directly from a node)
  • node.<id>.msg
    • used when a node sends a message (SMS, email, phone call, etc). This is typically initiated by a notification.
  • node.<id>.file (not currently implemented)
    • Is used to transfer files to a node in chunks, which is optimized for unreliable networks like cellular and is handy for transferring software update files.
• Auth
  • auth.user
    • Used to authenticate a user. Send a request with email/password points, and the system will respond with the User nodes if valid. There may be multiple user nodes if the user is instantiated in multiple places in the node graph. A JWT node will also be returned with a token point. This JWT should be used to authenticate future requests. The frontend can then fetch the parent node for each user node.
  • auth.getNatsURI
    • This returns the NATS URI and Auth Token as points. This is used in cases where the client needs to set up a new connection to specify the no-echo option, or other features.
• Admin
  • admin.error (not implemented yet)
    • Any errors that occur are sent to this subject
  • admin.storeVerify
    • Used to initiate a database verification process. This currently verifies hash values are correct and responds with an error string.
  • admin.storeMaint
    • Corrects errors in the store (current incorrect hash values)

HTTP

For details on data payloads, it is simplest to just refer to the Go types which have JSON tags. HTTP APIs currently return JSON payloads.

Most APIs that do not return specific data (update/delete) return a standard response

• Nodes
  • data structure
  • /v1/nodes
    • GET: return a list of all nodes
    • POST: insert a new node
  • /v1/nodes/:id
    • GET: return info about a specific node. Body can optionally include the id of parent node to include edge point information.
    • DELETE: delete a node
  • /v1/nodes/:id/parents
    • POST: move node to new parent
    • PUT: mirror/duplicate node
    • body is JSON api/nodes.go:NodeMove or NodeCopy structs
  • /v1/nodes/:id/points
    • POST: post points for a node
  • /v1/nodes/:id/cmd
    • GET: gets a command for a node and clears it from the queue. Also clears the CmdPending flag in the Device state.
    • POST: posts a cmd for the node and sets the node CmdPending flag.
  • /v1/nodes/:id/not
    • POST: send a notification to all node users and upstream users
• Auth
  • /v1/auth
    • POST: accepts email and password as form values, and returns a JWT Auth token

HTTP Examples

You can post a point using the HTTP API without authorization using curl:

curl -i -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '[{"type":"value", "value":100}]' http://localhost:8118/v1/nodes/be183c80-6bac-41bc-845b-45fa0b1c7766/points

If you want HTTP authorization, set the SIOT_AUTH_TOKEN environment variable before starting Simple IoT and then pass the token in the authorization header:

curl -i -H "Authorization: f3084462-3fd3-4587-a82b-f73b859c03f9" -H "Content-Type: application/json" -H "Accept: application/json" -X POST -d '[{"type":"value", "value":100}]' http://localhost:8118/v1/nodes/be183c80-6bac-41bc-845b-45fa0b1c7766/points

Frontend

Elm Reference Implementation

The reference Simple IoT frontend is implemented in Elm as a Single Page Application (SPA) and is located in the frontend/ directory.

Code Structure

The frontend is based on elm-spa, and is split into the following directories:

• Api: contains core data structures and API code to communicate with backend (currently REST).
• Pages: the various pages of the application
• Components: each node type has a separate module that is used to render it. NodeOptions.elm contains a struct that is used to pass options into the component views.
• UI: Various UI pieces we used
• Utils: Code that does not fit anywhere else (time, etc.)

We’d like to keep the UI optimistic if possible.

Creating Custom Icons

SIOT icons are 24x24px pixels (based on feather icon format). One way to create them is to:

• Create a 24x24px drawing in Inkscape, scale=1.0
• draw your icon
• if you use text
  • Convert text to path: select text, and then menu Path -> Object to Path
  • Make sure fill is set for path
• save as plain SVG
• set up a new Icon in frontend/src/UI/Icon.elm and use an existing custom icon like variable as a template.
• Copy the SVG path strings from the SVG file into the new Icon
• You’ll likely need to adjust the scaling transform numbers to get the icon to the right size

(I’ve tried using: https://levelteams.com/svg-to-elm, but this has not been real useful, so I usually end up just copying the path strings into an elm template and hand edit the rest)

File upload

The File node UI has the capability to upload files in the browser and then store them in a node point. The default max payload of NATS is 1MB, so that is currently the file size limit, but NATS can be configured for a payload size up to 64MB. 8MB is recommended.

Currently the payload is stored in the Point String field for simplicity. If the binary option is selected, the data is base64 encoded. Long term it may make sense to support JetStream Object store, local file store, etc.

The elm/file package is used upload a file into the browser. Once the data is in the browser, it is sent to the backup as a standard point payload. Because we are currently using a JSON API, binary data is base64 encoded.

The process by which a file is uploaded is:

• The NodeOptions struct, which is passed to all nodes has an onUploadFile field, which is used to triggers the UploadFile message which runs a browser file select. The result of this select is a UploadSelected message.
• This message calls UploadFile node.node.id in Home_.elm.
• File.Select.file is called to select the file, which triggers the UploadContents message.
• UploadContents is called with the node id, file name, and file contents, which then sends the data via points to the backend.

SIOT JavaScript library using NATS over WebSockets

This is a JavaScript library available in the frontend/lib directory that can be used to interface a frontend with the SIOT backend.

Usage:

import { connect } from "./lib/nats"

async function connectAndGetNodes() {
	const conn = await connect()
	const [root] = await conn.getNode("root")
	const children = await conn.getNodeChildren(root.id, { recursive: "flat" })
	return [root].concat(children)
}

This library is also published on NPM (in the near future).

(see #357)

(Note, we are not currently using this yet in the SIOT frontend we still poll the backend over REST and fetch the entire node tree, but we are building out infrastructure so we don’t have to do this.)

Custom UIs

The current SIOT UI is more an engineering type view than something that might be used by end users. For a custom/company product IoT portal where you want a custom web UI optimized for your products, there are several options:

1. Modify the existing SIOT frontend.
2. Write a new frontend, mobile app, desktop app, etc. The SIOT backend and frontend are decoupled so that this is possible.

Passing a custom UI to SIOT

There are ways to use a custom UI with SIOT at the app and package level:

1. Application: pass a directory containing your public web assets to the app using: siot serve -customUIDir <your web assets>
2. Package: populate CustomUIFS with a fs.FS in the SIOT server options`.

In both cases, the filesystem should contain a index.html in the root directory. If it does not, you can use the fs.Sub function to return a subtree of a fs.FS.

Rules

Rules are defined by nodes and are composed of additional child nodes for conditions and actions. See the node/point schema for more details.

All points should be sent out periodically, even if values are not changing to indicate a node is still alive and eliminate the need to periodically run rules. Even things like system state should be sent out to trigger device/node offline notifications.

Notifications

(see notification user documentation

Notifications are messages that are sent to users. There several concerns when processing a notification:

1. The message itself and how it is generated.
2. Who receives the messages.
3. Mechanism for sending the message (Twilio SMS, SMTP, etc.)
4. State of the notification
   1. Sequencing through a list of users
   2. Tracking if it was acknowledged and by who
5. Distributed concerns (more than one SIOT instance processing notifications)
   1. Synchronization of notification state between instances.
   2. Which instance is processing the notification.

For now, we assume all notifications will be handled on a single SIOT instance (typically in the cloud) - distributed aspects of notifications will implemented later.

Notifications can be initiated by:

1. Rules
2. Users sending notifications through the web UI

Notification Data Structures

Elsewhere in the system, configuration and sensor data are represented as Points. The Point data structure is optimized for synchronization and algorithms where simplicity and consistency allows us to easily process points with common code. But a Point does not have enough fields to represent a message or notification. We could encode the message as JSON in the Point text field, but it would be nice to have something a little more descriptive. Additionally, all notifications and messages should be stored in the time series database so there is a history of everything that was sent.

Time series databases like InfluxDB store records with the following attributes:

• Timestamp
• Measurement (similar to collection, bucket, or table in other databases)
• Keys (string only, indexed)
• Values (can be a variety of data types: float, integer, string, boolean)

Notifications will be handled by two data structures:

• Notification
  • Typically generated by a rule or a node that is directly sending a message
  • Stored in main database as they may contain state that needs to be processed over time
• Message
  • An individual message to a user (SMS, Email, voice call)
  • Stored in time series database as they are transient

Notifications are initiated (as is all write data in the system) by sending a message through NATS. The typical flow is as follows:

Rule -> Notification -> Message

Integration

This page discusses ways you can integration Simple IoT into your system. At its core, SIOT is a distributed graph database optimized for storing, viewing, and synchronizing state/config in IoT systems. This makes it very useful for any system where you need distributed state/config.

With SIOT, you run the same application in both the cloud and edge devices, so you can use any of the available integration points at either place.

The SIOT API

This primary way to interact with Simple IoT is through a NATS API. You can add additional processes written in any language that has a NATS client. Additionally, the NATS wire protocol is fairly simple so could be implemented from scratch if needed. If your most of your system is written in C++, but you needed a distributed config/state store, then run SIOT along side your existing processes and add a NATS connection to SIOT. If you want easy scripting in your system, consider writing a Python application that can read/modify the SIOT store over NATS.

SIOT Data Structures

The SIOT data structures are very general (nodes and points) arranged in a graph, so you can easily add your own data to the SIOT store by defining new node and point types as needed. This makes SIOT very flexible and adaptable to about any purpose. You can use points in a node to represent maps and arrays. If your data needs more structure, then nested nodes can accomplish that. It is important with SIOT data to retain CRDT properties. These concepts are discussed more in ADR-1.

The requirement to only use nodes and points may seem restrictive at first, but can be viewed as a serialization format with CRDT properties that are convenient for synchronization. Any distributed database requires meta data around your data to assist with synchronization. With SIOT, we have chosen to make this metadata simple and accessible to the user. It is typical to convert this data to more convenient data structures in your application - much the same way you would deserialize JSON.

The architecture page discusses data structures in more detail.

Time series data and Graphing

If you need history and graphs, you can add InfluxDB and Grafana. This instantly provides history and graphs of all state and configuration changes that happened in the system.

Embedded Linux Systems

Simple IoT was designed with Embedded Linux systems in mind, so it is very efficient - a single, statically linked Go binary with all assets embedded that is ~20MB in size and uses ~20MB of memory. There are no other dependencies required such as a runtime, other libraries, etc. This makes SIOT extremely easy to deploy and update. An Embedded Linux system deployed at the edge can be synchronized with a cloud instance using a sync connection.

Integration with MCU (Microcontroller) systems

MCUs are processors designed for embedded control and are typically 32-bit CPUs that run bare-metal code or a small OS like FreeRTOS or Zephyr and don’t have as much memory as MPUs. MCUs cannot run the full SIOT application or easily implement a full data-centric data store. However, you can still leverage the SIOT system by using the node/point data structures to describe configuration and state and interacting with a Simple IoT like any other NATS client. Points use a compact binary encoding (see data/point.go) that is straightforward to implement on MCUs.

If your MCU supports MQTT, then it may make sense to use that to interact with Simple IoT as MQTT is very similar to NATS, and NATS includes a built-in MQTT server. The NATS wire protocol is also fairly simple and can also be implemented on top of any TCP/IP stack.

If your MCU interfaces with a local SIOT system using USB, serial, or CAN, then you can use the SIOT serial adapter.

Serial Devices

Contents

(see also user documentation and SIOT Firmware)

It is common in embedded systems architectures for an MPU (Linux-based running SIOT) to be connected via a serial link (RS232, RS485, CAN, USB serial) to an MCU.

See this article for a discussion on the differences between an MPU and MCU. These devices are not connected via a network interface, so can’t use the SIOT NATS API directly, thus we need to define a proxy between the serial interface and NATS for the MCU to interact with the SIOT system.

State/config data in both the MCU and MPU systems are represented as nodes and points. An example of nodes and points is shown below. These can be arranged in any structure that makes sense and is convenient. Simple devices may only have a single node with a handful of points.

SIOT does not differentiate between state (ex: sensor values) and config (ex: pump turn-on delay) - it is all points. This simplifies the transport and allows changes to be made in multiple places. It also allows for the granular transmission and synchronization of data - we don’t need to send the entire state/config anytime something changes.

SIOT has the ability to log points to InfluxDB, so this mechanism can also be used to log messages, events, state changes, whatever - simply use an existing point type or define a new one, and send it upstream.

Data Synchronization

By default, the serial client synchronizes any extra points written to the serial node. The serial UI displays the extra points as shown below:

Alternatively, there is an option for the serial client to sync its parent’s points to the serial device. When this is selected, any points received from the serial device are posted to the parent node, and any points posted to the parent node that were not sent by the serial device are forwarded to the serial client.

Protocols

Two wire protocols are available, selected with the protocol point on the serial node:

• binary (or empty) — COBS-framed packets, described below. Compact, and the only protocol supporting high-rate data and file transfer.
• shell — lines of ASCII exchanged with a Zephyr console shell, described in Shell Protocol.

Binary Protocol

The SIOT serial protocol mirrors the NATS PUB message with a few assumptions:

• we don’t have mirrored nodes inside the MCU device
• the number of nodes and points in a MCU is relatively small
• the payload is always an array of points
• only the following SIOT NATS API subjects are supported:
  • blank (assumes ID of Serial MCU client node
  • p.<id>.<type>.<key> (used to send node points)
  • ep.<id>.<parent> (used to send edge points)
  • phr (specifies high-rate payload)
• We don’t support NATS subscriptions or requests - on startup, we send the entire dataset for the MCU device in both directions (see On connection section), merge the contents, and then assume any changes will get sent and received after that.

subject can be left blank when sending/receiving points for the MCU root node. This saves some data in the serial messages.

The point type nodeType is used to create new nodes and to send the node type on connection.

All packets are ack’d (in both directions) by an empty packet with the same sequence number and subject set to ‘ack’. If an ack is not received in X amount of time, the packet is retried up to 3 times, and then the other device is considered “offline”.

Encoding

Packet Frame

All packets between the SIOT and serial MCU systems are framed as follows:

sequence (1 byte, rolls over)
subject (16 bytes)
payload (binary encoded point array or HR repeated point payload)
crc (2 bytes) (Currently using CRC-16/KERMIT) (not included on log messages)

Protocols like RS232 and USB serial do not have any inherent framing; therefore, this needs to be done at the application level. SIOT encodes each packet using COBS (Consistent Overhead Byte Stuffing).

Log payload

The log message is specified with log in the packet frame subject. The payload is ASCII characters and CRC not included.

Point payload

Points are encoded using a compact binary format (see data/point.go Encode/DecodePoints). The format is:

count (4 bytes, little-endian uint32)
repeated:
  type (2 byte length prefix + string)
  key (2 byte length prefix + string)
  time (8 bytes, little-endian int64, nanoseconds since epoch)
  dataType (1 byte: 0=unknown, 1=float, 2=int, 3=string, 4=JSON)
  data (2 byte length prefix + bytes)
  tombstone (4 bytes, little-endian int32)
  origin (2 byte length prefix + string)

This encoding is used for low-rate samples, config, state, etc.

High-rate payload

A simple payload encoding for high-rate data can be used to avoid the overhead of Protobuf encoding and is specified with phr in the packet frame subject.

type (16 bytes) point type
key (16 bytes) point key
starttime (uint64) starting time of samples in ns since Unix Epoch
sampleperiod (uint32) time between samples in ns
data (variable, remainder of packet), packed 32-bit floating point samples

This data bypasses most of the processing in SIOT and is sent to a special phr NATS subject. Clients that are interested in high-rate data (like the InfluxDB client) can listen to these subjects.

File payload

This payload type is for transferring files in blocks. These files may be used for firmware updates or other transfers where large amounts of data need to be transferred. An empty block with index set to -1 is sent at the end of the transfer.

name (16 bytes) filename
index (4 bytes) file block index
data (variable, remainder of packet)

On connection

On initial connection between a serial device and SIOT, the following steps are done:

• The MCU sends the SIOT system an empty packet with its root node ID
• The SIOT systems sends the current time to the MCU (point type currentTime)
• The MCU updates any “offline” points with the current time (see offline section).
• The SIOT acks the current time packet.
• All the node and edge points are sent from the SIOT system to the MCU, and from the MCU to the SIOT system. Each system compares point time stamps and updates any points that are newer. Relationships between nodes are defined by edge points (point type tombstone).

Timestamps

The Simple IoT uses a 64-bit nanosecond since Unit epoch value for all timestamps.

Fault handling

Any communication medium has the potential to be disrupted (unplugged/damaged wires, one side off, etc.). Devices should continue to operate and when re-connected, do the right thing.

If an MCU has a valid time (RTC, sync from SIOT, etc.), it will continue operating, and when reconnected, it will send all its points to re-sync.

If an MCU powers up and has no time, it will set the time to 1970 and start operating. When it receives a valid time from the SIOT system, it will compute the time offset from the SIOT time and its own 1970 based time. It then indexes through all points and adds the offset to any points with time less than 2020, and then send all points to SIOT.

When the MCU syncs time with SIOT, if the MCU time is ahead of the SIOT system, then it set its time, and look for any points with a time after present, and reset these timestamps to the present.

Shell Protocol

Many Zephyr applications already expose a shell on their console UART and model their data as points on it. The shell protocol talks to that shell directly, rather than requiring the firmware to implement the binary framing above.

The zephyr-siot library implements the MCU side. Enable CONFIG_SIOT_POINT_SHELL and the firmware gains a point cache, a siot shell command, and point streaming.

Framing

Lines terminated by \n, optionally preceded by \r. No COBS and no CRC: the link is a console, and the shell already defines the framing.

The reader strips VT100 escape sequences and any leading shell prompt, then classifies what remains:

• a pt line that parses is a point
• a line matching the Zephyr log format ([HH:MM:SS.mmm,uuu] <lvl> module: text) becomes a log point
• anything else is ignored

Unrecognized lines are not errors. A console legitimately carries a boot banner, prompts, and command output, and tolerating all of it is what lets the protocol work on a link that was never meant to be machine-only. A line longer than maxMessageLength is dropped and counted in errorCount.

Point line

Both directions use the same fields and differ only in the verb:

pt <type> <key> <INT|FLT|STR|JSN> <data> [<time>]     MCU to SIOT
p  <type> <key> <INT|FLT|STR|JSN> <data> [<time>]     SIOT to MCU

p is the command the Zephyr firmware already registers, so anything SIOT writes could have been typed by hand. pt differs so that an echoed command is never mistaken for a point report.

The MCU uses 0 for a keyless point where SIOT uses an empty key; the client translates in both directions.

Quoting

Fields are separated by single spaces. A field is emitted bare unless it contains a space, a double quote, a backslash, or a control character, in which case it is wrapped in double quotes with \", \\, \r, \n, and \t escaped. This matches what the Zephyr shell tokenizer accepts, which is the constraint that fixes the rules — SIOT must satisfy it when writing p commands, so the same rules apply to pt.

Zephyr’s CONFIG_SHELL_CMD_BUFF_SIZE defaults to 256 bytes. SIOT refuses to send a longer command rather than letting the shell truncate it silently.

Timestamps

SIOT stamps every point it writes, and the MCU stores that value and hands it back unchanged when the point is emitted. The MCU needs no clock of its own for this; it is a carrier, not a timekeeper.

That round trip is what makes an echo identifiable. The MCU’s p handler publishes to the same channel its emitter subscribes to, so every point SIOT writes comes straight back. SIOT drops an inbound point whose value and timestamp match one it just wrote. Without this the two sides would trade the same point forever: a point with no timestamp is stamped on arrival, so each lap looks newer than the last and the store keeps accepting it.

Points the MCU originates carry no timestamp until the firmware has a real clock, and SIOT stamps those on arrival.

The format is RFC 3339 UTC with a fixed nine-digit fractional second (2026-07-31T12:00:00.000000000Z). The width is fixed deliberately: trimming trailing zeros, as Go’s time.RFC3339Nano does, makes the encoding non-canonical and the strings sort incorrectly. Parsing accepts shorter forms, so a hand-typed command still works; only the formatter is strict.

On connection

<newline>              clear any partial line in the shell input buffer
shell echo off         stop the shell echoing our writes
shell colors off       stop VT100 color sequences
siot stream on         start point streaming
siot dump              request every cached point

There is no time synchronization step, since the firmware has no clock to set. connected becomes true when the first line arrives, not when the port opens, and reverts after timeout seconds of silence.

Not supported

High-rate data (phr), file transfer, and packet acknowledgement have no shell equivalent, and encoding them would mean base64 over a console. Nodes needing those should use the binary protocol.

RS485

Status: Idea

RS485 is a half duplex, prompt response transport. SIOT periodically prompts MCU devices for new data at some configurable rate. Data is still COBS encoded so that is simple to tell where packets start/stop without needing to rely on dead space on the line.

Simple IoT also supports Modbus, but the native SIOT protocol is more capable - especially for structured data.

Addressing: TODO

CAN

Status: Idea

CAN messages are limited to 8 bytes. The J1939 Transport Protocol can be used to assemble multiple messages into a larger packet for transferring up to 1785 bytes.

Implementation notes

Both the SIOT and MCU side need to store the common set of nodes and points between the systems. This is critical as the point merge algorithm only uses an incoming point if the incoming point is newer than the one currently stored on the device. For SIOT NATS clients, we use the NodeEdge data structure:

type NodeEdge struct {
        ID         string
        Type       string
        Parent     string
        Points     Points
        EdgePoints Points
	Origin     string
}

Something similar could be done on the MCU.

If new nodes are created on the MCU, the ID must be an UUID, so that it does not conflict with any of the node IDs in the upstream SIOT system(s).

On the SIOT side, we keep a list of Nodes on the MCU and periodically check if any new Nodes have been created. If so, we send the new Nodes to the MCU. Subscriptions are set up for points and edges of all nodes, and any new points are sent to the MCU. Any points received from the MCU simply forwarded to the SIOT NATS bus.

DFU

Status: Idea

For devices that support USB Device Firmware Upgrade (DFU), SIOT provides a mechanism to do these updates. A node that specifies USB ID and file configures the process.

• DFU Specification
• Windows Implementation

Version

The Simple IoT app stores and uses three different version values:

• App Version
• OS Version
• HW Version

The App version is compiled into the application Go binary by the build (see the envsetup.sh file). This version is based on the latest Git tag plus hash if there have been any changes since the tag.

On Linux, the OS version is extracted from the VERSION field in /etc/os-release. The field can be changed using the OS_VERSION_FIELD environment variable.

The versions are displayed in the root node as shown below:

Security

Users and downstream devices will need access to a Simple IoT instance. Simple IoT currently provides access via HTTP and NATS.

Server

For cloud/server deployments, we recommend installing a web server like Caddy in front of Simple IoT. See the Installation page for more information.

Edge

Simple IoT Edge instances initiate all connections to upstream instances; therefore, no incoming connections are required on edge instances and all incoming ports can be firewalled.

HTTP

The Web UI uses JWT (JSON web tokens).

Devices can also communicate via HTTP and use a simple auth token. Eventually may want to switch to JWT or something similar to what NATS uses.

NOTE, it is important to set an auth token - otherwise there is no restriction on accessing the device API.

NATS

Currently devices communicating via NATS use a common auth token. It would be nice to move to something where each device has its own authentication (TODO, explore NATS advanced auth options).

Long term we plan to leverage the NATS security model for user and device authn/authz.:

• NATS authentication
• NATS authorization

Research

This document contains information that has been researched during the course of creating Simple IoT.

Status: this page is a historical record of the exploration that shaped the design; it is not a description of how Simple IoT works today. The hash-tree synchronization discussed below was implemented and later replaced by replicating JetStream streams, which carry their own sequence numbers and avoid the moving-target comparison problem entirely. See ADR-7 for the analysis and decision, and the store and synchronization references for the current design.

Synchronization

An IoT system is inherently distributed. At a minimum, there are three components:

1. Device (Go, C, etc.)
2. Cloud (Go)
3. Multiple browsers (Elm, JavaScript)

Data can be changed in any of the above locations and must be seamlessly synchronized to other locations. Failing to consider this simple requirement early in building the system can make for brittle and overly complex systems.

The moving target problem

As long as the connection between instances is solid, they will stay synchronized as each instance will receive all points it is interested in. Therefore, verifying synchronization by comparing Node hashes is a backup mechanism - that allows us to see what changed when disconnected. The root hashes for a downstream instance changes every time anything in that system changes. Only one value needs to be compared to ensure your entire config is synchronized, but it is also a disadvantage in that the top level hash is changing more often so you are trying to compare two moving targets. This is not a problem if things are changing slow enough that it does not matter if they are changing. However, this also limits the data rates to which we can scale.

Some systems use a concept called Merkle clocks, where events are stored in a Merle DAG and existing nodes in the DAG are immutable and new events are always added as parents to existing events. An immutable DAG has an advantage in that you can always work back in history, which never changes. The SIOT Node tree is mutable by definition. Actual budget uses a similar concept in that it uses a Merkle Trie to represent events in time and then prunes the tree as time goes on.

We could create a separate structure to sync all events (points), but that would require a separate structure on the server for every downstream device and seems overly complex.

Is it critical that we see all historical data? In an IoT system, there are essentially two sets of date - current state/config, and historical data. The current state is most critical for most things, but historical data may be used for some algorithms and viewed by users. The volume of data makes it impractical to store all data in resource constrained edge systems. However, maybe it’s a mistake to separate these two as synchronizing all data might simplify the system.

One way to handle the moving target problem is to store an array of previous hashes for the device node in both instances - perhaps for as long as the synchronization interval. The downstream could then fetch the upstream hash array and see if any of the entries match an entry in the downstream array. This would help cover the case where there may be some time difference when things get updated, but the history should be similar. If there is a hash in history that matches, then we are probably OK.

Another approach would be to track metrics on how often the top level hash is updating - if it is too often, then perhaps the system needs tuned.

There could also be some type of stop-the-world lock where both systems stop processing new nodes during the sync operation. However, if they are not in sync, this probably won’t help and definitely hurts scalability.

Resgate

resgate.io is an interesting project that solves the problem of creating a real-time API gateway where web clients are synchronized seamlessly. This project uses NATS.io for a backbone, which makes it interesting as NATS is core to this project.

The Resgate system is primarily concerned with synchronizing browser contents.

Couch/pouchdb

Has some interesting ideas.

Merkle Trees

• https://en.wikipedia.org/wiki/Merkle_tree
• https://jack-vanlightly.com/blog/2016/10/24/exploring-the-use-of-hash-trees-for-data-synchronization-part-1
• https://www.codementor.io/blog/merkle-trees-5h9arzd3n8
  • Version Control Systems Version control systems like Git and Mercurial use specialized Merkle trees to manage versions of files and even directories. One advantage of using Merkle trees in version control systems is we can simply compare hashes of files and directories between two commits to know if they’ve been modified or not, which is quite fast.
  • No-SQL distributed database systems like Apache Cassandra and Amazon DynamoDB use Merkle trees to detect inconsistencies between data replicas. This process of repairing the data by comparing all replicas and updating each one of them to the newest version is also called anti-entropy repair. The process is also described in Cassandra’s documentation.

Scaling Merkel trees

One limitation of Merkel trees is the difficulty of updating the tree concurrently. Some information on this:

• how to scale blockchains
• Angela: A Sparse, Distributed, and Highly Concurrent Merkle Tree

Distributed key/value databases

• etcd
• NATS key/value store

Distributed Hash Tables

• https://en.wikipedia.org/wiki/Distributed_hash_table

CRDT (Conflict-free replicated datatype)

• https://en.wikipedia.org/wiki/Conflict-free_replicated_data_type
• Yjs
  • https://blog.kevinjahns.de/are-crdts-suitable-for-shared-editing/
• https://tantaman.com/2022-10-18-lamport-sufficient-for-lww.html

Databases

• https://tantaman.com/2022-08-23-why-sqlite-why-now.html
  • instead of doing: select comment.* from post join comment on comment.post_id = post.id where post.id = x and comment.date < cursor.date and comment.id < cursor.id order by date, id desc limit 101
  • we do: post.comments().last(10).after(curosr);

Timestamps

• Lamport timestamp
  • used by Yjs

Other IoT Systems

AWS IoT

• https://www.thingrex.com/aws_iot_thing_attributes_intro/
  • Thing properties include the following, which are analogous to SIOT node fields.
    • Name (Description)
    • Type (Type)
    • Attributes (Points)
    • Groups (Described by tree structure)
    • Billing Group (Can also be described by tree structure)
• https://www.thingrex.com/aws_iot_thing_type/
  • Each type has a specified attributes - kind of a neat idea

Architecture Decision Records

This directory is used to capture Simple IoT architecture and design decisions.

For background on ADRs see Documenting Architecture Decisions by Michael Nygard. Also see an example of them being used in the NATS project. The Go proposal process is also a good reference.

Process

When thinking about architectural changes, we should lead with documentation. This means we should start a branch, draft a ADR, and then open a PR. An associated issue may also be created.

ADRs should used primarily when a number of approaches need to be considered, thought through, and we need a record of how and why the decision was made. If the task is a fairly straightforward implementation, write documentation in the existing User and Reference Guide sections.

When an ADR is accepted and implemented, a summary should typically be added to the Reference Guide documentation.

See template.md for a template to get started.

ADRs

Point Data Type Changes

• Author: Cliff Brake Last updated: 2023-06-13
• Issue at: https://github.com/simpleiot/simpleiot/issues/254
• PR/Discussion:
  • https://github.com/simpleiot/simpleiot/pull/279
  • https://github.com/simpleiot/simpleiot/pull/565
  • https://github.com/simpleiot/simpleiot/pull/566
• Status: Review

Contents

Problem

The current point data type is fairly simple and has proven useful and flexible to date, but we may benefit from additional or changed fields to support more scenarios. It seems in any data store, we need at the node level to be able to easily represent:

1. arrays
2. maps

IoT systems are distributed systems that evolve over time. If can’t easily handle schema changes and synchronize data between systems, we don’t have anything.

Context/Discussion

Should we consider making the point struct more flexible?

The reason for this is that it is sometimes hard to describe a sensor/configuration value with just a few fields.

Requirements

• IoT systems are often connected by unreliable networks (cellular, etc). All devices/instances in a SIOT should be able to functional autonomously (run rules, etc) and then synchronize again when connected.
• all systems must converge to the same configuration state. We can probably tolerate some lost time series data, but configuration and current state must converge. When someone is remotely looking at a device state, we want to make sure they are seeing the same things a local operator is seeing.

evolvability

From Martin Kleppmann’s book:

In a database, the process that writes to the database encodes the data, and the process that reads from the database decodes it. There may just be a single process accessing the database, in which case the reader is simply a later version of the same process—in that case you can think of storing something in the database as sending a message to your future self.

Backward compatibility is clearly necessary here; otherwise your future self won’t be able to decode what you previously wrote.

In general, it’s common for several different processes to be accessing a database at the same time. Those processes might be several different applications or services, or they may simply be several instances of the same service (running in parallel for scalability or fault tolerance). Either way, in an environment where the application is changing, it is likely that some processes accessing the database will be running newer code and some will be running older code—for example because a new version is currently being deployed in a rolling upgrade, so some instances have been updated while others haven’t yet.

This means that a value in the database may be written by a newer version of the code, and subsequently read by an older version of the code that is still running. Thus, forward compatibility is also often required for databases.

However, there is an additional snag. Say you add a field to a record schema, and the newer code writes a value for that new field to the database. Subsequently, an older version of the code (which doesn’t yet know about the new field) reads the record, updates it, and writes it back. In this situation, the desirable behavior is usually for the old code to keep the new field intact, even though it couldn’t be interpreted.

The encoding formats discussed previously support such preservation of unknown fields, but sometimes you need to take care at an application level, as illustrated in Figure 4-7. For example, if you decode a database value into model objects in the application, and later re-encode those model objects, the unknown field might be lost in that translation process. Solving this is not a hard problem; you just need to be aware of it.

Some discussion of this book: https://community.tmpdir.org/t/book-review-designing-data-intensive-applications/288/6

CRDTs

Some good talks/discussions:

I also agree CRDTs are the future, but not for any reason as specific as the ones in the article. Distributed state is so fundamentally complex that I think we actually need CRDTs (or something like them) to reason about it effectively. And certainly to build reliable systems. The abstraction of a single, global, logical truth is so nice and tidy and appealing, but it becomes so leaky that I think all successful systems for distributed state will abandon it beyond a certain scale. – Peter Bourgon

CRDTs, the hard parts by Martin Kleppmann

Infinite Parallel Universes: State at the Edge

Wikipedia article

Properties of CRDTs:

• Associative (order in which operations are performed does matter)
• Commutative (changing order of operands does not change result)
• Idempotent (operation can be applied multiple times without changing the result, tolerate over-merging)

The existing SIOT Node/Point data structures were created before I know what a CRDT was, but they happen to already give a node many of the properties of a CRDT – IE, they can be modified independently, and then later merged with a reasonable level of conflict resolution.

For reliable data synchronization in distributed systems, there has to be some metadata around data that facilitates synchronization. This can be done in two ways:

1. add meta data in parallel to the data (turn JSON into a CRDT, example automerge or yjs)
2. express all data using simple primitives that facilitate synchronization

Either way, you have to accept constraints in your data storage and transmission formats.

To date, we have chosen to follow the 2nd path (simple data primitives).

Operational transforms

There are two fundamental schools of thought regarding data synchronization:

1. Operation transforms. In this method, a central server arbitrates all conflicts and hands the result back to other instances. This is an older technique and is used in applications like Google docs.
2. CRDTs – this is a newer technique that works with multiple network connections and does not require a central server. Each instance is capable of resolving conflicts themselves and converging to the same point.

While a classical OT arrangement could probably work in a traditional SIOT system (where all devices talk to one cloud server), it would be nice if we are not constrained to this architecture. This would allow us to support peer synchronization in the future.

Other Standards

Some reference/discussion on other standards:

Sparkplug

https://github.com/eclipse/tahu/blob/master/sparkplug_b/sparkplug_b.proto

The sparkplug data type is huge and could be used to describe very complex data. This standard came out of the industry 4.0 movement where a factory revolves around a common MQTT messaging server. The assumption is that everything is always connected to the MQTT server. However, with complex types, there is no provision for intelligent synchronization if one system is disconnected for some amount of time – its all or nothing, thus it does not seem like a good fit for SIOT.

SenML

https://datatracker.ietf.org/doc/html/draft-ietf-core-senml-08#page-9

tstorage

The tstorage Go package has an interesting data storage type:

type Row struct {
	// The unique name of metric.
	// This field must be set.
	Metric string
	// An optional key-value properties to further detailed identification.
	Labels []Label
	// This field must be set.
	DataPoint
}

type DataPoint struct {
	// The actual value. This field must be set.
	Value float64
	// Unix timestamp.
	Timestamp int64
}

type Label struct {
	Name  string
	Value string

In this case there is one value and an array of labels, which are essentially key/value strings.

InfluxDB

InfluxDB’s line protocol contains the following:

type Metric interface {
	Time() time.Time
	Name() string
	TagList() []*Tag
	FieldList() []*Field
}

type Tag struct {
	Key   string
	Value string
}

type Field struct {
	Key   string
	Value interface{}
}

where the Field.Value must contain one of the InfluxDB supported types (bool, uint, int, float, time, duration, string, or bytes).

time-series storage considerations

Is it necessary to have all values in one point, so they can be grouped as one entry in a time series data base like influxdb? Influx has a concept of tags and fields, and you can have as many as you want for each sample. Tags must be strings and are indexed and should be low cardinality. Fields can be any datatype influxdb supports. This is a very simple, efficient, and flexible data structure.

Example: location data

One system we are working with has extensive location information (City/State/Facility/Floor/Room/Isle) with each point. This is all stored in influx so we can easily query information for any location in the past. With SIOT, we could not currently store this information with each value point, but would rather store location information with the node as separate points. One concern is if the device would change location. However, if location is stored in points, then we will have a history of all location changes of the device. To query values for a location, we could run a two pass algorithm:

1. query history and find time windows when devices are in a particular location.
2. query these time ranges and devices for values

This has the advantage that we don’t need to store location data with every point, but we still have a clear history of what data come from where.

Example: file system metrics

When adding metrics, we end up with data like the following for disks partitions:

Filesystem     Size Used Avail Use% Mounted on
tmpfs          16806068224 0 16806068224   0% /dev
tmpfs          16813735936 1519616 16812216320   0% /run
ext2/ext3      2953064402944 1948218814464 854814945280  70% /
tmpfs          16813735936 175980544 16637755392   1% /dev/shm
tmpfs          16813740032 3108966400 13704773632  18% /tmp
ext2/ext3      368837799936 156350181376 193680359424  45% /old3
msdos          313942016 60329984 253612032  19% /boot
ext2/ext3      3561716731904 2638277668864 742441906176  78% /scratch
tmpfs          3362746368 118784 3362627584   0% /run/user/1000
ext2/ext3      1968874332160 418203766784 1450633895936  22% /run/media/cbrake/59b35dd4-954b-4568-9fa8-9e7df9c450fc
fuseblk        3561716731904 2638277668864 742441906176  78% /media/fileserver
ext2/ext3      984372027392 339508314112 594836836352  36% /run/media/cbrake/backup2

It would be handy if we could store filesystem as a tag, size/used/avail/% as fields, and mount point as text field.

We already have an array of points in a node – can we just make one array work? The size/used/avail/% could easily be stored as different points. The text field would store the mount point, which would tie all the stats for one partition together. Then the question is how to represent the filesystem? With the added Key field in proposal #2, we can now store the mount point as the key.

Representing arrays

With the key field, we can represent arrays as a group of points, where key defines the position in the array. For node points to be automatically decoded into an array struct fields by the SIOT client manager, the key must be an integer represented in string form.

One example where we do this is for selecting days of the week in schedule rule conditions. The key field is used to select the weekday. So we can have a series of points to represent Weekdays. In the below, Sunday is the 1st point set to 0, and Monday is the 2nd point, set to 1.

[]data.Point{
  {
    Type: "weekday",
    Key: "0",
    Value: 0,
  },
  {
    Type: "weekday",
    key: "1",
    Value: 0,
  },
}

In this case, the condition node has a series of weekday points with keys 0-6 to represent the days of the week.

The SIOT data.Decode is used by the client manager to initialize array fields in a client struct. The following assumptions are made:

• the value in the key field is converted to an int and used as the index into the field array.
• if there are missing array entries, they are filled with zero values.
• the data.MergePoints uses the same algorithm.
• if a point is inserted into the array or moved, all array points affected must be sent. For example, if you have an array of length 20, and you insert a new value at the beginning, then all 21 points must to be sent. This can have implications for rules or any other logic that use the Point key field.

This does not have perfect CRDT properties, but typically these arrays are generally small and are only modified in one place.

If you need more advanced functionality, you can bypass the data Decode/Merge functions and process the points manually and then use any algorithm you want to process them.

Point deletions

To date, we’ve had no need to delete points, but it may be useful in the future.

Consider the following sequence of point changes:

1. t1: we have a point
2. t2: A deletes the point
3. t3: B concurrently change the point value

The below table shows the point values over time with the current point merge algorithm:

In this case, the point becomes undeleted because the last write wins (LWW). Is this a problem? What is the desired behavior? A likely scenario is that a device will be continually sending value updates and a user will make a configuration change in the portal that deletes a point. Thus it seems delete changes should always have precedence. However, with the last write wins (LWW) merge algorithm, the tombstone value could get lost. It may make sense to:

• make the tombstone value an int
• only increment it
• when merging points, the highest tombstone value wins
• odd value of tombstone value means point is deleted

Thus the tombstone value is merged independently of the timestamp and thus is always preserved, even if there concurrent modifications.

The following table shows the values with the modified point merge algorithm.

Duration, Min, Max

The current Point data type has Duration, Min, and Max fields. This is used for when a sensor value is averaged over some period of time, and then reported. The Duration, Min, Max fields are useful for describing what time period the point was obtained, and what the min/max values during this period were.

Representing maps

In the file system metrics example below, we would like to store a file system type for a particular mount type. We have 3 pieces of information:

data.Point {
  Type: "fileSystem",
  Text: "/media/data/",
  ????: "ext4",
}

Perhaps we could add a key field:

data.Point {
  Type: "fileSystem",
  Key: "/media/data/",
  Text: "ext4",
}

The Key field could also be useful for storing the mount point for other size/used, etc points.

making use of common algorithms and visualization tools

A simple point type makes it very nice to write common algorithms that take in points, and can always assume the value is in the value field. If we store multiple values in a point, then the algorithm needs to know which point to use.

If an algorithm needs multiple values, it seems we could feed in multiple point types and discriminated by point type. For example, if an algorithm used to calculate % of a partition used could take in total size and used, store each, and the divide them to output %. The data does not necessarily need to live in the same point. Could this be used to get rid of the min/max fields in the point? Could these simply be separate points?

• Having min/max/duration as separate points in influxdb should not be a problem for graphing – you would simply qualify the point on a different type vs selecting a different field.
• if there is a process that is doing advanced calculations (say taking the numerical integral of flow rate to get total flow), then this process could simply accumulate points and when it has all the points for a timestamp, then do the calculation.

Schema changes and distributed synchronization

A primary consideration of Simple IoT is easy and efficient data synchronization and easy schema changes.

One argument against embedded maps in a point is that adding these maps would likely increase the possibility of schema version conflicts between versions of software because points are overwritten. Adding maps now introduces a schema into the point that is not synchronized at the key level. There will also be a temptation to put more information into point maps instead of creating more points.

With the current point scheme, it is very easy to synchronize data, even if there are schema changes. All points are synchronized, so one version can write one set of points, and another version another, and all points will be sync’d to all instances.

There is also a concern that if two different versions of the software use different combinations of field/value keys, there could be information lost. The simplicity and ease of merging Points into nodes is no longer simple. As an example:

Point {
  Type: "motorPIDConfig",
  Values: {
    {"P": 23},
    {"I": 0.8},
    {"D": 200},
  },
}

If an instance with an older version writes a point that only has the “P” and “I” values, then the “D” value would get lost. We could merge all maps on writes to prevent losing information. However if we have a case where we have 3 systems:

Aold -> Bnew -> Cnew

If Aold writes an update to the above point, but only has P,I values, then this point is automatically forwarded to Bnew, and then Bnew forwards it to Cnew. However, Bnew may have had a copy with P,I,D values, but the D is lost when the point is forwarded from Aold -> Cnew. We could argue that Bnew has previously synchronized this point to Cnew, but what if Cnew was offline and Aold sent the point immediately after Cnew came online before Bnew synchronized its point.

The bottom line is there are edge cases where we don’t know if the point map data is fully synchronized as the map data is not hashed. If we implement arrays and maps as collections of points, then we can be more sure everything is synchronized correctly because each point is a struct with fixed fields.

Is there any scenario where we need multiple tags/labels on a point?

If we don’t add maps to points, the assumption is any metadata can be added as additional points to the containing node. Will this cover all cases?

Is there any scenario where we need multiple values in a point vs multiple points?

If we have points that need to be grouped together, they could all be sent with the same timestamp. Whatever process is using the points could extract them from a timeseries store and then re-associate them based on common timestamps.

Could duration/min/max be sent as separate points with the same timestamp instead of extra fields in the point?

The NATS APIs allow you to send multiple points with a message, so if there is ever a need to describe data with multiple values (say min/max/etc), these can simply be sent as multiple points in one message.

Is there any advantage to flat data structures?

Flat data structures where the fields consist only of simple types (no nested objects, arrays, maps, etc). This is essentially what tables in a relational database are. One advantage to keeping the point type flat is it would map better into a relational database. If we add arrays to the Point type, then it will not longer map into a single relational database table.

Design

Original Point Type

type Point struct {
	// ID of the sensor that provided the point
	ID string `json:"id,omitempty"`

	// Type of point (voltage, current, key, etc)
	Type string `json:"type,omitempty"`

	// Index is used to specify a position in an array such as
	// which pump, temp sensor, etc.
	Index int `json:"index,omitempty"`

	// Time the point was taken
	Time time.Time `json:"time,omitempty"`

	// Duration over which the point was taken. This is useful
	// for averaged values to know what time period the value applies
	// to.
	Duration time.Duration `json:"duration,omitempty"`

	// Average OR
	// Instantaneous analog or digital value of the point.
	// 0 and 1 are used to represent digital values
	Value float64 `json:"value,omitempty"`

	// Optional text value of the point for data that is best represented
	// as a string rather than a number.
	Text string `json:"text,omitempty"`

	// statistical values that may be calculated over the duration of the point
	Min float64 `json:"min,omitempty"`
	Max float64 `json:"max,omitempty"`
}

Proposal #1

This proposal would move all the data into maps.

type Point struct {
    ID string
    Time time.Time
    Type string
    Tags map[string]string
    Values map[string]float64
    TextValues map[string]string
}

The existing min/max would just become fields. This would map better into influxdb. There would be some redundancy between Type and Field keys.

Proposal #2

type Point struct {
	// The 1st three fields uniquely identify a point when receiving updates
	Type string
	Key string

	// The following fields are the values for a point
	Time time.Time
	(removed) Index float64
	Value float64
	Text string
	Data []byte

	// Metadata
	Tombstone int
}

Updated 2023-06-13: removed the Index field. We will use the Key field for array indices.

Notable changes from the first implementation:

• removal of the ID field, as any ID information should be contained in the parent node. The ID field is a legacy from 1-wire setups where we represented each 1-wire sensor as a point. However, it seems now each 1-wire sensor should have its own node.
• addition of the Key field. This allows us to represent maps in a node, as well as add extra identifying information for a point.
• the Point is now identified in the merge algorithm using the Type and Key. Before, the ID, Type, and Index were used.
• the Data field is added to give us the flexibility to store/transmit data that does not fit in a Value or Text field. This should be used sparingly, but gives us some flexibility in the future for special cases. This came out of some comments in an Industry 4.0 community – basically types/schemas are good in a communication standard, as long as you also have the capability to send a blob of data to handle all the special cases. This seems like good advice.
• the Tombstone fields is added as an int and is always incremented. Odd values of Tombstone mean the point was deleted. When merging points, the highest tombstone value always wins.

Decision

Going with proposal #2 – we can always revisit this later if needed. This has minimal impact on the existing code base.

Objections/concerns

(Some of these are general to the node/point concept in general)

• Q: with the point datatype, we lose types
  • A: in a single application, this concern would perhaps be a high priority, but in a distributed system, data synchronization and schema migrations must be given priority. Typically these collections of points are translated to a type by the application code using the data, so any concerns can be handled there. At least we won’t get JS undefined crashes as Go will fill in zero values.
• Q: this will be inefficient converting points to types
  • A: this does take processing time, but this time is short compared to the network transfer times from distributed instances. Additionally, applications can cache nodes they care about so they don’t have to translate the entire point array every time they use a node. Even a huge IoT system has a finite # of devices that can easily fit into memory of modern servers/machines.
• Q: this seems crude not to have full featured protobuf types with all the fields explicitly defined in protobuf. Additionally, can’t protobuf handle type changes elegantly?
  • A: protobuf can handle field additions and removal but we still have the edge cases where a point is sent from an old version of software that does not contain information written by a newer versions. Also, I’m not sure it is a good idea to have application specific type fields defined in protobuf, otherwise, you have a lot of work all along the communication chain to rebuild everything every time anything changes. With a generic types that rarely have to change, your core infrastructure can remain stable and any features only need to touch the edges of the system.
• Q: with nodes and points, we can only represent a type with a single level of fields
  • A: this is not quite true, because with the key/index fields, we can now have array and map fields in a node. However, the point is taken that a node with its points cannot represent a deeply nested data structure. However, nodes can be nested to represent any data structure you like. This limitation is by design because otherwise synchronization would be very difficult. By limiting the complexity of the core data structures, we are making synchronization and storage very simple. The tradeoff is a little more work to marshall/unmarshall node/point data structures into useful types in your application. However, marshalling code is easy compared to distributed systems, so we need to optimize the system for the hard parts. A little extra typing will not hurt anyone, and tooling could be developed if needed to assist in this.

Generic core data structures also opens up the possibility to dynamically extend the system at run time without type changes. For instance, the GUI could render new nodes it has never seen before by sending it configuration nodes with declarative instructions on how to display the node. If core types need to change to do this type of thing, we have no chance at this type of intelligent functionality.

Consequences

Removing the Min/Max/Duration fields should not have any consequences now as I don’t think we are using these fields yet.

Quite a bit of code needs to change to remove ID and add Key to code using points.

Additional Notes/Reference

We also took a look at how to resolve loops in the node tree:

https://github.com/simpleiot/simpleiot/issues/294

This is part of the verification to confirm our basic types are robust and have adequate CRDT properties.

Authorization

• Author: Blake Miner
• Issue: https://github.com/simpleiot/simpleiot/issues/268
• PR / Discussion: https://github.com/simpleiot/simpleiot/pull/283
• Status: Brainstorming

Problem

SIOT currently does not prevent unauthorized NATS clients from connecting and publishing / subscribing. Presently, any NATS client with access to the NATS server connection can read and write any data over the NATS connection.

Discussion

This document describes a few mechanisms for how to implement authentication and authorization mechanisms within Simple IoT.

Current Authentication Mechanism

Currently, SIOT supports upstream connections through the use of upstream nodes. The connection to the upstream server can be authenticated using a simple NATS auth token; however, all NATS clients with knowledge of the auth token can read / write any data over the NATS connection. This will not work well for a multi-tenant application or applications where user access must be closely controlled.

Similarly, web browsers can access the NATS API using the WebSocket library, but since they act as another NATS client, no additional security is provided; browsers can read / write all data over the NATS connection.

Proposal

NATS supports decentralized user authentication and authorization using NKeys and JSON Web Tokens (JWTs). While robust, this authentication and authorization mechanism is rather complex and confusing; a detailed explanation follows nonetheless. The end goal is to dynamically add NATS accounts to the NATS server because publish / subscribe permissions of NATS subjects can be tied to an account.

Background

Each user node within SIOT will be linked to a dynamically created NATS account (on all upstream nodes); each account is generated when the user logs in. Only a single secret is stored in the root node of the SIOT tree.

NATS has a public-key signature system based on Ed25519. These keypairs are called NKeys. Put simply, NKeys allow one to cryptographically sign and verify JWTs. An NKey not only consists of a Ed25519 private key / seed, but it also contains information on the “role” of the key. In NATS, there are three primary roles: operators, accounts, and users. In SIOT, there is one operator for a given NATS server, and there is one account for each user node.

Start-up

When the SIOT server starts, an NKey for the operator role is loaded from a secret stored as a point in the root node of the tree. This point is always stripped away when clients request the root node, so it’s never transmitted over a NATS connection. Once the NATS server is running, SIOT will start an internal NATS client and connect to the local NATS server. This internal client will authenticate to the NATS server with a superuser, whose account has full permissions to publish and subscribe to all subjects. Unauthenticated NATS clients only have permission to publish to authsubject and listen for a reply.

Authentication / Login

External NATS clients (including web browsers over WebSockets) must first log into the NATS server anonymously (using the auth token if needed) and send a request to the auth subject with the username and password of a valid user node. The default username is admin, and the default password is admin. The internal NATS client will subscribe to requests on the auth subject, and if the username / password is correct, it will respond with a user NKey and user JWT Token, which are needed to login. The user JWT token will be issued and signed by the account NKey, and the account NKey will be issued and signed by the operator NKey. The NATS connection will then be re-established using the user JWT and signing a server nonce with the user’s NKey.

JWT expiration should be a configurable SIOT option and default to 1 hour. Optionally, when the user JWT token is approaching its expiration, the NATS client can request re-authenticate using the auth subject and reconnect using the new user credentials.

Storing NKeys

As discussed above, in the root node, we store the seed needed to derive the operator NKey. For user nodes, account and user NKeys are computed as-needed from the node ID, the username, and the password.

Authorization

An authenticated user will have publish / subscribe access to the subject space of $nodeID.> where $nodeID is the node ID for the authenticated user. The normal SIOT NATS API will work the same as normal with two notable exceptions:

• The API subjects are prepended with $nodeID.
• The “root” node is remapped to the set of parents of the logged in user node

Examples

Example #1

Imagine the following a SIOT node tree:

• Root (Device 82ad…28ae)
• Power Users (Group b723…008d)
  • Temperature Sensor (Device 2820…abdc)
  • Humidity Sensor (Device a89f…eda9)
  • Blake (User ab12…ef22)
• Admin (User 920d…ab21)

In this case, logging in as “Blake” would reveal the following tree with a single root node:

• Power Users (Group b723…008d)
  • Temperature Sensor (Device 2820…abdc)
  • Humidity Sensor (Device a89f…eda9)
  • Blake (User ab12…ef22)

To get points of the humidity sensor, one would send a request to this subject: ab12...ef22.p.a89f...eda9.

Example #2

Imagine the following a SIOT node tree:

• Root (Device 82ad…28ae)
  • Temperature Sensor (Device 2820…abdc)
    • Blake (User ab12…ef22)
  • Humidity Sensor (Device a89f…eda9)
    • Blake (User ab12…ef22)
  • Admin (User 920d…ab21)

In this case, logging in as “Blake” would reveal the following tree with two root nodes:

• Temperature Sensor (Device 2820…abdc)
  • Blake (User ab12…ef22)
• Humidity Sensor (Device a89f…eda9)
  • Blake (User ab12…ef22)

To get points of the humidity sensor, one would send a request to this subject: ab12...ef22.p.a89f...eda9.

Implementation Notes

// Note: JWT issuer and subject must match an NKey public key
// Note: JWT issuer and subject must match roles depending on the claim NKeys

import (
	"github.com/nats-io/jwt/v2"
	"github.com/nats-io/nkeys"
	"github.com/nats-io/nats-server/v2/server"
)

// Example code to start NATS server
func StartNatsServer(o Options) {
	op, err := nkeys.CreateOperator()
	if err != nil {
		log.Fatal("Error creating NATS server: ", err)
	}
	pubKey, err := op.PublicKey()
	if err != nil {
		log.Fatal("Error creating NATS server: ", err)
	}
	acctResolver := server.MemAccResolver{}
	opts := server.Options{
		Port:                     o.Port,
		HTTPPort:                 o.HTTPPort,
		Authorization:            o.Auth,
		// First we trust all operators
		// Note: DO NOT USE conflicting `TrustedKeys` option
		TrustedOperators:         []{jwt.NewOperatorClaims(pubKey)},
		AccountResolver:          acctResolver,
	}
}

// Create an Account
acct, err := nkeys.CreateAccount()
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}
pubKey, err := acct.PublicKey()
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}
claims := jwt.NewAccountClaims{pubKey}
claims.DefaultPermissions = Permissions{
	// Note: subject `_INBOX.>` allowed for all NATS clients
	// Note: subject publish on `auth` allowed for all NATS clients
	Pub: Permission{
		Allow: StringList([]string{userNodeID+".>"}),
	},
	Sub: Permission{
		Allow: StringList([]string{userNodeID+".>"}),
	},
}
claims.Issuer = opPubKey
claims.Name = userNodeID
// Sign the JWT with the operator NKey
jwt, err := claims.Encode(op)
if err != nil {
	log.Fatal("Error creating NATS account: ", err)
}

acctResolver.Store(userNodeID, jwt)

Node Lifecycle

• Author: Cliff Brake, last updated: 2022-02-16
• PR/Discussion:
• Status: discussion

Context

In the process of implementing a feature to duplicate a node tree, several problems have surfaced related to the lifecycle of creating and updating nodes.

Node creation (< 0.5.0)

• if a point was sent and node did not exist, SIOT created a “device” node as a child of the root node with this point. This was based on this initial use of SIOT with 1-wire devices.
• there is also a feature where if we send a point to a Device node that does not have an upstream path to root, or that path is tombstoned, we create this path. This ensures that we don’t have orphaned device nodes in an upstream if they are still active. This happens to the root node on clear startup.
• by the user in the UI – Http API, /v1/nodes POST, accepts a NodeEdge struct and then sends out node points and then edge points via NATs to create the node.
• node is sent first, then the edge

The creation process for a node involves:

1. sending all the points of a node including a meta point with the node type.
2. sending the edge points of a node to describe the upstream connection

There are two problems with this:

1. When creating a node, we send all the node points, then the edge points. However this can create an issue in that an upstream edge for a device node does not exist yet, so in a multi-level upstream configuration A->B->C, if B is syncing to C for the first time, multiple instances of A will be created on C.
2. If a point is sent for a node that does not exist, a new device node will be created.

An attempt was made to switch the sending edge points of new nodes before node points, however this created other issues (this was some time ago, so don’t recall exactly what they were).

Node creation (>= 0.5.0)

With the switch to a SQLite store, a lot of code was rewritten, and in the process we changed the order of creating nodes to:

1. send the edge points first
2. then send node points

(See the SendNode() function).

discussion

Sending node and edge points separately for new nodes creates an issue in that these don’t happen in one communication transaction, so there is a period of time between the two where the node state is indeterminate. Consideration was given to adding a NATS endpoint to create nodes where everything could be sent at once. However, this is problematic in that now there is another NATS subject for everyone to listen to and process, rather than just listening for new points. It seems less than ideal to have multiple subjects that can create/modify node points.

It seems at this point we can probably deprecate the feature to create new devices nodes based on a single point. This will force new nodes to be explicitly created. This is probably OK as new nodes are created in several ways:

1. by the user in the UI
2. by the upstream sync mechanism – if the hash does match or a node does not exist upstream, it is sent. This is continuously checked so if a message does not succeed, it will eventually get resent.
3. plug-n-play discovery mechanisms that detect new devices and automatically populate new nodes. Again, it is not a big deal if a message gets lost as the discovery mechanism will continue to try to create the new device if it does not find it.

Sending edge before parents can be problematic for things like the client manager that might be listening for tombstone points to detect node creation/deletion. (Why is this???)

Decision

Consequences

Time storage/format considerations

• Author: Cliff Brake, last updated: 2023-02-11
• PR/Discussion:
• Status: accepted

Contents

Problem

How can we store timestamps that are:

• efficient
• high resolution (ns)
• portable
• won’t run out of time values any time soon

We have multiple domains:

• Go
• MCU code
• Browser (ms resolution)
• SQLite
• Protbuf

Two questions:

• How should we store timestamps in SQLite?
• How should we transfer timestamps over the wire (typically protobuf)?

Context

We currently use Go timestamps in Go code, and protobuf timestamps on the wire.

Reference/Research

Browsers

Browsers limit time resolution to MS for security reasons.

64-bit nanoseconds

2 ^ 64 nanoseconds is roughly ~ 584.554531 years.

https://github.com/jbenet/nanotime

NTP

For NTP time, the 64bits are broken in to seconds and fraction of seconds. The top 32 bits is the seconds. The bottom 32 bits is the fraction of seconds. You get the fraction by dividing the fraction part by 2^32.

Linux

64-bit Linux systems are using 64bit timestamps (time_t) for seconds, and 32-bit systems are switching to 64-bit to avoid the 2038 bug.

• musl
• glibc discussion

The Linux clock_gettime() function uses the following datatypes:

struct timeval {
	time_t          tv_sec;
	suseconds_t     tv_usec;
};

struct timespec {
	time_t          tv_sec;
	long            tv_nsec;
};

Windows

Windows uses a 64-bit value representing the number of 100-nanosecond intervals since January 1, 1601 (UTC).

Go

The Go Time type is fairly intelligent as it uses Montonic time when possible and falls back to wall clock time when needed:

https://pkg.go.dev/time

If Times t and u both contain monotonic clock readings, the operations t.After(u), t.Before(u), t.Equal(u), and t.Sub(u) are carried out using the monotonic clock readings alone, ignoring the wall clock readings. If either t or u contains no monotonic clock reading, these operations fall back to using the wall clock readings.

The Go Time type is fairly clever:

type Time struct {
        // wall and ext encode the wall time seconds, wall time nanoseconds,
        // and optional monotonic clock reading in nanoseconds.
        //
        // From high to low bit position, wall encodes a 1-bit flag (hasMonotonic),
        // a 33-bit seconds field, and a 30-bit wall time nanoseconds field.
        // The nanoseconds field is in the range [0, 999999999].
        // If the hasMonotonic bit is 0, then the 33-bit field must be zero
        // and the full signed 64-bit wall seconds since Jan 1 year 1 is stored in ext.
        // If the hasMonotonic bit is 1, then the 33-bit field holds a 33-bit
        // unsigned wall seconds since Jan 1 year 1885, and ext holds a
        // signed 64-bit monotonic clock reading, nanoseconds since process start.
        wall uint64
        ext  int64

        // loc specifies the Location that should be used to
        // determine the minute, hour, month, day, and year
        // that correspond to this Time.
        // The nil location means UTC.
        // All UTC times are represented with loc==nil, never loc==&utcLoc.
        loc *Location
}

Go provides a UnixNano() function that converts a Timestamp to nanoseconds elapsed since January 1, 1970 UTC.

To go the other way, Go provides a UnixMicro() function to convert microseconds since 1970 to a timestamp. The source code could probably be modified to create a UnixNano() function.

// UnixMicro returns the local Time corresponding to the given Unix time,
// usec microseconds since January 1, 1970 UTC.
func UnixMicro(usec int64) Time {
	return Unix(usec/1e6, (usec%1e6)*1e3)
}

// Unix returns the local Time corresponding to the given Unix time,
// sec seconds and nsec nanoseconds since January 1, 1970 UTC.
// It is valid to pass nsec outside the range [0, 999999999].
// Not all sec values have a corresponding time value. One such
// value is 1<<63-1 (the largest int64 value).
func Unix(sec int64, nsec int64) Time {
	if nsec < 0 || nsec >= 1e9 {
		n := nsec / 1e9
		sec += n
		nsec -= n * 1e9
		if nsec < 0 {
			nsec += 1e9
			sec--
		}
	}
	return unixTime(sec, int32(nsec))
}

Protobuf

The Protbuf time format also has sec/ns sections:

message Timestamp {
  // Represents seconds of UTC time since Unix epoch
  // 1970-01-01T00:00:00Z. Must be from 0001-01-01T00:00:00Z to
  // 9999-12-31T23:59:59Z inclusive.
  int64 seconds = 1;

  // Non-negative fractions of a second at nanosecond resolution. Negative
  // second values with fractions must still have non-negative nanos values
  // that count forward in time. Must be from 0 to 999,999,999
  // inclusive.
  int32 nanos = 2;
}

MQTT

Note sure yet if MQTT defines a timestamp format.

Sparkplug does:

timestamp

• This is the timestamp in the form of an unsigned 64-bit integer representing the number of milliseconds since epoch (Jan 1, 1970). It is highly recommended that this time is in UTC. This timestamp is meant to represent the time at which the message was published

CRDTs

LWW (last write wins) CRDTs often use a logical clock. crsql uses a 64-bit logical clock.

Do we need nanosecond resolution?

Many IoT systems only support MS resolution. However, this is sometimes cited as a deficiency in applications where higher resolution is needed (e.g. power grid).

Decision

• NATS messages
  • stick with standard protobuf Time definition in NATS packets
  • this is most compatible with all the protobuf language support out there
• Database
  • switch to single time field that contains NS since Unix epoch
  • this is simpler and allows us to easily do comparisons on the field

objections/concerns

Consequences

Migration is required for database, but should be transparent to the user.

Time Validation

• Author: Cliff Brake
• PR/Discussion:
• Status: discussion

Contents

Problem

To date, SIOT has been deployed to systems with RTCs and solid network connections, so time is fairly stable, thus this has not been a big concern. However, we are looking to deploy to edge systems, some with cellular modem connections and some without a battery backed RTC, so they may boot without a valid time.

SIOT is very dependent on data having valid timestamps. If timestamps are not correct, the following problems may occur:

• old data may be preferred over newer data in the point CRDT merge algorithm
• data stored in time series databases may have the wrong time stamps

Additionally, there are edge systems that don’t have a real-time clock and power up with an invalid time until a NTP process gets the current time.

We may need some systems to operate (run rules, etc) without a valid network connection (offline) and valid time.

Context/Discussion

Clients affected

• db (InfluxDB driver)
• sync (sends data upstream)
• store (not sure ???)

The db and sync clients should not process points (or perhaps buffer them until) until we are sure the system has a valid time. How does it get this information? Possibilities include:

1. creating a broadcast or other special message subject that clients can optionally listen to. Perhaps the NTP client can send this message.

• synchronization may be a problem here if NTP client sends messages before a client has started.

1. query for system state, and NTP sync status could be a field in this state.

• should this be part of the root device node?
• or a special hard-coded message?
• it would be useful to track system state as a standard point so it gets synchronized and stored in influxdb, therefore as part of the root node would be useful, or perhaps the NTP node.

Offline operation

System must function when offline without valid time. Again, for the point merge algorithm to work correctly, timestamps for new points coming into the store must be newer than what is currently stored. There are two possible scenarios:

• Problem: system powers up with old time, and points in DB have newer time.
  • Solution: if we don’t have a valid NTP time, then set system time to something later than the newest point timestamp in the store.
• Problem: NTP sets the time “back” and there are newer points in the DB.
  • Solution: when we get a NTP time sync, verify it is not significantly earlier than the latest point timestamp in the system. If it is, update the point timestamps in the DB that are newer than the current time with the current time - 1yr. This ensures that settings upstream (which are likely newer than the edge device) will update the points in the edge device. This is not perfect, but if probably adequate for most systems.

We currently don’t queue data when an edge device is offline. This is a different concern which we will address later.

The SIOT synchronization and point merge algorithm are designed to be simple and bandwidth efficient (works over Cat-M/NBIOT modems). There are design trade-offs. It is not a full-blown replicated, log-based database that will work correctly in every situation. It is designed so that changes can be made in multiple locations while disconnected and when a connection is resumed, that data is merged intelligently. Typically, configuration changes are made at the portal, and sensor data is generated at the edge, so this works well in practice. When in doubt, we prioritize changes made on the upstream (typically cloud instance), as that is the most user accessible system and is where most configuration changes will be made. Sensor data is updated periodically, so that will automatically get refreshed typically within 15m max. The system works best when we have a valid time at every location so we advise ensuring reliable network connections for every device, and at a minimum have a reliable battery backed RTC in every device.

Tracking the latest point timestamp

It may make sense to write the latest point timestamp to the store meta table.

Syncing time from Modem or GPS

Will consider in future. Assume a valid network connection to NTP server for now.

Tracking events where time is not correct

It would be very useful to track events at edge devices where time is not correct and it requires a big jump to be corrected.

TODO: how can we determine this? From systemd-timedated logs?

This information could be used to diagnose when a RTC battery needs replaced, etc.

Verify time matches between synchronized instances

A final check that may be useful is to verify time between synchronized instances are relatively close. This is a final check to ensure the sync algorithm does not wreak havoc between systems, even if NTP is lying.

Reference/Research

NTP

• https://wiki.archlinux.org/title/systemd-timesyncd
• timedatectl status produces following output:

               Local time: Thu 2023-06-01 18:22:23 EDT
           Universal time: Thu 2023-06-01 22:22:23 UTC
                 RTC time: Thu 2023-06-01 22:22:23
                Time zone: US/Eastern (EDT, -0400)
System clock synchronized: yes
              NTP service: active
          RTC in local TZ: no

There is a systemd-timedated D-Bus API.

Decision

what was decided.

objections/concerns

Consequences

what is the impact, both negative and positive.

Additional Notes/Reference

Time storage in rule schedules

• Author: Cliff Brake, last updated: 2023-07-10
• PR/Discussion:
• Status: discussion

Problem

When storing times/dates in rule schedules, we store time as UTC, but this can be problematic when there is a time change. In once application, SIOT plays a chime at a certain time of day, but when time changes (daylight savings time), we need to adjust the time in the rule and this is easy to forget.

Context/Discussion

UTC was chosen as the storage format for the following reasons:

• it is universal – it always means the same thing everywhere
• typically in UI or reports, times are translated to users local times
• server and edge devices can operate in UTC without needing to worry about local time
• rules run on cloud instances have a common timebase to work from. In a highly distributed system, you may have device in one timezone trigger an action in another time zone.

However, must applications (building automation, etc.) run in a single location, and the loss or gain of an hour when the time changes is very inconvenient.

Reference/Research

• TMPDIR Forum discussion

Decision

what was decided.

objections/concerns

Consequences

what is the impact, both negative and positive.

Additional Notes/Reference

JetStream SIOT Store

• Author: Cliff Brake, last updated: 2026-08-07
• Status: in progress (stages 1-2 complete; stage 3 initial implementation complete, follow-on work remaining)

Problem

SQLite has worked well as a SIOT store. There are a few things we would like to improve:

• Synchronization of history
  • Currently, if a device or server is offline, only the latest state is transferred when connected. We would like all history that has accumulated when offline to be transferred once reconnected.
• We want history at the edge as well as cloud
  • This allows us to use history at the edge to run more advanced algorithms like AI
• We currently have to re-compute hashes all the way to the root node anytime something changes
  • This may not scale to larger systems
  • Is difficult to get right if things are changing while we re-compute hashes
    • it requires some type of coordination between the distributed systems, which we currently don’t have.

Context/Discussion

The purpose of this document is to explore storing SIOT state in a NATS JetStream store. SIOT data is stored in a tree of nodes and each node contains an array of points. Note, the term “node” in this document represents a data structure in a tree, not a physical computer or SIOT instance. The term “instance” will be used to represent devices or SIOT instances.

Nodes are arranged in a directed acyclic graph.

A subset of this tree is synchronized between various instances as shown in the below example:

The tree topology can be as deep as required to describe the system. To date, only the current state of a node is synchronized and history (if needed) is stored externally in a time-series database like InfluxDB and is not synchronized. The node tree is an excellent data model for IoT systems.

Each node contains an array of points that represent the state of the node. The points contain a type and a key. The key can be used to describe maps and arrays. We keep points separate so they can all be updated independently and easily merged.

With JetStream, we could store points in a stream where the head of the stream represents the current state of a Node or collection of nodes. Each point is stored in a separate NATS subject.

NATS JetStream is a stream-based store where every message in a stream is given a sequence number. Synchronization is simple in that if a sequence number does not exist on a remote system, the missing messages are sent.

NATS also supports leaf nodes (instances) and streams can be synchronized between hub and leaf instances. If they are disconnected, then streams are “caught up” after the connection is made again.

Several experiments have been run to understand the basic JetStream functionality in this repo.

1. Storing and extracting points in a stream
2. Using streams to store time-series data and measure performance
3. Syncing streams between the hub and leaf instances

Advantages of JetStream

• JetStream is built into NATS, which we already embed and use.
• History can be stored in a NATS stream instead of externally. Currently, we use an external store like InfluxDB to store history.
• JetStream streams can be synchronized between instances.
• JetStream has various retention models so old data can automatically be dropped.
• Leverage the NATS AuthN/AuthZ features.
• JetStream is a natural extension of core NATS, so many of the core SIOT concepts are still valid and do not need to change.

Challenges with moving to JetStream

• Streams are typically synchronized in one direction. This is a challenge for SIOT as the basic premise is data can be modified in any location where a user/device has proper permissions. A user may change a configuration in a cloud portal or on a local touch-screen.
• Sequence numbers must be set by one instance, so you can’t have both a leaf and hub nodes inserting data into a single stream. This has benefits in that it is a very simple and reliable model.
• We are constrained by a simple message subject to label and easily query data. This is less flexible than an SQL database, but this constraint can also be an advantage in that it forces us into a simple and consistent data model.
• SQLite has a built-in cache. We would likely need to create our own with JetStream.

JetStream consistency model

From this discussion:

When the doc mentions immediate consistency, it is in contrast to eventual consistency. It is about how ‘writes’ (i.e. publishing a message to a stream).

JetStream is an immediately consistent distributed storage system in that every new message stored in the stream is done so in a unique order (when those messages reach the stream leader) and that the acknowledgment that the storing of the message has been successful only happens as the result of a RAFT vote between the NATS JetStream servers (e.g. 3 of them if replicas=3) handling the stream.

This means that when a publishing application receives the positive acknowledgement to it’s publication to the stream you are guaranteed that everyone will see that new message in their updates in the same order (and with the same sequence number and time stamp).

This ‘non-eventual’ consistency is what enables ‘compare and set’ (i.e. compare and publish to a stream) operations on streams: because there can only be one new message added to a stream at a time.

To map back to those formal consistency models it means that for writes, NATS JetStream is Linearizable.

Currently SIOT uses a more “eventually” consistent model where we used data structures with some light-weight CRDT proprieties. However, this has the disadvantage that we have to do things like hash the entire node tree to know if anything has changed. In a more static system where not much is changing, this works pretty well, but in a dynamic IoT system where data is changing all the time, it is hard to scale this model.

Message/Subject encoding

In the past, we’ve used the Point data structure. This has worked extremely well at representing reasonably complex data structures (including maps and arrays) for a node. Yet it has limitations and constraints that have proven useful it making data easy to store, transmit, and merge.

// Point is a flexible data structure that can be used to represent
// a sensor value or a configuration parameter.
// ID, Type, and Index uniquely identify a point in a device
type Point struct {
	//-------------------------------------------------------
	//1st three fields uniquely identify a point when receiving updates

	// Type of point (voltage, current, key, etc)
	Type string `json:"type,omitempty"`

	// Key is used to allow a group of points to represent a map or array
	Key string `json:"key,omitempty"`

	//-------------------------------------------------------
	// The following fields are the values for a point

	// Time the point was taken
	Time time.Time `json:"time,omitempty" yaml:"-"`

	// Instantaneous analog or digital value of the point.
	// 0 and 1 are used to represent digital values
	Value float64 `json:"value,omitempty"`

	// Optional text value of the point for data that is best represented
	// as a string rather than a number.
	Text string `json:"text,omitempty"`

	// catchall field for data that does not fit into float or string --
	// should be used sparingly
	Data []byte `json:"data,omitempty"`

	//-------------------------------------------------------
	// Metadata

	// Used to indicate a point has been deleted. This value is only
	// ever incremented. Odd values mean point is deleted.
	Tombstone int `json:"tombstone,omitempty"`

	// Where did this point come from. If from the owning node, it may be blank.
	Origin string `json:"origin,omitempty"`
}

With JetStream, the Typeand Key can be encoded in the message subject:

p.<node id>.<type>.<key>

Message subjects are indexed in a stream, so NATS can quickly find messages for any subject in a stream without scanning the entire stream (see discussion 1 and discussion 2).

Over time, the Point structure has been simplified. For instance, it used to also have an Index field, but we have learned we can use a single Key field instead. At this point it may make sense to simplify the payload. One idea is to do away with the Value and Text fields and simply have a Data field. The components that use the points have to know the data-type anyway to know if they should use the Value or Textfield. In the past, Protobuf encoding was used as we started with quite a few fields and provided some flexibility and convenience. But as we have reduced the number of fields and two of them are now encoded in the message subject, it may be simpler to have a simple encoding for Time, Data, Tombstone, and Origin in the message payload. The code using the message would be responsible for convert Data into whatever datatype is needed. This would open up the opportunity to encode any type of payload in the future in the Data field and be more flexible for the future.

Message payload:

• Time (uint64)
• Tombstone (byte)
• OriginLen (byte)
• Origin (string)
• Data Type (byte)
• Data (length determined by the message length subtracted by the length of the above fields)

Examples of types:

• 0 - unknown or custom
• 1 - float (32, or 64 bit)
• 2 - int (8, 16, 32, or 64 bit)
• 3 - unit (8, 16, 32, or 65 bit)
• 4 - string
• 5 - JSON
• 6 - Protobuf

Putting Origin in the message subject will make it inefficient to query as you will need to scan and decode all messages. Are there any cases where we will need to do this? (this is an example where an SQL database is more flexible). One solution would be to create another stream where the origin is in the subject.

There are times when the current point model does not fit very well - for instance when sending a notification - this is difficult to encode in an array of points. I think in these cases encoding the notification data as JSON probably makes more sense and this encoding should work much better.

Can’t send multiple points in a message

In the past, it was common to send multiple points in a message for a node - for instance when creating a node, or updating an array. However, with the type and key encoded in the subject this will no longer work. What is the implication for having separate messages?

• Will be more complex to create nodes
• When updating an array/map in a node, it will not be updated all at once, but over the time it takes all the points to come into the client.
• There is still value in arrays being encoded as points - for instance a relay devices that contains two relays. However, for configuration are we better served by encoding the struct in a the data field as JSON and updating it as an atomic unit?

UI Implications

Because NATS and JetStream subjects overlap, the UI could subscribe to the current state changes much as is done today. A few things would need to change:

• Getting the initial state could still use the NATS nodes API. However, the Value and Text fields might be merged into Data.
• In the p.<node id> subscription, the Type and Key now would come from the message subject.

Bi-Directional Synchronization

Bi-directional synchronization between two instances may be accomplished by having two streams for every node. The head of both incoming and outgoing streams is looked at to determine the current state. If points of the same type exist in both streams, the point with the latest timestamp wins. In reality, 99% of the time, one set of data will be set by the Leaf instance (ex: sensor readings) and another set of data will be set by the upstream Hub instance (ex: configuration settings) and there will be very little overlap.

The question arises - do we really need bi-directional synchronization and the complexity of having two streams for every node? Every node includes some amount of configuration which can flow down from upstream instances. Additionally, many nodes are collecting data which needs to flow back upstream. So it seems a very common need for every node to have data flowing in both directions. Since this is a basic requirement, it does not seem like much of stretch to allow any data to flow in either stream, and then merge the streams at the endpoints where the data is used.

Does it make sense to use NATS to create merged streams?

NATS can source streams into an additional 3rd stream. This might be useful in that you don’t have to read two streams and merge the points to get the current state. However, there are several disadvantages:

• Data would be stored twice
• Data is not guaranteed to be in chronological order - the data would be inserted into the 3rd stream when it is received. So you would still have to walk back in history to know for sure if you had the latest point. It seems simpler to just read the head of two streams and compare them.

Timestamps

NATS JetStream messages store a timestamp, but the timestamp is when the message is inserted into the stream, not necessarily when the sample was taken. There can be some delay between the NATS client sending the message and the server processing it. Therefore, an additional high-resolution 64-bit timestamp is added to the beginning of each message.

Edges

Edges are used to describe the connections between nodes. Nodes can exist in multiple places in the tree. In the below example, N2 is a child of both N1 and N3.

Edges currently contain the up and downstream node IDs, an array of points, and a node type. Putting the type in the edge made it efficient to traverse the tree by loading edges from a SQLite table and indexing the IDs and type. With JetStream it is less obvious how to store the edge information. SIOT regularly traverses up and down the tree.

• Down: to discover nodes
• Up: to propagate points to up subjects

Because edges contain points that can change over time, edge points need to be stored in a stream, much like we do the node points. If each node has its own stream, then the child edges for the node could be stored in the same stream as the node as shown above. This would allow us to traverse the node tree on startup and perhaps cache all the edges. The following subject can be used for edge points:

p.<up node ID>.<down node ID>.<type>.<key>

Again, this is very similar to the existing NATS API.

Two special points are present in every edge:

• nodeType: defines the type of the downstream node
• tombstone: set to true if the downstream node is deleted

One challenge with this model is much of the code in the SIOT uses a NodeEdge data structure which includes a node and its parent edge. This collection of data describes this instance of a node and is more useful from a client perspective. However, NodeEdge’s are duplicated for every mirrored node in the tree, so don’t really make sense from a storage and synchronization perspective. This will likely become more clear after some implementation work.

NATS up.* subjects

In SIOT, we partition the system using the tree structure and nodes that listen for messages (databases, messaging services, rules, etc.) subscribe to the up.*stream of their parent node. In the below example, each group has it’s own database configuration and the Db node only receives points generated in the group it belongs to. This provides an opportunity for any node at any level in the tree to listen to messages of another node, as long as:

1. It is equal or higher in the structure
2. Shares an ancestor.

The use of “up” subjects would not have to change other than the logic that re-broadcasts points to “up” subjects would need to use the edge cache instead of querying the SQLite database for edges.

AuthN/AuthZ

Authorization typically needs to happen at device or group boundaries. Devices or users will need to be authorized. Users have access to all nodes in their parent group or device. If each node has its own stream, that will simplify AuthZ. Each device or user are explicitly granted permission to all the Nodes they have access to. If a new node is created that is a child of a node a user has permission to view, this new node (and the subsequent streams) are added to the list.

Are we optimizing the right thing?

Any time you move away from an SQL database, you should think long and hard about this. Additionally, there are very nice time-series database solutions out there. So we should have good reasons for inventing yet-another-database. However, mainstream SQL and Time-series databases all have one big drawback: they don’t support synchronizing subsets of data between distributed systems.

With system design, one approach is to order the problems you are solving by difficulty with the top of the list being most important/difficult, and then optimize the system to solve the hard problems first.

1. Synchronizing subsets of data between distributed systems (including history)
2. Be small and efficient enough to deploy at the edge
3. Real-time response
4. Efficient searching through history
5. Flexible data storage/schema
6. Querying nodes and state
7. Arbitrary relationships between data
8. Data encode/decode performance

The number of devices and nodes in systems SIOT is targeting is relatively small, thus the current node topology can be cached in memory. The history is a much bigger dataset so using a stream to synchronize, store, and retrieve time-series data makes a lot of sense.

On #7, will we ever need arbitrary relationships between data? With the node graph, we can do this fairly well. Edges contain points that can be used to further characterize the relationship between nodes. With IoT systems your relationships between nodes is mostly determined by physical proximity. A Modbus sensor is connected to a Modbus, which is connected to a Gateway, which is located at a site, which belongs to a customer.

On #8, the network is relatively slow compared to anything else, so if it takes a little more time to encode/decode data this is typically not a big deal as the network is the bottleneck.

With an IoT system, the data is primarily 1) sequential in time, and 2) hierarchical in structure. Thus, the streaming/tree approach still appears to be the best approach.

Questions

Still open:

• How chatty is the NATS Leaf-node protocol? Is it efficient enough to use over low-bandwidth Cat-M cellular connections (~20-100Kbps)? Bandwidth on constrained links has not been measured.
• Are there any other features of NATS/JetStream that we should be considering?

Resolved by the 2026-08-06 revision:

• Is it practical to have 2 streams for every node? Per-node streams were replaced by boundary-origin streams; stream count now scales with instance count rather than fleet node count.
• Would it make sense to create streams at the device/instance boundaries rather than node boundaries? Yes — this is the adopted model. AuthZ within an instance is preserved because boundaries fall where authorization already happens (devices and groups).
• How robust is the JetStream store compared to SQLite in events like power loss? The file store fsyncs on a 2-minute interval by default, comparable to the prior SQLite WAL exposure; --storeSyncInterval shortens the window or forces an fsync on every write.

Stream Granularity and Synchronization Model (2026-08-06 revision)

The initial Stage 2 implementation used one stream per node. A design review before merging the store raised two structural concerns with that layout and with the original Stage 3 synchronization sketch, and led to a revised model.

Echo in merge-on-receive synchronization. The original Stage 3 sketch fed points received from a remote instance into the local store’s merge logic, which writes them into local streams. With bi-directional sync, each side then replays the other’s points back to it: the hub writes leaf points into hub streams, and the leaf’s consumer on those streams receives its own points again. Preventing the loop requires origin-based echo suppression on every message, and any defect in that suppression circulates points between instances indefinitely. Merge-on-receive also gives up the single-writer property that motivated JetStream in the first place: each stream becomes a mixture of local writes and republished remote writes, with arrival-order interleaving in the history.

Hub scaling. Per-node streams scale with the total number of nodes in the fleet, not with the number of instances. A hub serving 500 devices with 30 nodes each holds roughly 15,000 streams, each with its own file store and accounting, plus a durable consumer per synced stream per connected leaf. Node creation and deletion become stream administration operations rather than message publishes, and startup enumeration touches every stream.

Alternatives considered:

1. One stream per origin instance (an oplog per writer). Sync becomes one consumer per peer and hub storage scales with instance count. MaxMsgsPerSubject retention still works because it applies per subject, not per stream. However, node IDs are UUIDs, so the subject space is flat: selecting a subtree to sync requires maintaining an explicit filter list, and read-side AuthZ inside a single stream depends on filter-constrained consumer permissions, which have sharp edges (single-filter form only, legacy API forms must be denied).
2. Streams at sync/AuthZ boundaries. Authorization in SIOT naturally happens at device or group boundaries (see AuthN/AuthZ above), and a device subtree syncs as a unit. Making the stream the boundary aligns storage, sync, and permissions, and drops hub stream count to a small multiple of the device count.
3. Merge at read instead of on receive. Keep every stream single-writer and replicate remote streams locally (JetStream sourcing or durable consumers). Current state is the merge of subject tips across the local and replica streams, which is exactly the two-stream comparison described in the Bi-Directional Synchronization section above. The in-memory edge and point caches already perform this merge once at load time, so the read-path cost is negligible. Echo is impossible by construction because no instance ever writes remote data into its own streams.

Revised model (adopted): boundary-origin streams, combining 2 and 3:

• A boundary is a node that represents a SIOT instance: the local instance’s root node and any device node that corresponds to a (potentially synced) remote instance. Every node is owned by the nearest boundary found walking up the tree. Nodes above all device boundaries are owned by the instance root boundary.
• Each (boundary, origin instance) pair gets one stream, named inst_<boundaryID>_<originID> (stream names cannot contain dots, so the subject separator becomes an underscore; node IDs are UUIDs and carry dashes of their own). The inst prefix identifies both tokens as instances — a boundary is a node representing an instance — and keeps “node” reserved for the data tree. Only instance <originID> ever appends to that stream.
• Storage subjects carry both routing tokens so stream subject spaces never overlap: inst.<boundaryID>.<originID>.<nodeID>.p.<type>.<key> for node points and inst.<boundaryID>.<originID>.<parentID>.ep.<childID> for edge points. The stream captures inst.<boundaryID>.<originID>.>. Core NATS wire subjects (p.>, ep.>) are unchanged.
• Current state of a node is the merge of subject tips across all inst_<boundaryID>_* streams present locally, newest timestamp wins. The edge and point caches hold the merged state; merging happens at cache load and as messages arrive.
• Trade-offs accepted with this layout: retention (MaxMsgsPerSubject) is tuned per boundary rather than per node; moving a node across boundaries requires republishing its subject tips into the new stream and purging the old subjects; reads consult one stream per origin that has written to the boundary. Nodes mirrored under multiple parents resolve to a single owner (the instance root boundary when more than one boundary can reach them); mirroring across device boundaries remains an open design point for Stage 3.

Experiments

Several proof-of-concept experiments have been run to prove the feasibility of this:

https://github.com/simpleiot/nats-exp

Decision

Implementation is broken down into 3 stages:

1. message/subject encoding changes — COMPLETE (plan, branch feat/js-subject-point-changes). Point struct now uses DataType/Data instead of Value/Text. Protobuf replaced with binary encoding for point wire format. NATS subjects include type/key (p.<nodeId>.<type>.<key>, ep.<nodeId>.<parentId>). One point per NATS message for node points; edge points remain batched for atomicity.
2. switch store from SQLite to JetStream — initial implementation COMPLETE with per-node streams (plan, branch feat/js-store); layout revision to boundary-origin streams COMPLETE (plan). See the Stream Granularity and Synchronization Model section for the analysis behind the revision.
   • Boundary-origin streams: each (boundary, origin instance) pair gets stream inst_<boundaryID>_<originID> capturing subjects inst.<boundaryID>.<originID>.<nodeID>.p.<type>.<key> (node points) and inst.<boundaryID>.<originID>.<parentID>.ep.<childID> (edge points, stored with the parent node’s boundary). Only the origin instance appends to a stream.
   • Streams retain full history (time-series). Current state = merge of subject tips (via GetLastMsgForSubject) across the streams for a boundary, newest timestamp wins. Retention uses MaxMsgsPerSubject (not MaxAge or stream-level MaxBytes/MaxMsgs) so current state is always preserved, including rarely-updated config points that time/size-based policies could silently drop.
   • Retention is resolved per stream: the default is 5000 messages per subject (about a month of 10-minute data, effectively unlimited for configuration subjects, bounded disk on unattended devices), and the server option --storeMaxMsgsPerSubject / SIOT_STORE_MAX_MSGS_PER_SUBJECT overrides it (-1 = unlimited). Stage 3 adds per-boundary overrides at the same resolution point. Each instance’s store owns the configuration of every stream on its own disk: sync pumps create replica streams bare and never update existing stream configuration, and the store applies local retention when it discovers a replica, so hub and device retain independently. Changing the value applies to each existing stream the first time it is ensured or discovered after a restart, and JetStream trims existing subjects to the new limit.
   • Durability: the JetStream file store fsyncs on a 2-minute interval by default, which is the accepted power-loss window for typical deployments (comparable exposure to the prior SQLite WAL configuration). --storeSyncInterval / SIOT_STORE_SYNC_INTERVAL accepts a Go duration to shorten the window, or always to fsync every write for edge devices with unreliable power, trading write throughput.
   • META KV bucket for instance metadata (rootID, jwtKey).
   • In-memory edge and point caches hold the merged current state, populated on startup by reading stream tips.
   • Hash tree removed; JetStream sequence numbers replace it.
   • SQLite removed entirely; migration via siot export/siot import.
3. Use JetStream to sync between systems — initial implementation COMPLETE (plan, branch feat/js-store-boundary-stream), with follow-on work remaining (see the end of this section).
   • Each instance runs its own NATS server and owns its origin streams. The single-writer invariant holds globally: instance R appends only to inst_*_R streams.
   • Instances connect via NATS leaf/client connections. Each instance keeps local replicas of the remote-origin streams for the boundaries it participates in, using JetStream sourcing (durable consumers as a fallback if sourcing proves unsuitable across leaf connections). Replication is sequence-tracked, so reconnect after network loss delivers only missed messages. No rescan or hash comparison.
   • Replicated data stays in the replica streams. There is no merge-on-receive: current state is merged at read in the edge and point caches. Echo cannot occur because no instance writes remote data into its own streams.
   • Example: device X (root node ID X, hub root ID R) owns inst_X_X. The hub writes configuration for X’s subtree to its own inst_X_R. The hub replicates inst_X_X from the device; the device replicates inst_X_R from the hub. Multi-hop topologies chain sourcing through intermediate instances.
   • AuthZ: writes are enforced with core NATS subject permissions (unchanged by stream layout); reads with per-stream JetStream API permissions. Device X may replicate inst_X_* and export only inst_X_X. Grants are issued dynamically (NATS auth callout) as the tree changes.
   • Real-time point delivery continues via core NATS subjects (p.>, ep.>) as today. Replica catch-up covers only the offline/startup gap.
   • Prerequisite spikes before implementation: verify JetStream sourcing behavior across leaf connections/domains, and verify the filter-carrying consumer-create permission form ($JS.API.CONSUMER.CREATE.<stream>.<consumer>.<filter>) on the NATS version SIOT pins.
   • Spike results (2026-08-06): JetStream sourcing across a leaf connection with distinct JetStream domains works, including catch-up after the sourced server restarts (only missed messages delivered); see store/leafnode_spike_test.go. Chained (multi-hop) sourcing and the consumer-create permission form remain to be verified.
   • Initial implementation (2026-08-06, plan) uses durable-consumer replication over the existing upstream client connection rather than sourcing: the sync client copies messages between same-named streams subject-for-subject, acknowledging only after the receiving side confirms the write, so reconnects resume with only missed messages. This needs no leafnode listener and no static JetStream domain configuration (domains are server config, while instance identity is only known once the store initializes), and it chains through intermediate instances naturally. Sourcing over leaf connections remains the intended replacement once identity/domain configuration is worked out.
   • The receiving store consumes replica streams, merges tips into its caches, and re-broadcasts changed tips on the core NATS wire subjects tagged with a Siot-Origin header; a store never persists a wire message tagged with a remote origin. After an offline gap, broadcasts are held until the backlog drains and only final tips are sent, so rules do not replay intermediate values.
   • Deleting a device node on the hub now detaches it: the device does not force itself back into the tree (the old hash sync re-created it); only the hub can restore the edge.
   • Follow-on work is listed in the Remaining Work section below.

Remaining Work

Stage 3 is functional end to end — two instances replicate in both directions, survive disconnection, and converge — but the items below are still outstanding. They are grouped by area and roughly ordered by priority within each group. The Stage 3 plan tracks progress.

Sync coverage

1. Nested device boundaries: only the root boundary replicates today, so a device beneath another device’s boundary does not yet sync.
2. Multi-hop chaining test: each hop is independent and expected to work, but this is unverified.
3. Nodes mirrored across device boundaries: a node reachable from more than one boundary resolves to the instance root boundary. How mirroring should behave across a sync boundary is an open design point (see the Stream Granularity section).
4. Moving a node between boundaries: requires republishing subject tips into the new stream and purging the old subjects. Not implemented.

Transport

5. JetStream sourcing over leaf connections remains the intended replacement for durable-consumer replication, pending a way to drive server domain configuration from instance identity (identity is known only after the store initializes).
6. Chained (multi-hop) sourcing is unverified; the single-hop spike passed (store/leafnode_spike_test.go).

Security

7. AuthZ tightening: instances share a token today. The target is per-stream JetStream permissions issued dynamically via NATS auth callout, so a device may replicate inst_X_* and export only inst_X_X.
8. The filter-carrying consumer-create permission form ($JS.API.CONSUMER.CREATE.<stream>.<consumer>.<filter>) is unverified on the NATS version SIOT pins. Item 7 depends on it.

Operations and observability

9. Per-replica retention overrides: replica streams are currently unlimited. The resolution point exists in maxMsgsForStream.
10. History sinks: the Db client consumes boundary-origin streams with a durable consumer, so node points are gap-free across restarts (client/db.go), and external sinks can follow the same pattern. Remaining: edge points are excluded by the consumer filter and are not stored, and sink lag is not surfaced. High-rate (phrup) data stays a core NATS subscription by design.
11. Sync status points: per-replica lag and last-delivered sequence. SyncCount currently counts replication sessions.
12. Frontend sync status UI: surface lag rather than the former hash and SyncCount values.

Consequences

Positive:

• Every stream is a single-writer, linearizable log with provenance intact. Bi-directional sync cannot echo points between instances.
• Hub storage and consumer counts scale with the number of instances, not with total fleet node count. Node creation and deletion are message publishes, not stream administration.
• Stream boundaries align with sync boundaries and with the natural AuthZ boundaries (devices and groups), so device-level permissions are one rule per device.
• History is retained locally per boundary and synchronizes with the same mechanism as current state.

Negative:

• The store must resolve which boundary owns a node (an edge cache walk) on every write, and boundary resolution rules must be identical on every instance.
• Moving a node across boundaries requires republishing subject tips and purging old subjects; per-node streams handled moves for free.
• Retention is tuned per boundary rather than per node.
• Reads merge tips across one stream per origin instance that has written to the boundary; the in-memory caches hide this cost but must be correct.
• No SQLite fallback; existing users migrate via siot export/siot import.

Additional Notes/Reference

ADR Title

• Author: NAME, last updated: DATE
• PR/Discussion:
• Status: discussion

Problem

What problem are we trying to solve?

Context/Discussion

background, facts surrounding this discussion.

Reference/Research

links to reference material that may be

Decision

what was decided.

objections/concerns

Consequences

what is the impact, both negative and positive.

Additional Notes/Reference