Sol3 OS

Sol3 base units run a customized Android (Ubuntu 18.04) operating system, Sol3 OS.

Sol3 OS includes a set of core services that expose the capabilities of the Sol3 hardware.

Multi-image System

The Sol3 OS is a two layer system imaging scheme that supports over-the-air updates using a combination of customized tooling and OSTree. The system can store an arbitrary number of different system images in a local repository, any of which can be booted into.

OSTree repository

The heart of Sol3 OS is OSTree which allows for managing multiple versions of a system image much like git managed multiple versions of source code. It can be thought of as an “image repository”, in that it can efficiently store many versions of Sol3 OS in one place.

Commits

OSTree commits can be thought of much like commits in git. For our use cases each commit in the repository is a unique version of the system image, and contains metadata containing extra details about that particular version.

Two key differences between the OSTree commits and normal git commits:

  1. OSTree commits do not need to belong to any branch

  2. As a result of 1. OSTree commits are not ordered

In other words there is no concept of the “history” of a commit: each commit exists on its own. This means that when adding a commit to the OS repository we don’t need to worry about the current state of the repository commit

Checkouts

OSTree commits can be checked out much like git commits. A checkout of an OSTree repo is a directory containing the full file-system at a particular commit. One notable difference between typical git checkouts and OSTree checkouts is that it is expected that there are multiple checkouts of an OSTree repo at the same time. To avoid wasting data on checking out the same files over and over again, OSTree uses hard linking. This means that you don’t have to worry about running out of space due to old versions of Sol3 OS being checkout out on the device.

Note

The checkout out OSTree commits are what the system actually boots into during normal operation.

Layer 1: Recovery

This is the first layer of the system which is what is directly booted into by the Android kernel. During nominal operation the user will never interact with this layer in any capacity, but if layer 2 enters a bad state, then the recovery layer can be used to try to recover the system.

Contents

The layer 1 system image is contains the tools needed to manage the OS, but not any of the tools or services needed to interact with the cameras. The entry point of the recovery image is a custom init script which attempts to boot into layer 2 of the system, and falls back to booting the recovery image as normal.

Updating

The recovery layer cannot receive over-the-air updates, but can be re-flashed if a direct ADB connection is possible.

Layer 2: OSTree Checkouts

The second layer of Sol3 OS itself is made of multiple layers, the base of which is a checked out OSTree commit. In order to ensure that the checkout out OSTree commit is not modified while the system is running, layer 2 uses an overlay filesystem. The directory containing the checked out OSTree commit becomes the “lower” directory of the overlayfs, and an “upper” directory is created in a different directory on a persisted partition. The upper directory contains only the changes made to the lower directory, and these changes are overlayed onto the lower directory, hence the name overlayfs.

Contents

The “lower” portion layer 2 system image contains the tools and services needed to manage and interact with the OS and system hardware, including the cameras.

The “upper” portion of the layer 2 system image contains all the changes made to the system image during operation. These changes are persisted by default even during an update, but can be configured to reset on reboot in a granular manner.

Updating

Layer 2 supports over-the-air updates using OSTree’s concept of “static deltas”. For out purposes a static delta is a single file which contains the minimal set of changes needed to get from one OSTree commit to another. The static deltas are pre-generated for each Sol3 OS version and stored in a Sol3 OS registry.

The generation of the static deltas follows a pattern that attempts to minimize the size of each deltas by leveraging the idea that versions closer together in time are likely to require smaller updates. To avoid the number of deltas growing quadratically with respect to the number of versions static deltas are not generated between every pair versions. As a result, sometimes multiple deltas will be chained together to get from one version to another.

The update process looks like the following:

  1. Figure out what commits (versions) of Sol3 OS are already present on the device.

  2. Find a sequence of static deltas which gets from any currently present commit to the desired commit.

  3. Apply each static delta to the OSTree repository on the device

  4. Checkout the desired commit to a known location

  5. Tell the system to attempt booting into the newly checked out commit on the next boot

  6. Restart the system

  7. Set the new commit as the default if everything is as expected.

Note that the static deltas are the only files transferred during an update.