Ubiquiti is the Apple of networking

I wrote an article a few months ago explaining how to configure SSDP and mDNS forwarding between isolated VLANs on Unifi hardware. Ubiquiti, in their infinite wisdom, broke everything almost immediately that article was published with the release of UnifiOS v2.

The steps related to mDNS forwarding and firewall rules still hold — the UI and functionality of the Network application didn’t change. However, the process required to set up the multicast-relay application no longer apply. UnifiOS v1 ran its services in containers, so it was relatively simple to set up another service as an OCI (read: Podman) container and use the on-boot-script to configure the container to start … on boot. UnifiOS v2 abandons the containerized architecture in favor of running directly on the metal, leaning heavily into the systemd ecosystem.

The UnifiOS 2.4.27 release notes says this change is for “Enhanced device stability due to more efficient resource usage”. I believe it, but it’s frustrating that community addons are so casually invalidated — especially when “auto-update everything” is the system default. If you’ve spent a significant amount of time and research, it’s borderline soul-crushing to have that work procedurally wiped out (and your research invalidated) in the middle of the night.

Solutions

`on-boot-script`

on-boot-script depended on Podman being installed. Its current iteration has a new install script compatible with UnifiOS v2, but it’s only a few files, so it’s relatively easy to set up manually.

To follow along, just SSH into your console; there’s no need to enter the unifi-os shell. (Actually, the unifi-os command no longer exists.)

The new OS has a few notable persistent directories:

/data
/etc/systemd

We can use these directories to our advantage when architecting our new framework. First, the systemd service file, defined to run as soon as the network comes online:

A 0644 /etc/systemd/system/on-boot.service

[Unit]
After=network.target
Description=Run scripts on boot
Wants=network.target

[Service]
ExecStart=/data/on-boot
Type=forking

[Install]
WantedBy=multi-user.target

Let’s then create the script referenced by the ExecStart line:

A 0755 /data/on-boot

#!/usr/bin/env bash

mkdir -p /data/on-boot.d

while IFS= read -r -d '' script; do
    "$script"
done < <(
    find /data/on-boot.d -type f -executable -print0 \
        | sort -z
)

There’s definitely some obscurity happening here… just Bash things. This script finds all executable files under /data/on-boot.d and executes them in order.

Finally, let’s inform systemd of the new service and enable it.

# systemd daemon-reload
# systemd enable --now on-boot.service

From now on, any files placed under /data/on-boot.d will be executed on boot — and (at least until Ubiquiti deems it so) this configuration will persist across firmware updates.

`multicast-relay`

We have a few options to set up additional services like multicast-relay.

Install Podman and re-use the configuration from UnifiOS v1. This is a great option short-term, but unfortunately UnifiOS v3, currently in preview, will contain a kernel change that apparently breaks Podman entirely, so using this option is delaying the inevitable.
Install systemd services directory, as we did above. Since the root filesystem is wiped on firmware updates (except for the directories above), we’ll need to follow the same pattern as above. Note that since the root filesystem is wiped, system dependencies (e.g. packages installed with apt) will also be nuked; ensure that any dependencies are present on the system in the ExecStart script (or perhaps using ExecStartPre). I’ll show an example of this below.
Utilize systemd-nspawn to create “containers” to run our services in. These aren’t the same as OCI containers; it’s more like an overpowered chroot. Not only does this provide a greater level of isolation for your services, effectively preventing them from messing with UnifiOS itself, it means we’ll have a static number of system dependencies.

`systemd-nspawn`

Buried in unifios-utilities’ GitHub Issues, I found that GitHub user peacey had created an additional utility to partially automate this setup. I used their fork as a guide, customizing as necessary.

The general idea is this: create a subdirectory tree representing the persistent filesystem of an entire OS — analogous to the FROM line in a Dockerfile. Then, install your desired functionality in the virtual distro — again, same as a Dockerfile. Finally, enable a service that “boots” that directory as a “container” using systemd-nspawn + on-boot-script. Simple enough!

First, let’s install our system dependencies:

# apt install -t stretch-backports \
    debootstrap \
    systemd-container

UnifiOS 2.5.7 uses Debian Stretch (an old version) but ships with a newer version of systemd. We need to specify the stretch-backports release so we get the version of systemd-container that matches the installed systemd release. This might change with newer releases of UnifiOS, so YMMV. Try without the -t option first.

Next, let’s create our “machine” (the equivalent of an OCI image) based on Debian stable in the persistent /data directory:

# export machine=multicast-relay

# mkdir -p /data/custom/machines
# deboostrap --include dbus,systemd stable /data/custom/machines/$machine

Feel free to use whatever Debian release you’d like! You can also use another tool like pacstrap to create a machine based on a different distro, but it likely won’t be as plug-and-play as matching the host distro (Debian).

Since I’m targeting support for multicast-relay, I configured nspawn to use the equivalent of Podman’s host networking:

A 0644 /etc/systemd/nspawn/$machine.nspawn

[Network]
VirtualEthernet=no

Before we boot the machine as a container, we’ll need to symlink our persistent directory to /var/lib/machines, as this is the directory that machinectl uses when referring to a machine by name. (machinectl is the tool shipped with systemd-container used to manage machine & container actions.)

# mkdir -p /var/lib/machines
# ln -s /data/custom/machines/$machine /var/lib/machines/

# machinectl start $machine

systemd-nspawn supports imperatively creating users and setting their passwords pre-boot, but I figure if anyone has shell access to the console, I’m screwed anyways.

Finally, we can do the equivalent of podman exec:

# machinectl shell $machine
($machine) # _

Okay, this is the potentially difficult bit. Depending on what service you’re trying to add to your stack, there may or may not be a Debian package for it. For multicast-relay, there’s not, so I basically followed the steps from the scyto/multicast-relay image’s Dockerfile then created a systemd service. This works because since machines are booted with systemd as PID 1, all the normal systemd bullsh— *cough* — conventions apply inside containers.

($machine) # apt install git python3 python3-netifaces
($machine) # git clone --depth 1 \
               https://github.com/alsmith/multicast-relay.git
($machine) # apt remove git && apt autoremove

These lines are pulled directly from the Dockerfile, translating its pacman invocations to their apt equivalents.

A 0644 $machine/etc/systemd/system/multicast-relay.service

[Unit]
After=network.target
Wants=network.target

[Service]
ExecStart=/usr/bin/python3 /root/multicast-relay/multicast-relay.py --foreground --interfaces $VLAN_INTERFACES --noMDNS
Restart=on-failure

[Install]
WantedBy=multi-user.target

As in the Podman version, replace $VLAN_INTERFACES with your VLAN network interfaces and append any $OPTS you may have configured. In my case, I appended the --noMDNS flag.

($machine) # systemctl daemon-reload
($machine) # systemctl enable --now multicast-relay.service

And… that’s it! multicast-relay is now running in an nspawn container, and it’s configured to run whenever the machine boots. Feel free to exit the container:

($machine) # ^D
# _

We do have a few more steps, though. Let’s make sure the machine is automatically booted with the console:

# machinectl enable $machine

This will be preserved across reboots, but not across firmware updates, since systemd-nspawn will have been nuked. The /var/lib/machines directory we symlinked above is not persistent, either. Let’s write a quick on-boot script to manage that:

A 0755 /data/on-boot.d/00-containers.sh

#!/usr/bin/env bash

if ! dpkg -l systemd-container | grep ii >/dev/null; then
    apt-get install -t stretch-backports -y systemd-container
fi

mkdir -p /var/lib/machines

while IFS= read -r -d '' machine; do
    machine=$(basename $machine)

    if [ ! -e /var/lib/machines/$machine ]; then
        ln -s /data/custom/machines/$machine /var/lib/machines/

        machinectl enable $machine
        machinectl start $machine
    fi
done < <(
    find /data/custom/machines \
        -maxdepth 1 -mindepth 1 \
        -not -name '*.lck' \
        -print0
)

We can actually handle an additional error case; what if we have a service that doesn’t depend on the networking being online, and we want to be able to ensure that our system dependencies are installed even when we don’t have network access?

To accomplish this, we can cache the .deb files for systemd-container and its dependencies somewhere in /data (I chose /data/custom/dpkg) and use that as a fallback in our on-boot script.

# mkdir -p /data/custom/dpkg
# (
    cd /data/custom/dpkg && \
    apt download -t stretch-backports \
        btrfs-progs \
        libnss-mymachines \
        libzstd1 \
        systemd-container
)

diff --git a/data/on-boot.d/00-containers.sh b/data/on-boot.d/00-containers.sh
-     apt-get install -t stretch-backports -y systemd-container
+     if ! apt-get install -t stretch-backports -y systemd-container; then
+         yes | dpkg -i /data/custom/dpkg/*.deb
+     fi

This process is incredibly annoying, and I want to reiterate just how frustrating it is to be forced to re-learn systems that are already opaque. At the very least, I wish Ubiquiti wouldn’t auto-update across semver-incompatible UnifiOS releases. To mitigate this in the future, I’ve turned off all auto-update functionality, and I’ll update only after reading the release notes.