How to run a city-wide wireless network from a drawer

How do we test a wireless network that will be deployed like this:

With a test setup that looks like this:

The answer: MAC-level packet filtering. With a bit of software to help us set it up correctly.

How to test a large-scale IoT mesh network with a small-scale setup

At Thingsquare we build large Internet of Things (IoT) solutions that involve many thousands devices per system.

To cover large areas, we use sub-GHz radio frequencies with IPv6 mesh networking. In a mesh network, every device extend the range of the other devices. What do we mean by large? Sometimes that’s a department store. Sometimes it an entire city.

Is this challenging? Yes. The scale is a challenge by itself. To add to the challenge, every IoT solution is built with custom code.

Even with a solid IoT platform as a starting point, each solution is different. There is custom code running on the tiny microprocessors that are inside each device. There is custom software that controls the devices and handles the data that they produce. This software needs to be developed. And it needs to be tested. And its performance must be good.

IoT solutions are highly complex systems. In such complex systems, it is hard to predict the performance without actually running the system. Then measure its performance.

So we need to run the system as soon as possible. And the system needs to be as close to the real thing as we can make it.

A lab setup

We could just deploy a system in the field, before we even have any custom code written. And then progressively update the code, over the air, in the field as we develop it. That works. We have done this on occasion. But it is highly inefficient. And there is always a risk that we inadvertently deploy a version of the code that has a catastrophic bug that will make the entire system unusable.

So we need a lab setup. With a lab setup, we have immediate, physical access to all devices. This means that we can reprogram them if we would break something.

We also have control of the environment. Do we need to test it with a specific temperature? Do we need to generate radio interference on specific frequencies? In the lab, that’s easy. In the field? Not so much.

A lab setup can be simple and still be useful. All it takes is a bunch of hardware boards placed in the vicinity of each other. The next is up to the software.

How to make a small-scale system behave like a large-scale system

There are several factors that make a lab setup different from a field deployment. In the lab, all devices are close to each other. This means that they hear each other. In the field, devices are far away from each other. So they will only be able to hear the devices that are next to it.

We want to emulate this behavior in the lab. That is, we want to achieve this:

When we in reality have this:

So this is what we do:

• We program each device with the ability to select specific devices that it choses to hear – and which to ignore.
• We configure the that table for each device in a specific way – a way that matches what network structure we want to emulate.

This process is called MAC filtering, because it filters out specific devices based on their Medium Access Control (MAC) layer address. The MAC layer address is a 16-byte long address that is burned into the hardware of each device. This address is sometimes called the EUI – Extended Unique Identifier.

Manually configuring tables of devices and their MAC addresses are hard work though. So we make it easy by allowing devices to be placed on a map.

When we move devices around the map, the circle around the device shows the (emulated) communication range for that device. As the communication range covers neighboring devices, those devices are marked with a line. This makes it easy to see what devices it can communicate with.

Once all devices are placed on the map, we click the Apply button. This will cause all devices’ MAC layer filters to be configured according to their positions on the map.

But wait!

What if we have placed devices in positions where they are not reachable from other devices in the network? This could potentially cause devices to be unreachable.

Fortunately, the system helps by identifying this situation:

And allows us to remedy this by placing the problematic devices in better positions.

Once all devices are positioned, we can ask the system to activate the device filters.

The network will then start to activate their filters. Since the network now has a completely different layout, it might take a few minutes for the network to figure out its new structure.

Once all devices have reported their new parents, we can see the structure of the network in the mesh view.

We can now run the performance measurements that we want, with the full network structure that we can expect to have in the field!

How it works, under the hood

Under the hood, the devices form a wireless IPv6 mesh network. Each device has an IPv6 address, and a MAC-layer address. Each device also has the ability to enable a MAC layer filter, with a specific set of addresses.

By default, the MAC filter is disabled. Until we explicitly enable it.

When we place the devices on the map, and click the Apply button, the system figures out what devices are near each other. For each device, it takes the MAC addresses of its virtual neighbors and puts them in a variable called d.feui. These variables are then synchronized, so that all devices have a list of what MAC addresses to filter.

Each device will have a different number of neighbors in their filter table.

Before activating the filters for a network, it is crucial that all filter tables have been synchronized. Otherwise, we may end up in a situation that is asymmetric: device A may have device B in its table, but device B does not have device A in its table. The devices might then never be able to reach each other.

Once every filter is synchronized, the filters are activated simultaneously.

And then, the devices start to act like they are in a much larger network than they actually are.

What happens if we install a the filter table configuration that causes the network to become disconnected? Or if devices in the middle of the network become unresponsive? The answer: each device has a watchdog timer that disables the filters after a few hours, if the device sees no traffic. Thus the network will eventually recover, even if we were to mess up the filters.

Drawbacks

Although MAC-layer filtering is a standard way to emulate larger network structures, it is not without its drawbacks.

• Devices are still physically close to each other and their radio transmissions will, physically, interfere with each other. Even if they, virtually, don’t hear each others’ packets.
• In the field, devices that are distant from each other will sometimes be heard and sometimes not. MAC layer filtering will not emulate this.

So this method is not able to capture all effects that we see in the field.

But it allows us to do way more, in a chest drawer, than one might think we could.