Working From Home Gave Me an Excuse to Set Up Backup Internet (WAN Failover)
A week ago I was in the middle of presenting on a Teams call when I was suddenly disconnected. My Internet connection was down. By the time I got back online the meeting had ended. I rarely have Internet issues but this first world problem must have a reasonable solution.
Backup Internet has been on my personal wish-list for well over a decade, I just have not been able to justify the cost. The quick and simple solution is to tether or switch to a mobile phone when Internet connection issues strike… but in the middle of a MS Teams or Zoom presentation, time is limited.
Chasing the 0.1%
Anyone living near a large metropolitan area in the US will have stable Internet options. But stable does not equal 100% availability. Consumer-grade providers may get close to 100% but maintenance events are performed without notice. “Force majeure” type events such as a tree falling on a line during a storm may also take out a connection.
I am not actively logging when my Internet goes down and I do not care if it goes out for a few minutes at 3am on a Tuesday. If I had to guess, I am unable to use my Internet connection between 4 and 9 hours total per year.
There are 24 hours in a day and 365 days in a year; 24 hours/day * 365 days/year= 8,760 hours/year. An outage of 9 hours per year is 8,751 hours / 8,760 hours = roughly 99.9% uptime. I’m chasing the 0.1%.
Backup Options
I could go out and buy service from a second internet service provider (ISP) — the street I live on has both fiber and coax trenched. I am frugal, it would be stupid to pay another $60–100/month for a separate ISP just to cover the 0.1% use case. Other users online have hacked together mobile modems with prepaid data sim cards, but I do not have time to tinker around - I need something turnkey.
I wound up with the Ubiquiti U-LTE. For $15/month (+an extra $10 for every additional gigabyte) I am on the AT&T LTE data network. Of course owning other Ubiquiti equipment helped me make my decision.
Ubiquiti has always had an interesting set of professional and prosumer products, some with shorter lives than others. The U-LTE has been around for a couple of years and an EU model has been produced so I felt comfortable with the purchase.
Slightly Confusing Specifications and an OK Setup
The Ubiquiti U-LTE datasheet calls out 802.3at power over ethernet (PoE) but my unit appears to work fine with an 802.3af power injector. Perhaps Ubiquiti meant that the passthrough PoE would work up to 802.3at? (For those unfamiliar with the IEEE standards, think of 802.3at as the higher-power sibling of 802.3af). I wish Ubiquiti were clearer on this, but I am happy that I did not need a new switch or injector.
The setup experience for the U-LTE was OK. After turning it on, the unit would not automatically show up for adoption; I did need to SSH into the unit to manually provide the location of a .bin file for a firmware upgrade and then push it to adopt.
Testing with Microsoft Teams
My ultimate test is using my original problem — what happens during a teams call when my Internet fails over to backup? I set up a teams meeting with my phone on a different connection acting as a guest and presenting an online clock. In the middle of this test meeting I pulled the plug on my Internet connection to force a failover to backup Internet.
My screen recording of Teams shows that it takes about 19 seconds from “frozen call” to reconnecting with stability. I will use 19 seconds as my expected benchmark for a complete failover.
My data recording from Wireshark shows just over 12 seconds of no data followed by an “inrush current” of packets. The graph is filtered on data received from the Microsoft Teams server. This correlates well with my screen recording as stability is reached after the “inrush.” Technically one could argue that the failover was 12 seconds, but the 19 seconds includes return to stable operation.
I Am Happy With the Results
There is no way I could fumble around with my phone and reconnect or tether in less than 19 seconds. I’ll keep an eye on the number of automatic failovers over the next few months to determine if this is a good long-term investment; I think it will pay off.
Addendum
I tested my Internet failover again in the middle of a 12-person Teams meeting. I did not measure the failover time it but it felt a lot closer to 10 seconds than 19 seconds. Microsoft Teams handled the failover well. I am even happier than I was when I first posted this.
The one measure I will keep an eye on is bufferbloat. This issue is something that I may write a separate post on in the future. Let me know your thoughts!
Addendum 2 (January 20th 2022)
Maybe I’m getting old but it’s hard to beat terminal access. I have been keeping an eye on signal quality in case I need to adjust the installation location.
The U-LTE has a Qualcomm Atheros SOC (yup, MIPS brings back memories of college). Ubiquiti keeps their kernel up to date and has a rather featureful build of BusyBox.
mca-dump is a UI utility to dump status. If you SSH in, just grep for the values you are interested in: mca-dump | grep -e lte_network -e lte_signal
If you use the U-LTE, let me know how you monitor.
