e1e0.net

long-wireless-links-and-monitoring.md (24255B)

---

 Title: Long Wireless links and monitoring.
 Author: paco
 Date: 2019-07-31
 Type: article
 
 _Update 2021-05-29: This setup is quite outdated now.  One of the endpoints
 does not exist anymore.  Also, I replaced Zerotier by Wireguard, and the
 monitoring part changed quite a bit too.  All the research, materials, and
 build sections may be still useful so I left it all here.  I may revisit this
 document in the future and update it_
 
 ## Intro
 
 Some time ago I built 2 [P-t-P][1] links between some family members' buildings.
 
 Thing is that my brother and my sister live in an area with no coverage from
 traditional ISPs, but that is quite close (5.5km on a straight line, with no
 obstacles) to my parent's which have good coverage (even FTTH) and plenty of
 providers to choose from.
 
 This project has grown _organically_ so to speak, and the requisites kept
 changing.
 
 That, and my lack of experience on the subject make all this far from an
 optimal solution.
 
 In the end it has been working for almost 3 years now.  This is an attempt to
 document all the infrastructure and the bits and pieces used so I do not forget
 about them and maybe it can be of use to somebody else.
 
 ## First steps and research
 
 As I said, I knew nothing about this before tackling the project.  I have some
 solid knowledge about networking, but I knew little about long (for me)
 wireless links, antennas, propagation and a bunch of other stuff I never heard
 of.  So I had to do some research.
 
 If you want to do something like this, is better to plan ahead.  See what the
 requisites are and start digging.
 
 Some things to take into consideration are:
 
 * Budget.  This is an important one in this scenario, as this is for personal
     use only.
 * Distance between the endpoints of the link.  Modern hardware (more on my
     choice later), can easily cover 10km or maybe more, but read the
     manufacturer's datasheet and look for output power, antenna gain and
     sensitivity.  And always take their numbers with a grain of salt, as they
     are usually tested on ideal conditions you won't encounter.  You'll find
     later a way to calculate the ideal numbers to have an estimate.
 * Obstacles.  There has to be perfect clear vision between endpoints.  Wireless
     communications, especially WiFi either on 2.4GHz or 5GHz, are very
     sensitive to obstacles.  Even partial cover can have a big impact on link
     quality.  And clear vision does not mean _"I can see a single point in the
     distance"_, there's this thing called [Fresnel zone][2], under some
     atmospheric conditions or spectrum saturation it will give you a lot of
     trouble.
 * Materials.  Don't be cheap.  This will have to resists the outdoor conditions
     for as long as possible.
 * Neighbours and regulations.  There's the legal part (RF regulations in your
     country and things like that) and the _"social"_ part, in this case my
     family does not live in detached houses but on apartments, so that has to
     be taken into consideration if there are any rules about this.
 * Infrastructure.  And by that I mean all the necessary to be able to install
     the antennas, route the cables, install connectors, etc.  I'm not only
     talking about tools, but also access to the best spots to put the antennas,
     etc.
 * Antenna location.  As a rule of thumb, the higher the better.  But this
     depends a lot on your particular situation.  It deserves some thought.
 * Spectrum saturation.  Wifi is ubiquitous now.  That may be a challenge for
     any installation specially on urban areas.  Ideally, you should check how
     _crowded_ the spectrum is, but this is usually pretty difficult for
     amateurs without special equipment.  Some antennas have a built in spectrum
     analyser, but it may perform badly.
 
 ## Materials
 
 This is a list of materials I choose and why I choose them.  It is short, as it
 is really an easy installation.
 
 ### Antennas
 
 I ended up using [Ubiquity PowerBeams][3] to create the 2 links.  Four in
 total, 2 for each link.
 
 I was looking for some reputable manufacturer trying to avoid problems in the
 future.  Also, I wanted something as simple as possible.  This kind of antennas
 have the _"emitter/receiver"_ and the antenna all in the same device.  So no
 special connectors to be crimped, virtually no losses on cables, just an easy
 [PoE][4] setup from the house to the rooftop.
 
 Also, this antenna has an easy to setup web interface _and_ an SSH server that
 leaves you in a busybox with some proprietary commands that are pretty handy
 for automation and data collection.
 
 There are newer models now and other manufacturers.  Do your research, read on
 forums and all the usual stuff.  I can say those work for this setup with minor
 issues.
 
 If you know something about this subject you may be wondering why I did not use
 something with a wider angle on the _"access point"_ side and use just 3
 antennas instead of 4.  Truth is, I tried, but I had some problems with the 2nd
 link giving poor performance.  Not being an expert on this I can only guess
 that the partial obstruction on the LOS (line of sight) path for the second
 link was the cause of the poor performance, specially on bad weather days (WiFi
 is pretty sensitive to heavy rain) and episodes of spectrum saturation.
 
 Creating a separate link with a dedicated pair of antennas improved the
 situation a lot.
 
 ### Cables
 
 As the antennas only need a network connection, we only need Ethernet cable.
 Be sure that is CAT5e or better.
 
 Always use cable rated for outdoor use.  Regular network cable will not last
 long exposed to rain and the sun's UV.  I went for [this one][5] because it was
 available at the time on Amazon.
 
 ### Connectors
 
 Don't go extra cheap on this, but anything with reasonable quality will do
 here.  The antennas are built in a way that the connectors are never exposed,
 so this part is not that critical.
 
 ### Antenna pole and other hardware
 
 I cannot say much about this.  What to buy here depends a lot on your
 particular setup.  Remember that the higher the better for the antennas, and
 remember wind is a thing ... you do not want it to fly away like a plastic bag.
 
 ## Build steps
 
 This is a list of the build steps I took.  I started checking the list
 mentioned on the [First steps](#First steps and research) section.
 Specifically the location of the antennas and the clear line of sight.
 
 I have to admit that I did a sloppy job on the second link, because I did not
 know about the [Fresnel zone][2] back then, but there's some things you can do
 to mitigate its effects.
 
 ### Calculate signal strength
 
 There's a simple way to calculate the signal strength you should see on the
 other side of the link (on ideal conditions).  This can be taken as a reference
 to see if the setup is viable and what conditions and speed negotiation you can
 expect between the 2 endpoints of the link.
 
 The simplified formula to calculate the signal is:
 
 ```
 emitterPower + emitterGain - signalLoss + receiverGain
 ```
 
 I say this is the simplified formula, because it does not take into account
 loses on cables and connectors, that's because I choose to use a _"all in one
 packet"_ type of antenna, so that makes no sense in this case.  This is a huge
 advantage for a beginner.  Also, because I only take into account the free
 space loss and not any other kinds of loss, that would be a lot more difficult
 to calculate.  That was sufficient for me anyway, as the conditions of line of
 sight are pretty good.
 
 To calculate signal loss, this is the formula:
 
 ```
 loss = 20*log((4*π*d)/λ)
 ```
 
 Being `d` the distance between the 2 endpoints in meters and `λ` the
 wavelength, also in meters.  If you do not remember how to calculate the
 wavelength from the frequency is just:
 
 ```
 λ = C/f
 ```
 
 Being `C` the speed of light in meters per second and `f` the frequency in
 Hertz.
 
 So, as an example, let's say I choose channel `137` which is `5685 MHz`, and
 the 2 endpoints are 5.2km apart.  That gives us a signal loss of `121.85 dB`.
 
 According to the antenna datasheet the transmission power is `5 dBm`, the gain
 of the antenna is `25 dBi` (that's on average I guess across the whole range of
 channels).  So putting all that together I should get on the other end `-66.86
 dBm`.  This works both ways in this case, so now we have to check sensitivity.
 Again according to the datasheet, there's no problem in any modulation
 negotiation with this kind of signal strength (in theory, so to be on the safe
 side add at least `-3 dB` to your results).
 
 ### Physical setup and alignment
 
 With the theory calculations out of the way, knowing that was possible, the fun
 part started, I had to get on the roof and install the antennas.
 
 Of course I won't be saying much about this, as this is different for every
 single installation.  Suffice to say, I had a _"pretty fun time"_ up on ladders
 and climbing to places not meant to be climbed ...
 
 Before securing the antenna to the pole in its final position it has to be
 aligned.  I did this the best I could given the lack of specialised equipment.
 
 On the datasheet there are radiation plots for the chosen model.  The principle
 is simple, those are 2D representations of the radiation lobes of the antenna,
 and the loss referred to the total gain.  So basically you want to point them
 to one another as perfectly as possible, specially for parabolic antennas,
 which have a very narrow beam.
 
 Those radiation plots confused me at first as, in case of the PowerBeam there
 are 4 of them "Vertical Azimuth", "Vertical Elevation", "Horizontal Azimuth" and
 "Horizontal Elevation".  This did not make any sense for me in the beginning,
 as the azimuth is an horizontal angle and elevation is a vertical one.  It
 drove me nuts.  It turns out it refers to both polarisations of the signal that
 those devices create ... Once you understand that is easy, they are just the
 same measurement but times 2, one for each polarisation.
 
 Once I knew  how much of an angle I had before starting to loose signal, and
 with a bit of the good old trigonometry, I knew my margin of error when
 pointing the antennas to each other.
 
 I did this standing behind the antenna and looking as if my line of sight was
 the beam.  With some fiddling, that should be enough for the horizontal
 alignment.  For the vertical one, it was easier, because the error margin is
 pretty big compared to the distance to the ground, even if you're on a tall
 building (again, trigonometry, that angle at 5km is some meters ...).  Anyway
 with the help of some online tool I could calculate that easily to make it as
 precise as possible (search for "antenna downtilt calculator" on your favourite
 search engine).
 
 ### Network diagram and configuration
 
 With the antennas installed, it was time for some configuration.
 
 This is a basic diagram of the network setup I came up with:
 
 ```
                                                                192.168.1.6/24
                                                                 +--------+
                                                                 | Bro.   |
                             192.168.1.2/24      192.168.1.4/24  | Router |
                             +---------+         +----------+    +--------+
                             | Antenna |         | Antenna  |   / 192.168.10.1/24
                         ----| AP1     |+++++++++| ST1      |---           
      192.168.1.1/24 ---/    +---------+         +----------+              
               +---------+                                                 
 +---------+  -|  ISP    |                                                 
 |Internet |-/ |  Router |                                                 
 +---------+   +---------+                                                 
                  |  --\     +---------+         +-----------+             
                   \    --\  | Antenna |         |  Antenna  |             
                    \      --| AP2     |+++++++++|  ST2      |-\           
                    |        +---------+         +-----------+  -\ 192.168.1.7/24
                     \        192.168.1.3/24      192.168.1.5/24 +---------+
              +------------+                                     | Sis.    |
              | Rpi        |                                     | Router  |
              | Monitoring |                                     +---------+
              +------------+                                    192.168.10.1/24
              192.168.1.10/24
 ```
 
 All are cable connections but the `++++` ones, which are the 5km links.
 
 On the routers/APs at the end of the chain I used the same network segment for
 both, as hey will be isolated and do NAT.  I did this because I have little
 control over the ISP router.  It is _"reset to defaults"_ from time to time and
 that caused me problems before.  So setting static routes would be a pain to
 maintain.  That produces double NAT on my siblings', but that's a small price
 to pay for having a stable setup.
 
 Yes, I know that's a shitty thing to do for an ISP (they break your dhcp
 reservations and port forwarding too ...), but most of the ISPs where I live
 are the biggest idiots and do the dumbest stuff you can imagine, so that's not
 even something for them.
 
 The PowerBeams are configurable via a web interface that is pretty intuitive.
 They can also be configured via an SSH access and editing a text file + some
 commands.
 
 Some things I did:
 
 * Enable WDS (transparent bridge mode), so I could see the MAC addresses of all
   the chain from my monitoring station.  That helps on debugging if something
   network goes wrong.
 * I enabled SNMP for monitoring, SSH server for access (with public keys) and
   NTP so the antennas have the right time (good for logs).
 * All 4 antennas were set up on bridge mode.
 * The ones connected to the ISP router were set up as "Access Point" and the
   other 2 as "Stations"
 * The antenna startup wizard asks you for country location.  That's because
   they apply the necessary regulation restrictions automatically.  Do not cheat
   here, you can have problems with your local authorities.  Besides, if you do
   not have good signal within the power output regulations chances are you're
   doing something wrong or the conditions of line of sight, etc. are not really
   good, so it won't matter and you'll be breaking the law for nothing (and
   probably causing problems to other antennas and installations).
 
 If you prefer the command line to configure the antennas, log into them via SSH
 and edit the file `/tmp/system.cfg`.  Then save to `NVRAM` with the command
 `cfgmtd -w`.  Then reset with `/usr/etc/rc.d/rc.softrestart force`.
 
 I do not recommend that method at the beginning, until you get familiar with
 all the options and configurations possible.  You can make a pretty big mess.
 
 As I said earlier, those antennas have a sort of spectrum analyser you can use
 to determine which channel is less busy.  It uses some java applet (yes, I know
 ...) and it has been broken in 2 occasions on some firmware updates.  But it
 can be of assistance if your spectrum is really busy.
 
 
 ### Performance tests
 
 There are 2 ways to easily test the throughput of the links.  The web interface
 has a "speed test" built in.  You have to put the credentials of the other end
 and it can test TX, RX or both.
 
 The other way (that I like the most) is `iperf(1)`.  The antennas have installed
 a basic implementation of that tool, so log into the antenna on the other end,
 and use `iperf(1)` either as server or client to test both sides of the
 communication.
 
 Play a bit with the channel width.  More channel width allows for faster
 transfer rates, but a narrow channel increases stability.
 
 I ended up using `20 MHz` for one of the links and `10 MHz` for the other.
 That last one is the one with less than ideal LOS situation.  In the end
 reducing the channel width and choosing the least busy channel did the trick
 and I could get a stable link.
 
 In the end for the first link I get around `32Mbps` symmetrical.  The second
 link is a lot more variable depending on the conditions and the interferences
 from other stations.  I get up to `17Mbps` symmetrical, and is usually more
 than `12Mbps`, but on worst case scenario it can get as low as `6Mbps`.  Which
 is still enough to watch online videos at `1080p` with today's compressions and
 is more than enough to do any kind of browsing, email and whatever ...  so
 I guess is enough.
 
 ### Monitoring and management
 
 For various reasons I wanted to monitor the whole thing.  My brother had some
 network outages and I did not know why (I'm pretty sure they are related to
 some firmware bug introduced on a recent update, but I have no proof).
 
 My idea for this was to put a Raspberry PI on my parent's network that I could
 connect to and install all the necessary software for monitoring.
 
 As I said earlier, I have little control over the ISP router.  Also, I did not
 want to setup a VPN at my house or something similar on a VPS ...  So I ended
 up using [Zerotier][6] to create a _"local network"_ between one of my hosts at
 my home office and the PI at my parent's.  This software creates an interface
 on the device with a private range, just like a VPN.  The main difference in
 this case is that the _server_ part is managed (you can host it yourself too)
 and it uses some clever tricks to find the best path between to endpoints so
 latency is always the least possible.  It falls back to relay servers if none
 of the direct strategies work.  Besides, is quite easy to add or remove devices
 to/from a given virtual network.
 
 They have some [documentation][10] to make this process easy.
 
 Having the monitoring PI on a local network segment, I could now use it as
 a jump box to ssh into the antennas and routers (using `ProxyJump`), making
 management easier.
 
 In the end I decided to have some data collection and graphing and, after some
 consideration, I choose [influxdb][7] + [telegraf][8] + [grafana][9].  That gives
 me also alerts (more on that later).
 
 InfluxDB for the database backend, telegraf as the _"agent collector"_ and
 grafana for graphing tool.
 
 I choose influxdb because is really [easy to setup][11] on the PI.  Check that
 the retention is enabled so you do not fill up the little SD card on the PI.
 Is also quite easy to [set up telegraf][12] and [grafana][13].
 
 With that running I set up the InfluxDB data source on Grafana.  I used the
 database named _"telegraf"_, which was automatically created by the telegraf
 process as soon as it started collecting data.
 
 Then I configured telegraf to get snmp data from the "Access point" antennas
 and also from the routers at my siblings'.
 
 To do this I had to add a file to the configuration folder
 (something `/etc/telegraf/telegraf.d/snmp.conf`) with the snmp config
 parameters:
 
 ```
 [[inputs.snmp]]
   agents = [ "192.168.1.2", "192.168.1.3", "192.168.1.6", "192.168.1.7" ]
   version = 1
   community = "mycommunity"
   interval = "60s"
   timeout = "10s"
   retries = 3
 
   [[inputs.snmp.field]]
     name = "hostname"
     oid = "RFC1213-MIB::sysName.0"
     is_tag = true
 
   [[inputs.snmp.field]]
     name = "uptime"
     oid = "DISMAN-EXPRESSION-MIB::sysUpTimeInstance"
 
   # IF-MIB::ifTable contains counters on input and output traffic as well as errors and discards.
   [[inputs.snmp.table]]
     name = "interface"
     inherit_tags = [ "hostname" ]
     oid = "IF-MIB::ifTable"
 
     # Interface tag - used to identify interface in metrics database
     [[inputs.snmp.table.field]]
       name = "ifDescr"
       oid = "IF-MIB::ifDescr"
       is_tag = true
 ```
 
 The info that comes from this is basically network traffic for all interfaces
 and uptime.
 
 I also set up telegraf to collect pings to the remote routers.  That gives me
 info about the health of the link, and I based some alerts on that.
 
 The needed config was:
 
 ```
 [[inputs.ping]]
   ## List of urls to ping
   urls = ["192.168.1.6", "192.168.1.7"]
 
   ## Number of pings to send per collection (ping -c <COUNT>)
   count = 3
   ## Per-ping timeout, in s. 0 == no timeout (ping -W <TIMEOUT>)
   timeout = 1.0
 ```
 
 And finally, I wanted to have some info the devices provide, but only through
 some internal commands.  For instance, the number of connected devices.
 
 There are 2 commands that run on those devices that provide some internal
 information (like signal strength, connected devices, and much more).  They are
 `mca-status` and `wstalist`.
 
 It turns out telegraf can execute commands and store that as metrics data, no
 problem.  The configuration looks like this:
 
 ```
 [[inputs.exec]]
   ## Commands array
   commands = [ "/usr/local/bin/get_connected_devices.sh router1" ]
   interval = "300s"
 
   name_override = "conn_devices"
   tag_keys = [ "hostname" ]
   timeout = "5s"
   data_format = "json"
 ```
 
 The script is this:
 
 ```
 #!/bin/sh
 
 set -eu
 
 device=${1:-router1}
 device_info=$(ssh "ubnt@$device" mca-status | tr -d "\r")
 connected_devices=$(echo "$device_info" |grep wlanConnections| cut -d'=' -f 2)
 
 printf '{"hostname": "%s", "devices": %d }' "$device" "$connected_devices"
 ```
 
 It outputs some JSON that telegraf understands.
 
 After this it was just a matter of setting up some grafana dashboards to see
 what I wanted to see.  I think there is enough information on the internet on
 how to do that, so I won't be explaining it here.
 
 As I mentioned my brother was having some outages that I still cannot explain.
 They are fixed rebooting the "access point" part of the link (I'm pretty sure
 they would go away simply kicking out the client, but I could not be bothered
 in looking how to do that programatically ...).
 
 So I thought on automating the reboot process as a mitigation for the
 inconveniences it produces.  I set up an alert on grafana for the ping metric
 that, when it triggers calls a webhook.
 
 I did it that way because I wanted to be notified and also automatically take
 action based on those alerts.  The setup I came up with may seem a bit
 complicated, but it works with simple tools and it has been on service for some
 months now.
 
 For the webhook, I found [this][14], which is meant to be a sort of gateway
 from webhook to XMPP.  It only accepts grafana calls but it can be adapted
 pretty easily.
 
 I did [some modifications][15] to not only send an xmpp message, but also to write
 a flag file on disk on a specified folder if it gets an alert with a specific
 string on it.  Then, there's a cron job running that checks for those flags
 and, if it finds any, executes the script of the same name and deletes the flag
 on success.  All pretty simple to do with shell script.
 
 On the ping alert case, the shell scripts just connect to the "access point"
 antenna and perform a `reboot(8)`.
 
 With that done, outages do not last more than 5 minutes, and they are pretty
 rare anyway.  So I think is a good solution until the day I take the time to
 dig into it (if I ever do it ...).
 
 I also created a custom handler with super simple payload, so I could use it
 from other scripts (not necessarily from this project) to just be notified via
 xmpp.
 
 ## Conclusion
 
 And that's the whole setup.  Without using anything too complicated or
 expensive I could connect those isolated flats, have some insight on what
 happens on the network, have alerts on the most interesting metrics and even
 automate responses if I need to.
 
 I hope this may serve as a source of ideas for similar projects.
 
 [1]: https://en.wikipedia.org/wiki/Point-to-point_(telecommunications)
 [2]: https://en.wikipedia.org/wiki/Fresnel_zone
 [3]: https://www.ui.com/airmax/powerbeam/
 [4]: https://en.wikipedia.org/wiki/Power_over_Ethernet
 [5]: https://www.konigelectronic.com/computer/networking/network-cable-reel-cat5e-futp-100-m-black-solid-55896639
 [6]: https://www.zerotier.com/
 [7]: https://www.influxdata.com/time-series-platform/
 [8]: https://www.influxdata.com/time-series-platform/telegraf/
 [9]: https://grafana.com/
 [10]: https://zerotier.atlassian.net/wiki/spaces/SD/pages/8454145/Getting+Started+with+ZeroTier
 [11]: https://docs.influxdata.com/influxdb/v1.7/introduction/installation/
 [12]: https://docs.influxdata.com/telegraf/v1.11/introduction/installation/
 [13]: https://grafana.com/docs/installation/debian/
 [14]: https://github.com/opthomas-prime/xmpp-webhook/
 [15]: https://git.e1e0.net/xmpp-webhook/log.html