The tremendous growth of our ADS-B network has required us to scale our stats processing backend server to more processors in order to keep up with the realtime inflow rate. Things are back up now, but there will be a noticeable irregularity in the stats data (slightly lower totals) around Sep 3 00:00:00 UTC 2016 (and possibly the last hour of Sep 2) for some users. Thanks for your support!
Rapid growth is the kind of problem to have! Congratulations and may this kind of problem be the one we have most often!
bob k6rtm
That might explain what I saw as planes seen and position reports were not counted - reported in my new thread here. When ACA087 (Toronto to Shanghai) went over the time was 2016-09-02 22:00 UTC. (I don’t know how many reports I sent to FlightAware but 6 of them were used by FlightAware.com … but none appeared in the statistics.) Perhaps, part of this disruption covered the last 3 hours of Sep 2.
Yesterday I started getting errors in the log is this related? My Mlats stopped.
Site 2827:
[2016-09-03 07:39 BST] 108185 msgs recv’d from dump1090-fa (1135 in last 5m); 108185 msgs sent to FlightAware
[2016-09-03 07:39 BST] mlat-client(694): Receiver status: connected
[2016-09-03 07:39 BST] mlat-client(694): Server status: clock unstable
[2016-09-03 07:39 BST] mlat-client(694): Receiver: 950.8 msg/s received 121.7 msg/s processed (13%)
[2016-09-03 07:39 BST] mlat-client(694): Server: 0.0 kB/s from server 0.0kB/s TCP to server 1.0kB/s UDP to server
[2016-09-03 07:39 BST] mlat-client(694): Aircraft: 2 of 16 Mode S, 20 of 56 ADS-B used
[2016-09-03 07:39 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1387: [Errno 22] Invalid argument
[2016-09-03 07:39 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1740: [Errno 22] Invalid argument
[2016-09-03 07:39 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1613: [Errno 22] Invalid argument
[2016-09-03 07:40 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1807: [Errno 22] Invalid argument
[2016-09-03 07:40 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1677: [Errno 22] Invalid argument
[2016-09-03 07:40 BST] mlat-client(694): Accepted Beast-format results connection from ::ffff:192.168.1.65:1146
[2016-09-03 07:40 BST] mlat-client(694): Beast-format results connection with ::ffff:192.168.1.65:1872: [Errno 22] Invalid argument
[2016-09-03 07:40 BST] mlat-client(694): Accepted Beast-format results connection from ::ffff:192.168.1.65:1222
[2016-09-03 07:40 BST] mlat-client(694): Accepted Beast-format results connection from ::ffff:192.168.1.65:1292
[2016-09-03 07:41 BST] mlat-client(694): Accepted Beast-format results connection from ::ffff:192.168.1.65:1360
Not related. Oliver will need to give feedback on that.
It is an error that is local to your system. What did you change at around the time it started?
edit: also, see the anomaly report for that site: it is running out of CPU.
edit2: you restarted the feeder recently, but before that, it was not doing mlat because the clock was very unstable; either this is because of the CPU problem (did something change there?) or you have deteriorating hardware.
20160903 01:25:43.977 ...]: severity 60 (feed disabled), 27 pairings, 3 in sync, 20 unstable
looking further back, this is a problem that this site has had for weeks, at least, gradually getting worse.
20160823 19:52:04.175 ...]: severity 60 (feed disabled), 40 pairings, 2 in sync, 37 unstable
It’s almost certainly the CPU overload causing it. Work out what’s taking the CPU if you are running other software there, or upgrade to a newer Pi.
Thanks for the response Oliver. A restart seems to have cleared it for now.
However, the system is overloaded by an excessive number of feeders in heavily populated areas all feeding large amounts of data from high traffic flows that don’t add any new tracking points.
At what point does FA decide to not allow any more feeders in areas that are oversubscribed?
Two of us in Yukon, Canada, both see only 100 flights per day. From the 80 or so that are ADS-B equipped, we both contribute to the plots for each aircraft. The area covered is only about 150 nm radius because of mountains in all directions but a couple more feeders here could add to range and enable MLAT. More than that and FA is getting into redundancy.