I’ve got a little mobile setup to install in my car, gps is working, range is good, and I’ve got a little screen to display a local feed. Question is how does the piaware connection deal with mobile data. If a hotspot disconnects does the flight data get held in a buffer and uploaded when the connection comes back on? Will it continue searching for the connection indefinitely to start the live data feed again? What are people experiencing with their mobile feeds? I can’t find any definitive answers on this, I just see that others have them installed in cars and rvs but no real detailed experience. Thanks!
No. Data more than a small number of second old isn’t of value, so it isn’t buffered.
That’s good, shouldn’t be hard to resume connections then. Any idea what kind of delay it has when a network comes back online?
Hi,
I just had an internet outage here yesterday, so it was easy to check the logs.
When losing the connection, it will reattempt to connect every 5 seconds for usually three attempts. Then it will wait for approximately 60 seconds before trying again. That wait period seems to vary a bit, but does not increase to a larger value as I had expected it to.
So, I would expect that you would be back online within a minute or two.
I had expected to see 5 second reconnects, and then that period gradually get larger until it reached a max value. Then stay at that max value. The downside of that would be that it would take longer, which is probably why they went with approximately one minute as the max. The last thing software wants to do is hammer the network with reconnects every second or so 24/7 during an outage. What FA is doing seems like a good compromise, that allows for a quick recovery.
Jan 23 18:28:39 raspberrypi piaware[4110]: reconnecting in 62 seconds...
Jan 23 18:29:41 raspberrypi piaware[4110]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Jan 23 18:29:51 raspberrypi piaware[4110]: Connection to adept server at piaware.flightaware.com/1200 failed: couldn't open socket: Temporary failure in name res>
Jan 23 18:29:51 raspberrypi piaware[4110]: reconnecting in 6 seconds...
Jan 23 18:29:57 raspberrypi piaware[4110]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Jan 23 18:30:07 raspberrypi piaware[4110]: Connection to adept server at piaware.flightaware.com/1200 failed: couldn't open socket: Temporary failure in name res>
Jan 23 18:30:07 raspberrypi piaware[4110]: reconnecting in 5 seconds...
Jan 23 18:30:12 raspberrypi piaware[4110]: Connecting to FlightAware adept server at 206.253.80.196/1200
Jan 23 18:30:27 raspberrypi piaware[4110]: Connection attempt with adept server at 206.253.80.196/1200 timed out
Jan 23 18:30:27 raspberrypi piaware[4110]: reconnecting in 4 seconds...
Jan 23 18:30:31 raspberrypi piaware[4110]: Connecting to FlightAware adept server at 206.253.84.194/1200
Jan 23 18:30:46 raspberrypi piaware[4110]: Connection attempt with adept server at 206.253.84.194/1200 timed out
Jan 23 18:30:46 raspberrypi piaware[4110]: reconnecting in 50 seconds...
Jan 23 18:31:24 raspberrypi piaware[4110]: 1739479 msgs recv'd from dump1090-fa (141 in last 5m); 1702313 msgs sent to FlightAware
Jan 23 18:31:36 raspberrypi piaware[4110]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Jan 23 18:31:46 raspberrypi piaware[4110]: Connection to adept server at piaware.flightaware.com/1200 failed: couldn't open socket: Temporary failure in name res>
Jan 23 18:31:46 raspberrypi piaware[4110]: reconnecting in 5 seconds...
Jan 23 18:31:51 raspberrypi piaware[4110]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Jan 23 18:32:01 raspberrypi piaware[4110]: Connection to adept server at piaware.flightaware.com/1200 failed: couldn't open socket: Temporary failure in name res>
Jan 23 18:32:01 raspberrypi piaware[4110]: reconnecting in 6 seconds...
Jan 23 18:32:07 raspberrypi piaware[4110]: Connecting to FlightAware adept server at 206.253.80.201/1200
Jan 23 18:32:22 raspberrypi piaware[4110]: Connection attempt with adept server at 206.253.80.201/1200 timed out
Jan 23 18:32:22 raspberrypi piaware[4110]: reconnecting in 6 seconds...
Jan 23 18:32:28 raspberrypi piaware[4110]: Connecting to FlightAware adept server at 206.253.84.197/1200
Jan 23 18:32:43 raspberrypi piaware[4110]: Connection attempt with adept server at 206.253.84.197/1200 timed out
Jan 23 18:32:43 raspberrypi piaware[4110]: reconnecting in 70 seconds...
Regards,
-Dan
Hi,
Before someone asks, the outage was over 24 hours, and the max reconnect value never increased during that period.
Here is the last reconnect before the network came back up.
Jan 24 18:59:36 raspberrypi piaware[4110]: reconnecting in 59 seconds...
Jan 24 19:00:35 raspberrypi piaware[4110]: Connecting to FlightAware adept server at piaware.flightaware.com/1200
Jan 24 19:00:40 raspberrypi piaware[4110]: 1783164 msgs recv'd from dump1090-fa (120 in last 5m); 1702313 msgs sent to FlightAware
Jan 24 19:00:40 raspberrypi piaware[4110]: Connection with adept server at piaware.flightaware.com/1200 established
Regards,
-Dam
Seems like fairly random wait times there. 5 seconds. 6 seconds. 70 seconds. I guess as long as it always keeps trying that should work. Not like this is a critical application. Maybe I need to go digging in code. 
yes, we randomize the reconnect interval somewhat to reduce thundering-herd problems if a server goes down. (i.e. we don’t want 5000 clients all trying to reconnect at exactly the same time…)
The logic here is essentially to do this:
- try twice via DNS (piaware.flightaware.com)
- try once by IP from a pool of hardcoded fallback IPs in our main datacenter
- try once by IP from a pool of hardcoded fallback IPs in our secondary datacenter
with 5 second delays between retries. If all fail, wait 60 seconds and go back to the start. All delays get +/- 20% jitter added.
The assumption here is that likely failure modes are (in rough order of probability):
- random internet glitch disconnected us, the first reconnect will just work immediately;
- local DNS resolution is broken (a very common failure mode!), we need to use a fallback IP;
- local connectivity is entirely dead;
- the particular adept server IP we chose is dead, but others are up;
- one datacenter (or connectivity to it) is entirely dead, but the other datacenter is up
Hi @obj,
Thank you for the details. That looks like a well-designed system and definitely recovers quickly without constantly hammering the network.
I can appreciate the +/- 20% jitter randomness being used. I assumed it was intentional and that’s why I showed 3 of those varying cycles.
Thanks,
-Dan
This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.