MLAT keeps disappearing

So for the last few days I notice randomly when watching the radar on the webpage my MLAT will just disappear. Anyone else seeing this?

You can see it in the graph when the aircraft with positions will just drop suddenly.

https://dl.dropboxusercontent.com/u/4910921/Screen%20Shot%202015-10-21%20at%2011.00.45%20AM.png

Same here.

http://i.imgur.com/lUpYZbU.png


10/23/2015 14:30:55 lost connection to FlightAware, reconnecting...
10/23/2015 14:30:55 multilateration data no longer required, disabling mlat client
10/23/2015 14:30:55 reconnecting after 60s...
10/23/2015 14:30:56 mlat(1046): Disconnecting from localhost:30005: **Lost connection to multilateration server**, no need for input data
10/23/2015 14:30:56 mlat(1046): Exiting on connection loss
10/23/2015 14:30:56 the system told us that process 1046 exited cleanly
10/23/2015 14:30:56 the system confirmed that process 1046 exited with an exit status of 0
10/23/2015 14:31:55 connecting to FlightAware piaware.flightaware.com/1200
10/23/2015 14:32:45 FlightAware server SSL certificate validated
10/23/2015 14:32:45 encrypted session established with FlightAware
10/23/2015 14:32:45 autoUpdate is not configured in /etc/piaware or by piaware-config
10/23/2015 14:32:45 manualUpdate is not configured in /etc/piaware or by piaware-config
10/23/2015 14:32:45 multilateration support enabled (use piaware-config to disable)
10/23/2015 14:32:45 logged in to FlightAware as user jepolch
10/23/2015 14:32:46 multilateration support enabled (use piaware-config to disable)
10/23/2015 14:32:46 multilateration data requested, enabling mlat client
10/23/2015 14:32:46 Starting multilateration client: /usr/lib/piaware/helpers/fa-mlat-client --input-connect localhost:30005 --results beast,connect,localhost:30104 --results ext_basestation,listen,30106 --udp-transport 70.42.6.197:8818:388947278
10/23/2015 14:32:46 mlat(11076): fa-mlat-client 0.2.4 starting up
10/23/2015 14:32:46 mlat(11076): Using UDP transport to 70.42.6.197:8818
10/23/2015 14:32:46 mlat(11076): Listening for Extended Basestation-format results connection on port 30106
10/23/2015 14:32:46 mlat(11076): Input connected to localhost:30005
10/23/2015 14:32:47 mlat(11076): Beast-format results connection with localhost:30104: connection established
10/23/2015 14:32:51 mlat(11076): Accepted Extended Basestation-format results connection from 192.168.1.5:52870
10/23/2015 14:33:45 server is sending alive messages; we will expect them
10/23/2015 14:35:55 45858 msgs recv'd from dump1090-mutabi (1156 in last 5m); 45412 msgs sent to FlightAware
10/23/2015 14:36:00 lost connection to FlightAware, reconnecting...
10/23/2015 14:36:00 multilateration data no longer required, disabling mlat client
10/23/2015 14:36:00 reconnecting after 60s...
10/23/2015 14:36:00 mlat(11076): Disconnecting from localhost:30005: **Lost connection to multilateration server**, no need for input data
10/23/2015 14:36:01 mlat(11076): Exiting on connection loss
10/23/2015 14:36:01 the system told us that process 11076 exited cleanly
10/23/2015 14:36:01 the system confirmed that process 11076 exited with an exit status of 0
10/23/2015 14:37:00 connecting to FlightAware piaware.flightaware.com/1200
10/23/2015 14:37:01 error during tls handshake: handshake failed: connection reset by peer, will try again soon...


… and here. thank you jepolch for your great thread about these stats i’m now running too :slight_smile:

https://dl.dropboxusercontent.com/u/39745369/mlat.jpg

You’re welcome, although I’m not an expert on the subject. I was just passing along some useful info. Can you share the code lines for that graph you posted? I like the added detail. Thanks. :slight_smile:

anyway - without your post i wouldn’t never learned the graph-thing :slight_smile:

of course - here is the code - but all glory belongs to paulmerkx - i did just tiny changes to the bottomline data display.

you need the newest dump1090.db and dump1090.py from here:
https://raw.githubusercontent.com/mutability/dump1090-tools/master/collectd/dump1090.db
https://raw.githubusercontent.com/mutability/dump1090-tools/master/collectd/dump1090.py

p.s. sorry - i forgot to mention running mutability 1.15~dev (don’t know whether other flavors would work too - guess obj knows)

and this replaces the standard aircraft_graph function in make-graphs.sh


aircraft_graph() {
  rrdtool graph \
  "$1" \
  --start end-$4 \
  --width 480 \
  --height 200 \
  --step "$5" \
  --title "$3 Aircraft Seen / Tracked" \
  --vertical-label "Aircraft" \
  --lower-limit 0 \
  --units-exponent 0 \
  "TEXTALIGN:center" \
  "DEF:all=$2/dump1090_aircraft-recent.rrd:total:AVERAGE" \
  "DEF:pos=$2/dump1090_aircraft-recent.rrd:positions:AVERAGE" \
  "DEF:mlat=$2/dump1090_mlat-recent.rrd:value:AVERAGE" \
  "CDEF:noloc=all,pos,-" \
  "VDEF:avgac=all,AVERAGE" \
  "VDEF:maxac=all,MAXIMUM" \
  "AREA:all#00FF00:seen  " \
  "LINE1:pos#0000FF:w/ pos  " \
  "LINE1:noloc#FF0000:w/o pos  " \
  "LINE1:mlat#000000:mlat  " \
  "GPRINT:avgac:avg\:%3.0lf  " \
  "GPRINT:maxac:max\:%3.0lf  "
}

kind regards
tom

TomMuc - thanks to you and paulmerkx! Works like a charm. :smiley:

perfect :slight_smile: and i have to say thanx to paulmerkx too

Yeah, mlat is having a few internal hiccups.

(jepolch, looks like you might be having some network trouble too, as you’re completely losing the connection to FA, not just mlat)

Jepolch, are you on Centurylink? Something seems to be up with them for the past couple days…

Nope, Time Warner cable. And during those times I lost connection to FA my internet was able to connect to everything else. I was sitting here watching it and I had no other network issues.

10/27 @ 2126 EDT - Is MLAT down/hiccuping?

No changes on my end, and neither of my receivers are indicating locations from non ADS-B tracks (in either Dump1090 or VRS), though thePiAware log file is indicating MLAT sync with >40 receivers.

I have no MLAT either

Try restarting piaware. The mlat servers are fine at the moment but there may be some residual hiccups if you have not reconnected in the last ~6 hours.

rebooting fixed my MLAT issue

Thanks obj, I sent the restart for PiAware and not my mlat is showing as working. It’s weird because it was saying I was synchronized with 39 receivers before, but they weren’t showing on my graphs or dump1090 screen.

Thanks again.

The problem last night was that the result-delivery path was broken for some connections, so you’d be synchronized and generating results but the results would never back it back to piaware.

Yesterday (10/27) the mlat was down on one of my Pi’s most of the day, but the other was up all day. Pretty much identical hardware and software. I restarted Piaware on the one that was down, but it didn’t fix mlat. Finally I rebooted that Pi and mlat started working again. I hate to have to babysit it like this. It used to be rock solid. I guess we can chalk this up to “growing pains”.

Agreed, seems like some scale-out hiccups in the past month+. But don’t take that as complaining, the price is right :slight_smile:

Restarted one PiAware remotely just now, and I’ll have to do the other one locally tonight. Thanks Oliver.

i restart my raspis every night via crontab - when traffic is low. doing so the worst case is loosing one day of mlat …

CRONJOB REBOOT 04:00:


sudo crontab -e
0 4 * * * /sbin/shutdown -r now

I lost mine again this morning

and now its back