Multilateration (MLAT) Beta

Just for my knowledge, what does this means :question:
06/01/2015 20:10:27 multilateration data no longer required, disabling mlat client
06/01/2015 20:10:27 mlat: Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data

Phew! Ignored the message about upgrading to 2.0 (beta) to my own peril after installing the new MLAT client and, of course, I had email notification set to 12 hours and not the desirable 6 hours. Anyway, corrected my mistakes and I should be back online shortly after successfully installing the 2.0 beta. Note to myself: check the logs after an update. Rgds.

Noticed that mlat related logging (presumably by fa-mlat client) is continuing to output to the same inode (i.e., now logging to piaware.out.yesterday).

(Edit: horrid grammar, less horrid.)

It will be fixed in the next release.

I see in the logs that I’m providing MLAT data to the server.


06/02/2015 06:13:14 366553 msgs recv'd from dump1090-muta (2307 in last 5m); 366470 msgs sent to FlightAware
06/02/2015 06:18:00 mlat: Receiver connection: ready
06/02/2015 06:18:00 mlat: Server connection:   ready
06/02/2015 06:18:00 mlat: Receiver:  939.0 msg/s received     17.0kB/s from receiver
06/02/2015 06:18:00 mlat: Server:      0.0 kB/s from server    6.3kB/s TCP to server   0.0kB/s UDP to server
06/02/2015 06:18:00 mlat: Aircraft: 154 known, 32 requested by server

but on the stats page, it says that my receiver is not supported


Feeder Type: PiAware 2.0
Multilateration (MLAT): Receiver Not Supported

Is this because I’m not on a Pi, but using different platform?

I seem to be getting lots of errors in the log. Pi B+ only running Piaware, Dump mutability and now MLAT client.
[Edit] forgot to add collectd too !


[2015-06-02 06:43 BST] got EOF from multilateration client
[2015-06-02 06:45 BST] 54014 msgs recv'd from dump1090-mutab (1762 in last 5m); 53964 msgs sent to FlightAware
[2015-06-02 06:45 BST] got EOF from multilateration client
[2015-06-02 06:48 BST] Malformed message from multilateration client ('typ'), restarting..
[2015-06-02 06:49 BST] error handling message 'type	mlat_sync	hexid	3C6571	m_sync	0055951' from multilateration client (42), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 06:50 BST] 55990 msgs recv'd from dump1090-mutab (1976 in last 5m); 55940 msgs sent to FlightAware
[2015-06-02 06:51 BST] error handling message 'type	mlat_sync	hexid	4CAA5E	m_sync	0055d7251baf 8d4caa5e58756281c7c362b8de45 0055d6c07' from multilateration client (86), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 06:53 BST] got EOF from multilateration client
[2015-06-02 06:54 BST] error handling message 'type	mlat_sync	hexid	405636	m_sync	00565a3920e5 ' from multilateration client (48), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 06:55 BST] 58026 msgs recv'd from dump1090-mutab (2036 in last 5m); 57976 msgs sent to FlightAware
[2015-06-02 06:56 BST] got EOF from multilateration client
[2015-06-02 06:59 BST] error handling message 'type	mlat_mlat	hexid	484C51	m_short	' from multilateration client (36), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:00 BST] 59981 msgs recv'd from dump1090-mutab (1955 in last 5m); 59931 msgs sent to FlightAware
[2015-06-02 07:02 BST] error handling message 'type	mlat_sync	hexid	40655D	m_sync	0057af3149c3 8d' from multilateration client (50), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:04 BST] Malformed message from multilateration client ('type	mlat_mlat	hexid	489526	'), restarting..
[2015-06-02 07:05 BST] 62057 msgs recv'd from dump1090-mutab (2076 in last 5m); 62007 msgs sent to FlightAware
[2015-06-02 07:07 BST] got EOF from multilateration client
[2015-06-02 07:09 BST] error handling message 'type	mlat_mlat	hexid	39A149	m_long	' from multilateration client (35), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:10 BST] 64209 msgs recv'd from dump1090-mutab (2152 in last 5m); 64159 msgs sent to FlightAware
[2015-06-02 07:12 BST] error handling message 'type	mlat_mlat	hexid	400CB0	m_short	00594f1db87d' from multilateration client (48), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:14 BST] Malformed message from multilateration client ('ty'), restarting..
[2015-06-02 07:15 BST] 66749 msgs recv'd from dump1090-mutab (2540 in last 5m); 66699 msgs sent to FlightAware
[2015-06-02 07:16 BST] error handling message 'type	mlat_mlat	hexid	405C04	m_long	005a0e596872' from multilateration client (47), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:18 BST] got EOF from multilateration client
[2015-06-02 07:20 BST] 69442 msgs recv'd from dump1090-mutab (2693 in last 5m); 69392 msgs sent to FlightAware
[2015-06-02 07:20 BST] got 'error writing "file9": broken pipe' writing to multilateration client, restarting..
[2015-06-02 07:21 BST] Malformed message from multilateration client ('type	mlat_mlat	hexid	4CA2C3	m_sho'), restarting..
[2015-06-02 07:23 BST] error handling message 'type	mlat_mlat	hexid	400A78	m_short	005b33a3' from multilateration client (44), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:25 BST] Malformed message from multilateration client ('type	mlat_mlat	hexid	405F78	m_short'), restarting..
[2015-06-02 07:25 BST] 72166 msgs recv'd from dump1090-mutab (2724 in last 5m); 72116 msgs sent to FlightAware
[2015-06-02 07:26 BST] error handling message 'type	mlat_mlat	hexid	400983	m_short	005bb' from multilateration client (41), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..
[2015-06-02 07:28 BST] Malformed message from multilateration client ('type	mlat_mlat	hexid'), restarting..
[2015-06-02 07:30 BST] 74932 msgs recv'd from dump1090-mutab (2766 in last 5m); 74882 msgs sent to FlightAware
[2015-06-02 07:30 BST] got EOF from multilateration client
[2015-06-02 07:32 BST] got EOF from multilateration client
[2015-06-02 07:33 BST] error handling message 'type	mlat_mlat	hexid	405F67	m_short	005ce7cc' from multilateration client (44), (not enough arguments for all format specifiers
 while executing
"binary format $format {*}$row($var)"
 (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
 invoked from within
"compress_array row"
 (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
 invoked from within
"adept send_array row"
 (procedure "process_mlat_message" line 8)
 invoked from within
"process_mlat_message message"), restarting..

It is CPU overload in piaware. piaware is not keeping up with fa-mlat-client’s output; eventually fa-mlat-client gives up mid-message which looks like a malformed message to piaware.

There will be a new beta version at some stage which tries to deal with the CPU problems.

doesn’t it restrict flexibility shoveiing the MLAT data out through PiAware - could it be more flexible if the mlat client wrote to different servers (possibly geographically local - same continent- to the clients)

will the B+ be able to keep up with the processing or should we look to upgrade now?

I have the B+ and it is running out of CPU. I do live in an area with lots of planes so dump1090 is working hard. Maybe if you live somewhere with fewer planes you may be OK?

My B seems OK for CPU … apart from the issue where the new MLAT client isn’t fast enough for the server (I assume that will be fixed by next version - it’s what betas are for)


top - 18:56:42 up 12 days,  5:18,  1 user,  load average: 1.01, 1.19, 1.13
Tasks:  68 total,   2 running,  66 sleeping,   0 stopped,   0 zombie
%Cpu(s): 45.8 us, 16.3 sy,  0.0 ni, 33.3 id,  0.0 wa,  0.0 hi,  4.6 si,  0.0 st
KiB Mem:    445740 total,   237652 used,   208088 free,    80996 buffers
KiB Swap:   102396 total,        0 used,   102396 free,   100980 cached

  PID USER      PR  NI  VIRT  RES  SHR S  %CPU %MEM    TIME+  COMMAND
24727 dump1090  15  -5 19872 9904 1808 R  46.0  2.2  43:29.51 dump1090-mutabi
24904 root      20   0 73984 6628 3388 S  10.1  1.5   8:28.99 fr24feed
16385 root      20   0 90368 7476 2796 S   3.9  1.7 266:07.68 pfclient
24815 root      20   0 18840 8204 4788 S   2.6  1.8   2:21.36 piaware
25106 root      20   0  4672 2444 2100 R   1.0  0.5   0:00.11 top
    3 root      20   0     0    0    0 S   0.3  0.0  27:13.82 ksoftirqd/0


Installed piaware 2.0-2 via ssh on Pi2. Installed OK. Stats show piaware 2.0 installed with MLAT on and enabled. Updated coords. Waiting for something spectacular to happen!!! :stuck_out_tongue:

Some “lower” activity here at BRU/EBBR at this nightly hour …
Seems my B+ can keep up right now :wink:


 1962 root      20   0 15912 6980 1788 R  33.5  1.6  70:32.43 dump1090
 2014 mlat      25   5 10576 9104 5456 S  26.9  2.0  58:14.38 mlat-client
 2506 root      20   0  9656 8160 4024 S  20.2  1.8   1:59.67 fa-mlat-client
 2192 root      20   0 17472 7880 4440 S  11.4  1.8   5:01.68 piaware
 1969 pi        20   0 22756 5384 2748 S   5.7  1.2  12:38.81 fr24feed_arm-rp
 2535 pi        20   0  4688 2500 2116 R   1.3  0.6   0:00.96 top


Not possible at daytime load, with the actual version.

Beta v2.0-3 is now out, which addresses the CPU usage issue, logging problems, and other issues:



sudo service piaware stop
sudo wget http://flightaware.com/adsb/piaware/files/piaware_2.0-3_armhf.deb
sudo dpkg -i piaware_2.0-3_armhf.deb


Updated successfully. Need more updated feeders in the Ohio area so my MLAT will kick in.

Cheers!
LitterBug

2.0-3 updated successfully at K4AA-1 site (KIAD) :stuck_out_tongue:

I updated. I’m in the Rochester, NY area. Anyone else in my area running the beta?

I’ve updated, and am still seeing errors in the log and high cpu usage:


06/03/2015 13:46:27 mlat: fa-mlat-client 0.1.14.2 starting up
06/03/2015 13:46:28 mlat: Input connected to localhost:30005
06/03/2015 13:46:37 mlat: Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Exception BlockingIOError: BlockingIOError(11, 'write could not complete without blocking') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ascii'> ignored
06/03/2015 13:46:37 the system told us that process 19855 exited cleanly
06/03/2015 13:46:37 the system confirmed that process 19855 exited with an exit status of 0
06/03/2015 13:46:42 got EOF from multilateration client
06/03/2015 13:47:45 mlat: fa-mlat-client 0.1.14.2 starting up
06/03/2015 13:47:45 mlat: Input connected to localhost:30005
06/03/2015 13:47:56 mlat: Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Exception BlockingIOError: BlockingIOError(11, 'write could not complete without blocking') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ascii'> ignored
06/03/2015 13:47:56 the system told us that process 19911 exited cleanly
06/03/2015 13:47:56 the system confirmed that process 19911 exited with an exit status of 0
06/03/2015 13:48:02 error handling message 'type	mlat_sync	hexid	4068B7	m_sync	1b2c159' from multilateration client (42), (not enough arguments for all format specifiers
    while executing
"binary format $format {*}$row($var)"
    (object "::adept" method "::fa_adept::AdeptClient::compress_array" body line 29)
    invoked from within
"compress_array row"
    (object "::adept" method "::fa_adept::AdeptClient::send_array" body line 7)
    invoked from within
"adept send_array row"
    (procedure "process_mlat_message" line 8)
    invoked from within
"process_mlat_message message"), restarting..
06/03/2015 13:49:04 mlat: fa-mlat-client 0.1.14.2 starting up
06/03/2015 13:49:05 mlat: Input connected to localhost:30005
06/03/2015 13:49:15 mlat: Disconnecting from localhost:30005: Lost connection to multilateration server, no need for input data
Exception BlockingIOError: BlockingIOError(11, 'write could not complete without blocking') in <_io.TextIOWrapper name='<stdout>' mode='w' encoding='ascii'> ignored
06/03/2015 13:49:16 the system told us that process 19990 exited cleanly
06/03/2015 13:49:16 the system confirmed that process 19990 exited with an exit status of 0
06/03/2015 13:49:21 got EOF from multilateration client


Piaware cpu is around 3-4% when the mlat client is not running, but jumps to 30% or more when the mlat client starts. The mlat client itself uses 14-15%:

http://i.imgur.com/QJ7J9mS.png

It looks like you have several duplicated processes. Do you know why piaware is running 2x, fr24feed 3x, pfclient 2x?

They aren’t duplicates, it’s just the way htop is displaying threads. If I switch it to tree view you can see the difference:

http://i.imgur.com/RwGJPpp.png