Notice from Adept Server: xx% of messages are not

[foxhunter]

Hi all,

since a few hours i have these messages in the logs and also the profile is reporting an anomaly:

Aug 10 05:35:52 AirSquitter piaware[354]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 12:07:06 AirSquitter piaware[354]: NOTICE from adept server: 10% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 12:08:06 AirSquitter piaware[354]: NOTICE from adept server: 11% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 12:14:09 AirSquitter piaware[354]: NOTICE from adept server: 12% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:16:11 AirSquitter piaware[354]: NOTICE from adept server: 11% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:23:13 AirSquitter piaware[354]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:24:14 AirSquitter piaware[354]: NOTICE from adept server: 12% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:30:15 AirSquitter piaware[354]: NOTICE from adept server: 23% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:35:17 AirSquitter piaware[354]: NOTICE from adept server: 10% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:40:18 AirSquitter piaware[354]: NOTICE from adept server: 10% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:41:18 AirSquitter piaware[354]: NOTICE from adept server: 16% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:42:19 AirSquitter piaware[354]: NOTICE from adept server: 15% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:48:20 AirSquitter piaware[354]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:49:21 AirSquitter piaware[354]: NOTICE from adept server: 12% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:53:23 AirSquitter piaware[354]: NOTICE from adept server: 13% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:54:23 AirSquitter piaware[354]: NOTICE from adept server: 18% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:55:25 AirSquitter piaware[354]: NOTICE from adept server: 14% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 15:56:24 AirSquitter piaware[354]: NOTICE from adept server: 11% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 16:03:27 AirSquitter piaware[354]: NOTICE from adept server: 18% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 16:04:27 AirSquitter piaware[354]: NOTICE from adept server: 16% of multilateration messages (UDP) are not reaching the server - check your network?
Aug 10 16:05:27 AirSquitter piaware[354]: NOTICE from adept server: 11% of multilateration messages (UDP) are not reaching the server - check your network?

There are no networking issues the device is running properly.
Time and Position are coming via GPS which is also fully operational.

Thoughts?

[obj]

@foxhunter

There are no networking issues the device is running properly.

The message is telling you that there are networking issues somewhere between the device and the FlightAware servers.

<20% isn’t really anything to worry about though.

[foxhunter]

Ok, many thanks

So it does not necessarily mean it’s something i can manage in my home network, but it can be also somehwere on the route to the servers?

Beside the message in my profile and the entries in the logs i cannot see that something is not working.
I will leave it as it is and check it from time to time.

Meanwhile the message has disappeared only the log entries every few minutes.

[foxhunter]

Issues are back again without changing the network setup:

Sep 03 11:23:08 Raspy piaware[13677]: NOTICE from adept server: 14% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:24:09 Raspy piaware[13677]: NOTICE from adept server: 15% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:26:09 Raspy piaware[13677]: NOTICE from adept server: 21% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:27:09 Raspy piaware[13677]: NOTICE from adept server: 14% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:29:10 Raspy piaware[13677]: NOTICE from adept server: 21% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:33:11 Raspy piaware[13677]: NOTICE from adept server: 24% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 11:34:12 Raspy piaware[13677]: NOTICE from adept server: 17% of multilateration messages (UDP) are not reaching the server - check your network?

[foxhunter]

Strange, it jumps up and down. Tried different network settings without improvement:

Sep 03 11:19:07 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 11:25:54 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 11:26:10 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 11:36:02 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 11:36:17 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 12:16:30 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 12:17:04 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 12:26:51 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 12:27:08 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 12:37:13 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 12:37:54 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 12:47:41 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 12:47:58 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1476
Sep 03 12:52:54 Raspy piaware[13677]: mlat-client(13700): Route MTU changed to 1492
Sep 03 12:53:25 Raspy piaware[13677]: mlat-client(15945): Route MTU changed to 1492
Sep 03 12:53:40 Raspy piaware[13677]: mlat-client(15945): Route MTU changed to 1476
Sep 03 12:55:54 Raspy piaware[16780]: mlat-client(16823): Route MTU changed to 1476
Sep 03 13:13:41 Raspy piaware[16780]: mlat-client(16823): Route MTU changed to 1492
Sep 03 13:13:57 Raspy piaware[16780]: mlat-client(16823): Route MTU changed to 1476
Sep 03 13:18:43 Raspy piaware[16780]: mlat-client(16823): Route MTU changed to 1492
Sep 03 13:18:59 Raspy piaware[16780]: mlat-client(16823): Route MTU changed to 1476
Sep 03 13:19:12 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1492
Sep 03 13:19:44 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1476
Sep 03 13:29:43 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1492
Sep 03 13:30:18 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1476
Sep 03 13:40:13 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1492
Sep 03 13:40:31 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1476
Sep 03 13:50:18 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1492
Sep 03 13:50:33 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1476
Sep 03 14:00:34 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1492
Sep 03 14:00:52 Raspy piaware[16780]: mlat-client(24421): Route MTU changed to 1476

Changing the MTU size of the network interface to 1476 or 1492 also did not help

[LawrenceHill]

@foxhunter

Strange, it jumps up and down.

Mine is the same. Constantly jumping between 1492 and 1476.

[foxhunter]

At least i am not alone.

My WiFi and the internet connection are running fine. I’ve also connected the device to a repeater which connects with 130 Mbit, also the Internet is 250 Mbit.

This shouldn’t be a problem

[foxhunter]

wlan0     IEEE 802.11  ESSID:"xxx"  
          Mode:Managed  Frequency:2.437 GHz  Access Point: xx:xx:xx:xx:xx:xx   
          Bit Rate=135 Mb/s   Tx-Power=31 dBm   
          Retry short limit:7   RTS thr:off   Fragment thr:off
          Power Management:on
          Link Quality=49/70  Signal level=-61 dBm  
          Rx invalid nwid:0  Rx invalid crypt:0  Rx invalid frag:0
          Tx excessive retries:50  Invalid misc:0   Missed beacon:0

Sep 03 15:43:31 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 15:53:31 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 15:53:50 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:03:47 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:04:04 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:13:50 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:14:07 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:23:56 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:24:11 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:34:03 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:34:19 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:44:16 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:44:32 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476
Sep 03 16:54:17 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1492
Sep 03 16:54:33 Raspy piaware[11337]: mlat-client(15584): Route MTU changed to 1476

Sep 03 16:40:24 Raspy piaware[11337]: NOTICE from adept server: 23% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:41:24 Raspy piaware[11337]: NOTICE from adept server: 31% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:42:25 Raspy piaware[11337]: NOTICE from adept server: 22% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:43:25 Raspy piaware[11337]: NOTICE from adept server: 25% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:44:25 Raspy piaware[11337]: NOTICE from adept server: 32% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:45:26 Raspy piaware[11337]: NOTICE from adept server: 52% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:46:26 Raspy piaware[11337]: NOTICE from adept server: 48% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:47:26 Raspy piaware[11337]: NOTICE from adept server: 52% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:48:27 Raspy piaware[11337]: NOTICE from adept server: 53% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:49:27 Raspy piaware[11337]: NOTICE from adept server: 56% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:50:28 Raspy piaware[11337]: NOTICE from adept server: 53% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:51:28 Raspy piaware[11337]: NOTICE from adept server: 43% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:52:28 Raspy piaware[11337]: NOTICE from adept server: 26% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:53:29 Raspy piaware[11337]: NOTICE from adept server: 24% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:54:29 Raspy piaware[11337]: NOTICE from adept server: 38% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:55:29 Raspy piaware[11337]: NOTICE from adept server: 40% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:56:29 Raspy piaware[11337]: NOTICE from adept server: 50% of multilateration messages (UDP) are not reaching the server - check your network?
Sep 03 16:57:30 Raspy piaware[11337]: NOTICE from adept server: 34% of multilateration messages (UDP) are not reaching the server - check your network?

[foxhunter]

So, for confirmation purposes i’ve connected my Radarcape device (AirSquitter) to that network connection and it shows the same problem.

As the AirSquitter was there before running for months, i would expect, it’s not my network.

`Sep 03 15:18:04 AirSquitter piaware[353]: NOTICE from adept server: 47% of multilateration messages (UDP) are not reaching the server - check your network`?

This should be plenty of bandwidth from and to the device

Also the ping to the piaware servers is looking normal

PING piaware.flightaware.com(2620:13d:c000:11::196 (2620:13d:c000:11::196)) 56 data bytes
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=1 ttl=50 time=131 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=2 ttl=50 time=157 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=3 ttl=50 time=129 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=4 ttl=50 time=131 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=5 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=6 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=7 ttl=50 time=133 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=8 ttl=50 time=127 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=9 ttl=50 time=135 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=10 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=11 ttl=50 time=129 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=12 ttl=50 time=136 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=13 ttl=50 time=133 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=14 ttl=50 time=133 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=15 ttl=50 time=128 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=16 ttl=50 time=137 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=17 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=18 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=19 ttl=50 time=129 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=20 ttl=50 time=129 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=21 ttl=50 time=134 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=22 ttl=50 time=130 ms
64 bytes from 2620:13d:c000:11::196 (2620:13d:c000:11::196): icmp_seq=23 ttl=50 time=130 ms
^C
--- piaware.flightaware.com ping statistics ---
23 packets transmitted, 23 received, 0% packet loss, time 53ms
rtt min/avg/max/mdev = 127.384/132.214/157.295/5.946 ms

[obj]

Are you still seeing it now? (After 1528Z)

[LawrenceHill]

@foxhunter

This should be plenty of bandwidth

Bandwidth isn’t the problem. It’s the maximum size a packet can be without fragmentation.
I’m certain the problem is external to my home network. It looks as if one of the paths to the Flightaware server is going through a tunnel which adds an extra data header and thus reduces the maximum payload that can be sent in a single packet. This could be almost anywhere but as we are in different countries with different ISPs my suspicion is that it is occurring in the FlightAware data centre.

[LawrenceHill]

@obj

Are you still seeing it now?

Yes, still seeing it:

[2021-09-03 16:37 BST] mlat-client(5761): Route MTU changed to 1492
[2021-09-03 16:38 BST] mlat-client(5761): Route MTU changed to 1476
[2021-09-03 16:40 BST] mlat-client(5761): Route MTU changed to 1492
[2021-09-03 16:41 BST] mlat-client(5761): Route MTU changed to 1476

[hphillip]

I am getting the anomaly warning about UDP multilateration traffic on both my PiAware version 5.0 sites. The first is IPv4 connected an the second of IPv6 connected.
Only the IPv4 connected site is showing the bouncing around of MTU size.
However, I am more concerned about the high percentages of UDP not reaching the FlightAware servers on both…

[foxhunter]

Same here on my Raspberry device.

I’ve reconnected again the Jetvision device checking it.
Last check on my Raspberry was 5 Minutes ago, so approx 1600Z

[foxhunter]

@obj

Are you still seeing it now? (After 1528Z)

Hi @obj
I have continued in the already existing thread where others reported the issues as well.
Doesn’t make sense having two for the same case:

MLAT issues after 6.0 upgrade

Just upgraded my PiZero to 6.0 about 5 hours ago and I noticed most of the time my homepage reports MLAT in Orange “local clock source is unstable” most of the time. It sometimes turns green but only lasts a minute or two before going back to orange. I configured adaptative gain both on and off, but the issue persists. Any idea? Thanks.

[esmathews]

We are aware of the MLAT UDP issues and are working on a fix.

[hphillip]

While I have not seen the “XX% of UDP multilateration traffic” message recently (so I assume that issue has been resolved) I am still seeing the Route MTU bouncing around. Most recent observed incidents:

Site 117925
[2021-09-04 08:35 BST] mlat-client(32672): Route MTU changed to 1492
[2021-09-04 08:36 BST] mlat-client(32672): Route MTU changed to 1476

Site 123922
[2021-09-04 09:07 BST] mlat-client(6633): Route MTU changed to 1492
[2021-09-04 09:07 BST] mlat-client(6633): Route MTU changed to 1476

[foxhunter]

Same here, every 10 Minutes:

Sep 04 08:38:46 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 08:39:02 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 08:48:53 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 08:49:12 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 09:08:59 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 09:09:15 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 09:19:08 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 09:19:42 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 09:29:39 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 09:29:55 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 09:40:03 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 09:40:19 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 09:50:18 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 09:50:33 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 10:00:24 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 10:00:39 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 10:20:37 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 10:20:53 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 10:30:52 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 10:31:06 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 10:40:59 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 10:41:17 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 11:01:22 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 11:01:38 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 11:11:25 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 11:11:40 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 11:31:32 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 11:31:48 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476
Sep 04 12:01:55 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1492
Sep 04 12:02:12 Raspy piaware[608]: mlat-client(1438): Route MTU changed to 1476

[obj]

So long as you’re not actually seeing mlat loss, the route MTU flapping is harmless (but annoying)

[foxhunter]

Today i have reactivated my AirSquitter device (Radarcape) and there the route MTU changes are not given.

The last three entries are coming from a restart:

Sep 12 16:19:57 Airsquitter piaware[1453]: mlat-client(1495): Route MTU changed to 1500
Sep 12 18:24:14 Airsquitter piaware[7289]: mlat-client(7328): Route MTU changed to 1500
Sep 12 18:26:01 Airsquitter piaware[7878]: mlat-client(7918): Route MTU changed to 1500