USB usage of flightaware pro stick - does this look normal?

[rickined1]

Guys – I wiped one of my FA systems this weekend and restinalled everything, no software or hardware changes, but now my FA pro stick is causing strange USB problems – for about 5 min i’ll get mlat, and then 5 min later FA will report “local clock unstable” and is unable to process MLAT, and alternate between these ad infinitum. As I found out before, no MLAT from “local clock unstable” 99% of the time is a USB problem. My system is a Ubuntu 22.04.3 LTS NUC 10i7 with 64 gb memory. Overall I only use 3% of CPU. Flightaware stick only uses 20% of CPU max. This system is vastly overpowered – i’ve had other systems where heavy temporary CPU usage will make MLAT unstable for a bit.

I swapped FA Pro sticks – no change. I tried all different types of USB ports, with FA Pro being the only thing on there – no change. I swapped USB cables – no change. I re-added a powered USB hub – no change.

Before I nuke the whole thing again (Which usually fixes these strange one-off Ubuntu bugs), i’m trying to see if there’s a way to figure out what’s going on.

I used usbtop and I got this data – does this look good, or is the FA Pro stick passing too much data and maybe overwhelming the computer/some buffer, causing it to go into an unstable mode every few minutes?

  Device ID 1 :                 0.00 kb/s       0.00 kb/s
  Device ID 2 :                 0.00 kb/s       0.00 kb/s
  Device ID 7 :                 0.00 kb/s       0.00 kb/s
  Device ID 8 :                 0.00 kb/s       0.00 kb/s
  Device ID 9 :                 1.21 kb/s       4949.30 kb/s
Bus ID 1 (Raw USB traffic, bus number 1)        To device       From device
  Device ID 1 :                 0.00 kb/s       0.00 kb/s
  Device ID 2 :                 0.00 kb/s       0.00 kb/s
  Device ID 7 :                 0.00 kb/s       0.00 kb/s
  Device ID 8 :                 0.00 kb/s       0.00 kb/s
  Device ID 9 :                 1.21 kb/s       4949.32 kb/s
Bus ID 2 (Raw USB traffic, bus number 2)        To device       From device
  Device ID 1 :                 0.00 kb/s       0.00 kb/s
Bus ID 3 (Raw USB traffic, bus number 3)        To device       From device
  Device ID 1 :                 0.00 kb/s       0.00 kb/s
Bus ID 4 (Raw USB traffic, bus number 4)        To device       From device
  Device ID 1 :                 0.00 kb/s       0.00 kb/s

My understanding is that USB 2.0s max data is 480 Mbps. This is about 5 Mbps which shouldn’t overwhelm anything…

top -i shows nothing exciting

top - 07:46:58 up 15:23,  1 user,  load average: 0.33, 0.33, 0.31
Tasks: 358 total,   1 running, 357 sleeping,   0 stopped,   0 zombie
%Cpu(s):  1.7 us,  0.3 sy,  0.0 ni, 98.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st
MiB Mem :  64006.5 total,  60694.6 free,    999.4 used,   2312.6 buff/cache
MiB Swap:   2048.0 total,   2048.0 free,      0.0 used.  62080.0 avail Mem

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
    796 dump1090  15  -5   99580  10752   5632 S  17.9   0.0 156:50.71 dump1090-fa
   1486 root      20   0 1455360 115216  19456 S   2.3   0.2  12:10.85 tailscaled
   1841 rslsync   20   0 1983612  40672  12288 S   1.3   0.1  11:04.24 rslsync
    809 combine+  20   0  453424   9856   3712 S   0.7   0.0   3:43.67 readsb
   1895 fr24      20   0  805668  19456   7936 S   0.7   0.0   2:08.37 fr24feed
 803020 rickined  20   0   13352   4224   3328 R   0.7   0.0   0:00.14 top
     16 root      20   0       0      0      0 I   0.3   0.0   1:29.78 rcu_preempt
    591 systemd+  20   0   14832   6784   6016 S   0.3   0.0   1:48.68 systemd-oomd
    799 adsbfi    19  -1  544108   9856   3712 S   0.3   0.0   3:55.68 feed-adsbfi
    802 adsbfi    19  -1   30344  20324   7424 S   0.3   0.0   3:16.25 mlat-client
   1796 piaware   20   0   33316  17456   8576 S   0.3   0.0   2:13.78 piaware
   2131 piaware   20   0   22200  12672   7168 S   0.3   0.0   3:04.62 fa-mlat-client

it looks like I need to nuke and start over (next weekend’s project), but wanted to double check. Ubuntu has done this to me before, last time with not having any audio output via HDMI. But then reinstalling Ubuntu fixed it…

[tomvdhorst]

doe you have other USB devices on the system ? If so remove them temporary to see if that’s improving the situation.

[rickined1]

yeah I tried that. Got rid of all the USBs except for the FA Pro. Tried FA pro only in a powered hub. Rebooted between attempts.

Here’s my mpstat output.

Linux 6.5.0-15-generic (pihole2)        01/29/2024      _x86_64_        (12 CPU)

07:50:15 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
07:50:25 AM  all    1.63    0.09    0.35    0.02    0.00    0.01    0.00    0.00    0.00   97.90
07:50:35 AM  all    1.70    0.08    0.29    0.03    0.00    0.00    0.00    0.00    0.00   97.89
07:50:45 AM  all    1.62    0.08    0.31    0.01    0.00    0.01    0.00    0.00    0.00   97.98
07:50:55 AM  all    1.35    1.50    0.82    0.03    0.00    0.01    0.00    0.00    0.00   96.29
07:51:05 AM  all    1.63    0.13    0.41    0.03    0.00    0.02    0.00    0.00    0.00   97.78
07:51:15 AM  all    1.71    0.08    0.32    0.05    0.00    0.01    0.00    0.00    0.00   97.83
07:51:25 AM  all    1.60    0.07    0.38    0.00    0.00    0.00    0.00    0.00    0.00   97.96
07:51:35 AM  all    1.71    0.08    0.28    0.02    0.00    0.00    0.00    0.00    0.00   97.92
07:51:45 AM  all    1.60    0.17    0.36    0.02    0.00    0.00    0.00    0.00    0.00   97.85
07:51:55 AM  all    1.36    1.12    0.81    0.02    0.00    0.01    0.00    0.00    0.00   96.68
07:52:05 AM  all    1.59    0.07    0.40    0.04    0.00    0.00    0.00    0.00    0.00   97.90
07:52:15 AM  all    1.95    0.08    0.45    0.02    0.00    0.05    0.00    0.00    0.00   97.45
07:52:25 AM  all    1.60    0.19    0.33    0.02    0.00    0.01    0.00    0.00    0.00   97.84
07:52:35 AM  all    1.75    0.08    0.31    0.03    0.00    0.03    0.00    0.00    0.00   97.82
07:52:45 AM  all    1.65    0.10    0.41    0.03    0.00    0.01    0.00    0.00    0.00   97.79
07:52:55 AM  all    1.35    1.26    0.70    0.01    0.00    0.00    0.00    0.00    0.00   96.68
07:53:05 AM  all    1.60    0.15    0.33    0.04    0.00    0.00    0.00    0.00    0.00   97.88
07:53:15 AM  all    1.60    0.09    0.29    0.02    0.00    0.02    0.00    0.00    0.00   97.99
07:53:25 AM  all    1.49    0.09    0.33    0.01    0.00    0.00    0.00    0.00    0.00   98.08
07:53:35 AM  all    1.62    0.08    0.25    0.02    0.00    0.00    0.00    0.00    0.00   98.04
07:53:45 AM  all    1.65    0.18    0.38    0.13    0.00    0.01    0.00    0.00    0.00   97.66
07:53:55 AM  all    1.39    1.23    0.64    0.03    0.00    0.03    0.00    0.00    0.00   96.68
07:54:05 AM  all    1.76    0.06    0.37    0.04    0.00    0.00    0.00    0.00    0.00   97.77
07:54:15 AM  all    1.73    0.09    0.33    0.03    0.00    0.02    0.00    0.00    0.00   97.81
07:54:25 AM  all    1.72    0.16    0.33    0.03    0.00    0.02    0.00    0.00    0.00   97.74
07:54:35 AM  all    1.71    0.09    0.30    0.03    0.00    0.01    0.00    0.00    0.00   97.85
07:54:45 AM  all    2.00    0.08    0.46    0.01    0.00    0.03    0.00    0.00    0.00   97.43
07:54:55 AM  all    1.37    1.24    0.68    0.01    0.00    0.01    0.00    0.00    0.00   96.70

07:54:55 AM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
07:55:05 AM  all    1.75    0.19    0.35    0.03    0.00    0.00    0.00    0.00    0.00   97.68
07:55:15 AM  all    1.77    0.09    0.29    0.04    0.00    0.01    0.00    0.00    0.00   97.79
Average:     all    1.63    0.30    0.41    0.03    0.00    0.01    0.00    0.00    0.00   97.62

I think it’s some weird ubuntu craziness. Sometimes I’ve found, being in expert in wiping my system and reinstalling everything, that sometimes Ubuntu seems to roll a dice and if I get an unlucky roll it won’t install everything correctly even though I’m doing the same process over and over again…
Although I haven’t done this before, I’m going to probably have to reinstall the system again but this time I’ll have everything attached USB-wise to the computer. Maybe that will get it to initialize the USB ports properly.

Up until this week I had both 1090 and 978 FA sticks on the computer with a problem, along with a USB mouse/keyboard, USB input from my uninterruptable power supply, USB that powers my bias tee…

Ubuntu can be fun, but frustrating at that same time. Then again it’s FOSS, so I shouldn’t complain.

BTW thank you (once again) for graphs1090 because it allowed me to see (graphically and easily) what the CPU on the FA stick was doing…

Since I have 2 setups, I could swap the hard drives between the setups and see if the problem follows the hard drive, but while that works 99% of the time, I’ve had GRUB/bootloaders freak out on me in the past and had to nuke systems before. It’ll be easier just to reload my current system…

[tomvdhorst]

Starting over again is usually the best option

It takes me max 1 hour to recreate an station from scratch with all the bells and whistles for my setups.
Oh and I didn’t create Graphs1090, that’s a great tool made by @wiedehopf

[obj]

@rickined1

  Device ID 9 :                 1.21 kb/s       4949.32 kb/s

That looks normal (2.4MSPS * 2 bytes/sample)

mlat instability is usually one of:

• wrong location; or
• USB data getting dropped

There are many reasons that USB data can get dropped:

• power issues
• noisy USB bus
• busy USB bus
• poor scheduling on the USB bus
• buggy USB controller or driver

(the issue is, essentially, that the RTL2832U has very limited buffer space, so if a USB packet gets lost for any reason, probably the buffer will overflow and data gets dropped before it gets re-polled; also, if there’s any long interval between polling due to e.g. controller quirks or USB bus contention or poor USB bus scheduling, that also makes the buffer overflow)

rtl_test -s 2400000 sometimes helps to verify if you’re dropping samples (you should not see any ongoing “lost at least X bytes”; a couple at the start is normal)

[rickined1]

wow, this worked. I’m dropping data like crazy. You can see the periodicity. I have 5 min of nothing (which is when the system reports MLAT is working and it shows me connected to 100+ receivers) and then it turns into drop data central. Anything I can do about this other than wipe the system and start over again since it was working fine (and with 2 sticks, 1090 and 978) until I wiped the computer on Saturday and re-set everything up again?
I honestly think it’s some craziness internal to Ubuntu, though I could sudo systemctl stop piaware dump1090fa and then sudo apt purge piaware dump1090-fa and then sudo reboot and reinstall… I’ve been fortunate enough to have to reinstall piaware many times using @abcd567a repositories many different ways and this is the first time I’ve had this craziness… And the periodicity makes it look like a buffering issue… Like buffer gets overloaded and then needs time to clear itself out.
graph1090 shows CPU levels for the stick being steady, and I also swapped sticks with it being intrinsic to the computer…

Here’s the output for
rtl_test -s 2400000

Found 1 device(s):
  0:  Realtek, RTL2832U, SN: 00001090

Using device 0: Generic RTL2832U
Found Rafael Micro R820T tuner
Supported gain values (29): 0.0 0.9 1.4 2.7 3.7 7.7 8.7 12.5 14.4 15.7 16.6 19.7 20.7 22.9 25.4 28.0 29.7 32.8 33.8 36.4 37.2 38.6 40.2 42.1 43.4 43.9 44.5 48.0 49.6
[R82XX] PLL not locked!
Sampling at 2400000 S/s.

Info: This tool will continuously read from the device, and report if
samples get lost. If you observe no further output, everything is fine.

Reading samples in async mode...
Allocating 15 zero-copy buffers
lost at least 96 bytes
lost at least 960 bytes
lost at least 752 bytes
lost at least 704 bytes
lost at least 932 bytes
lost at least 616 bytes
lost at least 1120 bytes
lost at least 864 bytes
lost at least 1516 bytes
lost at least 412 bytes
lost at least 1568 bytes
lost at least 1432 bytes
lost at least 628 bytes
lost at least 756 bytes
lost at least 1056 bytes
lost at least 964 bytes
lost at least 1992 bytes
lost at least 1380 bytes
lost at least 640 bytes
lost at least 1036 bytes
lost at least 1272 bytes
lost at least 1552 bytes
lost at least 840 bytes
lost at least 1396 bytes
lost at least 820 bytes
lost at least 1052 bytes
lost at least 680 bytes
lost at least 844 bytes
lost at least 496 bytes
lost at least 1024 bytes
lost at least 1408 bytes
lost at least 864 bytes
lost at least 676 bytes
lost at least 1340 bytes
lost at least 1076 bytes
lost at least 604 bytes
lost at least 576 bytes
lost at least 676 bytes
lost at least 556 bytes
lost at least 620 bytes
lost at least 1364 bytes
lost at least 368 bytes
lost at least 728 bytes
lost at least 672 bytes
lost at least 812 bytes
lost at least 1208 bytes
lost at least 1332 bytes
lost at least 1132 bytes
lost at least 660 bytes
lost at least 840 bytes
lost at least 1032 bytes
lost at least 584 bytes
lost at least 1076 bytes
lost at least 780 bytes
lost at least 80 bytes
lost at least 1140 bytes
lost at least 1224 bytes
lost at least 600 bytes
lost at least 240 bytes
lost at least 564 bytes
lost at least 372 bytes
lost at least 648 bytes
lost at least 1296 bytes
lost at least 776 bytes
lost at least 1564 bytes
lost at least 1168 bytes
lost at least 820 bytes
lost at least 1116 bytes
lost at least 548 bytes
lost at least 1204 bytes
lost at least 464 bytes
lost at least 1168 bytes
lost at least 1464 bytes
lost at least 900 bytes
lost at least 896 bytes
lost at least 1288 bytes
lost at least 720 bytes
lost at least 664 bytes
lost at least 1464 bytes
lost at least 1532 bytes
lost at least 1168 bytes
lost at least 788 bytes
lost at least 844 bytes
lost at least 868 bytes
lost at least 536 bytes
lost at least 808 bytes
lost at least 1052 bytes
lost at least 828 bytes
lost at least 1032 bytes
lost at least 520 bytes
lost at least 920 bytes
lost at least 1736 bytes
lost at least 480 bytes
lost at least 596 bytes
lost at least 608 bytes
lost at least 1008 bytes
lost at least 1064 bytes
lost at least 1104 bytes
lost at least 924 bytes
lost at least 944 bytes
lost at least 1444 bytes
lost at least 748 bytes
lost at least 1036 bytes
lost at least 996 bytes
lost at least 1276 bytes
lost at least 520 bytes
lost at least 1512 bytes
lost at least 1080 bytes
lost at least 868 bytes
lost at least 1740 bytes
lost at least 1224 bytes
lost at least 1088 bytes
lost at least 1152 bytes
lost at least 968 bytes
lost at least 864 bytes
lost at least 1028 bytes
lost at least 1360 bytes
lost at least 1344 bytes
lost at least 1256 bytes
lost at least 1672 bytes
lost at least 1376 bytes
lost at least 860 bytes
lost at least 1492 bytes
lost at least 864 bytes
lost at least 416 bytes
lost at least 2028 bytes
lost at least 588 bytes
lost at least 800 bytes
lost at least 880 bytes
lost at least 928 bytes
lost at least 928 bytes
lost at least 656 bytes
lost at least 1032 bytes
lost at least 1272 bytes
lost at least 976 bytes
lost at least 1120 bytes
lost at least 1680 bytes
lost at least 1184 bytes
lost at least 556 bytes
lost at least 1000 bytes
lost at least 948 bytes
lost at least 1220 bytes
lost at least 80 bytes
lost at least 196 bytes
lost at least 1000 bytes
lost at least 1068 bytes
lost at least 656 bytes
lost at least 1156 bytes
lost at least 656 bytes
lost at least 668 bytes
lost at least 968 bytes
lost at least 688 bytes
lost at least 732 bytes
lost at least 912 bytes
lost at least 1004 bytes
lost at least 1444 bytes
lost at least 936 bytes
lost at least 756 bytes
lost at least 808 bytes
lost at least 80 bytes
lost at least 368 bytes
lost at least 1408 bytes
lost at least 900 bytes
lost at least 668 bytes
lost at least 1128 bytes
lost at least 224 bytes
lost at least 32 bytes
lost at least 744 bytes
lost at least 1212 bytes
lost at least 1032 bytes
lost at least 760 bytes
lost at least 608 bytes
lost at least 856 bytes
lost at least 420 bytes
lost at least 912 bytes
lost at least 948 bytes
lost at least 544 bytes
lost at least 676 bytes
lost at least 1000 bytes
lost at least 728 bytes
lost at least 740 bytes
lost at least 1504 bytes
lost at least 1076 bytes
lost at least 804 bytes
lost at least 1000 bytes
lost at least 576 bytes
lost at least 16 bytes
lost at least 420 bytes
lost at least 744 bytes
lost at least 1028 bytes
lost at least 1388 bytes
lost at least 1204 bytes
lost at least 480 bytes
lost at least 520 bytes
lost at least 1016 bytes
lost at least 1152 bytes
lost at least 948 bytes
lost at least 996 bytes
lost at least 824 bytes
lost at least 592 bytes
lost at least 48 bytes
lost at least 876 bytes
^CSignal caught, exiting!

User cancel, exiting...
Samples per million lost (minimum): 114

I’m wondering if / when I re-wipe and re-install if I should just go back to Lubuntu from Ubuntu. I used Lubuntu for the first year, but then decided to be fancy and go with Ubuntu, but Ubuntu seems to bring more bugs with it’s complexity than Lubuntu… It’s basic but gets the job done, and since I only use it as a headless computer anyway, I don’t need the fance GNOME desktop…

[rickined1]

@obj I was reading an old post I made about mlat problems on another setup that fixed itself with a powered USB hub. And in that post I did mention that when my ssd did large transfers it would destabilize the clock for mlat but it was only temporary.

For this most recent system redo I did swap out my nvme for another one. Same capacity. But this new one is much faster…

I bet that this new SSD is pulling power probably every few minutes due to some timed job and that is destabilizing the USB ports
Only one way to find out- to swap hard drives again and rebuild the system.

But I’m 99 percent sure that is what it is. The drive I had in there forever that worked uses an average of 2 watts less than the new drive I have in there, so if the motherboard/usb hub is a bit fragile, that could put it over the edge, I guess. I’ve seen stranger things…

So you can add to the list of MLAT problems that it can be caused by super duper ssds that overpower the clock and that for some reason mlat/fa stick/USB is too sensitive to everything…

I’ll rebuild the computer in a few days and let you know the outcome.

Fun fun fun.

[rickined1]

@obj it was the too fast pci 4.0 SSD sucking up extra watts which was destabilizing the USB bus. Which is surprising given this is an Intel nuc 10i7 with a 120w power supply… oh well.