FYI New Pi Kernel causes dump1090-fa to run much harder

I rolled back to the last pre-5.4 update (4.19.118) and now CPU usage is back to normal levels.

Command used: sudo rpi-update e1050e94821a70b2e4c72b318d6c6c968552e9a2

Previously I tried 5.4.42 as an early 5.4 release, but that exhibited the same problems.

Information on the firmware at:

2 Likes

@jefftomlinson - Nice! I didnā€™t want to keep trying other reversions since this was a new thread and seemed like this was something more recent. Maybe going back that far dragged in the earlier version of whatever got changed to cause this? Good info!

Iā€™m at 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux now after restoring from backup.

@davidinjp - I think you are right. @obj was thinking the issue was somewhere in the USB code. Maybe in the kernel module or routines/libraries and not in the kernel? Iā€™m out of my comfort zone so if thatā€™s stupid, disregard. :grin:

Me too! At least we have the fix from @jefftomlinson now.

FWIW, I will not be running dist-upgrade any time soon just in case, but I did upgrade as recently as last week. For 1090-fa I use the FA blue Pro Stick Plus. My current kernel matches what Jeff rolled back to:

$ uname -a
Linux adsb2 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux

This post got me to finally take the time and make a perfect full install SDcard copy (after blowing up my Pi 10 days ago trying to do fancy things with Graphs1090ā€¦). It took a long time to rebuild everything to the way it was. Donā€™t want that late night headache againā€¦

1 Like

Yep! Good backups can save a lot of time and effort! Glad Jeff made that unnecessary for those bit by this without a backup, but itā€™s a good reminder to back up occasionally. :+1:

For our purposes, there is no true reason to need to update/upgrade unless itā€™s needed for another project running on the same machine. That said, should someone be in a pickle:

Confirmed, it breaks between commit 7a4e85f6e682a59f984e5a7b605f9bb90e047585 and b28e2bc11d7dd1aec923a389bc570eddb9343309

So the last commit that retains the lower CPU/USB use is:

sudo rpi-update 7a4e85f6e682a59f984e5a7b605f9bb90e047585 
Linux raspberrypi 5.4.40-v7l+ #1318 SMP Wed May 20 12:00:35 BST 2020 armv7l

This may or may not help someone smart to nail the issue down.

2 Likes

Thatā€™s good work! But there is a reason to update, though. Many updates fix security issues and itā€™s a race from being publicized to having systems patched to have systems protected by the time the attacks hit. Itā€™s script kiddie stuff once the vulnerabilities are incorporated into hacker attack tools. The Pi doesnā€™t even need to be the original way an attacker gets into the local network. All it needs to be is vulnerable and then it can serve as a beachhead for other attacks, a dead drop, a denial of service soldier, etc. Hackers recruit armies of vulnerable systems to use for all sorts of things like searching for and compromising other systems. Unfortunately, the only way to minimize the risk is by staying updated.

1 Like

I guess there are proā€™s and conā€™s and that argument will be valid for as long as computers exist for the most part. If your machine isnā€™t behind a firewall and itā€™s forward facing, there is more risk granted, but the only ones here who have that purposely set it up that way and they should realize those risks already. Buyer beware. The bulk of users here, I venture to guess, use the pre-canned image and itā€™s already wound relatively tight. I definitely hear what you are saying, but years of Winblows use has many brainwashed in that department as well.

Problem is, and for the sake of argument, Raspbian (or whatever the name dujour is) is relatively unstable, especially on the Pi4. dist-upgrade/rpi-update sometimes ends up breaking more than they fix hence the ā€œThis is your problem if you hit Y popupsā€. Fine for the advanced user, but many Microsoftianā€™s are conditioned to think they need updates every week where that really isnā€™t true in this environment.

One really slick upgrade that they recently reintroduced is the ability to boot from USB again, no SD card necessary. IF the USB stack can handle the data without dropping too many samples as it did when they only had a USB2.0 controller shared with LAN (MLAT would simply blow up), this could be the way to go for our use here - much less worry of worn-out SD Cards over time and much faster. PI4 users may see some benefit from some of the updates, so they arenā€™t all bad, donā€™t get me wrong.

1 Like

I keep my devices up to date for this reason but keep an SD card clone as a snapshot just in case. I usually do an apt update and have a nosey at what is going to go on with apt list upgradable and paused when I saw what was going to install on the PiAware. Running just apt update and apt upgrade on another less important Pi, I noticed the kernel had a significant version update and so decided to leave the PiAware well alone for now.

I donā€™t have that much experience of Pi devices but felt somewhat surprised a whole kernel version upgrade could go on via just apt update/upgrade.

1 Like

Definitely agree about the snapshot. Having one is what allowed me to get back to normal before a specific solution was posted.

@Nitr0 - I debated about posting about the security aspects but decided it was worth noting. If referring to an unpatched Pi running PiAware as being ok in this environment, the only difference I see from any other unpatched, network-aware, active network using, system is this is running Linux which in general is pretty robust - but when you get down to it, isnā€™t that different from Windows once you own it. Also, some of the remote attacks that have appeared for Linux in recent years have been downright scary. Itā€™s not as invulnerable as many want to believe. In addition, these Piā€™s pump significant and variable amounts of data that is a nice data stream to help hide other data movements. Plus, most if not all leave them on 24/7 and many ā€œset it and forget itā€. If I was looking for good cover, PiAware systems are actually pretty desirable.

The risk grows for any new, known, vulnerability as time passes from discovery and more hackers update their tools. And to be clear, I donā€™t know if any security updates were even involved in this trouble. In general, staying patched is the best way to go, plus installing and configuring other security measures like not using default passwords, installing iptables/ufw and fail2ban. I donā€™t want to get too far off topic so will leave it there.

I canā€™t disagree with any pointā€¦like I said, there are valid arguments for each side, itā€™s all good and this is a discussion board afterall. :+1:

@obj: These may be of interest, but I could be in the wrong house:
https://gitlab.0pointer.org/mirrors/linux-stable/-/commit/0432f7632a24f15332ba36228fe9d055f4a5a771

EDIT: Donā€™t know, perhaps I am barking up the wrong tree here. Compiled rtl-sdr using the -DENABLE_ZEROCOPY=ON switch, recompiled dump1090-fa, upgraded to the most recent kernel and use is still high. Figured the increased load was from interrupts being tossed, but this is out of my wheelhouse and above my paygrade.

1 Like

That is a different bug, it was discussed a while back in another topic somewhere; the symptom was that using librtlsdr with zerocopy support enabled meant that dump1090 would die with a SIGKILL early during startup. Good to see that itā€™s getting fixed in the kernel though.

1 Like

Coincidentally, that commit was on kernel 5.4.42 and something between 5.4.40 and 5.4.42 causes the increased stress. Figured I may have been onto something. /me cowers in shame

Well, I mean, it could be relatedā€¦ maybe the zerocopy support started working thanks to the bugfix (it would invariably disable itself on Pis because of yet another bug in that area, IIRC) but ends up being slower? I havenā€™t poked at it yet, so this is guesswork.

FYIā€¦ there was another new kernel distribution last night. No joy! dump1090-fa misbehavior is the same.

I can confirm it.

Today i needed to restart my Raspberry 4B because it became unresponsive.

Since this reboot, the CPU utilization for the ADS-B thread went up from 10 to > 45% and increased the overall usage:

As i did not change anything else and the device was up and running for 60 days now, there seem to be an issue.with dump1090-fa on that kernel, or?

my current kernel:
Linux Raspi4 5.4.51-v7l+ #1327 SMP Thu Jul 23 11:04:39 BST 2020 armv7l GNU/Linux

I stopped and started the dump1090-fa service to get a confirmation and itā€™s clearly this process

Downgraded the kernel now two times.
First time to a lower version of 5.x ā†’ same issue
second time to 4.19.119 ā†’ CPU load back to normal.

Same here. Itā€™s very clear, when I updated my system (updated twice)
grafik

Looks like it is, indeed, a problem with zerocopy.

Running 5.4.51 on a Pi 4B, dump1090-fa 3.8.1 + librtlsdr c1faae295cb523d34ebfd1e007f244e8ee624ea3:

librtlsdr built with ENABLE_ZEROCOPY=ON (and zerocopy is actually used):

CPU load: 46.3%
  4031 ms for demodulation
  23800 ms for reading from USB
  8 ms for network input and background tasks

librtlsdr built with ENABLE_ZEROCOPY=OFF:

CPU load: 11.8%
  5490 ms for demodulation
  1614 ms for reading from USB
  11 ms for network input and background tasks

On the older kernels, librtlsdr would try to use zerocopy but would bail out and not actually use it because of a kernel bug.

I would speculate that when zerocopy is enabled and actually used, the write cache characteristics of the memory region the buffers lie in interacts badly with what dump1090 does with those buffers (it does the initial conversion work in-place, writing the modified data back to the same buffer, and this is where the extra CPU seems to be coming from)

1 Like

I thought default was OFF in librtlsdr (IE this would be the default for repo installs), was their workaround for the kernel issue. Is why I tried with zerocopy =ON with the latest kernel and was still seeing the increased use. I had it backwards?

CMAKE:

option(ENABLE_ZEROCOPY "Enable usbfs zero-copy support" OFF)
if (ENABLE_ZEROCOPY)
    message (STATUS "Building with usbfs zero-copy support enabled")
    add_definitions(-DENABLE_ZEROCOPY=1)
else (ENABLE_ZEROCOPY)
    message (STATUS "Building with usbfs zero-copy support disabled, use -DENABLE_ZEROCOPY=ON to enable")
endif (ENABLE_ZEROCOPY)

I prolly messed something upā€¦

This is true for current git, but not for the older versions provided by Raspbian. The default changed after the ā€œdies with SIGKILLā€ problem surfaced, and those changes havenā€™t made it into Debian stable so far AFAIK.