@jefftomlinson - Nice! I didnāt want to keep trying other reversions since this was a new thread and seemed like this was something more recent. Maybe going back that far dragged in the earlier version of whatever got changed to cause this? Good info!
Iām at 4.19.97-v7+ #1294 SMP Thu Jan 30 13:15:58 GMT 2020 armv7l GNU/Linux now after restoring from backup.
@davidinjp - I think you are right. @obj was thinking the issue was somewhere in the USB code. Maybe in the kernel module or routines/libraries and not in the kernel? Iām out of my comfort zone so if thatās stupid, disregard.
Me too! At least we have the fix from @jefftomlinson now.
FWIW, I will not be running dist-upgrade any time soon just in case, but I did upgrade as recently as last week. For 1090-fa I use the FA blue Pro Stick Plus. My current kernel matches what Jeff rolled back to:
$ uname -a
Linux adsb2 4.19.118-v7l+ #1311 SMP Mon Apr 27 14:26:42 BST 2020 armv7l GNU/Linux
This post got me to finally take the time and make a perfect full install SDcard copy (after blowing up my Pi 10 days ago trying to do fancy things with Graphs1090ā¦). It took a long time to rebuild everything to the way it was. Donāt want that late night headache againā¦
Yep! Good backups can save a lot of time and effort! Glad Jeff made that unnecessary for those bit by this without a backup, but itās a good reminder to back up occasionally.
For our purposes, there is no true reason to need to update/upgrade unless itās needed for another project running on the same machine. That said, should someone be in a pickle:
Confirmed, it breaks between commit 7a4e85f6e682a59f984e5a7b605f9bb90e047585 and b28e2bc11d7dd1aec923a389bc570eddb9343309
So the last commit that retains the lower CPU/USB use is:
sudo rpi-update 7a4e85f6e682a59f984e5a7b605f9bb90e047585
Linux raspberrypi 5.4.40-v7l+ #1318 SMP Wed May 20 12:00:35 BST 2020 armv7l
This may or may not help someone smart to nail the issue down.
Thatās good work! But there is a reason to update, though. Many updates fix security issues and itās a race from being publicized to having systems patched to have systems protected by the time the attacks hit. Itās script kiddie stuff once the vulnerabilities are incorporated into hacker attack tools. The Pi doesnāt even need to be the original way an attacker gets into the local network. All it needs to be is vulnerable and then it can serve as a beachhead for other attacks, a dead drop, a denial of service soldier, etc. Hackers recruit armies of vulnerable systems to use for all sorts of things like searching for and compromising other systems. Unfortunately, the only way to minimize the risk is by staying updated.
I guess there are proās and conās and that argument will be valid for as long as computers exist for the most part. If your machine isnāt behind a firewall and itās forward facing, there is more risk granted, but the only ones here who have that purposely set it up that way and they should realize those risks already. Buyer beware. The bulk of users here, I venture to guess, use the pre-canned image and itās already wound relatively tight. I definitely hear what you are saying, but years of Winblows use has many brainwashed in that department as well.
Problem is, and for the sake of argument, Raspbian (or whatever the name dujour is) is relatively unstable, especially on the Pi4. dist-upgrade/rpi-update sometimes ends up breaking more than they fix hence the āThis is your problem if you hit Y popupsā. Fine for the advanced user, but many Microsoftianās are conditioned to think they need updates every week where that really isnāt true in this environment.
One really slick upgrade that they recently reintroduced is the ability to boot from USB again, no SD card necessary. IF the USB stack can handle the data without dropping too many samples as it did when they only had a USB2.0 controller shared with LAN (MLAT would simply blow up), this could be the way to go for our use here - much less worry of worn-out SD Cards over time and much faster. PI4 users may see some benefit from some of the updates, so they arenāt all bad, donāt get me wrong.
I keep my devices up to date for this reason but keep an SD card clone as a snapshot just in case. I usually do an apt update and have a nosey at what is going to go on with apt list upgradable and paused when I saw what was going to install on the PiAware. Running just apt update and apt upgrade on another less important Pi, I noticed the kernel had a significant version update and so decided to leave the PiAware well alone for now.
I donāt have that much experience of Pi devices but felt somewhat surprised a whole kernel version upgrade could go on via just apt update/upgrade.
Definitely agree about the snapshot. Having one is what allowed me to get back to normal before a specific solution was posted.
@Nitr0 - I debated about posting about the security aspects but decided it was worth noting. If referring to an unpatched Pi running PiAware as being ok in this environment, the only difference I see from any other unpatched, network-aware, active network using, system is this is running Linux which in general is pretty robust - but when you get down to it, isnāt that different from Windows once you own it. Also, some of the remote attacks that have appeared for Linux in recent years have been downright scary. Itās not as invulnerable as many want to believe. In addition, these Piās pump significant and variable amounts of data that is a nice data stream to help hide other data movements. Plus, most if not all leave them on 24/7 and many āset it and forget itā. If I was looking for good cover, PiAware systems are actually pretty desirable.
The risk grows for any new, known, vulnerability as time passes from discovery and more hackers update their tools. And to be clear, I donāt know if any security updates were even involved in this trouble. In general, staying patched is the best way to go, plus installing and configuring other security measures like not using default passwords, installing iptables/ufw and fail2ban. I donāt want to get too far off topic so will leave it there.
EDIT: Donāt know, perhaps I am barking up the wrong tree here. Compiled rtl-sdr using the -DENABLE_ZEROCOPY=ON switch, recompiled dump1090-fa, upgraded to the most recent kernel and use is still high. Figured the increased load was from interrupts being tossed, but this is out of my wheelhouse and above my paygrade.
That is a different bug, it was discussed a while back in another topic somewhere; the symptom was that using librtlsdr with zerocopy support enabled meant that dump1090 would die with a SIGKILL early during startup. Good to see that itās getting fixed in the kernel though.
Coincidentally, that commit was on kernel 5.4.42 and something between 5.4.40 and 5.4.42 causes the increased stress. Figured I may have been onto something. /me cowers in shame
Well, I mean, it could be relatedā¦ maybe the zerocopy support started working thanks to the bugfix (it would invariably disable itself on Pis because of yet another bug in that area, IIRC) but ends up being slower? I havenāt poked at it yet, so this is guesswork.
Looks like it is, indeed, a problem with zerocopy.
Running 5.4.51 on a Pi 4B, dump1090-fa 3.8.1 + librtlsdr c1faae295cb523d34ebfd1e007f244e8ee624ea3:
librtlsdr built with ENABLE_ZEROCOPY=ON (and zerocopy is actually used):
CPU load: 46.3%
4031 ms for demodulation
23800 ms for reading from USB
8 ms for network input and background tasks
librtlsdr built with ENABLE_ZEROCOPY=OFF:
CPU load: 11.8%
5490 ms for demodulation
1614 ms for reading from USB
11 ms for network input and background tasks
On the older kernels, librtlsdr would try to use zerocopy but would bail out and not actually use it because of a kernel bug.
I would speculate that when zerocopy is enabled and actually used, the write cache characteristics of the memory region the buffers lie in interacts badly with what dump1090 does with those buffers (it does the initial conversion work in-place, writing the modified data back to the same buffer, and this is where the extra CPU seems to be coming from)
I thought default was OFF in librtlsdr (IE this would be the default for repo installs), was their workaround for the kernel issue. Is why I tried with zerocopy =ON with the latest kernel and was still seeing the increased use. I had it backwards?
CMAKE:
option(ENABLE_ZEROCOPY "Enable usbfs zero-copy support" OFF)
if (ENABLE_ZEROCOPY)
message (STATUS "Building with usbfs zero-copy support enabled")
add_definitions(-DENABLE_ZEROCOPY=1)
else (ENABLE_ZEROCOPY)
message (STATUS "Building with usbfs zero-copy support disabled, use -DENABLE_ZEROCOPY=ON to enable")
endif (ENABLE_ZEROCOPY)
This is true for current git, but not for the older versions provided by Raspbian. The default changed after the ādies with SIGKILLā problem surfaced, and those changes havenāt made it into Debian stable so far AFAIK.