So for my own education… it basically boils down to a USB stack issue causing excessive buffer/“memcopy” interrupts? Not wanting to confuse the thread, just trying to validate. I would assume the SIGKILL problem was more or less a timeout, or a GCC compilation bug. There are so many memory access layers, I get lost.
There’s a bunch of stuff going on here.
The idea behind zerocopy is to allow userspace to directly map the actual memory buffer used for talking to the USB controller. So the USB controller fills that memory, and then userspace can directly use the data, without the kernel needing to copy the data around further.
There was a cache coherency or mapping problem of some type (I’m not familiar with the exact mechanics) that meant that in older kernels, the userspace zerocopy buffer didn’t actually point at the right place. So userspace would see garbage data, not the expected USB data. librtlsdr built with ENABLE_ZEROCOPY=ON tests for this and disables zerocopy if that kernel bug is present. This is what happens on the 4.x kernels on a Pi.
Separately (I’m not sure if it had the same cause or not) there was a kernel bug that meant on some architectures (maybe aarch64? I forget), trying to access a zerocopy buffer at all resulted in a SIGKILL. librtlsdr would trigger this while trying to test for the previous zerocopy bug. I think this was, again, a mapping problem where the mapping pointed to the wrong place, perhaps outside physical memory. The existence of this problem lead to librtlsdr changing the default to ENABLE_ZEROCOPY=OFF
Finally, with both kernel bugs fixed and ENABLE_ZEROCOPY=ON (this is the situation on Raspbian with a 5.x kernel), zerocopy is actually used. But it performs terribly; if I had to guess, given the previous cache coherency issues, this might be due to something like having the memory region underlying the zerocopy buffers set to uncacheable.
AFAIK userspace is not doing anything really wrong here, it’s just that the zerocopy performance is bad.
dump1090 can probably avoid the problem by ensuring that it only reads from the zerocopy buffer, and doesn’t try to do conversion work in-place there.
3 Likes
Now that the discussion is ongoing, are there plans from FA side trying to fix that? Or is it out of scope and more related to the operating system?
Rolling back to an old kernel should not be the final solution but only a workaround, or am i wrong?
Yes (but it’ll be at least a week before I can start on it)
1 Like
No worries, at least you keep it alive 
Thanks!
I won’t update my feeder Pis until this is resolved.
I was running the upgrade weeks ago where the problem was not known, but it was pending until yesterday where i needed to reboot because the device was not responding (60 days uptime) and then the trouble began by using the new kernel
I followed the guide to downgrade to the last 4.19 kernel, but this is then a mixed system with old kernel and new components. As long as it works, but i like to have this cleaned up
So I guess this is why my dump1090 process keeps crashing?
I also have the USB dongle and my dump1090 process is using about 25% - 30% of my Rasberry’s CPU.
Seems like a fix is a while away. Is that correct?
For me, dump1090 is not crashing. Only more CPU power is needed (Pi3 B+) with passive cooling, temperature around 50°C now (before about 48°C)
Nope. Different symptoms. Start by looking at syslog / journalctl.
After downgrading to the earlier kernel, the high CPU usage problem went away, but the system seemed unstable (kept losing WiFi and occasionally froze). I reinstalled the current version Pi OS Lite from the Raspberry Pi Imager (version 4.19.118) and then PiAware and the rest of the software.
It seems a lot more stable now (no noticeable freezes or loss of WiFi) for the past few days since the re-install.
I’m not sure the issue is a huge problem in the thick of things (thanks @obj for clarifying). The Pi doesn’t have an issue dealing with the extra work. No reason to panic IMHO.
I have no issues with UAT978(using rtl-sdr dongles) only, Mode S Beast only and airspy only RPIs.
I think it could be related to dump1090fa and the rtl-sdr dongles.
Lets keep things in perspective… My system is a Pi 4 with lots of memory. For me, the CPU increase is certainly acceptable for the few weeks necessary for the FlightAware people (@obj et al) to repair and deploy whatever is necessary. That means that I’ll wait for things to be fixed and keep my eyes open.
But I’m sure there are more than a few people out there that have Pi 3 or lower that may be impacted a lot more. Their CPU jump could cause other hardware flakiness in turn. It may be better for these people to downgrade the kernel but I’m not in a position to tell one way or the other.
I’m just glad that I picked up on this issue early on and decided to post it. I’m not patting my own back; just relieved that what I brought up is a real issue.
3 Likes
I am not saying it’s not acceptable. And as stated earlier you can return to a 4.xx kernel which does not produce this CPU load.
I am operating a 3B outdoor and currently i do not want to have that higher CPU load as it also increases the temperature.
But as long as i am able to use a working kernel, i am good until FA has it resolved.
Makes it more bearable, performance is still not great.
(simple memcpy before doing any work in the rtl-sdr callback)
Edit:
Note the above graphic might have other factors …
Removing zerocopy actually didn’t remove much more cycles than than the work-around.
But i was testing with readsb … maybe that just has some inefficiencies that dump1090-fa doesn’t have.
I am seeing the same, no discernible difference, but like you I’m not running a Flightaware dongle.
Slightly off topic.
Could you post your ADS-B signal level graph for your Mode-S Beast please?
Mine seems to have extremely low dynamic range.
I think maybe the front end is damaged.
@Humpfrey,
Here is my graph. Note that the Radarcape and Mode S beast have a different signal scale compared to the RTL-SDR dongles. Note this setup is in my bedroom, so doesn’t have the best range.
I have found that both the Radarcape and Mode S beast (and even the airspy)require pre-amps(and in my case a cavity filter). They seem really deaf without one.
@[jonhawkes2030
Thank you Jon
I too have a cavity filter and LNA, but mounted on a chimney.
It looks like I haven’t broken it then!
Me too… I recently had to reboot my RPi 3 and noted the big jump in resource usage.
At least once a week, I do “sudo apt update; sudo apt upgrade”, so I assume I missed some log message asking me to reboot in order to have a kernel upgrade take effect.
In any case, I’ve read over this thread and do hope that a fix will come relatively soon. My RPi3 is set up to feed to 12 organizations, so it’s moderately loaded. It’s also in an outdoor weatherproof box, with CPU and case fans going full out, but the summer heat does present challenges. I’m just hoping this temporary situation won’t push it over the edge.