FYI New Pi Kernel causes dump1090-fa to run much harder

[tgmct]

Yesterday a new kernel was deployed by the RPi people. This morning when the machine rebooted, dump1090-fa went from a typical 8% utilization to 41% (according to atop). The ads-b CPU Utilization output from graphs1090 shows the same thing. Reviewing the atop output carefully shows that it is only dump1090-fa that has jumped, all other processes are running at normal cpu levels, including dump978-fa (was 8% and still is). This is a Pi 4 Buster system with 4 GB of memory. A second reboot did not help. For the moment, CPU heating has gone up 5 degrees (C) and the machine is not getting bogged down with anything. But who knows how this may impact Pi 3 machines.

This is the install history of upgraded modules:

Start-Date: 2020-07-20 04:46:20
Commandline: apt-get -y install libopenmpt-modplug1 libopenmpt0 redis redis-server redis-tools
Upgrade: libopenmpt0:armhf (0.4.3-1, 0.4.3-1+deb10u1), redis:armhf (5:5.0.3-4+deb10u1, 5:5.0.3-4+deb10u2), libopenmpt-modplug1:armhf (0.4.3-1, 0.4.3-1+deb10u1), redis-tools:armhf (5:5.0.3-4+deb10u1, 5:5.0.3-4+deb10u2), redis-server:armhf (5:5.0.3-4+deb10u1, 5:5.0.3-4+deb10u2)
End-Date: 2020-07-20 04:46:47

Start-Date: 2020-07-20 09:55:13
Commandline: apt-get -y install libraspberrypi-bin libraspberrypi-dev libraspberrypi-doc libraspberrypi0 raspberrypi-bootloader raspberrypi-kernel rpi-eeprom
Upgrade: libraspberrypi-bin:armhf (1.20200601-1, 1.20200717-1), libraspberrypi-dev:armhf (1.20200601-1, 1.20200717-1), libraspberrypi-doc:armhf (1.20200601-1, 1.20200717-1), rpi-eeprom:armhf (7.7-1, 7.8-1), raspberrypi-kernel:armhf (1.20200601-1, 1.20200717-1), raspberrypi-bootloader:armhf (1.20200601-1, 1.20200717-1), libraspberrypi0:armhf (1.20200601-1, 1.20200717-1)
End-Date: 2020-07-20 10:05:42

Rebooted 2020-07-21 01:03

[Humpfrey]

Updated 2 Pi4s 2Gb running Buster yesterday. CPU usage for ADS-B CPU usage did not jump.
I’m not running FA dongles on the Pi4s, which could make a difference.

[tgmct]

I’m not using FA dongles either… Have NooElec NESDR Smart v4 SDR’s instead on both 1090-es and UAT-978 dongles.

[tgmct]

Also, I’m running the 64bit version of the kernel due to some other software running on this machine.

[foxhunter]

I had the kernel installed (not the rest of the software) as 64 bit version for testing purposes and i did not notice any significant change in CPU usage.

[Humpfrey]

Mine are AirSpy running 64bit Buster.
I only feed FlightAware and ADS-B Exchange.

[tgmct]

I’ve uploaded the ADS-B CPU Utilization from graphs1090 and the current atop output for you to look at.

[jefftomlinson]

I updated a Rpi 3B yesterday and have seen similar results. Total ADS-B CPU went from ~30% to 70% and Pi CPU from ~15% to 30%. This is using a FA dongle.

[obj]

Given that the increase is all in the USB thread, that looks like whatever the change was, it affected librtlsdr / libusb. The USB thread does little beyond:

• telling librtlsdr to provide samples (which in turn runs a loop calling into libusb to fetch data from the dongle);
• when librtlsdr provides some data, doing some basic data conversion on the provided samples;
• handling the converted data to another thread

[tgmct]

So why does this not impact dump978 in the same way?

[Nitr0]

Can not confirm for 64bit, but no perceptible change on 32bit dist-upgrade.
Edit: I’m using an Airspy, so I’m in Net mode for Dump1090, but no change that I can see with Dump978 or on USB utilization with airspy_adsb - so more than likely something changed enough in the usb stack to thwart librtlsdr. I’m also using package install and librtlsdr built from source, so some differences may lie there as well.

[SFRobert]

On a 32-bit Buster / Pi 3b+, I did the latest updates just now and checked top before and after. Dump1090-fa went from around 11% CPU to 40% now. I’ve also got an sh zombie that increments its PID.

It seems to be tracking aircraft fine but something is up. I should have duplicated the SD card before I did the upgrade. I have a copy from a month ago. Might revert tomorrow.

[davidinjp]

You can update to an older kernel version. I did it last year on an RPI that got weird. Here’s one of many suggestions. Raspberry Pi Firmware and kernel update with rpi-update - KaliTut(downgrade%20%2F%20rollback,want%20to%20go%20back%20to.

Also, be sure to use apt dist-upgrade, never use rpi-update on any Pi you care about stability. If rpi-update is what you did and ended up with an experimental kernel, this is one of many suggestions raspbian - How to revert from rpi-update to stable build - Raspberry Pi Stack Exchange

[tgmct]

More info… Swapped around dongles to different USB ports (2.0 vs 3.0, etc). No improvement or changes with either type of dump utilization. We still need an answer why dump1090 has the issue BUT dump978 does NOT. I suspect there must be something specific. It seems that which brand of dongle does not make any difference from the comments so far. It seems that someone who supports these processes should get involved with some testing, etc. I looked into some apt history and libusb hasn’t been upgraded since before April 2020 and librtlsdr was upgraded on April 10th.

Before I attempt to downgrade the kernel, I definitely want to perform a complete backup of storage. I may even recover a previous mirror backup (about a month old) and then perform some controlled upgrades (painful). From what I’ve read, the RPi people have a 64bit full install in beta. I may wait for it to go into full production and do a staged rebuild of all my twiddles. Thankfully, I just received an order of microSD chips.

[obj]

@tgmct

We still need an answer why dump1090 has the issue BUT dump978 does NOT.

I’d start by comparing strace output (be sure to use -f since they’re both multithreaded). You’re probably looking for something spinning on syscalls. Ideally you’d compare strace output between the two kernel versions.

dump978-fa talks to the dongle via SoapySDR (which then uses librtlsdr), while dump1090-fa uses librtlsdr directly, so the setup and call patterns will be a little different.

[jefftomlinson]

I thought I would try the upgrade on my test Rpi 4 running dump1090-fa. I use this PI for experimenting with different software, rather than using one of my active feeders, so it doesn’t matter if I break it. It uses a cheap no-name SDR from eBay, which is a lot less effective than a FA dongle.
The upgrade (straight-forward sudo apt update sudo apt dist-upgrade) had the same effect - ADS-B CPU Usage more than doubled (all due to USB) and RPi CPU also doubled. This highlighted a problem with my under-powered power supply, so the Pi was no longer stable.
Doubling the CPU usage is an annoyance, it doesn’t seem to affect data collection, but the slightly higher power requirements could cause some odd failures.

[SFRobert]

I tried @davidinjp’s suggestion to try a rollback to 5.4.50 from 5.4.51 since that page recommended even kernel versions. The command I used was:

sudo rpi-update bafd743eeb3e8a2a863936594cd7201a0af136fa

That rolled me back to Linux FAPi 5.4.50-v7+ #1324 SMP Wed Jul 1 17:00:22 BST 2020 armv7l GNU/Linux but no joy. Still have the 40% CPU on dump1090-fa. Going to roll back to my backup later.

Thanks anyway, @davidinjp. That’s a good trick to know! Maybe a different version would do it but I’m just going to flash back to a saved SD image and hold there until this gets sorted.

[pretoescuro]

Just checked mine and have the same behavior, it was updated today.
Now I have kernel 5.4.51-v7l+

Previous Kernel
Jul 22 08:32:28 piaware kernel: [ 0.000000] Linux version 4.19.118-v7l+

New Kernel
Jul 22 10:17:45 piaware kernel: [ 0.000000] Linux version 5.4.51-v7l+

[SFRobert]

I reverted to my last saved image and dump1090-fa is now back to normal at 11-15%.

I shall make a suitable offering to the gods of backups.

[davidinjp]

@jefftomlinson

The upgrade (straight-forward sudo apt update sudo apt dist-upgrade ) had the same effect - ADS-B CPU Usage more than doubled (all due to USB) and RPi CPU also doubled.

That’s a bad sign if a proper upgrade causes this. A whole lot of people are probably going to have the same problem in the coming days.

@SFRobert

I tried @davidinjp’s suggestion to try a rollback to 5.4.50 from 5.4.51 since that page recommended even kernel versions. The command I used was:

sudo rpi-update bafd743eeb3e8a2a863936594cd7201a0af136fa

That rolled me back to Linux FAPi 5.4.50-v7+ #1324 SMP Wed Jul 1 17:00:22 BST 2020 armv7l GNU/Linux but no joy. Still have the 40% CPU on dump1090-fa.

Either that didn’t rollback far enough, or more likely the problem isn’t the kernel but something else eeprom or other software/firmware update.

I wonder if this was a regression in Pi USB or power handling or intentional change and now we have to wait for a librtlsdr update. I just browsed rtl-sdr and even dump1090 repositories and searched for a history related to cpu utilization. I’m at a loss and not helping!