I have been flight feeding for sometime and am running in beast mode and feeding multiple sites for the most part things are stable unless flightradar botches something etc, but I have had some issues with USB stability. I run both 1090 and 978, both flightaware dongles on a rpi 3b but have gone to a powered usb hub and things appear to be more stable but from time to time things will go south. Yesterday I was doing upgrades on my router and it was offline for a length of time, and it looks like that messed things up, got a email about stats not being updated. I ssh’d to the rpi and saw that things were in fact not working properly and tried to restart 1090 and 978 and that didn’t work and then reset the usb hub and still had issues. I rebooted and all came back up but I really hate rebooting, this is not windooze, lol.
Here is my question, I am a linux guy so I am really comfortable on the cli but I am not sure when things go south what services need to be reset and then restarted to get things working again, hoping @abcd567 or others could point me in the right direction and I will make a script to stop and start all the services that I need to make things happy. If anyone has any suggestions on what I can do to make things even more stable please feel free, it is on a UPS and I only reboot when I have too…
Here is one of the error messages that I was seeing in journalctl when I reset the USB and restarted the main services:
dvb_usb_rtl28xxu 1-1.2.1:1.0: chip type detection failed -71
Oct 24 08:43:37 adsbpi kernel: dvb_usb_rtl28xxu: probe of 1-1.2.1:1.0 failed with error -71
bishop@adsbpi:~ $ lsb_release -a
No LSB modules are available.
Distributor ID: Raspbian
Description: Raspbian GNU/Linux 10 (buster)
Release: 10
Codename: buster
us 001 Device 007: ID 0bda:2832 Realtek Semiconductor Corp. RTL2832U DVB-T
Bus 001 Device 006: ID 0bda:2832 Realtek Semiconductor Corp. RTL2832U DVB-T
Bus 001 Device 004: ID 2109:2813 VIA Labs, Inc.
Bus 001 Device 005: ID 0424:7800 Standard Microsystems Corp.
Bus 001 Device 003: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 002: ID 0424:2514 Standard Microsystems Corp. USB 2.0 Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
I can provide any information that may be needed, just let me know, thanks.
Unfortunately I do NOT have enough depth of Linux knowledge to give you proper guidance. Hopefully more knowledged person like @obj or @wiedehopf or @caius will drop in and give you right advise. There are many other Linux experienced members here whome I have not listed, my appologies. I hope some of them will give yiu good advice.
I can only tell you the crude way I adopt. First I reboot. If that does not solve the issue, I reimage the microSD card with latest image, and reinstall latest versions of all software.
Thanks, I’m just looking for what I can reset/restart when things stop working. Looks like the USB sticks did not get reset even when I pulled the power etc. I need to figure out what all I am feeding, I did this so long ago that probably a clean start would not be a bad idea but no idea how to figure out all the feeders that I am feeding, lol.
dump1090 or dump978 are the services which talk to the dongles, so they are the ones which might need restarting. I suspect problem you have might be hardware related though - the OS is losing connection to the dongle for some reason. Are there any other messages from dump1090 in the log around the same time that might indicate a problem?
It’s possible that there is a component failure on the dongle or a dodgy USB connector. As you are using 2 dongles on the same pi, you also need to make sure your power supply is of decent quality - if you get voltage drop that could put the dongle into an unrecoverable state that needs power cycling to clear. A powered USB hub won’t necessarily solve this if it also has a crappy power supply.
Ideally test them with a dummy load to see how the voltage sags - some supposedly 2A or 3A chargers and PSUs are poorly regulated and drop down to 4.5V or less when approaching their rated output.
The various feeding clients all connect to dump1090 via network sockets, so they shouldn’t have any bearing on it - if one of them fails it shouldn’t take out the rest. If dump1090 (or 978) is unable to talk to the hardware properly, then none of them will have any data to feed.
Yeah I restarted both 1090 and 978 but that did not fix the issue, reset the usb dongles and that appeared to work but I did see the error message. I have the dongles on a powered USB hub and last time I looked I did not see any voltage issues.
sudo dmesg --ctime | grep voltage
[sudo] password for bishop:
bishop@adsbpi:~ $
I do think its a hardware related somehow to one of the dongles, but again it can run for months on end, not sure why having the router offline yesterday that would have caused any issues but it appeared to happen at the same time that I was doing the upgrade. Just would like to figure out steps to reset everything vs rebooting, hate rebooting stuff. I know I restarted 978 and 1090 and still had issues, reset the usb dongles and still had issues, reboot and everything happy. I need to set up a notification that will email me sooner vs waiting for one of the feeders to complain, and try to poke around a little more next time.
Looks like that works I need to read some more and see how to use it, this is a powered hub so not sure how that would work.
sudo uhubctl
Current status for hub 1-1.2 [2109:2813 VIA Labs, Inc. USB2.0 Hub, USB 2.10, 4 ports, ppps]
Port 1: 0503 power highspeed enable connect [0bda:2832 Realtek RTL2832U 00001090]
Port 2: 0100 power
Port 3: 0503 power highspeed enable connect [0bda:2832 Realtek RTL2832U 00000978]
Port 4: 0100 power
Current status for hub 1-1.1 [0424:2514, USB 2.00, 3 ports, ppps]
Port 1: 0503 power highspeed enable connect [0424:7800]
Port 2: 0100 power
Port 3: 0100 power
Current status for hub 1-1 [0424:2514, USB 2.00, 4 ports, ppps]
Port 1: 0503 power highspeed enable connect [0424:2514, USB 2.00, 3 ports, ppps]
Port 2: 0503 power highspeed enable connect [2109:2813 VIA Labs, Inc. USB2.0 Hub, USB 2.10, 4 ports, ppps]
Port 3: 0100 power
Port 4: 0100 power
Current status for hub 1 [1d6b:0002 Linux 5.10.103-v7+ dwc_otg_hcd DWC OTG Controller 3f980000.usb, USB 2.00, 1 ports, ppps]
Port 1: 0503 power highspeed enable connect [0424:2514, USB 2.00, 4 ports, ppps]
Long term unix user here, Sun, Silicon Graphics, and Linux back to the early 90’s. Rebooting the old systems was always iffy, as the hard disks would sometimes fail to spin up or bad sectors on the drive would fail to load properly. I understand your reluctance on rebooting. I hated rebooting servers especially, and never on a Friday or weekend. Those lessons are hard learned.
That said, the raspberry pi’s are much more reliable, assuming your SD card has not failed. I use the same philosophy as @abcd567, reboot first, because the number of things we can think of is by definition limited, and the number of things possibly messed up is unlimited.
Restarting with a different SD card and new install is a very good diagnostic. This removes virtually all likely problems, and you can reinstall any other programs as necessary. I had to start a log of what I install and when so I can recover when necessary or I mess something up. Hope everything works out soon for you.
Yeah I wished I would take notes and log what I do, but I never do, lol. Yeah I run quite a few rpi’s and the SD cards are all high endurance types thats all good, I need to spend some time and try to figure out what all feeders I am using etc…I do think its a USB hardware issue when it does flake out, just not sure how to address it, for the most part it just runs, the powered USB hub helps for sure since I am running 2 dongles.
Funny, I’ve been around also, my son went to college and took my Silicon Graphics mouse pad with him, lol.
If I read that right, that’s your external hub and it has per-port-power-switching (“ppps”) support, so you should be able to cycle the power on the external hub ports individually.
Thanks fot that. Used a shorter power lead and “Undervoltage detected!” went away
That was on another test system I use for IoT type detection and it also confirmed that my flight detection system is not undervoltage as it has a proper Raspberry Pi power supply vs the old Ipad power block on the other one.
