Why do my Pis keep dying?

FlightAware ADS-B Flight Tracking
1

Hi all,

Like most, I have a humble little feeder site. I have had it running several years with a Pi 4 without issue, and then one day last month my device went unresponsive. I assumed it was the SD card failing, and purchased another endurance SD card and made a new image of PiAware. But it wasn’t. The Pi itself was dead.

I was annoyed but I had a backup Pi 4, and plus, it gave me an opportunity to clean out my setup. I previously had the antenna terminate into a waterproof box, which connected to the Pro Stick, and the Pi 4 lived in there as well. I drilled a hole for the power cord and sealed well.

I was confident that the box was truly waterproof, but to rule out any issue with water, I purchased another waterproof box of another style and mounted the replacement Pi in there, and modified my original box with an extra hole for a USB extension cable to connect to the Pro Stick. All good there, and the benefit of that was my performance actually increased by about 20%, so I was pretty satisfied.

Until today when my replacement Pi died. Both show the same symptoms, no video, with a solid red and green light. It seems to be a bootloader issue, but I have created new SD cards to no avail.

I suppose it could just be coincidence… but it’s pretty odd that two Pis died a week apart. I am using the official Raspberry Pi USB-C power supply. Does anybody have any thoughts on why my setup would be killing them? Unfortunately, with Pis being impossible to find these days, I’m not exactly sure when I’ll be able to get my feeder back up and running, if at all.

2

I had a similar experience recently with my original Pi 4 failing after nearly three years of operation, and the replacement Pi 4 starting to starting to fail to restart reliably. My setup is also mounted remotely but uses POE. While not being able to offer a diagnosis of your problem, I can suggest something to try to get one or both of your Pi 4’s running - reinstalling the eeprom firmware as described in this link:

https://www.raspberrypi.com/documentation/computers/raspberry-pi.html#raspberry-pi-4-boot-eeprom

This got my original Pi going again for a while but it has since died completely. The second Pi is still running but often refuses to restart after a reboot or power-off, so I have a dedicated eprom reinstaller SD card to get it going again.

Luckily, I managed to get a third Pi 4 for the feeder, so that Pi has been retired to acting as a GPS based time server and isn’t “mission critical”. One thing to note is that where the instructions say “The green activity LED will blink with a steady pattern” - it is actually rapid flashing to signify that it has worked. Sometimes you have to try a few times to get the install to work. (power off, remove and insert the SD card and power on again.)

If the install doesn’t work, you may see other LED patterns - in case you are interested, this link provides a good summary of what they all mean:

https://support.pishop.ca/article/33-raspberry-pi-act-led-error-patterns

3

I had very tricky results with the type of POE injector used on my mast-mounted RPi4. I switched to this one and all has worked flawlessly since:

Someting with the 802.3 af type injector.

4

Thanks. I did try to flash the EEPROM using a new SD card from the Raspberry Pi Imager, but sadly both red and green lights stay on solid. On one of my Pis, the green LED is dimmer than the other, but I’m unsure if that’s in any way significant or just component deviation. From the searching that I have done, it seems like if both lights come on and stay on after a firmware upgrade attempt, that the unit is pretty well broke.

Luckily, I was able to score a Pi Zero W from Pishop, which I used to replace one at home that I was using for a project, so I have my feeder back running again. It’s noticeably slower than the Pi 4 that it replaced, but it works.

1 Like
5

When running PiAware the RPi 4 is definitely over kill. I run RPi’s 1, 2, 3A+, 3B all without problems. The RPi 4 takes more power and runs hotter ( a Pi killer ) costs more and are currently hard to get. Try what you can get. Pi zero W might be fun.

1 Like
6

Well, it’s currently doing the job, albeit at 95% CPU. I just used a Pi 4 because at the time that’s all I had.

7

@vannossc thanks for the suggestion. What are you using at the feeder end?

I have been using a Netgear POE switch (af out) as the injector, connected to the original Raspberry Pi POE hat which is an af type. I chose it because it includes a programmable cooling fan and also allows pass through of all the GPIO pins on the Pi - this allowed me to also mount a GPS hat.

The af POE hat has been in use for over three years but I suspect it may be the cause of my damaged Pi 4s - the symptoms appear after a power interruption (like accidentally disconnecting the ethernet cable delivering POE) where the Pi has not been shutdown first - maybe some kind of transient voltage spike?

When it first happened, the Pi would still boot but had lost all of its USB ports - the lsusb command returned nothing. Researching that problem led to discovering the re-flashing the eeprom solution.

The af POE hat is no longer available having been replaced by an at-type. I got one of those as a backup but wouldn’t recommend it - it draws much more power, and the 4-pin POE header connection is prone to breaking off if you connect and disconnect it regularly. Mine is waiting for me to attempt some micro-soldering to fix that.

I’ve just bought a waveshare POE fan HAT as a spare - the fan runs all of the time (but is very quiet), and it blocks some of the GPIO pins. As I’ve stopped using the GPS hat on the feeder Pi, that’s not a problem. I’ve only tested it and thrown it in the parts draw, so I can’t comment on durability.

I’m sticking with the old af HAT for the moment due to effort required to access the remote feeder. I have the POE injector system on an UPS and have a Pi running NUT connected to the UPS, the feeder Pi also runs NUT, so in theory, I will have a graceful shutdown avoiding the problem if there is a prolonged power outage.

Also, it is a bit of experiment to see if the problem recurs with the new Pi 4. Older Pi 4s used a separate eeprom for the VL805 USB controller and it was this that was corrupted on my older Pi - as you can see here - the Pi can’t read the current firmware version in the eeprom.

Model		: Raspberry Pi 4 Model B Rev 1.1
jrg@raspi4-2:~ $ lsusb
Bus 002 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Bus 001 Device 002: ID 2109:3431 VIA Labs, Inc. Hub
Bus 001 Device 001: ID 1d6b:0002 Linux Foundation 2.0 root hub
jrg@raspi4-2:~ $ rpi-eeprom-update
BOOTLOADER: up to date
   CURRENT: Wed 11 Jan 17:40:52 UTC 2023 (1673458852)
    LATEST: Wed 11 Jan 17:40:52 UTC 2023 (1673458852)
   RELEASE: default (/lib/firmware/raspberrypi/bootloader/default)
            Use raspi-config to change the release.

  VL805_FW: Dedicated VL805 EEPROM
     VL805: version unknown. Try sudo rpi-eeprom-update
   CURRENT: 
    LATEST: 000138c0

Since Hardware Rev 1.4 the Pi 4 has used the main eeprom for the VL805 firmware and my new Pi is a rev 1.5 maybe the corruption issue may have been eliminated.

Apologies for such a long-winded reply to your short comment :slightly_smiling_face:

8

Sorry to hear that this didn’t work.

Before you junk the old Pi s, try repeating the the procedure a few times. On one occasion it took for attempts to get the re-flashing to work and see the green LED flashing rapidly.

Also after inserting the SD caed with the OS on it and powering up it can take quite a while before anything seems to happen - I’ve been confronted with the steady red and green LEDs and concluded that Pi was dead, but have come back a few minutes later to discover that it had booted successfully.

9

Different SDs can also impact. I find flash new UHC/S models and camera endurance etc may be… too smart. or have odd sector sizes etc for larger capacity

Finding el cheapo older smaller ones better recognised or handled by devices that don’t really need fast/large capacity

Could try dialling back to a 2nd hand reliable 8/16Gb card for kicks. Cheaper than replacing units.

10

Sadly in my experience, the el cheapo cards tend to die in spectacular fashion after about two years, unless you take mitigation strategies such as turning off syslog.

Thanks for the tip, but I have already tried a few cards. These things are dead.

11

I’m using this on the Pi end:

1 Like
12

Just an update, I noticed that MLAT was disabled because of timing delays caused by high CPU load. Apparently a Zero W may not be up to the task after all.

1 Like
13

Hi @zoomie38 , i notice you say you run RPi’s 1,2 without problems. IS that running them with PiAware?

I’m keen to set up a feeder site, but have not done so as the docs say Pi 3,4,zero.

I’d be super keen to get started if i can do it with what i already have available!

Thanks

14

A Raspberry Pi 2 will be fine for feeding Flightaware

2 Likes
15

I’m currently running PiAware 8.2 on the following: Pi 1b, 2b, 3a, and 3b without problems.
I’m using RTL-SDR V3, FlightAware orange, and NooElec NESDR Nano 3 sdrs
Very important to have good antenna like the outdoor ones FlightAware sells.
Good luck.

16

That’s why. Pi are not supposed to live outside in extreme heat and cold.
Run a coax cable to inside the house. Keep only a LNA at the antenna.

17

Maybe we can agree to disagree with this one. At the time of my issue, average outdoor temps were about 15C/60F here. Hardly anything a Pi can’t handle and cooler than indoor temp. Looking back at my CPU temp history, with a fan and heatsink, it rarely went over 50C, and in the summer maybe 60C briefly. But never higher.

But, sure, maybe the one that was outdoors for over two years died because of cumulative effects. But what about about the replacement (that also died) that was outside for an only a week, in a different waterproof enclosure? That’s the weird one for me.

Anyway, running coax indoors is not possible in my application, so we will just deal with the consequences. The Le Potato I purchased seems to be doing better, just with higher CPU temps.

18

Thermal cycles of expansion and contraction affect the internals of electronic components.
Also outside they might be exposed to static electricity. The coax that I bring in the house has an arrester on it. Grounded.

So you can run power over there and/or Ethernet but not coax?

19

Yes, I have an exterior power receptacle and use wifi.

Living in an urban environment, not all of us live in single-family homes where modifications of that sort are permitted.

20

This topic was automatically closed 365 days after the last reply. New replies are no longer allowed.