What kind of luck are you boys having lately keeping your SD cards healthy? Between my two sites I’ve reformated/replaced the SD cards about a dozen times in the last couple of weeks. Maybe I’m just overdue for some bad luck. I’ve been buying the well regarded Sandisk and Samsung cards, among others. Same result typically on each. Not surprisingly, the problem was more pronounced after I switched to Mutability, performance graphs, and all that (which is fantastic) given the increased I/O. Recent power outages in this area because of high winds haven’t helped either. The latter can be resolved with a UPS obviously, but isn’t practical on my mast mounted rooftop installs.
A quick session with Dr. Google indicated that I might see some benefit with making the OS on the SD card read-only and pointing the rest of the disk I/O to a more stable volume like SSD or similar. Anyone messed with that type of approach?
In the meantime, I have both my sites dismantled and shut down since I’m tired of climbing on the roof. I was over my data cap anyway. Doing nearly 1GB per day on 4G cellular. That gets expensive. LOL!
I would take a good look at your power supplies if you’re getting that sort of failure rate. I’ve had one sdcard failure (out of a half dozen systems) in the last 18 months.
I’ve had success (in a slightly different context) running the root fs as an overlayfs that combines a base readonly filesystem on the sdcard with an in-memory readwrite tmpfs filesystem to store changes; there is no simple out-of-the-box way to set this up though, I ended up writing my own initramfs-tools hooks to handle it.
Someone on here has done this in the last 9 months or so. Since I wasn’t successful in locating the thread, suggest your own forum search or maybe someone else will remember the thread.
I have had no SD problems in over a year of operation. I primarily use uhs-1 16GB pro cards from Samsung. The higher end cards supposedly do better wear leveling and also allow me to complete offline backups very quickly on my windoze box with a uhs-1 reader. One thing I notice is that boxes I started with a FlightAware image have /tmp mounted on RAM. This can greatly reduce the amount of SD IO. My main feeder is running with mutabillity, collectd, lighttpd, etc… and has been on the same SD card with my current feeding streak of 415 days. Current uptime 54 days with very clean and stable power on a UPS. (no reboots etc.)
I also use these cards and have a similar experience. I am new to ADS-B but have been a RPi fan for years.
As I bought lots of 4GB large cards for the original RPi, I only expand the file system to 4GB (so can move images between systems) so I have 12G of spare sector for repairs - don’t know if this is relevant
That’s a good call on the power supply, obj. I had not previously considered that, but now that the whole enclosure is down here on the ground I fired it up last night and measured some pretty clean power. I’m getting 5.04VDC with less than a millivolt of AC ripple under load (albeit a small load from just the Pi and dongle). I’m using a DC/DC buck converter to create the 5VDC from a larger 48VDC supply that powers the rest of the enclosure (cooler, cell modem, router, etc). All in all, the enclosure has been a great performer - along with the Pi up until recently. As mentioned in my previous post, numerous short duration power outages last week are likely my biggest problem. No UPS on this rig.
I’m optimistic that there’s hope in these lightweight SD cards given LitterBug’s streak. Not bad. I may have a dud or two, but in the large stack I have I don’t believe they are all bad. I’m wondering if I may have a suspect card socket to complicate matters? I’ve been a little bit hard on it the last few trips on the roof. I’m not exactly friendly with hardware when I’m pissed. I should be able to eliminate that question with a fresh Pi.
All my gear is in a cool basement with fairly constant temperature and humidity. If your gear is all outdoors, it is probably seeing large swings in temperature and possibly humidity as well. That can take a toll on gear and connectors. SD cards are fairly resistant to that. I accidentally left an SD card in my pocket once and it went through the washer and dryer. Card is still working fine several years later. LOL
I’m not too worried about the environmental in this case. The gear is all roof mounted, but it lives in a sealed steel IP66 enclosure with around 200 btu of closed loop cooling. The temp has been hanging steady at around 80F up there. I have a desiccant bag I picked up at Graybar in there that has yet to change color after 5 months or so, so its very dry. Regulated DC is clean. I think the biggest issue here is equal parts of bad luck, bad SD cards, and bad card management on my part. Maybe a wonky card socket on my big site. Obviously the rapid fire AC mains outages last week did no favors and blew up both sites. I’ll probably get a new Pi on the way soon - will also give me an excuse to try a pro stick when they are back in stock. Just need a neighbor kid to carry all this junque back up on the roof.
I wouldn’t mind scaring up some efforts to take the I/O load off the SD Card as has been mentioned here, but will need some college credits in Linuxology to get anywhere with that…I’ve been in the boutique world of windows too long. Maybe Ubuntu on an old laptop is the way to go. My buddy k5ted has has some luck with the Intel stick, but they are hard to find at 35 clams a pop.
After trying a number of things I ended up moving the file system root to an old 240GB laptop hard drive connected via USB. The SD card just handles /boot now so receives virtually no I/O
Maybe its over kill for a lot of systems but so far its rock solid and opens up other interesting possibilities like samba server etc.
Big Thanks to @Ieand and @JProchazka for the procedure on setting up an external volume for disk I/O. Using JProchazka’s procedure, I’ve put a couple of Pi’s together on the shop bench with a variety of different USB external drives. In all cases, the lash up has been successful. To give it a real world test, I’ve thrown all sorts of abuse at the system to see if I can kill it. Pulling power, pulling the disk, even browning it out a few times with the bench supply resulted in no problems. The Pi booted back up with no complaints in all cases. I/O on the SD Card appears to be essentially zero (except at boot) while the external volume flickers away. So my confidence is high I can move forward with little dependence on these sickly SD Cards. I’ll use a 32GB SATA SSD on one site, and an older SATA rotational drive on my other site. The latter draws about 850ma, so I’ll power it externally. Both drives tested out nicely on the bench.
Apparantly there are also industrial microSD cards from suppliers like Kingston, Micron, SanDisk and others. There are not especially fast compared to other more expensive cards, but are supposed to be more durable with regards to temperatures and climate. Not sure if they are worth their money, but maybe they make more sense than buying more expensive normal cards whose main benefit is increased speed, which a feeder does not really need.
I have good experience with several Adata and Kingston 8Gb class 10 microSD cards.
All these are in constant use for last 2 to 3 years. Each has undergone a large number of formatting and re-writing operations. None has yet failed or gave any trouble. All are used inside my apartment in normal enviroment comfortable for humans.
I am in favor of USB key.
I have tried SD cards made by Kingston, Intenso, Sandisk, Transcend and had no issues.
However due to their sensitivity, I used them to boot the RPi and
redirect to an “el-cheapo” USB key which is faster, has more space, and is more durable.
The same image burned to both SD card and the USB key (Adata, Transcend etc).
With regard to industrial micro SD cards depends how good you want to go you can spend up to and over a £100 for a 8gig card. It will be bullet proof, but not worth it. They tend to do a lot more internally than a normal SD card things like proper wear leveling (like an ssd), smart functions etc…
Most SD cards touted as industrial are normally just the better ones from a yield and deal with a wider temperature range (most industrial stuff is rated from -40 to 85c), and the industrial could also just mean they guarantee to make it four the next four years. This is especially true if they are not much more expensive than normal ones.
Looking at two again, it seems the higher temperature range (and more robust connectors) are the main differences of the “consumer” industrial cards.
Not sure USB sticks are a better choices compared cards. I remember building a NAS, and both cards and sticks were not recommended as the system device as they are not made fo constant read/write operations.
Still, haven’t had any problems with microSD cards yet, apart from breaking one in half
SSD have a lot of variation in how the data cells are made. Some are made to last 100,000 write cycles and other are measuring the millions. This is mostly due to how close they are stacking memory cells. MicroSD cards are usually rated in the 100k write cycles range since they use the cheaper memory cells.
microSD cards have a small processor next to the memory cells. The processor is the thing that gets hot on the microSD card since it handles all the data transfer, wear leveling, and other things. There is a big variation in processor quality. https://www.bunniestudios.com/blog/?page_id=1022
Most microSD cards have a bunch of extra cells that are swapped in as bad cells are found. I don’t know if there is a way to know how many bad cells are on a microSD card since there are no public programs to talk to the processor directly.
The most common failure is NOT cells going bad but power problems causing corruption of the data. During a write or read cycle if the power is cut it can cause bad data to be put into a cell. This will corrupt the data but the cell itself is still good. There is no easy way to detect corrupted data. We have seen many many more corrupted data problems with Piaware and FlightFeeders.
BACK UP YOUR DATA!!! (clone the SD card). This makes restoring your site very easy when it does happen.
If you don’t have any special data on the SD card then just restore from the current Piaware image.
