13

A data center employee told me a story about 25 years ago, explaining why that data center took some time to assemble a RAID array. Recently, I wasn't able to verify the facts by googling, and I may misremember some details, but maybe someone here has relevant information.

The story went like this: the data center had assembled a RAID array from brand new disks, which were among the first to support the then-new SMART features. After some time, all those disks failed at the same time, taking the RAID with them. After a lot of debugging and discussion with the manufacturer, it turned out that the firmware had a bug in one of the SMART counters (probably power on hours?) which led to the disk becoming unresponsive when the counter overflowed. As all the disks were powered up/down simultaneously, the bug manifested in all of them at the same time.

This led to the data center taking time to assemble new RAIDs: power on one disk, wait a day, power on a second, wait a few hours, then add the others, always with a few hours between disks.

I wasn't able to find anything on the internet about the exact time, brand of hard drives, and detailed information about this bug - does anyone remember this a bit better?

Improve this question
2
  • 11
    More recently, Crucial’s M4 SSDs had a firmware bug which caused them to crash after ~5000 hours of operation, so this sort of problem is certainly possible. (The M4 bug was fixed by a firmware update.)
    – Stephen Kitt
    Commented Jul 31, 2022 at 21:05
  • Some Seagate 1TB drives had faulty firmware and would eventually lock up and fail to be detected by BIOS effectively bricking them. This was an issue late 2008 early 2009. How to fix article (if firmware wasn't updated before the drive bricked itself): sites.google.com/site/seagatefix
    – Brian
    Commented Mar 7, 2023 at 23:23

1 Answer 1

Reset to default
8

25 year old recalls may be a bit hard to search but the problem is as true today:

SanDisk SSDs had a 40,000 hour failure

HPE had some that died on 32,768 hours

The more paranoid data centre people used to build arrays with a mix of vendors and models because even without firmware bugs a box of sequential serial numbered drivers were probably going to have the same environment, contaminants and faults as one another.

Big storage management today is a whole different world and scale so the rules are rather different anyway.

Improve this answer

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .