Alright its been a little over a month since the crash.
Main server is running perfectly again, this time hardware raid on Enterprise grade SSDs, with the old consumer ones reinstalled after being checked over multiple times, they are also under hardware raid but not willing to host anything outside of projects I am willing to lose.
This web server still runs across six Raspberry Pi 3b(+) and I plan to keep it this way for the unseen future. Its quite capable, running https, and multiple web apps.
So the system is back up, most systems got through unscathed, it seems my FreeNas box which hosts my in home Plex server is worse for the wear, I’m backing it up and getting ready to pull the 3tb HDDs it sits on and put them through a battery of tests to see if they are indeed failing as well or if they were just corrupted from the crash.
After pulling the other SSDs from the system and putting them through their paces it seems they are unscathed, I plan to put them back in, likely in the 3tbs place.
The title says it all, I went on Vacation a few weeks back, everything was running fine, I decided to brings 80% of the systems down as no one would be here to use them, I.E: Me.
I get back and start booting them all up only to start having issues left an right in different Machines, Background most of my ‘Lab’ is virtualized, maintained on one Main Server, Dual CPU 6 core, 128GB ram, over 3TB SSD and over 3TB HDD, sad part is under a money crunch the 3TB of SSD were consumer grade and thats what ended up being my downfall. It seems one began to fail and ESXi just freaked out.. Now the entire system is down, I have made back ups of what I could, ordered new Enterprise Drives, and await there.
This is running on 6 Raspberry PIs, you may remember my old cluster project, 7 raspberry pis running Gentoo for a single goal: self sufficiency that goal was met with one building the kernel, another storing everything outside of root, slowly they all started to die without a will to rebuild it was reborn into this. Gluster backend, MariaDB galera split over the six of them.