FireZen

Its been a hell of a ride.

So within recent memory I pulled the consumer drives from the Main server as they were causing issues again and kept making the system unstable.

Both the Main and Sub server updated to ESXi 7 during a routine update, which didn’t go well, The main servers age proved to be a detriment as its hardware raid, the raid I just got done putting the new enterprise SSDs in, was no longer supported in 7, nor was the 10/40GB SFP+ Nic cards, I ended up downgrading the Main server back to 6.7 but keeping the Sub on 7, The main server needed the raid but I figured I could always test the Sub and see if those Nics ever came back.

I bought a Raspberry pi 4, after a few different Project, Retro pi, Zabbix Server, Cluster Head, It settled into the role of the NAS for a few Months, I got a four Drive USB 3 “Toaster” that I put four of the consumer SSDs in and backed it with a 5TB USB 3 HDD, This ran great, low power usage and the SSDs never gave me any grief, I had the system rsync nightly the changes from the flash “raid” to the HDD, Until..

Introduction of the i7! Ok so its nothing new, This is my fathers old system and by old I mean 3960X, it had 64GB of RAM but before giving me the system he pulled half for other systems, Not a big deal, First thing I did was yank it out of the case it was in and put it in a 4U Rack case, Little tricky with it having a 240mm water cooler but eventually I got it all to fit, Stole the drives back from the PI, All of them. Built the system using Manjaro Architect, Things were ok for a while, I bought the missing 32 GB for the i7, was running a Minecraft server on it for my kids and family, slowly instability crept in along with corruption, I shut it down for months as I didn’t have time to troubleshoot what was going on, I bought some new drives and decided to tear it all apart and test components, found one of the 8GB RAM sticks was bad, rebuit it with couple 2TB Ent SSDs, a 120GB ENT Boot SSD, the 4X500 Consumer toaster, and the 5TB Rust. it been running Arch since last weekend and after tuning the Network setting its been stable.

The Raspberry pi 4 /4GB became my Zabbix system, I bought one of the newer pi4 8GB and ran it as my desktop for about a month before calling it quits, Thinking of moving my (this) Web site and mail server to it so its easier to shutdown the servers without interrupting services.

Alright its been a little over a month since the crash.

Main server is running perfectly again, this time hardware raid on Enterprise grade SSDs, with the old consumer ones reinstalled after being checked over multiple times, they are also under hardware raid but not willing to host anything outside of projects I am willing to lose.

This web server still runs across six Raspberry Pi 3b(+) and I plan to keep it this way for the unseen future. Its quite capable, running https, and multiple web apps.

So the system is back up, most systems got through unscathed, it seems my FreeNas box which hosts my in home Plex server is worse for the wear, I’m backing it up and getting ready to pull the 3tb HDDs it sits on and put them through a battery of tests to see if they are indeed failing as well or if they were just corrupted from the crash.

After pulling the other SSDs from the system and putting them through their paces it seems they are unscathed, I plan to put them back in, likely in the 3tbs place.

All for now!

Bonus! Picture of the pi rack this runs on:

Don’t mind the stray pi2.

The title says it all, I went on Vacation a few weeks back, everything was running fine, I decided to brings 80% of the systems down as no one would be here to use them, I.E: Me.

I get back and start booting them all up only to start having issues left an right in different Machines, Background most of my ‘Lab’ is virtualized, maintained on one Main Server, Dual CPU 6 core, 128GB ram, over 3TB SSD and over 3TB HDD, sad part is under a money crunch the 3TB of SSD were consumer grade and thats what ended up being my downfall. It seems one began to fail and ESXi just freaked out.. Now the entire system is down, I have made back ups of what I could, ordered new Enterprise Drives, and await there.

This is running on 6 Raspberry PIs, you may remember my old cluster project, 7 raspberry pis running Gentoo for a single goal: self sufficiency that goal was met with one building the kernel, another storing everything outside of root, slowly they all started to die without a will to rebuild it was reborn into this. Gluster backend, MariaDB galera split over the six of them.

Edit!: And now with https!