FireZen

I realized after the last Tech update I haven’t “had time” between work and play to do as much home lab work as I used to, this is partially due to striving for a more remote capable* work life and partially due to not really having a project currently to hyperfixate on.

Also the Gym, going three to four times a week in the evenings takes even more time.

*I’m already WFH, I’m tired of actually working from my home, my office is in my basement, over the winter I built essentially a work outdoors go bag with everything I need to hop in the truck and work from anywhere.

There’s another post I’m seeing I forgot to finish talking about my ereader(s)/tablet etc.., so I will finish that one for the more tech side of:

READING

I have gotten back into reading since the Holiday season, started with Manga, of which I have read 25,689 pages to date. Then my s/o got me into the Empyrean series (Fourth Wing, Iron Flame and Onyx Storm) by Rebecca Yarros.

After going through that series I went back to my Manga that it interrupted and my brain said, “nope you dont remember whats going on” and not wanting to start them over I moved on to re-reading Cassandra Clare’s Shadow Hunters as its been years and I know I’m behind.

GAMING

Not sure if I ever updated on the games I play in posts of the past, I enjoy RPG’s like Final Fantasy, Dragon Quest and so on, to the point of recent memory I have beaten:
Final Fantasy 4 Pixel Remaster
Final Fantasy 5 Pixel Remaster
Final Fantasy 6 Pixel Remaster
Final Fantasy 8 Remaster(?)
Final Fantasy 7 Remake and Rebirth
FINAL FANTASY TACTICS – The Ivalice Chronicles
*(have beaten 9-13 previously, played but never finished 15 and 16)
Dragon Quest 11
Dragon Quest 3 HD-2D
Dragon Quest 1&2 HD-2D
Visions of Mana (with s/o)
Working on: Dragon Quest VII Reimagined

Ontop of all the Atelier games from Gust from Iris up with my s/o.

I also enjoyed Armored Core 6 but have lost alot of my reflexes since started with AC2 on the PS2 and it was a struggle.

More recently I found out how to use https://retroachievements.org/ with my older game Library, this makes for quick ADD breaks from longer games/books I am engaged with for example:

I have owned Final Fantasy Legends since I was in grade school, couldn’t ever remember beating it this made it the perfect test game.

As you can tell I’m not going for 100%, this would require for example beating FFL with a single character (Human I think) or Pokemon Yellow with just Pikachu, doable but not enjoyable, I am trying to get the easier/enjoyable optional trophies though.

I beat each of these over the past two weekends, one each starting Friday afternoon/night.

Other than that, I may start posting more posts like these, may post random quotes I find interesting/funny/inspiring.

Pretty much going from what I once envisioned as a tech blog, to a random place I can loose the fire of my mind.

Take it easy and Thanks for reading.

So around October of last year I got back into reading Comics, started with a trip down memory lane with MegaTokyo and, while I’m still working on it, Hardcore Leveling Warrior. I had looked up other web comics as I was attached to reading quite a few in my teens, but alot were either gone or I had forgotten them.

Then came the holidays and being asked what I wanted, I decided on an e-ink e-reader, but never having had one before I did some research and found the cheapest I (assumed) I would be able to do the most with.

Enter the Musnap Neo C a 6″ little color e-reader that runs android under it all, I also got the official case because it has the physical page buttons the actual device lacks (that I have already returned once for one of the buttons going bad) I had already done a bit of research before I was physically handed it to know how to sideload apps and such, something this little device MUCH needed as its stock launcher and “library management” left much to be desired.

*Also, I run WPA2/3-Enterprise at home, however this version of android has been modified for whatever reason I cannot install certificate into it, it say the password is wrong even if I strip the cert of a password.

My setup came down to:
No App store: install F-Droid
Replace the Launcher: from F-Droid install inkOS
Comics: from F-Droid install Kahon
Browser: from F-Droid install EinkBro (mainly for, and to keep updated, the next entry)
Books: install FBReader Premium (I do own a license on my phone/google play and appreciate their model of making the Premium APK available.)

With all that done, I did add some extras I will go over later when its of interest but for now:

Setup a Komga/Komf duology for self hosting, metadata and progress tracking, tie that to Kahon.
Setup Calibre, tie that to FBreader.
repeat on phone*

Ok, now we have a central repository for reading material that the Musnap can pull from..while its on the local.. network..

VPN? VPN. Same one whats on the phone, no need to expose those new services to the internet, I like it better that way.

Alright, this setup I have been running for the last few months and it works well, only note I have is the 6″ turned out to be smaller than I realized, most Manga and Manhwa (HCLW*) read fine, some Manga print is really small and I have to zoom it to read it, if that becomes a series norm I just moved on from the series until..

Enter Samsung Tab A 2016! Wifes old Tablet, found it last week in a drawer while looking for something else and while the case was worse for wear the tablet was pristine if outdated, quick google shockingly (I hadn’t had good luck with custom firmwares and Samsung devices in the past) showed there was an unofficial build of LineageOS 21 over on XDA’s forums. Took a few tries getting TWRP flashed (you have to flash it then immediately go into it or else it will be over written by stock again next boot) but after that installing felt like doing all the custom roms I did ages ago, install wipe everything, boot.

Installed pretty much everything I did on the Musnap including inkOS, because I like its simplicity for a primarily reading device, with a few additions/changes:

Did not install any Google apps/Gapp’s package, on the forum it said to use mini/micro for performance reasons, I decided like the Musnap to just forgo Google all together.

Yokai * instead of Kahon, Primarily because when in landscape it offers the ability to show two pages at a time.
Wifi-Automatic from F-Droid, the Musnap has a nice feature of when the device is “off”/sleeping it automatically turns off wifi/bt to save power, This handles the wifi, bt is just always off.
RetroArch Emulation

This post was started on 2/23/26 and never finished, I am going to push it out with only a few more lines as it pertains to another post I’m working on.

FB reader also has a linux appimage: https://fbreader.org/linux/packages

Waydroid on Linux will run Yokai.

and a really geeky outcome of both of those: with a cheap HDMI2USB-C you can pipe the output of the laptop to a VR headset like the Meta Quest, meaning reading in VR :).

All on this topic,
Thanks for reading

I haven’t updated my blog..

Lots has changed in the lab:

Now running a cluster of 4-5 Proxmox Servers to play with HA and CEPH, both have been truly eye opening in the way of maintaining 0.99x up time. With this combination I can update them, set one to reboot, watch as it shunts all its HA VMs/CTs to other systems, reboots and takes them back. no more services down, no more wife/kids screaming or waiting until the dead of night for updates.

A note before I continue, these “Servers” the only real server grade is the one I built in Lab runneth over, its been upgraded over the years, Dual  Intel(R) Xeon(R) CPU E5-2667 v2 and now 256GB of RAM. The rest are used/old hand me downs and a repurposed gaming PC I built. You don’t need the best to learn in a home lab.

The cluster also came with a need for 10GB networking, I beta tested the cluster before making the switch to live and the biggest problem I ran into (also the internet long had told me it would be an issue but again beta) was running CEPH over 1GB. Before bringing it live I bought them all 10GB SFP+ nics and a new eight port SFP+ switch.

New switch brought about a completely network design overhaul, Originally it was:

WAN>(1GB E)Router>(10GB SFP+)24/2P Switch>Everything
And I could have added the new 8 Port SFP+ after the 24/2P as it had a spare SFP+ port and daisy chained it.

Instead I split them:
WAN>(1GB E)Router>(10GB SFP+,VLANs 1 and 2)24/2P Switch

WAN>(1GB E)Router>(10GB SFP+,VLANs 3 and 99)8P SFP+ (Also self containted VLANs 10 and 11*)

Then both switches are linked and pass their respected VLAN’s (*other than 10 and 11) between them.

*The new switch brought about new fun, the beta ran CEPH and Proxmox’s Cluster traffic over my default “Lan” network, Yeah not what I wanted and I also didn’t want the Firewall/IDS/IPS anywhere near that traffic to cause any latency after the 1GB test, so they both got their own VLANs that only traverse the SFP+ switch.

Tied AD to Authentik, originally for giggles after a random youtube video, now everything that can uses it and while having a single SSO solution by its self was awesome, the real magic came when I figure out MFA then shortly after how to use my Yubikey. Now logging into anything is simply: Punch Yubikey pin, touch Yubikey, Done.

Also a check in from one of other posts where I got Linux tied to AD: Another from the It Bucket list, using a combo of that and the above, I have a Debian box tied to AD that I (and a friend who also toys in my lab) can SSO into.

Tried out various other projects that I’ll give short blurbs about:

Immich: I like it but it needs a way to stay synced with my Google photos until I’m ready to actually make the switch.
Graphene (Android Rom): Ran it for a month, had Work and Google profiles separated from the “default” got exasperated with the constant profile switching, went back to stock.
Bunch of random AI/LLM stuff: Things raining from starting with following a tutorial on getting Openwebui started to falling down the rabbit hole of writing python scripts to make an AI sort my mailbox, also N8N.

I know I’m probably missing things, its been a long time since I posted.

Thats it for now, Thanks for reading.
FireZen

Most may have Guessed I am not a Windows Guy, I prefer Linux, Mainly Debian for Servers and Arch for my Desktop and Laptops.

I do dabble to keep my window skills sharp enough to be dangerous, From spinning up a new Active Directory Server every major Windows Server Revision, Installing and getting a fully functioning Exchange environment last fall and a Windows Distribution Server (PXE boot Windows installs over the network) Most of these don’t last longer then there intended projects and all being virtual I keep them around and updated for around 6 months to a year before killing them and starting over.

One thing I haven’t been able to do over the years was tie a Linux box to MS AD for authentication, I have tried for years (read once, for maybe a a max of a week per year for years) and could just never get it to work.

Cant say that anymore:

I got the Server tied to AD, a test user created:

Initial Login was successful with the only quirk being the home folder being broken.

I fixed that and made a new group for “LinuxAdmins” to give anyone in it ‘sudo’ access:

End result:

Its seems I update this page yearly, lets see whats happened over the last year.

I upgraded the Networks switching from the old 1G Dell Power connect 48 port I had gotten free to a combination of Mikrotiks, one 24/1G Eth 2/10G SFP+(CSS326-24G-2S+) and a 4/10G SFP+ 1/1G Eth SFP (CRS305-1G-4S+)

With this network upgrade I got both the “Servers” and the “Wireless” on VLans as well as going over the 10G link coming out of the OPNsense router now to the New switches, “Wireless” is split out to the access points.

I was forced off of ESXi at the end of last year with the announcement that ESXi was ending support for version 6.7 in November of 2021, I have just read they actually extended it to October 2022.

The end of support hurt as 7.0 dosent support the hardware ware in my server nor did it support the 10G SFP+ card inside.

To be ahead of the End of Support and to make sure I didn’t get lost in security holes I started seeking out alternatives.

The one I landed on and have been running since roughly October of last year is Proxmox VE, 95% of my Virtual Machines were converted without a problem. The two from memory that did cause an issue were my ansible control Linux system and a Windows 10 box. These were recreated and have been functioning happily.

I decided to forgo the Hardware raid and return to my long lost love, ZFS Originally opting for Raid Z3 until I saw a lot of performance degradation so I dropped it down to Raid Z2 and have been running it like that since.

I also went through a time of testing NVME drives as Log and Cache for the SSD this ended up being more detrimental then helpful, since then I instead have created a NVME mirrored pool that hosts a single VM I use for work.

My server is starting to get aged.. I first bought it in February 10, 2016, even then the Motherboard Z9PA-D8 was 3 years old coming out atleast in February 7, 2013. Which Next year will make then a Decade old.

The CPUs in it are not the best it can handle, currently it has Intel(R) Xeon(R) CPU E5-2620 v2 @ 2.10GHz and the best I can find for it is Intel Xeon E5-2690 v2 @ 3.00 GHz, This would give me a 900MHz Boost per core (5.4GHz though I know thats not how it works) on top of 4 extra cores, Per cpu as this is a Dual CPU system. Both Processors are Discontinued and came out Q3’13.

On the ram side I upgraded the Ram from 128GB to 256GB not long ago with the switch to Proxmox and ZFS, ZFS likes yummy ram.

Looking over Amazon for a new server, Matching but allowing for expand-ability is roughly five thousand dollars:

Price

• ASUS Z11PA-D8 Server CEB Motherboard Socket-P LGA3647 for Intel Xeon Skylake Scalable Processors Featuring DDR4ASUS Z11PA-D8 Server CEB Motherboard Socket-P LGA3647 for Intel Xeon Skylake Scalable Processors Featuring DDR4

$619.97

• Intel Xeon Gold 6226 Processor 12 Core 2.70GHZ CPU CD8069504283404 (OEM Tray Processor)Intel Xeon Gold 6226 Processor 12 Core 2.70GHZ CPU CD8069504283404 (OEM Tray Processor)

Qty:1    

$1,628.88

• Samsung 64GB/4Gx4 DDR4-2666 ECC/REG Load Reduced CL19 Server Memory Model M386A8K40BM2-CTD7QSamsung 64GB/4Gx4 DDR4-2666 ECC/REG Load Reduced CL19 Server Memory Model M386A8K40BM2-CTD7Q

Qty:4      

$325.67

• Noctua NH-D9 DX-3647 4U, Premium CPU Cooler for Intel Xeon LGA3647 (Brown)Noctua NH-D9 DX-3647 4U, Premium CPU Cooler for Intel Xeon LGA3647 (Brown)

$99.95

• 10Gb PCI-E Network Card NIC Compatible for Intel X520-DA2(Intel E10G42BTDA), Dual SFP+ Port, with Intel 82599EN Controller, 10G PCI Express LAN Adapter Support Windows Server/Windows, Linux, Vmware10Gb PCI-E Network Card NIC Compatible for Intel X520-DA2(Intel E10G42BTDA), Dual SFP+ Port, with Intel 82599EN Controller, 10G PCI Express LAN Adapter Support Windows Server/Windows, Linux, Vmware

Qty:1      

$175.00

Save 5% 

Clip Coupon

• ASUS – MOTHERBOARDS TPM SPI Module System Components MOTHERBOARDSASUS – MOTHERBOARDS TPM SPI Module System Components MOTHERBOARDS

Qty:1      

$19.97

• CORSAIR RM1000X 80+ 1000w GOLD MODULAR PSUCORSAIR RM1000X 80+ 1000w GOLD MODULAR PSU

Qty:1      

$265.99

• RROYJJ 4U Rackmount Server Case Chassis with 24 Hot-Swappable SATA/SAS Drive BaysRROYJJ 4U Rackmount Server Case Chassis with 24 Hot-Swappable SATA/SAS Drive Bays

Qty:1      

Subtotal (11 items): $4,802.43

Thats my Next Goal.

So Over the last months the ‘Router’ ESXi system was screaming its head off whenever it got close to 100% cpu usage, this would be generally when anything in the house would cause a spike in the Opnsense VM.

Along with this the drives in there were the last of the consumer drives from the V1 lab and it seems I was in prefailure of another drive, No data lost but ESXi wouldn’t allow me to update as the drive would disappear any time I tried.

I had a small 256GB Enterprise drive left over from the last set so I migrated all the VMs from the Router to the man system, So much for Disaster Mode, and set about ripping all the drives out of the Router systems chassis, I put in the single 256GB and install OPNSense on the baremetal, worries me come upgrades but for the moment the backup restored easily with only having to reassign the NICs to the physical interfaces rather then the virtuals and my 10G NIC came back after a config tweak so my ‘Servers’ branch (Most of my VMs run on this as its my DMZ) of my big server runs over a 10G link again.

It could be placebo but it feels alot faster, the system runs cool, I cleaned it at the same time so I bet that helped.

Its been a hell of a ride.

So within recent memory I pulled the consumer drives from the Main server as they were causing issues again and kept making the system unstable.

Both the Main and Sub server updated to ESXi 7 during a routine update, which didn’t go well, The main servers age proved to be a detriment as its hardware raid, the raid I just got done putting the new enterprise SSDs in, was no longer supported in 7, nor was the 10/40GB SFP+ Nic cards, I ended up downgrading the Main server back to 6.7 but keeping the Sub on 7, The main server needed the raid but I figured I could always test the Sub and see if those Nics ever came back.

I bought a Raspberry pi 4, after a few different Project, Retro pi, Zabbix Server, Cluster Head, It settled into the role of the NAS for a few Months, I got a four Drive USB 3 “Toaster” that I put four of the consumer SSDs in and backed it with a 5TB USB 3 HDD, This ran great, low power usage and the SSDs never gave me any grief, I had the system rsync nightly the changes from the flash “raid” to the HDD, Until..

Introduction of the i7! Ok so its nothing new, This is my fathers old system and by old I mean 3960X, it had 64GB of RAM but before giving me the system he pulled half for other systems, Not a big deal, First thing I did was yank it out of the case it was in and put it in a 4U Rack case, Little tricky with it having a 240mm water cooler but eventually I got it all to fit, Stole the drives back from the PI, All of them. Built the system using Manjaro Architect, Things were ok for a while, I bought the missing 32 GB for the i7, was running a Minecraft server on it for my kids and family, slowly instability crept in along with corruption, I shut it down for months as I didn’t have time to troubleshoot what was going on, I bought some new drives and decided to tear it all apart and test components, found one of the 8GB RAM sticks was bad, rebuit it with couple 2TB Ent SSDs, a 120GB ENT Boot SSD, the 4X500 Consumer toaster, and the 5TB Rust. it been running Arch since last weekend and after tuning the Network setting its been stable.

The Raspberry pi 4 /4GB became my Zabbix system, I bought one of the newer pi4 8GB and ran it as my desktop for about a month before calling it quits, Thinking of moving my (this) Web site and mail server to it so its easier to shutdown the servers without interrupting services.

Alright its been a little over a month since the crash.

Main server is running perfectly again, this time hardware raid on Enterprise grade SSDs, with the old consumer ones reinstalled after being checked over multiple times, they are also under hardware raid but not willing to host anything outside of projects I am willing to lose.

This web server still runs across six Raspberry Pi 3b(+) and I plan to keep it this way for the unseen future. Its quite capable, running https, and multiple web apps.

So the system is back up, most systems got through unscathed, it seems my FreeNas box which hosts my in home Plex server is worse for the wear, I’m backing it up and getting ready to pull the 3tb HDDs it sits on and put them through a battery of tests to see if they are indeed failing as well or if they were just corrupted from the crash.

After pulling the other SSDs from the system and putting them through their paces it seems they are unscathed, I plan to put them back in, likely in the 3tbs place.

All for now!

Bonus! Picture of the pi rack this runs on:

Don’t mind the stray pi2.