This is very awesome. For those interested, the Reddit.com/r/homelab section is super great for this kind of stuff.
I just got a Dell R720 off Craigslist to do an experiment with the Firecracker MicroVM of all things. It's got 48gb of ram and dual hex-core Xeons. It's a lot of fun to play with this kind of gear.
Since I got fiber gbit up and down I try to relocate as much as possible of my homelab to real data centers (Hetzner) since I can't stand the noise. (I have 10ms ping to my server)
Then I use wireguard to connect it to my home network and make it act like a server in the home network. (All the traffic even gets directed through home)
I still have the main storage server locally though because otherwise I'd not be able to even watch movies if my internet went down.
My dream would be an all SSD storage server being near silet.
I recently set up Pfsense on an old box I had lying around. The practical uses i’ve found are: setting up the PfsenseNG DNS adblocker and setting up an OpenVPN server so I can VPN into my LAN away from home instead of port forwarding. I also just set up Dynamic DNS so that I can do this with a domain.
Can anyone share other use cases they’ve found to be practical with their homelab setup?
To expand on the VPN benefits you already have in place... grab a vnc app for Android and VPN in then vnc to a machine from your phone. All of the sudden you are sitting at an Linux desktop on your phone. With DLNA casting you can literally now use your phone as a full blown computer casting to a monitor (assuming you have some sort of peripherals). That was always my holy grail.
Can you expand on the QoS setup you’re using? I’ve recently read about the fq_codel addition to Pfsense and wondering if anyone here has been using that.
I’m also looking for more performance related tunes as well. To that end, I’ve configured Cloudflares DNS but not sure what else yields results. It sounds like squid caching is mostly useless due to most traffic being HTTPS.
It's actually a pretty simple queue-based setup right now. My primary intent was to ensure that bulk downloads and uploads don't interfere with my latency while gaming.
I did look into fq_codel but I don't believe that was a practical option on PfSense back when I was setting everything up. I might have to revisit that now that it has native support in PfSense 2.4.4.
fq_codel is working pretty well for me to keep my ping consistent with big downloads going. There are some straightforward tutorials out there now too, since pfSense has native support.
Not my lab but it varies from homelab to homelab. A lot of it is using it as a safe place to understand/experiment with new tech without breaking anything in a production/work environment.
Some people also use them to run Plex/Pihole/Pfsense and the like as well.
I think this is a pretty good summation. I'm fairly active at the r/homelab subreddit and this is pretty on point. Usually there is some combination of upgraded home media/routing and then on top of that a learning and experimenting section based on your specific goals.
For others though it really is a hobby in the sense of building to build.
These days, I mainly use three old i5 boxes with 8GB RAM and SSD RAID10. Two with 4x128GB and one with 6x256GB. They're all VirtualBox hosts, and run various pfSense VPN-gateway VMs, Whonix VMs, and other workspace VMs.
I also have some old rack-mount servers. But their fan-control systems are all Windows-based, and they're impossibly loud with Linux. I keep meaning to do some crude electrical hack -- basically a manual speed control -- but just haven't gotten around to it. And there's also the fact that I don't need that much muscle locally. Because I like to keep my network activity minimal.
Everything is on a 0.25kWh UPS (3kW maximum load). With a kill switch on my desk. The UPS and a pfSense perimeter router/firewall (and a rack with the old servers) live in a well-ventilated closet.
Nice write up and photos. I used to maintain a lab like this only my cable management was nowhere near as nice as his. I've replaced this all with a combination of AWS and an Antsle. I miss the hands-on hardware time but don't miss the fan noise.
As others have said, it can be done cheaply. The electricity costs more than the hardware at least for most people's setups that I've seen. (Exception is people using stuff like RPi boards from solar panels; that's almost a sub-hobby in itself.)
I run mine in a basement in place of a small utility heater. If I'm going to burn the watts anyway, might as well get some processing cycles out of it.
Be patient and wait for deals you can snipe. You can get a server that was considered high-end circa 2010 for $50.
Some high-end Xeon chips are dirt cheap, like $50 each or less, as parts. Certain models were made by the millions and are still good, but got tossed as more energy efficient versions came to market. It's just not economical to use them at scale.
I used to run a full size rack with servers and gear but ended up downsizing to three NUCs with 32GB in each, a simple managed Ubiquiti switch, and an 8 bay Synology like the one in the article. Doesn't have the "wow factor" of the big rack of equipment, but it's a lot easier to work on, quieter, and cheaper on the electric bill.
Worried to see the rackmount hardware close to the carpeted floor with a wall directly behind them. Potential for dust and airflow issues.
I've had bits and pieces of a home setup over the years (currently just an HP microserver with disks and VMs) but lots of cost and noise for something I probably won't use a lot of.
Cool. This puts my homelab to shame. I've basically got an old i5 server running Linux (non SSD), an unstable HP DL160 (never buy HP, kids!) and a Raspberry Pi for home automation. I've also got lots of additional hardware that I rarely power up such as Sun Blade 2000's, SGI's etc.
My problem with homelabs (mine and others) is the power management. Finding hardware that triangulates to energy efficient/good proxy for production/cost effective is really hard to do.
We had a small cluster in the basement, with a server for all the backups, a firewall etc. initially for our software development and later mainly for email.
I just decommissioned it all (still have a few network hard drives for local backup of laptops etc). The power it was drawing was about 300-400 watt. That is 7.2-9.6 kWh/day and 2600-3500 kWh/year. (At about $0.1/kWh.)
That is about equivalent of powering a Tesla Model 3 (160 Wh/km) 16000-22000 km, which is about what I drive per year.
Our garage has a south east facing roof which with solar panels could deliver about 3000 kWh/year. That installation isn’t terribly efficent due to scale, and would cost me about $15000 to install.
Instead we decided to move the email to a Swiss email provider (Migadu) that cost about $4/month. We may install solar panels later amyway, but moving the servers into the cloud cost $48/year and gave us enough equivalent electricity to drive the whole year.
Noise is the other issue. I found a nice fanless switch, but my server was noisy for years until I got serious about fans and switched everything to 4-pin Noctuas with PWM set at 50%.
I made my server near silent (it's basically just a desktop computer) but now the biggest noise comes from the disks. I can't stand the noise. Not sure how to make disks quiet though.
"After messing around with the ERL for months and going through flash drive after flash drive (What they boot from) I decided it was time to move on."
Hmm, if he is talking about the EdgeRouter Lite, I have one, going strong since ~2014, didn't have any clue it has problems with flash drives, and there certainly isn't any need to open the case and mess with internal components. It's a good little router and I have few complaints with my Ubiquiti gear. I have not placed mine in a rack without adequate ventilation, however.
I've heard the same complaint about the USB flash drives failing. I run an ER-8 but have had no issues. I think it may have been a bad batch or something.
Some people have also had issues after installing OpenBSD on them without limiting writes.
Anyone who is thinking of doing this should consider how loud and annoying 1RU sized servers and switches are. They're not designed to be used in the same room as people for any extended amount of time. If you have a whole room in your house that you can dedicate to this, or a corner of a garage, sure. But don't buy a used Dell R610 1U server off eBay and expect to put it comfortably in your home office.
Agreed. Workstation towers like the Z420 are probably a lot quieter for small homelabs. I have a TS140 and Microserver, while the old Microserver is nearly silent, the TS140 can be irritating if seeking a dead silent room.
I love this idea. You can get decent laptops (i7, 8gb rm) with broken displays for cheap on EBay/Craigslist, and install Linux. Best way to set up a homelab on the cheap.
I used to run a super loud HP server I got off craigslist for something like 70 bucks. Thing was a beast. Since have switched to a dual Xeon storinator with 30 8tb hdds. My next project is getting another storinator and finishing wiring the house for 10ge
There’s lots of perfectly good hardware out there for cheap that may be beyond it’s useful life for top end production workloads but is still great for other projects or just tinkering.
My little TS140 with it's Xeon running Proxmox does just about all I need. My Synology does the backups and my EdgeRouter-X does the routing to round things out.
I've always wanted to do something like this. Where would someone go about learning how? Would you need a background or are there other resources you could use to learn?
reddit.com/r/homelab has a decent wiki for the hardware side. For the software side, it's mostly DevOps/sysadmin work. DigitalOcean has extensive tutorials for that kind of stuff.
I just have a Skylake Xeon E3 quad core from 2016 as my main sever and an i3-6100 with 8GB of RAM and Intel NICs as my router, but I want more goodies :D
If you are actually interested in learning more about networking, i cannot recommend starting your own homelab enough.
networking seems to be a hit or miss for people, some people find it very logical and "easy" while others seem convinced it's black magic.
networking basics is rather easy, the problem is complexity that gets added when you scale. (although i doubt you would reach that point in a homelab though).
I just got a Dell R720 off Craigslist to do an experiment with the Firecracker MicroVM of all things. It's got 48gb of ram and dual hex-core Xeons. It's a lot of fun to play with this kind of gear.
Ikea sells a 'rack' (not the Lackrack, https://wiki.eth0.nl/index.php/LackRack) that is really great for holding gear: https://www.ikea.com/us/en/catalog/products/S39031411/
Some pix of my recent setup: https://imgur.com/a/xXc6720
Back in ~2014 I had a similar tiny setup in my basement with a UPS and an older (and much louder) Poweredge 2950: http://i.imgur.com/zxYxuLq.jpg