This is a minor inconvenience that I’ve dealt with for far too long. When using Virtualmin as a reverse proxy, it doesn’t handle Let’s Encrypt verification records correctly and forwards them to the upstream service. In my case, this would cause certificates to issue correctly initially, but then fail to renew after three months is up. Since every request that hits the server was getting sent to the upstream server (including any requests to ....
Through some Google-fu and some other great tutorials, I’ve successfully setup a groups of Redis machines with automatic failover detection via keepalived and sentinel. This sounds mundane, but lets you setup another layer of protection for your Redis cluster without lots of extra configuration. Background – Redis Sentinel and Keepalived Redis Sentinel makes it pretty easy to setup a group of replicated Redis machines and elect new master nodes when others are offline....
Not too long ago in the Linux world, firewall rules were complex. iptables did its job very well, but managing rules was daunting for a newcomer. Debian 9 introduces some changes that make it pretty simple to add a firewall rule. Usually firewall rules are taken care of automatically, when you install a program it takes care of opening up the required ports for itself. In some cases, software can conflict and that is what happened in my case....
One of the new tools I’ve discovered is Chef to manage the configuration and software on Storehouse’s fleet of virtual machines. Chef makes it really handy to update and track config changes, since everything can be tracked using Git or similar. One issue we ran into was having chef-client run at the same time for multiple machines. This issue is kinda subtle, but makes a lot of sense when you think about it....
This is a pretty straightforward thing I’ve wanted to do for some time. Basically, I have a number of sites that I use internally that I wanted to get certificates via Let’s Encrypt, but I also wanted to keep them restricted to only a few IP addresses. The solution is quite simple and works perfectly. We accomplish this with two .htaccess files. One at the site root to restrict IP address that can access the site, the second to disable that restriction on the directory where the Let’s Encrypt challenge is stored....
A subtle issue I ran into was the issue that Proxmox VE would sometimes unmount a GlusterFS volume and would fail to backup. This issue was a bit sneaky though, since the PVE backup program wouldn’t execute it wouldn’t send an email notifying me of the failure. This would make it so the backups would fail silently for some time, until I happened to login and see the errors in the cluster’s log....
Graph of free memory on a node with a leaking piece of software.
This is a scary problem when you’re recovering from an outage of your database machines. If you’re running a Galera cluster and they all go offline, you’ll need to do a bit of work to restart the cluster and make it safe. Galera relies on the fact that there’s at least one node running in your cluster at all times. If your entire cluster goes offline, you won’t be able to start it again, even with the –wsrep-new-cluster option....