After a long a deep think, I’ve decided to retire my physical hardware and migrate all of my machines to Amazon Web Services. It wasn’t an easy choice, and I feel like I need to spend some time explaining why, just get some things off my chest.
Through some Google-fu and some other great tutorials, I’ve successfully setup a groups of Redis machines with automatic failover detection via keepalived and sentinel. This sounds mundane, but lets you setup another layer of protection for your Redis cluster without lots of extra configuration.
Not too long ago in the Linux world, firewall rules were complex. iptables did its job very well, but managing rules was daunting for a newcomer. Debian 9 introduces some changes that make it pretty simple to add a firewall rule.
Today, The Storehouse experienced an outage that lasted approximately 12 hours. This was caused due to updates performed late the night before and services restarting during that process.
One of the new tools I’ve discovered is Chef to manage the configuration and software on Storehouse’s fleet of virtual machines. Chef makes it really handy to update and track config changes, since everything can be tracked using Git or similar. One issue we ran into was having `chef-client` run at the same time for multiple machines.