• 1 Post
  • 171 Comments
Joined 2 years ago
cake
Cake day: August 11th, 2023

help-circle

  • Definitely overkill lol. But I like it. Haven’t found a more complete solutions that doesn’t feel like a comp sci dissertation yet.

    The goal is pretty simple. Make as much as possible, helm values, k8s manifests, tofu, ansible, cloud init as possible and in that order of preference because as you go up the stack you get more state management for “free”. Stick that in git and test and deploy from that source as much as possible. Everything else is just about getting to there as fast as possible, and keeping the 3-2-1 rule alive and well for it all (3 backups, 2 different media, 1 off-site).


  • Fleet from Rancher to deploy everything to k8s. Baremetal management with Tinkerbell and Metal3 to management my OS deployments to baremetal in k8s. Harvester is the OS/K8S platform and all of its configs can be delivered in install or as cloudinit k8s objects. Ansible for the switches (as KubeOVN gets better in Harvester default separate hardware might be removed), I’m not brave enough for cross planning that yet. For backups I use velero, and shoot that into the cloud encrypted plus some nodes that I leave offline most of the time except to do backups and updating them. I user hauler manifests and a kube cronjob to grab images, helm charts, rpms, and ISO to local store. I use SOPS to store the secrets I need to boot strap in git. OpenTofu for application configs that are painful in helm. Ansible for everything else.

    For total rebuilds I take all of that config and load it into a cloudinit script that I stick on a Rocky or sles iso that, assuming the network is up enough to configure, rebuilds from scratch, then I have a manual step to restore lost data.

    That covers everything infra but physical layout in a git repo. Just got a pikvm v4 on order along with a pikvm switch, so hopefully I can get more of the junk on Metal3 for proper power control too and less IPXE shenanigans.

    Next steps for me are CI/CD pipelines for deploying a mock version of the lab into Harvester as VMs, running integrations tests, and if it passes merge the staged branch into prod. I do that manually a little already but would really like to automate it. One I do that I start running Renovate to grab the latest stable for stuff for me.













  • Another step up is the confidential computing project. Requires hardware that supports it though, which sucks, but takes the virtual hardware concept and adds multi key memory encryption on top.

    Remember though security without a threat model is just paranoia, so what level of hoops and investment you need really depends on what your threats actually look like.

    I personally love containers and Macsec. It limits most of my concerns. I want to mess with confidential containers next, which is to say lightweight VMs in containers with memory encryption set, but thats all future to me. The irony is that I then I have to figure out attestation better for those machines since from the host they are black boxes.


  • Worth paying for. Honestly guys selhost for yourself and family contribute to project you depend on, and pay for everything else you don’t have the time to do that too.

    Its free as in freedom, not free as in beer. Their still FOSS even so this isn’t even a rug pull imho. The worst thing is it requires an account which hurts the privacy aspect of that. One the few times something like Meta mask actually makes sense… Maybe some way to get an token generated and sent for use rather than a blanket account