From 1a325e7a64c1d6312f07bde74d9f160ebf10791b Mon Sep 17 00:00:00 2001 From: Alex Corcoles Date: Sat, 21 Oct 2023 22:18:39 +0200 Subject: [PATCH] Update PERSONAL_INFRA.md --- PERSONAL_INFRA.md | 93 ++++++++++------------------------------------- 1 file changed, 20 insertions(+), 73 deletions(-) diff --git a/PERSONAL_INFRA.md b/PERSONAL_INFRA.md index a589481..4e0441d 100644 --- a/PERSONAL_INFRA.md +++ b/PERSONAL_INFRA.md @@ -1,17 +1,22 @@ # My personal infrastructure -I'm migrating to a new Hetzner auction server with 128Gb RAM, 2x1Tb SSD. [New documentation here](personal_infra/README.md). +[Further documentation](personal_infra/README.md). -Many stuff below is slowly being deprecated, and I'm updating the doc above. - -* Hetzner auction server: 48Gb RAM, 2x2Tb HDD. Runs Proxmox, tinc/ocserv, Apache as reverse proxy - * LXC container running Dokku, hosting a few personal apps +* Hetzner auction server: 128Gb RAM, 2x1Tb SSD. Runs Proxmox, tinc/ocserv, Apache as reverse proxy * LXC container running Nagios * LXC container running Grafana * LXC container running Ipsilon * LXC container running PostgreSQL * LXC container running a workstation * LXC container running Gitolite + * LXC container running a FreeIPA replica + * LXC container running Miniflux + * LXC container running Nextcloud + * LXC container running FreeSWITCH + * LXC container running Bitwarden + * Two VMs running Talos, providing two Kubernetes clusters (production/test) + * My blog + * A CRUD system I ran two track my weight * Flat 1 * HP Proliant Microserver: 4Gb RAM, 2x4Tb HDD * DHCP/DNS @@ -53,7 +58,7 @@ It also has sudo integration, so I can sudo on all systems with a single passwor Many systems and services are integrated in FreeIPA. My laptop is joined to the domain so I can even log in to some web applications without typing a password. -Ipsilon adds SAML for some applications which do not support Kerberos. +Ipsilon adds OpenID for web application authentication. Ipsilon is backed by Red Hat, although they seem to have shifted their focus to KeyCloak. KeyCloak is much more featureful, but I prefer Ipsilon because: @@ -61,19 +66,16 @@ Ipsilon is backed by Red Hat, although they seem to have shifted their focus to * Integration with FreeIPA is a one-liner * It's still used by the Fedora Project infrastructure -FreeIPA and Ipsilon are running in CentOS 7- I will probably reconsider this stack around 2024 when CentOS gets close to EOL. +FreeIPA and Ipsilon are running on Rocky Linux 9. ## Mail -All systems are running Postfix configured to send emails through an account on my free G Suite account. +All systems are running Postfix configured to send emails. This way I get notifications on failed cronjobs or automated updates. ## TLS -I set up certificates using certbot-route53 on Ansible playbooks. -DNS verification allows me to run TLS on non-reachable hosts. - -I run the playbooks from my workstation periodically to renew certificates. +I set up certificates using [mod_md](https://httpd.apache.org/docs/2.4/mod/mod_md.html). ## Monitoring @@ -81,34 +83,22 @@ I run Nagios monitoring all hosts and services. I get alerts for hosts and services being down. I use https://github.com/alexpdp7/ragent as the monitor, which also means I get notifications when a host is updated and requires a reboot. -To monitor certain things, such as FreeIPA, I set up cronjobs which run health checks and drop the output somewhere in `/var/www/html/*`, which then I check using check_http. - -I also run Netdata on many hosts, which I can access via a reverse proxy at https://netdata.mydomain/$HOSTNAME with single sign on. - ## Configuration management -I use Ansible playbooks to provision VMs and LXC containers on Proxmox. -The playbooks add the new hosts automatically to FreeIPA, set up SSH, etc. See: - -https://github.com/alexpdp7/ansible-create-proxmox-host -https://github.com/alexpdp7/ansible-create-proxmox-centos7-ipa - -I also use Ansible for some orchestration tasks (such as deploying FreeIPA replicas, handling Letsencrypt certificates, etc.). - -I use an Ansible playbook using https://github.com/alexpdp7/ansible-puppet/ to run Puppet to configure individual systems. +[Further documentation](personal_infra/README.md). ### Operating systems I use: * Proxmox, as it provides LXC containers (and VMs if needed) and ZFS storage. I like ZFS for its protection about bitrot, and because send/recv and snapshots are great for backups -* EL7/EL8, due to the long life cycle and stability. Due to the CentOS 8 life cycle changes, I'm switching CentOS 8 hosts to Rocky Linux, while CentOS 7 remains. +* EL8/EL9, using Rocky Linux. FreeSWITCH is on EL8 (Rocky Linux too), because the OKay repo RPMs for EL9 don't seem to work * Rocky Linux for my server Raspberry. * LibreElec for my mediacenter Raspberry. Common distros are not an option, as they don't support hardware video acceleration. LibreElec sets up everything I need with minimal fuss, so while it's the system that doesn't use configuration management, it works fine. ### Software updates -I use `yum-cron` on EL7, `dnf-automatic` on EL8 and `unattended-upgrades` on Debian/Ubuntu so updates are automatically installed. +I use `dnf-automatic` on EL8 and EL9, and `unattended-upgrades` on Debian/Ubuntu so updates are automatically installed. `ragent` monitors when systems need a reboot and warns me through Nagios. @@ -137,55 +127,13 @@ The Hetzner server sends/receives datasets to the Proliant daily. I send/receive datasets from the Proliant to USB drives using ZFS. -## Dokku - -I use Dokku to host a few personal applications, so I can update them with `git push`. I also have Ansible playbooks to set up the applications and handle some of them which have more complex deployments. - -## Why not Docker/Kubernetes? - -Delivering applications as Docker images is massively popular right now, so it's worth explaining why I'm running VMs and LXC containers and I'm not more container-driven. - -#### Some things are not containerized - -FreeIPA has some support for running in containers, but it doesn't seem to be the most recommended way of running it. -While it's not a "core" service, I don't think I have an alternative way to get some of its benefits (single users/passwords database across other services and systems, single-sign on, sudo management, etc.). -It looks like integrating Ipsilon with FreeIPA if Ipsilon is running in a container would not be easy/supported either. +## Kubernetes -WordPress has Docker images, but like the EPEL packages I use, they don't seem to be officially supported by Wordpress. -However, both seem to be well maintained, but EPEL packages are automatically updated using the same process than the rest of my systems (`yum-cron`). - -I need to have non-container infrastructure in place, so I have the option of running additional things there or add the overhead of setting up container infrastructure on top. -Containerization has its advantages, so it's just an equation about how much you benefit from containers compared to the overhead of having container infrastructure. - -#### Some containerized things are special - -Dokku is its own special system. It could be replaced completely with Kubernetes, but with additional complexity. - -#### Containerization infrastructure has its cost - -There are lightweight ways to run containers; docker-compose, plain Docker, etc. -These require significant additional work to automate them to the level of my existing non-containerized infrastructure (automation, backups, etc.), but they consume little additional resources. -The cost analysis for those is that my existing automation works and the cost of re-implementing them makes them not worthwhile at this point. - -Heavyweight solutions like Kubernetes tend to consume more resources, but have better automation features built-in. -The cost analysis for those is that with the money I'm spending now (single 48gb RAM Hetzner dedicated server) I wouldn't be able to run the same amount of stuff. -If I was running a significantly greater amount of stuff or I had high-availability requirements, then this would change. - -#### Conclusion - -If I was starting from scratch, perhaps a light-weight container solution would have been worthwhile, as some services might be easier to provide using a container approach. Also perhaps setting up the automation/etc. would be easier and would give me some advantages. - -If I was running more services or had greater availability requirements, a cluster-ready solution like Kubernetes would probably be required. - -In my current situation, with the work already performed, I don't think investing more in containers is the most effective use of my limited resources. +I use Talos Linux to run Kubernetes. ## My blog -I was never a fan of Wordpress (I prefer other platforms to PHP and MySQL), Remi maintains very up to date EPEL 7 packages. Remi also maintains some packages for EL 8 (derived from Remi's own packages in Fedora, hosted on Remi's personal repositories). - -However, after reading about Geminispace, I decided to port my blog to Geminispace and skip migrating from EL 7 to EL 8. Right now I run some custom scripts that generate a static blog and serves them using Agate in my workstation container. I run a Kineto proxy on Dokku that makes the content available through conventional HTML/HTTP. See details at: - -https://github.com/alexpdp7/gemini_blog +See [blog](blog). ## Phones @@ -226,6 +174,5 @@ So you can do real fancy stuff with it, but I haven't looked at it, because real ## Possible improvements -* Better sync'ing of user files. NextCloud out of the box only works on systems with a graphical interface. There are solutions to mount NextCloud using WebDav, but I prefer to do a sync (so if the server is down I still can access my files) and to run the client headless, but I prefer to stay within supported solutions. Probably syncthing would be a good solution for headless systems to sync dotfiles, etc. * Add a lab so I can experiment with things in isolated environments. * Set up SSO on my smartphone, perhaps do some MDM -- 2.47.3