From 69f986d2bea221d3e5aed3c4f46a0b3e2749969d Mon Sep 17 00:00:00 2001 From: alex Date: Wed, 18 Mar 2020 20:24:34 +0100 Subject: [PATCH] Add more details --- PERSONAL_INFRA.md | 52 +++++++++++++++++++++++++++++++++++++++++++++-- 1 file changed, 50 insertions(+), 2 deletions(-) diff --git a/PERSONAL_INFRA.md b/PERSONAL_INFRA.md index 6ec20fc..0464887 100644 --- a/PERSONAL_INFRA.md +++ b/PERSONAL_INFRA.md @@ -29,7 +29,9 @@ I like having working DNS, so I run dnsmasq on both flats and for the Proxmox ne It also does integrated DHCP (mostly everything gets a DHCP IP and thus, a hostname). Every environment has a /24 network with DNS/DHCP and their own domain (hetzner.int.mydomain, flat1.int.mydomain, etc.). I've set up SRV records so DNS resolution works across networks (even reverse DNS). -I join all networks using tinc in a mesh. +I use Route 53 for DNS records (except those of my own networks). DNS records are created with Ansible playbooks. + +I join all networks using tinc in a mesh. Tinc keys are generated and distributed using an Ansible playbook. On every network I've also set up ocserv to provide remote access if I'm outside these networks; I can pick the closest access point and reach my entire network. @@ -44,6 +46,17 @@ My laptop is joined to the domain so I can even log in to some web applications Ipsilon adds SAML for some applications which do not support Kerberos. +# Mail + +All systems are running Postfix configured to send emails through an account on my free G Suite account. + +# TLS + +I set up certificates using certbot-route53 on Ansible playbooks. +DNS verification allows me to run TLS on non-reachable hosts. + +I run the playbooks from my workstation periodically. + ## Monitoring I run Nagios monitoring all hosts and services. @@ -51,9 +64,44 @@ I get alerts for hosts and services being down. I monitor some stuff like Nextcloud updates using Nagios and cron jobs. I use https://github.com/alexpdp7/ragent as the monitor, which also means I get notifications when a host is updated and requires a reboot. +I also run Netdata on many hosts, which I can access via a reverse proxy at https://netdata.mydomain/ with single sign on. + ## Configuration management -I use Ansible playbooks to provision VMs on Proxmox. +I use Ansible playbooks to provision VMs and LXC containers on Proxmox. +The playbooks add the new hosts automatically to FreeIPA, set up SSH, etc. + I also use Ansible for some orchestration tasks (such as deploying FreeIPA replicas, handling Letsencrypt certificates, etc.). I use an Ansible playbook using https://github.com/alexpdp7/ansible-puppet/ to run Puppet to configure individual systems. + +### Software updates + +I use `yum-cron` on CentOS and `unattended-upgrades` on Debian/Ubuntu so updates are automatically installed. + +`ragent` monitors when Debian/Ubuntu systems need a reboot and warns me through Nagios. + +## Storage + +I run Nextcloud on an LXC container, files are stored in a ZFS filesystem. + +Media and other non-critical files are stored in the Proliant and shared via Samba and NFS. + +### Backup + +Systems with valuable data dump databases, etc. to `/srv/backup/$HOSTNAME/`. This is rsynced to the Proliant Microserver. + +I have two external USB HDDs. Each one is a ZPOOL. I plug them in monthly and run a backup script that: + +* rsyncs `/srv/backup` and local storage folders +* Uses zfs send/receive and snapshots to backup some ZFS filesystems (Nextcloud). +* Creates snapshots + +## Dokku + +I use Dokku to host a few personal applications, so I can update them with `git push`. I also have Ansible playbooks to set up the applications and handle some of them which have more complex deployments. + +# Possible improvements + +* Adapt backup scripts so they can be run from everywhere and I can do "offsite" backups +* Use Grafana instead of pnp4nagios -- 2.47.3