1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
|
# My personal infrastructure
* OVH 2Gb RAM VPS running FreeIPA (also tinc/ocserv)
* Hetzner auction server: 48Gb RAM, 2x2Tb HDD. Runs Proxmox, tinc/ocserv, Apache as reverse proxy
* LXC container running FreeIPA replica
* LXC container running Nagios
* LXC container running NextCloud
* LXC container running Grafana
* LXC container running Ipsilon
* LXC container running my WordPress blog
* LXC container running PostgreSQL
* LXC container running a workstation
* VM running Jenkins, Gitolite and an old app I haven't migrated yet to another system
* VM running a Discourse forum
* VM running Dokku
* Flat 1
* HP Proliant Microserver: 4Gb RAM, 2x4Tb HDD
* DHCP/DNS
* Runs SMB/NFS
* ZFS backups on external USB drives
* tinc/ocserv
* Raspberry Pi running LibreElec + TVHeadend, records to NFS share on HP server
* Flat 2
* Raspberry Pi running Raspbian, runs DHCP/DNS, tinc/ocserv
## Networking
I like having working DNS, so I run dnsmasq on both flats and for the Proxmox network on the Hetzner server.
It also does integrated DHCP (mostly everything gets a DHCP IP and thus, a hostname).
Every environment has a /24 network with DNS/DHCP and their own domain (hetzner.int.mydomain, flat1.int.mydomain, etc.).
I've set up SRV records so DNS resolution works across networks (even reverse DNS).
I use Route 53 for DNS records (except those of my own networks). DNS records are created with Ansible playbooks.
I join all networks using tinc in a mesh. Tinc keys are generated and distributed using an Ansible playbook.
On every network I've also set up ocserv to provide remote access if I'm outside these networks; I can pick the closest access point and reach my entire network.
## Authentication
I run a two-node FreeIPA cluster.
It provides a user directory and centralized auth, with paswordless single-sign on.
It also has sudo integration, so I can sudo on all systems with a single password.
Many systems and services are integrated in FreeIPA.
My laptop is joined to the domain so I can even log in to some web applications without typing a password.
Ipsilon adds SAML for some applications which do not support Kerberos.
## Mail
All systems are running Postfix configured to send emails through an account on my free G Suite account.
## TLS
I set up certificates using certbot-route53 on Ansible playbooks.
DNS verification allows me to run TLS on non-reachable hosts.
I run the playbooks from my workstation periodically.
## Monitoring
I run Nagios monitoring all hosts and services.
I get alerts for hosts and services being down.
I monitor some stuff like Nextcloud updates using Nagios and cron jobs.
I use https://github.com/alexpdp7/ragent as the monitor, which also means I get notifications when a host is updated and requires a reboot.
I also run Netdata on many hosts, which I can access via a reverse proxy at https://netdata.mydomain/<hostname> with single sign on.
## Configuration management
I use Ansible playbooks to provision VMs and LXC containers on Proxmox.
The playbooks add the new hosts automatically to FreeIPA, set up SSH, etc.
I also use Ansible for some orchestration tasks (such as deploying FreeIPA replicas, handling Letsencrypt certificates, etc.).
I use an Ansible playbook using https://github.com/alexpdp7/ansible-puppet/ to run Puppet to configure individual systems.
### Software updates
I use `yum-cron` on CentOS and `unattended-upgrades` on Debian/Ubuntu so updates are automatically installed.
`ragent` monitors when Debian/Ubuntu systems need a reboot and warns me through Nagios.
## Storage
I run Nextcloud on an LXC container, files are stored in a ZFS filesystem.
Media and other non-critical files are stored in the Proliant and shared via Samba and NFS.
### Media
I run a Jellyfin server on the Proliant to serve media to phones, a MiBox and a Raspberry running LibreElec.
The Raspberry has a DVB-T tuner and TVHeadend, recordings are stored on the Proliant in an NFS share.
### Backup
Systems with valuable data dump databases, etc. to `/srv/backup/$HOSTNAME/`. This is rsynced to the Proliant Microserver.
I have two external USB HDDs. Each one is a ZPOOL. I plug them in monthly and run a backup script that:
* rsyncs `/srv/backup` and local storage folders
* Uses zfs send/receive and snapshots to backup some ZFS filesystems (Nextcloud).
* Creates snapshots
## Dokku
I use Dokku to host a few personal applications, so I can update them with `git push`. I also have Ansible playbooks to set up the applications and handle some of them which have more complex deployments.
## Possible improvements
* Right now I execute backups in the Proliant, plugging in small USB HDDs. As I'm often away from flat 1, I'd like to re-do my backup scripts so I can plug in the USB drives in any system (e.g. my laptop, the Proliant in flat 1 or the Raspberry Pi on flat 2) and run the backup wherever I am.
* Convert the Proliant to Proxmox so it uses ZFS (for even simpler backups and snapshotting) and the few misc services there can run isolated in LXC containers.
* Kill the VM running Jenkins and migrate to individual LXC containers (Git repo hosting, Jenkins and the other old app).
* Find a way to run the stuff that relies on Docker (Dokku, Discourse) nested in an LXC container using ZFS in a "correct" way, so I can drop more VMs and have more density.
* Deploy a good virtual workstation. I'd like to have a persistent desktop I can use when I'm not at my laptop, where I can run overnight jobs, etc. My current LXC workstation works well, but I'd like to have a good graphical remote desktop solution.
* Better sync'ing of stuff. NextCloud is not well-suited to headless systems.
* Add a lab so I can experiment with things in isolated environments.
* Set up SSO on my smartphone, perhaps do some MDM
|
