Devco.Net

Choria Hierarchical Data

Tue, 25 Nov 2025 00:00:00 +0100

As most are aware, I created the widely used Hiera system in Puppet. I introduced it in 2011, and it has since become essentially the only way to use Puppet in any meaningful fashion. Given its widespread adoption, I donated the code to Puppet, and it became integrated with Puppet core.

Unfortunately, during this integration we lost some key values—the command line and the ability to use it in scripts and elsewhere.

Meanwhile, our world is changing, and we are ever more focussing on small, single purpose compute. I intend to create a new kind of Configuration Management system that is focussed on the small, single, purpose needs. Filling the gap that has always existed - how to manage our applications rather than systems, an area Puppet has always been weak at.

There is then still the need for hierarchical data and given that I have the flexibility to start completely fresh I am at best taking some inspiration from Hiera.

So today I’ll introduce a new tool called Choria Hierarchical Data - current code name tinyhiera but that might change.

NOTE This post was first posted on the Choria blog.

Read on for the details

The big difference here is that this new system supports just one data file, you express a data structure that is the complete outcome of the query and then configure in the same file how to override and extend that data file.

Let’s look at it, here I’ll show the different sections, in reality it’s just one file.

This is the data we wish to manage:

data:
    log_level: INFO
    packages:
        - httpd
    web:
        listen_port: 80
        tls: false

Lets define the hierarchy for overriding this data; this should be familiar to Puppet users:

---
hierarchy:
    merge: deep # or first
    order:
        - env:{{ lookup('env') }}
        - role:{{ lookup('role') }}
        - host:{{ lookup('hostname') }}

And finally, we show the overrides, here we just specify how the data will be extended:

overrides:
    env:prod:
        log_level: WARN

    role:web:
        packages:
            - nginx

        web:
            listen_port: 443
            tls: true

    host:web01:
        log_level: TRACE
        web:
            listen_port: 8080

We can now call our CLI tool to resolve this entire structure.

We query the data setting the role fact to web, the data from the role:web section is merged into the data from the data section:

$ tinyhiera parse data.yaml role=web
{
  "log_level": "INFO",
  "packages": [
    "ca-certificates",
    "nginx"
  ],
  "web": {
    "listen_port": 443,
    "tls": true
  }
}
$ tinyhiera parse data.yaml role=web --query web.listen_port
443

Features and Differences

This is the basic feature, similar to what you might have used in Puppet except differing in a few areas:

It’s all one file and one data structure, it’s not key-lookup orientated. The entire outcome is rendered in one query
Expressions used in overrides and data lookups are a full query language called expr-lang, this means we can call many functions, generate data, derive data and easily extend this over time
The data is typed, if your facts have rich data the results can also be rich data, if value is a integer, array or map the result will be the same type x="{{ lookup("value") }}"
The CLI has built in system facts, can take facts on the CLI, read them from JSON and YAML files, parse your environment variables as facts or all at the same time
The CLI can emit data as environment variables for use in scripts
There is a Golang library to use it in your own code

Environment Variable Output

We want this to be usable in scripts, the easiest might be dig into the data and just get a single value:

PORT=$(tinyhiera parse data.yaml role=web --query web.listen_port)

But this involves many calls to the command, we can instead emit all the data as variables.

$ tinyhiera parse data.yaml role=web --env
HIERA_LOG_LEVEL=INFO

You can see the data is emitted as environment variables that you can just source into your shell script, obviously in this use case it benefits to have flat data.

Built-in facts

We have many ways to get facts into it such as files, environment variables, JSON and YAML files. The CLI also includes it’s own mini fact source called system which will return facts about the system it is running on.

To view the facts fully resolved, we can run the following, I am removing much of the details here but you can see this offers enough to make most required decisions:

$ tinyhiera facts --system-facts
{
  "host": {
    "info": {
      "hostname": "grime.local",
      "uptime": 1843949,
      "bootTime": 1762258234,
      "procs": 711,
      "os": "darwin",
      "platform": "darwin",
      "platformFamily": "Standalone Workstation",
      "platformVersion": "26.1",
      "kernelVersion": "25.1.0",
      "kernelArch": "arm64"
    }
  },
  "memory": {
    "swap": { },
    "virtual": { }
  },
  "network": {
    "interfaces": [ ]
  },
  "partition": {
    "partitions": [ ],
    "usage": [ ]
  }
}

Long-term view

Autonomous Agents

The goal here is to incorporate this into various other places, first we pull it into a Choria Autonomous Agents, these agents own the full lifecycle of an application:

Deploy dependencies
Configure the application
Run the application
Monitor the application
Restart the application on failure
Orchestrate rolling upgrades
Present APIs for interacting with the management layer

Agents can fetch data from a Key-Value store, but it’s kind of all the same data. With Hiera integrated into the KV feature, we get:

  - name: watch_tag
    type: kv
    interval: 10s
    success_transition: regional_update
    state_match: [RUN, RESTART]
    properties:
      bucket: NATS
      key: config
      hiera_config: true

Here the autonomous agent will check KV every 10 seconds for data, fully resolve it using Hiera and save the resulting data into the Autonomous Agent internal data store. If the KV data chanfes or the referenced facts change to the extent that the data change, the machine will update the stored data and fire the regional_update event.

This way we can create role-orientated specific data all in the same Key-Value key.

Configuration Management

I want to create a new CM system that takes the model we are used to in Puppet but brings it to scripts and reusable APIs.

The script use case would essentially be standalone commands:

$ marionette package ensure zsh --version 1.2.3
$ marionette service ensure ssh-server --enable --running
$ marionette service info httpd

This should be familiar to Puppet users, we’re basically pulling the resources and RAL into standalone commands. The commands will be fully idempotent like Puppet and support multiple Operating Systems.

For integration with other languages, you can also deal with JSON-in-JSON-out style:

# Can be driven by JSON in and returns JSON out instead
$ echo '{"name":"zsh", "version":"1.2.3"}'|marionette package apply
{
....
}

I do not want to create something huge like Puppet, but just basically enough that can enable the package-config-service trio pattern, ultimately a manifest would look like this:

cat <<EOF>manifest.yaml
data:
    resources:
        - package:
            name: myapp
            ensure: present
        - service:
            name: myapp
            ensure: running
            enabled: true

hierarchy:
    merge: deep
    order:
        - platform:{{ lookup('host.info.platform') }}
        - hostname:{{ lookup('host.info.hostname') }}

overrides:
    platform:darwin
        resources:
            - package:
                ensure: MyApp-1.2.3.dmg
EOF

$ marionette apply manifest.json --system-facts

Pull this into Autonomous Agents and you have self-contained, self-managing applications similar to Kubernetes Operators but anywhere.

Use this in scripts, simple setups, during container creation or orchestration, or even in CI pipelines.

We should even be able to compile this manifest, or a full autonomous agent, into a single, static, binary that you can just run as ./setup or ./manage and the entire lifecycle is managed with zero dependencies.

Availability and Status

You can get the command on GitHub choria-io/tinyhiera, we are pretty far along but expect some more breaking changes including a potential name change.

Lab Infra Rebuild Part 6

Wed, 31 Jul 2024 01:00:00 +0000

This is the final in a series of posts about rebuilding my lab infrastructure, see the initial post here.

Today we’ll wrap things up with a look at some SaaS tools I use and a general look through some small utilities and things I use to bring it all together.

I’ve been enjoying my summer for the last 3 months hence the hiatus of posts.

Email

Long ago I used to run my own Zimbra but I gave up on that a few years ago, been with Fastmail ever since and really happy with their service.

Delivering email from my 20+ VMs all over the world though is tricky, getting all the various DKIM and other settings right for a big set of IP addresses and ranges quickly becomes a maintenance nightmare. But ther’s a constant trickle of stuff from them - cron job, monitoring, backup statusses and more.

After some looking around at options I found SMTP2GO who have a very generous 1000 / month free tier. This is usually fine for me but I ended up paying them anyway for a annual account. This way I have just one egress point to consider in my various email policy setups and thus far, for delivering system emails, this has been a great time saver.

Read on about DNS, Git, SSO and more.

DNS

Till this rebuild I hosted my own DNS, I’ve had a set of quite stable DNS servers on the same IP addresses for over a decade so it’s worked well.

I’ve always wanted to be rid of hosting this myself but most services are request based billing and I always felt this would not go well since you have no way of controlling how many DNS requests you get.

After some research I found ClouDNS who have a 50 zone / 2000 record plan that allows 200 million queries a month for $5/month. This is plenty and incredibly cheap. They have Geo DNS servers, support is responsive and knowledgeable and have a decent API.

For comparison DNSimple is $30/month and charges per zone and 10c per million queries a month. That’s crazy.

My usage is around 20 million queries a month so I am very comfortable in this level of service and for $5/month there is simply no way to compete with this on any kind of self hosting setup.

They have a bind zone file import feature and I use their API to do daily backups of all the zones into my local Git server using a little tool I wrote call CloudDNS Backup

Git

Speaking of Git hosting, I quite like Gitea though should probably move to Foregejo. The split from Gitea happened just as I was building things up so I am still on that.

Gitea is pretty great, it does a reasonable job of being a GitHub facsimile and uses act (also great for local action testing) to provide reasonably compatible self hosted GitHub Actions.

Gitea and Foregejo are both single binary/single process tools so really easy to get going, for my needs even their SQLite support is ideal.

SSO

Getting all things authenticated and managing users is a nightmare, I use Okta free tier to front almost all my HTTP stuff and it’s been great. With Apache mod_auth_openidc it’s easy to stick that in-front of most things even static sites.

Object Storage

I used to use Digitalocean S3 compatible storage (still do a bit tbh), but I am slowly moving all my use of that over to a private Minio instance. I do not need it to be super high available but do need the data on it secure so this runs on my Hetzner Backup server with its 6 disk redundant storage setup.

To be honest I think Minio is just unusable. It’s not that the tool is bad, it’s really great, I really want to love it, it’s that the project is just crazy with releases. It’s in everyones interest to upgrade and run the latest software but with 50 releases THIS YEAR ALONE (IT IS AUGUST!), I do not know how anyone use this in any seriousness. This is not software made to be used by real teams in the real world with real pressures on their time. Further their Release Notes are of the git log --oneline variety which does not help, commit logs are developer UX not end user UX.

So I am probably looking at an alternative soon. This is kind of why my migration to it have stalled until I can figure out what to do with this.

Security

We all have to stay on top of security alerts, in the good old days you’d just read Bugtraq and know all there is to know. These days though things are much more complex with the amount of things we run and how much goes on the internet.

I use OpenCVE to track and alert me of any CVEs on the main tools I care about.

I’ve also for the first time set my Enterprise Linux machines to auto update themselves, it’s a bit scary to be honest and I exclude kernel updates, but so far it’s been ok. Once a httpd update messed me around a bit but Puppet soon fixed that on next run so was a small inconvenience. On the whole I’d strongly recommend using this.

Monitoring

Apart from the obvious Grafana/Prometheus pair that I do self-host I use a few other things. Graphite to store my IoT data in, it just seems a bit more suitable though alas it’s in a sad state and mostly dying. I might need to revisit what I do there in time.

I have deployed Apprise everywhere and it’s really great it can notify a huge list of services and having it on every machine ready to use is great. I have it integrated in a reusable Gitea action so any failing builds result in alerts.

Apprise sends some alerts and statuses to a dedicated private Mastodon account others to Slack and others to Victorops and to Pushover. This is well worth looking into. Geting Slack/Mastadon reach outs from cron or other tools is really good.

Victorops I use for my Prometheus alerts and some other things, I like its ability to silence and to give me a clear view of the current state of things when I am not around computers. Probably better options now but it’s cheap and just works.

I use Pulsetic to do external checks on my websites, they have a very generous free tier though I did recently upgrade to a paid plan. They allow you to make great external visible or private dashboard like this one for my Choria Project Infrastructure

Letsencrypt

I use LE for TLS like more or less everyone else. ClouDNS is supported for a DNS authenticator in acme.sh so that’s a good fit.

I have a difficult problem in that I want to do a globally redundant hosting of some static files on the same name but I do not want to pay for a GSLB. So I ended up making an Action that will manage those certificates daily, renew them and commit them to my Puppet repository from where they get rolled out to the webservers. This works great.

Awesome Lists

I’ll take liberty and do a plug for my Free-for-dev list that currently has 1500+ services listed. It focus on services that provide generous free tiers generally in the devopsey world many of them suited for home labs. Most of the ones above came from this list, this is literally why I maintain this list. Currently this is the 4th most popular Awesome List with nearly 90 000 stars. Built by 1600+ people collaboratively on GitHub

If you’re not aware I also strongly recommend subscribing to This Week in Self-Hosted if home labs is your thing this will be invaluable.

In general if you are not yet on board with the whole Awesome List movement you really should, check out a giant list in sindresorhus/awesome and you’ll find many tools to track, discover, search and more.

These are the new Yahoo at grass roots level, it’s amazing and one of the most real and relevant resources out there today.

Conclusion

Well that about sums it up.

After this we’ll get back into some general blogging, there’s a fair bit I did not get into like my recent intro into 3D printing and more but this series related to my Home Lab build has more or less run its course.

Lab Infra Rebuild Part 5

Thu, 25 Apr 2024 06:00:00 +0000

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today let’s look at VMs and Baremetals and Operating Systems.

Virtual Machines

As I mentioned I’ve been a Linode customer since essentially their day one and have had many 100s of machines there for personal and client needs.

I soured off them quite significantly after they botched their Kubernetes release by not having Europe side support past basic triage initially that led to multiple multi hour outages and so I have been making some changes.

My most recent incarnation had about 5 or 6 Linode machines - Puppet Server, Choria Repos * 3, 2 x DNS, General Services machine.

Today I have 2 Linode machines left. I was reluctant to move them as they were DNS servers for many domains but I changed my way of hosting domains also during this so less of a concern now.

They are now just RPM/Deb repos for Choria and I will move those elsewhere soon also. That’ll be the first time I am without Linode machines since basically 2003, such a shame. One is in the US to provide a nearby mirror there, I might keep it and just scale it down to lower spec. But with the recent changes at Linode it feels a bit like it’s time to consider alternatives.

Previously I had Digital Ocean droplets x 3 for my Kubernetes cluster, as discussed that’s all gone now too.

I used to have quite a selection of Vultr machines, I don’t recall why I left them really I think I felt Linode was just the rolls royce of this kind of Cloud provider and so consolidated to simplify my business accounting etc

Baremetal

In the previous iteration I had only 1 hosted physical machine and that was my Backups machine running Bacula on a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm. I do not need much from this machine, it wakes up, do backups then sleep again till tomorrow. So the spinning rust is fine for that, I just need lots of it. I rebuilt on a new one just to get a hardware and OS refresh.

Of course I do still use Virtual Machines just managed by Cockpit as per the 2nd part in this series.

I got a pair of Hetzner AX41-NVMe (64GB RAN Ryzen 5 3600) with 2 x 512 NVMe SSDs each. I was expecting to add a 3rd but really these 2 plus the Ryzen at my office turns out to be plenty for my needs. They have some upgrades available - more RAM, Extra Disks, SATA can be added etc. I don’t know if Hetzner supports upgrading running machines but this is nice little platform. At EUR37 per machine that puts them between a 4GB and 8GB shared CPU Linode. You really can’t complain. I might get a 3rd one just for the sake of it and spread my development machines out more. Something for after the summer.

Performance wise moving from my Droplets and other VMs to these machines have been amazing, for an investment equalling 1 Linode I can run several VMs with no additional cost to expand to another VM or to shuffle memory allocations - or just to allocate more since 64GB is way more than I need. This really is a no brainer for my needs.

I’ve been a Hetzner customer also since a very long time, it’s not clear how long but it feels like maybe 2008 or so. They’ve had their ups and downs and dodgy datacenters, dodgy connectivity and dodgy hardware, bad english support, but in the past few years I think they’re really firmed up quite nicely and my machines there have not given me trouble so I felt it’s safe to lean on them a bit more. A few months in now and I’ve not had one minute of problems.

Read on about Operating Systems and more.

Operating System

A bit of history is needed I guess. I started with Softlanding Linux System around 1993, after installing this on a 20MB HDD I removed from a Novell machine the disk promptly died. A bit of hits and misses later I eventually got it quite happy.

Then I moved to Slackware which was the successor to SLS of course. I tried a few things but once RedHat released their first preview in October 1994 on Halloween it was pretty much just RedHat all the way from there. I’ve used all the versions, even the 5.x series that was a total disaster after they moved to ELF but kept at it.

When RedHat removed their free editions I dabbled with Debian but I don’t like the inconsistencies, community, approach or really anything at all about Debian.

Luckily of course CentOS came to the rescue, I even donated hardware to them - a IBM Bladecenter full of Blades. Of course then for some reason they decided to join RedHat, I never did understand the thinking here. Needless to say that did go about as well as it was obvious to anyone it would.

This left a few choices, mainly Rocky Linux and AlmaLinux. I initially went with Rocky Linux it seemed to be on a good track and had the CentOS founder on board, they did seem to loose steam a bit and had a few fits and starts. Now seems fairly solid so I wouldn’t have qualms using them but I settled on AlmaLinux. AlmaLinux seemed at the time to have a bit more backers, a bit more polish and just worked better for my needs.

Then of course RedHat removed the source repos from CentOS Stream this left things quite difficult. Rocky Linux, then, stated they are going to get their SRPMs from a bunch of places to keep the bug-for-bug EL rebuild thing going by means which I really do not think is sustainable. From using paid-for cloud instances and then getting SRPMs using those or using the UBI container images etc, basically they go spelunking all over the place to find source RPMs and then somehow promise to be a complete like-for-like distribution of RHEL. I am not convinced.

Almalinux went another way, they decided to drop the 1:1 RHEL promise and instead moving to promising ABI/Binary compatible with RHEL. They probably also get their SRPMs from some interesting places but they are forging a path of innovation which already resulted in them returning some old hardware support for example. I am quite happy with my choice.

To complicate matters a bit more - or simplify them I guess - the Open Enterprise Linux Association was created to be a source of SRPMs for EL rebuilders like Alma and Rocky. Their mission is to deliver All sources necessary to achieve a 1:1 / bug-for-bug compatible version of EL which will be distributed via Git, encouraging community collaboration. Hopefully this will help matters along. This is supported by Oracle and SUSE. Oracle of course have their own Oracle Enterprise Linux that was dunked into the same problem - and no doubt the source of all the headaches for RedHat.

Anyway, so a lot of history, I like EL. I will keep using EL even if its a slightly different EL. I don’t think its inherently better or worse than other alternatives - and I do not care. The best tool for the job is often the one you know best. I have 30 years of EL experience and that works for me. It’s all just SystemdOS now anyway.

Conclusion

That’s about it really for hardware and OS. The consolidation, moving to Baremetals etc have ended up resulting in quite a bit of money saved. Even if I got a 3rd VM host at Hetzner I’d still be more than a $100 down on monthly expenses but got a lot more.

I have some travel coming up in the next few weeks so the next installment might be a while.

Lab Infra Rebuild Part 4

Thu, 11 Apr 2024 09:00:00 +0000

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk about my physical office and office hardware.

Office Space

When my son started going to school I did not look forward to all the driving so figured a office near his school would be good, I’d spend the days there and come home after pick up. I rented a nice place in a town called Mosta, it was nice and had ample storage and would have made a really great maker space as it had about 4 car garages worth of underground storage that was well lit and ventilated.

Unfortunately this place was opposite a school and parking was absolute hell. I ended up just not using it for months at a time since I could drive home in 12 minutes or spend 45 minutes finding parking, no thanks. I gave up trying to find a garage to rent around there, it’s just crazy.

When I started looking the 2nd place in Żebbuġ I saw seemed perfect, a fantastic bright 7m x 7m office with a underground 1.5 car garage at a reasonable price. I took that and just recently extended my lease to 5 years. My sister-in-law is also moving her business to the same street and there’s a 3D printer shop around the corner, bonus. Parking is always easy even on the street and it’s like 5 minutes from my house.

Here I am able to set up 3 workstations, my main desk, one for guests with a little Desktop for my son and big work station for soldering, assembling IoT projects and such. I also have a nice 2 seater sofa with a coffee table. With the garage I have space to put things like a lazer cutter and more in future - though I am eyeing a small space across the road as a workshop space also for that kind of thing.

Location wise I could not want more, it’s a easy walk or cycle from my house through a beautiful car-free valley and the town is pleasant enough with food options and a small corner shop nearby.

Read on for more about the hardware.

Desktop / Laptop

I’ve been using using Apple desktops and laptops almost exclusively since about 2007. I do like the graphical UI but loathe the BSD based shell, so generally my mantra is: MacOS Shells are for SSH to Linux machines.

I’ve had every iMac since the very first plastic white ones, I really liked that form factor, so I just kept buying each model that came out. I especially liked the time when Apple sold visually complimentary displays for these machines and you could have a quite pleasing dual screen setup.

Alas those days are gone, now to be honest, every iMac dual screen setup just looks like rubbish, so I just can’t with that anymore and so it was time to change. I now have 2 useless Display port old Cinema displays, guess they go to the trash.

I have a MacBook 16" M2 PRO 32GB/1TB SSD, which I think is pretty much perfect - despite coffee damage knocking out all the ports on the right side, it’s still awesome.

I’ve always had a disconnect between Laptop and Desktop speeds and this time round the stars aligned reasonably well that one can almost have parity between the 2, so after some looking around I figured a Mac Mini will give me near enough performance and good enough options for displays.

To be honest I was not too convinced about moving to a Mini - but the iMacs were never really speed daemons so maybe it would be fine? So I picked up a Mac Mini M2 PRO 32GB/1TB SSD and really moving between the 2 just feels exactly the same performance wise. This has been a bigger deal than I anticipated for my general happiness when switching setup - something I do all the time.

The mini sitting on my desk is a bit annoying, I’m considering options to mount it on the wall behind the display.

For my display I went with a LG 40" Curved UltraWide 5K2K Nano IPS Monitor. I’ve used a 32 inch ultra wide for a few years, this has been a great upgrade. It’s quite pricey around EUR1200, but it’s a really nice display and as it’s a Thunderbolt monitor it just works flawlessly with the Macs. I did buy BetterDisplay to get some more control, well worth it - I’d say essential if you have a monitor like this. I do wish it was brighter though.

I used to have a Moode Audio system with some high end DACs and Amp paired with JBL monitors for audio, I got those before HomePods existed. Now tossed all that out for just 2 HomePod Minis on my desk, they’re fantastic.

I use a Logitech Brio 4K Ultra HD Webcam - works well on MacOS including it’s management software. My previous camera did not handle the sync frequency of my new office lights well and so was instantly useless.

Today I am replacing my Razer Abyssus (2014) with a new Abyssus that doesn’t look like it’s been to war.

Keyboard wise I use the MS Natural 4000 and have 4 more in boxes so I don’t need to think about keyboard for years, they last about 3 to 4 years each.

Storage

I’ve been a Qnap user since forever, like maybe 2005 or so. The first Qnap I bought was some weird ARM CPU and they ran a patched Linux Kernel to get big partition support. When the machine died I had quite a bit of trouble to access my files but did get there eventually - but did make me reconsider my backup strategy.

Their next gen kit was Intel based and ran stock kernels so I gave them another go. What I got then was a 4 bay TS-439. This machine moved with me from London to Malta and now 14 years later it still got a security update just before I retired it, unbelievable.

Later I got a TS-451+ as the old ones CPU was a bit slow, the old one moved to my office and I synced to it daily as a backup. But as the old one is now 14 years old it seems I am just pushing my luck with this box, so I retired it. Moved the TS-451+ to the position of backup machine and got a new TS-453E.

The TS-453E is interesting because it’s a kind of LTS hardware that will have full hardware support till 2029 and probably software far past that. I got it with an extended warranty also so this should set me up for 5 years at least on this platform.

I set the primary NAS up in 2 x RAID-1 configuration and basically perform backups like this:

Real time RAID-1 mirror between 2 disk for the primary volume
Daily sync my Dropbox onto the primary NAS
Daily sync primary to the backup NAS that is running a RAID-5 setup in another location in Malta
Daily sync the primary to a 5TB external USB drive that I can take to a 3rd offsite when I leave the office for months over summer
Monthly I do a full bitrot check across all the files on the primary
Monthly I do a manual sync between the RAID-1 volumes of the primary NAS
Monthly I sync the full primary NAS to Finland on a machine with a RAID-6
My primary desktop does TimeMachine backups to the primary regularly

This achieves the following redundancy:

+1 disk redundancy for any change I made now thanks to the first RAID-1
daily same-country offsite to a RAID-5 storage
monthly OS-level integrity checks of all files and the chance to restore any to the primary
monthly on-site backup for long term recovery should I notice a accidental file deletion or corruption
monthly backup to another country.

In the end every file lands on about 11 disks in duplicate, 3-4 locations, 2 countries, different raid levels and with bitrot detection at multiple RAID levels both standard Linux kernel and NAS kernels and at the OS level. I could make the final Finland archive a 3 month rolling archive since I have enough space there for it.

The machine I mention in Finland is a Hetzner SX64 (64GB RAM Ryzen 5 3600) with 4 x 16 TB SATA Enterprise HDD 7200rpm drives it runs Bacula for my server backups, but for above I just rsync to it. It’s a slow crappy machine tbh but for the purpose of mostly just keepting files on disks mostly idle, it’s perfectly fine and the price is great. I’ve had similar machines for a decade or more, I cycle them every 3 years to new hardware.

Linux Shell

I mentioned earlier that I live by: MacOS Shells are for SSH to Linux machines, so I need a Linux machine.

I used to use the NUC machines but as they were ancient and dying I needed something else. Slack suggested I look at the Asus Mini PC PN51 and what a great suggestion that was.

I have the Ryzen 7 5700U based machine with 64GB RAM and 1TB M.2 SSD in it, this machine runs CentOS and later Almalinux flawlessly. I used to use it bare but that was a bit of a waste now it runs Cockpit as per my previous post and I have a 16GB VM there dedicated as my shell box.

Along with that I have 6-10 VMs elsewhere making up a dev environment.

Miscellaneous Hardware

At my office I have a few other things:

One of my old NUC machines is a Linux Desktop for my son, mainly for minecraft/youtube when he visits
A Krups Nescafe Dolce Gusto Infinissima Touch Black Automatic Coffee Machine - it’s rubbish, don’t buy these coffee machines
A bar fridge
Xiaomi air purifier
Xiaomi robot vacuum - the nice one with auto bag empty feature so the place keeps clean when I am away for months at a time over summer
A Prusa MK4 and it’s many accessories (this is a whole blog post to be made)
A 15 year old Brother printer that is enormous and has finally now probably met it’s end, probably fixable, but I want an excuse to buy a smaller printer
Ubiquity Dream Machine and a 8 port switch

Everything is new except the Brother printer, but we’ll fix that soon.

I try to not clutter the place up so I keep things to a minimum here and as mentioned I might move my workshop across the road to another unit and get some more 3D printers, Lazer cutters and so.

Conclusion

That’s about it for the physical location and physical hardware. I’ll need to redo my home office as I gutted it and brought most things here but that’s for later, it’s mainly used now for its real purpose of being a media room with a 100 inch 4K project Dolby Atmos audio setup.

Lab Infra Rebuild Part 3

Sun, 07 Apr 2024 09:00:00 +0100

This is an ongoing post about rebuilding my lab infrastructure, see the initial post here.

Today I’ll talk a bit about Configuration Management having previously mentioned I am ditching Kubernetes.

Server Management

The general state of server management is pretty sad, you have Ansible or Puppet and a long tail of things that just can’t work or are under terrible corporate control that you just can’t touch them.

I am, as most people are aware, a very long term Puppet user since almost day 1 and have contributed significant features like Hiera and the design of Data in Modules. I’ve not had much need/interest in being involved in that community for ages but I want to like Puppet and I want to keep using it where appropriate.

I think Puppet is more or less finished - there are bug fixes and stuff of course - but in general core Puppet is stable, mature and does what one wants and have extension points one needs. There’s not really any reason not to use if it fits your needs/tastes and should things go pear shaped its a easy fork. One can essentially stay on a current version for years at this point and it’s fine. There used to be some issues around packaging but even this is Apache-2 now. If you have the problem it solves it does so very well, but the industry have moved on so not much scope for extending it imo.

All the action is in the content - modules on the forge. Vox Pupuli are doing an amazing job, I honestly do not know how they do so much or maintain so many modules, it’s really impressive.

There’s a general state of rot though, modules I used to use are almost all abandoned with a few moved to Vox. I wonder if stats about this is available, but I get the impression that content wise things are also taking a huge dive there and Vox are holding everything afloat with Puppet hoping to make money from selling commercial modules - I doubt it will sustain a business their size but it’s a good idea.

Read on for more about Puppet today in my infra.

Given the general state of things in the server management world I decided to use Puppet again for this round of infrastructure rebuild. I can’t see this lasting alas. Most people are aware that Puppet have been bought by Perforce and have had a huge shift in people and such. It’s inevitable that revenue generation is the main push at the moment.

Unfortunately the way this play out is pretty unpleasant. Here’s an example: I have an ancient, and crappy, monitoring script that runs on the node and checks last_run_summary.yaml to infer the current status of runs - are they failing etc. It’s worked for a very long time and last_run_summary.yaml is a contract that’s to be maintained. Something in recent Puppet broke this script (the last_run_summary.yaml now behaves differently) so I thought I’d ask on their Slack if there is something newer/maintained before I fixed mine.

Immediately from Puppet people you get a message saying to use Puppet Enterprise for this. In a community Slack where people are just asking questions about a 200 line script. The suggestion is to move to a price-not-disclosed product vs a 200 line script. Without even so much as a question about needs or environment or if the suggestion would be relevant. It’s just corporate enshitification. Eventually I got some good answers from the community, despite Puppets best efforts.

This outcome is of course entirely predictable, I can only hope Vox doesn’t get burned in the inevitable slide into the sewer.

Managing Puppet

I have 2 old EL based machines that could not make the trip to Puppet 8 due some legacy there, the rest got moved to EL9 with a Puppet Server. I am though a big fan of running puppet apply based builds and will likely move to that instead of the server. Apply based workflows present a few problems though, primarily how you get the code on the nodes and how the workflow is around that.

I’d like a git based flow where I commit a change, CI package it and puts it on a repo and the fleet updates to it. Ideally the fleet updates to it asap. Further I want visibility into the runs, node-side monitoring which fits my event based world-view and allows me central control to trigger runs and do scheduled maintenance.

So I built a system to orchestrate Puppet that’ll release soon called Puppet Control.

Provides a nice cli for triggering runs, querying runs etc
Git+CI based flow handles fetching, validating and deploying code to nodes for apply, including tamper detection
Has concurrently control built in for fine-grained resource management of file servers used to deliver code bundles
Includes a run scheduler with concurrency controls that include ability to say for eg: only 1 database server out of all database server can run Puppet at a time but webservers can run 10 concurrently
Can do on-demand runs as soon as possible subject to concurrency control to ensure the shared infra performs at peak
Has various ways to find nodes in certain states like pctl nodes failing to find nodes with failing resources
Can show real-time events of Puppet runs
Can have maintenance declared that will stop all scheduled Puppet runs
Expose run statistics to Prometheus for runtimes, changes, health and more

There’s an optional video below with more details and show the code release flow etc:

I’ll release this eventually, it’s dependent on some work happening in Choria at the moment.

The concurrency control is a big deal. Scheduling Puppet runs is quite a difficult task. Usually the solution Puppet users do is just to spread the runs in cron by some random time distribution.

That leaves the problem of fast Puppet runs during maintenance windows though. In the past we had a command mco puppet runall 100 which would discover all the nodes then in a loop ask their state and schedule more as some stopped - the goal being to keep as close to 100 running at a time. The choice of 100 nodes is related to the capacity of the Puppet Server infrastructure.

This worked fine but it was very resource intensive on the Choria/MCollective network as 1000s of RPC requests have to be made to know the current state and it was not suited to using in an ongoing fashion. With Puppet Control every run is a run that happens at the desired concurrency, but without a central orchestrator trying to make all the choices. It’s significantly cheaper on the network.

More significantly by allowing the concurrency group name to be configured on a per node basis one can have different policies by type of machine. This is a big deal, let’s say we are using puppet apply but we do not wish to have our 5 database machines all do concurrent runs and potentially restarting at the same time. By creating a concurrency governor for just those machines set to 1 we prevent that.

With this in place triggering all the nodes to run at their groups configured concurrency is a simple pctl nodes trigger which takes 2 seconds to complete. From there the nodes will run without overwhelming the Servers.

Another interesting thing here is that this model maps well onto Ansible local mode as well. So in theory, unexplored theory, this same central controller and scheduler could be made for Ansible.

This is built on Choria Concurrency Governors which is an amazing distributed system building block.

Choria

No surprise that I am using Choria for a large part of this, with a bit of a twist though. Choria, as released on choria.io, is actually a distribution of a much larger system that is tailored for Puppet users. That official Choria release requires Puppet Agent and will not support unofficial builds or unsupported deviations from that.

With the writing being on the wall though for Puppet this leaves me with a problem, I have no easy to use Public distribution of Choria for non Puppet users. Puppet provides the following to Choria:

Deployment of the packages and files to nodes
Management of policies and plugins on nodes
Certificate Authority with certs on every node
Optional discovery source of truth in PuppetDB
Libraries for managing packages and services

These are actually quite significant hurdles to cross to create a fully Puppet-free Choria distribution.

That said for a long time Choria have had another life as a large scale orchestrator that is not Puppet related. This implies:

It can self-provision at a rate of 1000+ nodes / minute
Deploy its own plugins at a rate of multiple plugins delivered to millions of nodes in minutes
Manage its own security both integrated with a CA or using a new JWT+ed25519 based approach
Integrate with non Puppet data sources using external extension points
Can upgrade itself in place without any help from Puppet in an Over-The-Air type self-upgrade system
Has centralised RBAC integrated with tools like Open Policy Agent

In the last 2 years I was on a related contract and all these components have been firmed up and made much more capable, reliable and horizontally scalable and used in anger in the real world in some quite serious mission critical builds.

Thus, my Choria infrastructure is actually a highbrid between Puppet and Non Puppet. Puppet places the RPMs and plugins that require Puppet (package, service, legacy plugins), but Choria self provisions everything else and owns the life of the agent and more. I am running the new Protocol and with Open Policy Agent based RBAC. I’ve made some changes to the various Puppet models to enable this and will start looking for some early adopters.

This means my many Raspberry PI - from Xmas lights to sensors and HVAC control - are all now managed by Choria as well as the provisioner caters for them since those are without Puppet.

I wrote about the new protocol in an ADR and you can see there is scope for integration into TPMs and more. This is the future world view of Choria and already in use on some 100s of thousands of machines.

Conclusion

So that’s a bit about managing the machines without Kubernetes or a ISP managing them.

As I’ve been out of active Puppet use for a few years it’s been interesting to come back with some semi-fresh eyes and rethink some of the old things I believed was true when I used it constantly.

Choria will play a critical role in the path forward as I’ll move much into containers managed as per the previous blog post leaving the problem Puppet solves to quite a thin layer. I’ve some thoughts on doing something to at least replace the most basic package-config-service trio of CM with something in Choria long term.

Lab Infra Rebuild Part 2

Thu, 21 Mar 2024 09:00:00 +0100

Previously I blogged about rebuilding my personal infra, focussing on what I had before.

Today we’ll start into what I used to replace the old stuff. It’s difficult to know where to start but I think a bit about VM and Container management is as good as any.

Kubernetes

My previous build used a 3 node Kubernetes Cluster hosted at Digital Ocean. It hosted:

Public facing websites like this blog (WordPress in the past), Wiki, A few static sites etc
Monitoring: Prometheus, Grafana, Graphite
A bridge from The Things Network for my LoRaWAN devices
3 x redundant Choria Brokers and AAA
Container Registry backed by Spaces (Digital Ocean object storage)
Ingress and Okta integration via Vouch
Service discovery and automatic generation of configurations for Prom, Ingress etc

Apart from the core cluster I had about 15 volumes, 3 Spaces, an Ingress load balancer with a static IP and a managed MySQL database.

I never got around to go full GitOps on this setup, it just seemed too much to do for a one man infra to both deploy all that and maintain the discipline. Of course I am not a stranger to the discipline required being from Puppet world, but something about the whole GitOps setup just seemed like A LOT.

I quite liked all of this, when Kubernetes works it is a pleasant experience, some highlights:

Integration with cloud infra like LBs is an amazing experience
Integration with volumes to provide movable storage is really great and hard to repeat
I do not mind YAML and the diffable infrastructure is really great, no surprise there. I hold myself largely to blame for the popularity of YAML in infra tools at large thanks to Hiera etc, so I can’t complain.
Complete abstraction of node complexities is a double-edged sword but I think in the end I come to appreciate it
I do like the container workflow and it was compatible with some pre-k8s thoughts I had on this
Easy integration between CI and infrastructure with the kubectl rollout abstraction

Some things I just did not like, I will try to mention some things that not the usual gripes:

Access to managed k8s infra is great, but not knowing how its put together for the particular cloud can make debugging things hard. I had some Cilium failures that was a real pain
API deprecations are constant, production software rely on Beta APIs and will just randomly break. I expected this, but over the 3 years this happened more than I expected. You really have to be on top of all the versions of all the things
The complimentary management tooling is quite heavy like I mentioned around GitOps. Traditional CM had a quick on-ramp and was suitable at small scale, I miss that
I had to move from Linode K8s to Digital Ocean K8s. The portability promises of pure kubernetes is lost if you do not take a lot of care
Logging from the k8s infra is insane, ever-changing, unusable unless you really really are into this stuff like very deep and very on-top of every version change
Digital Ocean does forced upgrades of the k8s, this is fine. The implication is that all the nodes will be replaced so Prometheus polling source will change with big knock on effect. The way DO does it though involves 2 full upgrades for every 1 upgrade doubling the pain
It just seem like no-one wants to even match the features Hiera have in terms of customization of data
Helm

In the end it all just seemed like a lot for my needs and was ever slightly fragile. I went on a 3 month sabbatical last year and the entire infra went to hell twice, all on its own, because I neglected some upgrades during this time and when Digital Ocean landed their upgrade it all broke. It’s a big commitment.

See the full entry for detail of what I am doing instead.

Container Management

So let’s look at what I am working on for container management. My current R&D focus in Choria is around Autonomous Agents that can manage one thing forever, one thing like a Container. So I am dog-fooding some of this work where I need containers and will move things into containers as I progress down this path.

Looking at my likes list from the Kubernetes section above we can imagine where I am focussing with the tooling I am building, lets just jump right in with what I have today:

containers:
  tally:
    image: registry.choria.io/choria/tally
    image_tag: latest
    syslog: true
    kv_update: true
    restart_files:
      - /etc/tally/config/choria.conf
    volumes:
      - /etc/tally/config:/tally/config
    ports:
      - 9010:8080
    register_ports:
      - protocol: http
        service: tally
        ip: %{facts.networking.ip}
        cluster: %{facts.location}
        port: 9010
        priority: 1
        annotations:
          prometheus.io/scrape: true

This is Puppet Hiera data that defines:

A running Container for a Choria related service that passively listens to events and expose metrics to Prometheus
It will watch a file on the host file system and restart the container if the file changes, Puppet manages the file in question
It supports rolling upgrades via Key-Value store updates but defaults to latest
It exposes the port to service discovery with some port specific annotations

This creates a Choria Autonomous Agent that will forever manage the container. If health checks fail the port will not be published to Service Discovery anymore and remediation will kick in etc.

Of course Puppet is an implementation detail - anything that can press the YAML file and place it into Choria can do this. Choria can also download and deploy these automations as a plugin at runtime via securely signed artifacts. So this supports a fully CI/CD driven GitOps like flow that has no Puppet involvement.

To replace the kubectl rollout process I support KV updates like choria kv put HOIST container.tally.tag 0.0.4 (we have Go APIs for this also), the container will listen for this kv update and perform an upgrade. Upgrades support rolling strategies like say 2 at at time out of a cluster of 20 etc. The kv put is the only interaction needed to do this and no active orchestration is needed.

Service discovery is also in a Choria Key-Value bucket and I can query it:

$ sd find
[1] tally @ ve2 [http://192.168.1.10:9010]
   prometheus.io/port: 9010

Or generate Prometheus configurations:

$ sd prometheus
- targets:
    - 192.168.1.10:9010
  labels:
    __meta_choria_cluster_name: ve2
    __meta_choria_ip: 192.168.1.10
    __meta_choria_port: "9010"
    __meta_choria_priority: "1"
    __meta_choria_protocol: http
    __meta_choria_service: tally

So one can imagine how that integrates with Prometheus file based SD or HTTP based. In future I’ll add things to manage ingress configurations automatically etc, and of course Service Discovery -> file flow can ge managed using Autonomous Agents also.

Actual building of containers have not changed much from earlier thoughts about this and the above system - called Hoist - will focus on strengthening those thoughts.

Virtualization

Previously I ran some various mixes of KVM things: I had Puppet code to generate libvirt configuration files, later I got lazy and used the RedHat graphical machines manager - I just don’t change my VMs much to be honest so don’t need a lot of clever things.

As I was looking to run 3 or 4 baremetal machines with VMs on-top I wanted something quite nice with a nice UI and started looking around and found numerous Youtubers going on like crazy about Proxmox. I tried Proxmox for a week on some machines and had some thoughts about this experience.

It is nice with a broad feature set that is quite a good all round product in this space, I can see if done right this will be formidable tool to use and would consider it in future again.

Is seems like this is a company who has engineers, paid to engineer, and they will engineer things all day long. Ditto product owners etc. There’s a lot there and lots of it feels half-baked, awkward or incomplete or in-progress. Combined with there just being A LOT it seemed like a mess. I bet this is a company who just love sprint based work and fully embrace the sprints being quite isolated approach. It shows in obvious ways.

It’s Debian based. I have used EL from their first Beta. Slackware before. SLS before (1993!). In kubernetes you can get away with not caring so much about your hosts but in a virtualized environment you will need to make sure you have ways to manage updates, backups and such of those baremetals. I do not have any tooling specifically built for Debian so I really do not want that lift. I also just do not care even remotely for Debian or its community.

Ultimately I do not need integration with Ceph etc, I don’t need (oh hell no do I not need) a database-driven file system developed by Proxmox and for my needs I do not need SDN. All these things are useful but it seemed like it would tick some of the boxes I listed in Kubernetes conns.

After some looking around at options I came across Cockpit which comes integrated already into EL based distros and while it’s not as full featured as Proxmox I find that to be a feature rather than a shortcoming. It does just the right things and I can easily just not install things I do not want.

I think its firewall management needs a bunch of work still, but that’s ok I do not want to manage firewalls in this manner (more later) so that is just fine, I also do not need its package management - no problem, just uninstall the feature.

Really not much to say or complain about here - and having nothing to say is a huge feature - it makes virtual machines, allow me to edit their configurations, see their consoles etc. Just what I need and no more. One annoying thing with it is that I cannot figure out how to trigger a re-install of a machine, I did not look too deep though.

EDIT: I do have a few things to say about it after all:

It took like 2 minutes to install on an EL machine.
When you are not using it, there is nothing in the process tree. It’s fully based on socket activation. No resources wasted.
It does not take over and camp on everything. It uses the same commands and APIs Puppet/Ansible/You does, you can keep using CLI tools you know. Or progressively learn.
It uses your normal system users via PAM so not much to deploy regarding authentication etc.
It support EL/Debian/Ubuntu and more.

This is huge, it just is super lightweight and gets out of your way and does not prescribe much. It does not invent new things and does not invent new terminology. Huge.

Conclusion

So that is roughly what I am doing for Virtualization and Container management after Kubernetes. Container management is a work in progress so a lot of my components are now just stuff running on VMs but as I progress to improve Hoist a bit I’ll gradually move into it for more things, this has been something I’ve wanted to finish for a long time so I am glad to get the chance.

Further as my Lab is really that after all, a place for R&D, using the area I am focussing on in Choria in anger to solve some real problems has been invaluable and I wanted more of that, this influenced some of these decisions.

Lab Infra Rebuild Part 1

Wed, 20 Mar 2024 09:00:00 +0100

I’ve been posting on socials a bit about rebuilding my lab and some opinions I had on tools, approaches and more. Some people have asked for a way to keep up with my efforts, so I figured it might be time to post here for the first time since 2018!

In this post I’ll focus on what came before, a bit of a recap of my previous setup. Additionally, to a general software refresh I have also been in Malta now 8 years and a lot of my office hardware was purchased around the time of moving here, so we’ll also cover replacing NAS servers and more.

My previous big OS rebuilt was around CentOS 7 days, so that’s about 3 years ago now, high time to revisit some choices.

My infra falls in the following categories:

Development machines: usually 10 or so Virtual Machines that I use mainly in developing Choria.io
Office support equipment: NAS, Printers, Desktops, Laptops etc
Networking equipment: mainly home networking stuff for my locations
Hosting publicly visible items: This blog, DNS, Choria Package repos etc
Management infrastructure: Choria, Puppet, etc
Monitoring infrastructure: Prometheus and friends
Backups: for everything
General all-purpose things like source control etc
My actual office

Below I’ll do a quick run through all the equipment, machines, devices etc. I use regularly. I’ve largely replaced it all and will detail that in the following posts. It’s not huge infra or anything, all told about 20 to 30 instances in 5 or 6 locations.

Development Machines

I’ve had 3 x Intel NUC machines, each with 512GB solid state and 8GB RAM since 2016. They have served me well and really I could see them going for a while longer.

They were each a KVM host and had on them 3 to 8 CentOS VMs. One had some bigger ones as that was my general shell machine and the other were mainly there to add some servers to my node count for Choria development. Choria being inherently a distributed system having some more nodes help.

For my needs they were great but unfortunately they have a hardware problem, their BIOS battery die and it is soldered on to the board so not exactly user serviceable.

Two of these are just retiring - I will remove the SSD and see what it can be used for, more on that later - but otherwise they are done.

One of them is actually a bit newer so it’s finding a new home as a little desktop for my 6 y/o mainly for Scratch and Minecraft etc.

I have been using RedHat since version 0.9 Halloween Beta release then used CentOS for ages - even donated hardware - and after they seemed to be intent on self-destruction I started moving to Alma Linux.

Public Hosting and Management

I host my blog and a few other public bits myself. When I rebuilt all my things 3 years ago I wanted to get some more real Kubernetes experience so I got 3 x 8GB droplets from Digital Ocean with a Kubernetes cluster.

There I hosted the sites, used their managed MySQL, used their Object store and volumes.

It’s been fine generally, I don’t think I like the complexity for what I needed but it was good experience - more on this later. This cluster started with Linode managed KE but it was early days for them in Kubernetes and so I bailed - their support was terrible at the time wrt Kubernetes.

This was surprising as I’ve been a Linode customer since almost their day one and they have been consistently amazing. But when they launched managed Kubernetes it seemed their EU timezone support people were like level 1 only and I always ended up waiting till US time to get things resolved, several times many hour outages. It’s a sad outcome, at the time I moved all I felt comfortable away from them fearing it was a start of a downward slide in quality. Apart from things I am reluctant to move Linode has essentially lost me as a customer.

So I moved to Digital Ocean. It was fine, I have only really one pain with them - they do forced updates of the Kubernetes version (fine) but the way they do it always result in 2 full upgrades and node replacements. This means my outgoing IPs for things like Prometheus kept changing which was extremely annoying.

The Kubernetes infra also ran a 3 node Choria Broker cluster, Prometheus+Alert Manager and Graphite. Alerts to Victorops that I pay for.

Apart from that I had Linode machines for Puppet, Name Servers, Choria Package repos and some general use machines that I should have killed years ago.

I guess the bills for this came to like $400 a month, helps to have a company to bill this to.

Office Equipment

My main office desktop is one of the last Intel iMacs with an old, like 2009 era, Apple Thunderbolt display. I’ve had iMacs since the very first one and they were one of my favourite form factors. It’s a bit sad about this one as it was a really nice machine but Apple is moving fast to stop supporting Intel so now there are no more OS updates.

The screen being ridiculously old and Thunderbolt only - no HDMI - is now just useless.

The current iMacs do look nice but the problem is there is no screen I can put next to them that don’t look like crap and I want either a big monitor or 2, it’s just not working anymore to use iMacs.

I have an old tank Brother printer that is maybe 15 to 20 years old. It just refuses to die but I think its paper feed has now dried up so maybe time to go.

Other than things actually stuck to my office I use a range of new Macbooks - currently a 16 inch M2 Pro with 32GB RAM.

For file storage I use Dropbox of course but QNAP devices for larger storage. I have an old TS-439 that is now 14 years old and still getting security updates (!!!), this is at my office. At home I have a TS-451+. Each provide around 4TB of usable space.

At home I also have a 34 inch ultra-wide display that I put my Laptop on when I sit at the desk here. I got this 7 years ago now so getting a bit old.

Networking Equipment

I have 4 locations I work from often with bits of equipment scattered everywhere. Some time ago I had a mix of networking equipment but I’ve been standardising on Ubiquiti. I am still sad about Apple retiring their Wi-Fi range. Mikrotik was nice enough but I wanted something a bit more polished.

In Malta I have 3 Dream Machines (the round tube one) and around 16 switches, APs, etc. My house is 450 year old with 1.5 meter thick walls so every room gets an AP. I like the single vendor system as all have a single management pane and generally things like VPNs etc. just work.

In Latvia I have a Dream Machine Pro as I have a set of surveillance cameras there, I am considering updating to the Pro at my main house at least also as I’d like to add a camera outside.

In Malta I used to be on Melita for everything, now on Go who have a really great new all fibre to the house network. I have one location left to move and then I’ll have 1Gb links everywhere. In Latvia a 4G link that’s actually quite surprisingly good (190 Mbps down 11 up).

Firewalls tended to be a mix of the Ubiquiti devices and iptables.

General Infra

My mail has been hosted at Fastmail for years and they are amazing. The web ui is a bit of a mess to be honest but at least it doesn’t keep changing. Their mail hosting features are rock solid though.

I used GitHub for everything since they introduced unlimited Private repos. I do pay for GitHub though.

Backups

I have a 30TB Hetzner machine that runs Bacula. It does daily backups of everything I have and does a full 3 month rotation of Full backups with 1 month of Incrementals onto a RAID-10 disk setup.

The main QNAP syncs my Dropbox onto its disks daily.

The main QNAP is set up as 2 x RAID-1 volumes. The main volume is synchronised daily to the office QNAP and I do monthly manual disk rot checks and sync between the RAID-1 volumes. This gives me a 1 month old full backup of the entire QNAP at home and daily off-sites. Monthly I also sync the QNAP to the Hetzner machine.

This means every file hits 9 drives in 3 locations over 2 countries with access to file versions going back 3 months. Ample time to recover from user error like deleting the wrong files and also a lot of redundancy built in.

Physical Office

I’ve had an Office in a town called Mosta here for 4 years now, I have not used it much because parking around it was hell. On paper it was 5 minutes from School and I would stay there while the boy is at School - in practise it could take me 40 minutes to get parking and 12 to drive back home. And when I found parking it could be very far from the office - walking along pavements in Summer is no joke here.

It just never worked, I ended up working from home most of the time. I should have bailed out of it years ago but just never got around to it.

Conclusion

So that about rounds it up, keep reading to hear what literally everything is being replaced with and what new things are being added to the mix to boot!

Part 2 covers Kubernetes, Virtualization and Containers
Part 3 covers Server Management, Puppet and Choria
Part 4 covers My Office and Office Hardware
Part 5 covers VMs, Baremetals and Operating Systems
Part 6 wraps up the series with a look at SaaS and other tools