Infrastructure

My network is quite vast and spans many physical sites. The network diagram below will give you a good visual feel for it. All of my internet gateways are re-purposed thin clients running the purpose-built VyOS Linux distribution. My entire network is dual-stacked running native IPv4 and native IPv6 where possible with fallback to tunnel brokers. I interconnect my physical sites using WireGuard tunnels and run OSPFv2/3 over those tunnels to propagate routing tables. My core Brocade switch connects my kubernetes cluster to the internet via BGP using the Calico CNI on the kubernetes side to advertise all of the relevant subnets living in the cluster. Everything on the kubernetes side is pure layer-3 networking, there are no overlays or tunnels between nodes. Finally, my network is divided into several VLANs to isolate different types of traffic, such as home traffic, lab traffic, storage traffic, kubernetes traffic and OOB management traffic.

I am currently in the process of obtaining a personal ASN as well as my own PA and possibly PI IPv6 address space as part of a project to learn more about BGP and more importantly to avoid relying on unstable residential ISP-supplied PA prefixes. This will probably involve running BGP sessions on a VPS from a cloud provider that allows BGP peering and tunnelling that home. Unless I can convince my residential ISPs to let me establish a BGP session with them but that it more likely to be a pipe dream.

To be complete, this repo needs to be as self-contained as possible. What I mean by this is that to manipulate or apply any kind of configuration in this repo, you shouldn’t need anything on your system other than git to clone it and the Nix package manager. The repo contains a flake at the root, flake.nix which defines apps and development shells that are immediately accessible via Nix and don’t perform or require any alteration to the state of the system the repo is cloned onto. The development shell makes available to the user all of the CLI tools required to interact with the repository. It also decrypts and makes available in the environment all of the relevant secrets required to authenticate with the infrastructure, given that the user can provide private key material to decrypt them.

To enter the development shell, either have direnv configured for your shell and run direnv allow in the root of the project or else run nix develop. You may need to enter your PIN and touch your Yubikey to decrypt secrets during the shell hook execution.

This listing intentionally doesn’t cover the entire structure of the repo, it is meant to be a high-level overview of where various components are located.

You might notice at a glance that it appears that this repository is full of secret things like API keys and stuff that shouldn’t be publicly available. That is partially correct. This repository contains a variety of sensitive values but none of them are committed in plain text. Instead, they are encrypted with SOPS, which protects them from prying eyes at rest. However, with the right decryption keys, these secrets can be decrypted at runtime. Activating this project’s nix shell using nix develop or via direnv will automatically decrypt the secrets and place them in the appropriate environment variable to be used by tools and config in the repo.

The .sops.yaml file at the root of the repo defines creation rules for secrets to be encrypted with sops. Any files matching the defined creation rule paths will be encrypted with the PGP fingerprints specified. The user encrypting new files mut have all of the PGP public keys included in their keyring in order to encrypt the secret.

To encrypt a new file or edit an existing one in $EDITOR, run sops <filename>. To encrypt a file in place, run sops -i -e <filename>.

Occasionally, we might want to add a collaborator to the project, or a new tool that needs to be able to decrypt secrets. In that case, it will be necessary to re-key all existing files to encrypt them with the new public key after adding it to .sops.yaml. The following snippet can be used for this purpose.

I try to keep all configuration related the to deployment of a particular "application" or "unit" together. This is why, for example, many deployments in the k8s-235/apps folder have their own terraform folders for DNS and other cloud configuration. For underlying infrastructure, this is less obvious so the dns folder is a collection of DNS zones and records that don’t exactly fit with a particular project and are all applied in one collection of terraform.

The following zones, including their DNSSEC configuration are managed by this collection:

The 235 kubernetes cluster (referred to as k8s-235) is my only personal kubernetes cluster at the moment. It consists of 3 consolidated master, worker and etcd nodes. Etcd is the backing data store for kubernetes. The cluster is deployed on 3 physical (bare metal) servers running NixOS, these hosts are provisioned by the Nix expressions and modules located in the nixos folder at the root of the repo. The kubernetes cluster deployed using a combination of the kubernetes module included in upstream nixpkgs and a custom module that I wrote specific to my infrastructure. Most importantly, it handles end-to-end mutual TLS certification of all cluster components using Hashicorp Vault and vault-agent.

The physical servers providing compute for the kubernetes worker and master nodes are provisioned by Nix. The flake in this repo contains NixOS configurations for the pysical servers soarin, sassaflash and stormfeather.

The core components of the kubernetes cluster include the control-plane itself, etcd, the Cilium CNI networking plugin and the democratic-csi CSI storage plugin. Cilium is installed at cluster provisioning time using a manual invocation and democratic-csi is later installed by ArgoCD. Cilium can then be further upgraded by ArgoCD.