[{"title":"pvectl: A CLI for Managing Proxmox","href":"/blog/pvectl-a-cli-for-proxmox/","summary":"My home Proxmox cluster has grown into two nodes, a few dozen LXC containers, and a handful of VMs. Recently I’ve been exploring ways to make this experience more streamlined, especially when creating new containers, shelling into containers, running migrations, restoring from backups, etc. Usually when I need to do this, I either SSH into one of the Proxmox nodes or open up the GUI. When on a node, I usually run pct or qm, and then immediately run into the issue: which ID was the thing I actually wanted? Which node was it running on?\n","content":"My home Proxmox cluster has grown into two nodes, a few dozen LXC containers, and a handful of VMs. Recently I\u0026rsquo;ve been exploring ways to make this experience more streamlined, especially when creating new containers, shelling into containers, running migrations, restoring from backups, etc. Usually when I need to do this, I either SSH into one of the Proxmox nodes or open up the GUI. When on a node, I usually run pct or qm, and then immediately run into the issue: which ID was the thing I actually wanted? Which node was it running on?\nBash wrapper My initial attempt to relieve some of the complexity was a bash script that shelled out to pct and piped the result through fzf so I could fuzzy-search containers by name instead of scanning a list of IDs. It worked well enough.\nThe problem with this approach is that it only worked when I was already on the Proxmox node itself, since it called pct directly — no running it from my laptop, and it only knew about LXC containers.\nRewriting it against the API After years of using kubectl, I really wished there was something comparable for Proxmox. I decided to start making pvectl, which communicates directly with the Proxmox VE API instead of shelling out to pct/qm. It can work from any machine that can reach the cluster\u0026rsquo;s API. ct and qm are separate command trees under the same CLI (pvectl ct start web, pvectl qm start pihole).\nA few other things have since been implemented:\nTab completion suggests container/VM names as you type, so you never touch an ID unless you want to. Anything that runs as a background Proxmox task (start, migrate, backup, snapshot) shows a live spinner and a final pass/fail summary with timing, instead of leaving you guessing whether it\u0026rsquo;s still running. pvectl setup will try to store the API token in your OS keychain instead of a plaintext config file. Support for machine-readable output on list/summary commands instead of a table (--output json) A raw escape hatch for any Proxmox API endpoint using pvectl api get/post/put/delete \u0026lt;path\u0026gt; pvectl schema prints the full command tree (names, flags, descriptions) as JSON for introspection. Caveats Before starting out, I didn\u0026rsquo;t realize that certain commands had no REST equivalents and still required shelling out to ssh \u0026lt;node\u0026gt; .... This includes pct enter, pct exec, and some config updates that touch lxc.*. I still included these commands in pvectl for completeness, but it requires having a valid SSH config setup.\npvectl is not a drop-in replacement for any existing tool. Most of the tools that do exist (pct, qm, and pvesh) all have access to a vast API and need to run on the host. What it does handle well is the day-to-day lifecycle — start/stop, snapshots, backups, migrations, config edits, console access — because that\u0026rsquo;s what I mostly need on a repeating basis. It doesn\u0026rsquo;t touch cluster/storage/network configuration, and for anything with no REST equivalent, it still shells out to ssh \u0026lt;node\u0026gt; ... under the hood. If you need the full surface area of the Proxmox API, pvesh is likely the right tool.\nTrying it out macOS:\nbrew install davegallant/public/pvectl Linux:\ncurl -fsSL https://raw.githubusercontent.com/davegallant/pvectl/main/scripts/install.sh | sh Nix:\nnix profile install github:davegallant/pvectl The source and full documentation can be found at github.com/davegallant/pvectl.\nI\u0026rsquo;m actively using it against my own cluster. If you decide to try it against yours and something breaks — or behaves differently on a setup I haven\u0026rsquo;t tested — feel free to open an issue or PR.\n","tags":["proxmox","cli","go","homelab","pvectl"]},{"title":"Using Home Assistant and Grafana to Monitor Radon","href":"/blog/using-home-assistant-and-grafana-to-monitor-radon/","summary":"I recently became more concerned about radon since I live and work in a basement daily. I decided to explore ways not only to monitor radon levels but also to hook up the metrics to my existing homelab.\n","content":"I recently became more concerned about radon since I live and work in a basement daily. I decided to explore ways not only to monitor radon levels but also to hook up the metrics to my existing homelab.\nRadon Radon is a radioactive gas that can be found in homes, and at high levels and persistent exposure, can be extremely dangerous to breathe in. Radon gas comes from the natural decay of uranium in soil and rock, and it can seep into homes through cracks in the foundation, and can even permeate through concrete. Health Canada recommends taking action if radon levels exceed 200 Bq/m³, while the WHO guideline is 100 Bq/m³.\nHome Assistant My first thought was to try to plug into an ecosystem that is already robust. Home Assistant is an open-source home automation platform that allows you to monitor and control various aspects of your home. It supports a wide range of sensors and devices, including radon detectors. By integrating sensors with Home Assistant, it is easy to monitor radon levels in your home and receive alerts if they exceed safe thresholds.\nOf course, the first step is to get actual hardware that is designed to detect radon. I went with the Airthings 325 Corentium Home 2, which is the sequel to a well-trusted radon detector. It has a built-in display that shows the current radon levels, and it also has Bluetooth connectivity, which could unlock the ability to share metrics with Home Assistant. I was skeptical at first if this could work without having to integrate with a cloud subscription, but it turns out that Home Assistant has a built-in integration that can pull in the radon levels and other metrics from the device, and Corentium Home 2 is on the list of supported devices!\nOne of the downsides of the Corentium Home 2 is that it will not passively sync data over time by itself. It will only sync when you open the app. This is not a problem with the Home Assistant integration, because this integration will pull data periodically, eliminating the need to have to manually sync the data using a mobile app.\nInstalling home assistant is straightforward. In my case, I installed it on Proxmox, using this community script.\nAfter installing home assistant and passing through my bluetooth dongle to the VM, I was able to add the Airthings BLE integration and start to see metrics in the home assistant web ui:\nInfluxDB Okay, so I have a view of the sensor data in Airthings BLE integration in HA (Home Assistant). How do I export these metrics as time series data and integrate with grafana? I explored a few options: prometheus, influxdb, and statistics. I went with InfluxDB since it is a popular time series database that has good support for home assistant and grafana.\nFor simplicity\u0026rsquo;s sake, I installed it using InfluxDB Community Addon.\nAfter installing the addon, I had to configure Home Assistant to send the metrics to InfluxDB. This is done by adding the following configuration to the configuration.yaml file:\ninfluxdb: include: entities: - sensor.corentium_home_2_019191_radon_1_day_average - sensor.corentium_home_2_019191_radon_longterm_average - sensor.corentium_home_2_019191_temperature - sensor.corentium_home_2_019191_humidity - sensor.corentium_home_2_019191_battery I restarted HA but was unable to see any metrics in InfluxDB. After some troubleshooting, I realized that I had to create a database and then configure HA to use that database. I created a database called homeassistant and created a new user with appropriate permissions to this database. I can now see the metrics in InfluxDB, and I can query them using the InfluxDB web interface:\nGrafana Dashboard It\u0026rsquo;s nice to be able to visualize the data in InfluxDB, but I want to be able to create custom dashboards and alerts. Grafana\u0026rsquo;s alerting is powerful. It\u0026rsquo;s also easy to setup notifications via Slack, email, PagerDuty, etc. when radon spikes above a threshold.\nAfter exposing both grafana and homeassistant to my tailnet, I added InfluxDB as a datasource in Grafana and created a dashboard to visualize the radon levels over time:\nThe json export of this dashboard can be found here.\nGrafana Alerts Now that the dashboard is setup, I would prefer to setup an alerting rule to notify me when radon levels spike above a certain threshold.\nMake sure before you setup an alert, you create a contact point. I chose a web hook that sends to a gotify instance that I have running in my homelab, since I prefer this over email.\nOvercautiously, I setup an alert such that if the radon levels exceed 100 Bq/m³, I get a notification:\nI modified this to a lower threshold temporarily to simulate an alert, and it worked!\nConclusion It is reassuring to be able to monitor radon levels in my home and receive alerts if they exceed safe thresholds. The integration with Home Assistant, InfluxDB, and Grafana allows me to have a comprehensive view of the radon levels over time, and I can easily share this dashboard with family members. The metrics are still coming in, so seeing the data over 90 days will provide more accurate long-term averages that can be used to act on.\n","tags":["radon","grafana","home-assistant","homelab","tailscale"]},{"title":"Using a Realtek NIC with OPNsense","href":"/blog/using-a-realtek-nic-with-opnsense/","summary":"For the past few years, I’ve been running pfSense (and more recently OPNsense) in a virtual machine within Proxmox. This has been running fine with a single onboard Intel NIC. A few months ago, I upgraded to a machine that has a CPU that supports hardware-accelerated transcoding, has more SATA ports, and has more PCI slots for future expansion. With the goal of having a dedicated NIC for WAN, I bought an inexpensive 1Gbps PCIe NIC (TG-3468) despite reading about some of the concerns around Realtek NICs (sluggish performance, driver instability, and in some cases system crashes).\nI’ve been running a Realtek NICs reliably on Linux and Windows desktops, so I figured I could make it work without too much effort, but it turns out Realtek NICs really can be problematic when it comes to FreeBSD-based routers, and commonly documented workarounds did not solve my problems.\n","content":"For the past few years, I\u0026rsquo;ve been running pfSense (and more recently OPNsense) in a virtual machine within Proxmox. This has been running fine with a single onboard Intel NIC. A few months ago, I upgraded to a machine that has a CPU that supports hardware-accelerated transcoding, has more SATA ports, and has more PCI slots for future expansion. With the goal of having a dedicated NIC for WAN, I bought an inexpensive 1Gbps PCIe NIC (TG-3468) despite reading about some of the concerns around Realtek NICs (sluggish performance, driver instability, and in some cases system crashes).\nI\u0026rsquo;ve been running a Realtek NICs reliably on Linux and Windows desktops, so I figured I could make it work without too much effort, but it turns out Realtek NICs really can be problematic when it comes to FreeBSD-based routers, and commonly documented workarounds did not solve my problems.\nEnvironment My environment consists of:\nProxmox 8.4 OPNsense 25.1 (QEMU VM) Ethernet controller: Intel Corporation Ethernet Connection (5) I219-LM Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 15) Goal The goal is to upgrade the OPNsense router from a single NIC to two NICs. The NICs are responsible for:\nLAN: the internal network for computers, phones, cameras, printers, etc (NIC 1) WAN: the connection from the ISP (NIC 2) Having two separate physical interfaces for LAN and WAN creates clear, physical separation between the trusted internal network and the untrusted external network at the hardware level. This also should improve performance and throughput since the same physical connection is no longer shared between LAN and WAN.\nDevice Passthrough For maximum performance and reduced hypervisor overhead, passing through a physical NIC for WAN directly to the VM seemed to make the most sense, so I passed it through to the OPNsense VM.\nI added the PCI device and restarted the OPNsense VM and re-configured the WAN in OPNsense to use this device.\nI received the WAN IP and everything appeared to be working. I ran a few speed tests and noticed that the download speeds were much lower than normal from all of my devices. I checked my instance of speedtest-tracker noticed that the download speeds were significantly slower than historical records:\nThese speeds tests were going through Mullvad, which occasionally is inconsistent, but the results remained consistently lower than the previous configuration.\nI reverted the WAN back to the original NIC, and the download speeds returned to more average results immediately so it became obvious that something was not right with this setup.\nRealtek drivers I did some web searching / LLM prompting and discovered that some people have had improved results after installing the OPNsense plugin os-realtek-re.\nAfter installing the plugin and ensuring the kernel module was loaded at boot by following the post-install instructions, the throughput was still significantly slower than before adding a second NIC.\nI was starting to think that there might be a problem with the hardware and began the process to return it to the vendor.\nVirtualized NIC with a Linux bridge As one last shot, I created Linux Bridge in the Proxmox GUI with the Realtek NIC and passed it through to the OPNsense VM:\nI re-configured the WAN interface in OPNsense to use the newly added network device, and the download and upload speeds returned to the typical speeds. Another added benefit to this setup is that it bypasses the need for installing Realtek FreeBSD drivers on the OPNsense VM, since the network device is virtual and managed on the Proxmox host (debian-based).\nConclusion Although I am not sure why passing through a Realtek NIC to an OPNsense VM causes so much degradation in throughput, I am glad that there is a workaround. If I get ahold of another NIC, I would be interested in trying to reproduce the issue.\n","tags":["linux","freebsd","opnsense","pfsense","proxmox","realtek","nic"]},{"title":"Replicating TrueNAS Datasets to SFTPGo over Tailscale","href":"/blog/replicating-truenas-datasets-to-sftpgo-over-tailscale/","summary":"I’ve recently spun up an instance of TrueNAS SCALE after salvaging a couple hard drives from a past computer build and decided I could use additional network storage for various backups such as Proxmox VMs and home directory backups.\n","content":"I\u0026rsquo;ve recently spun up an instance of TrueNAS SCALE after salvaging a couple hard drives from a past computer build and decided I could use additional network storage for various backups such as Proxmox VMs and home directory backups.\nThe only app I\u0026rsquo;ve needed to install has been Tailscale, which has enabled me to access the TrueNAS Web UI from anywhere. I\u0026rsquo;ve set up a few datasets and NFS shares to store various backups, and the rest of the periodic backups have routinely been working without a hitch. Since my homelab is becoming more of a vital piece of infrastructure for my daily needs, I wanted to ensure that these datasets had Cloud Sync Tasks set up for offsite backups. These encrypted backups are mostly being stored in places such as Google Drive and other blob storage providers.\nMore recently, to reduce cloud costs, I\u0026rsquo;ve set up a small node at another physical location and installed both Tailscale and SFTPGo on it to facilitate offsite backups. After setting up the infrastructure and adding a Cloud Sync Task in TrueNAS SCALE to replicate these backups offsite to SFTPGo, I noticed that Tailscale\u0026rsquo;s MagicDNS was not working, nor was the Tailscale IPv4 address.\nAfter reading the Tailscale docs , it became clear that the Userspace box had to be unchecked in the Tailscale app settings. This is because the Tailscale app is running within a docker container on the TrueNAS SCALE VM. After unchecking the Userspace box, I was able to verify that the Backup Credentials created for sftpgo worked when specifying the host as a Tailscale IPv4 address. This was probably good enough since the IP won\u0026rsquo;t change unless the node is re-registered.\nTo get MagicDNS working, I went to Network \u0026gt; Global Configuration and set \u0026ldquo;Nameserver 1\u0026rdquo; to 100.100.100.100. After this, I was able to specify the FQDN in the Backup Credentials and the Cloud Sync Tasks started.\nThis method of adding MagicDNS can lead to issues with DNS when updating the Tailscale application in TrueNAS, so I ended up using the Tailscale IP directly.\n","tags":["tailscale","truenas","sftpgo"]},{"title":"Opting Out of HaveIBeenPwned","href":"/blog/opting-out-of-haveibeenpwned/","summary":"Data breaches are a concern for anyone trying to live a life of relative privacy. Last month, PowerSchool informed its customers that hackers stole data of 62 million students. This may not have impacted you, but unless you have been practicing Extreme Privacy techniques for decades, you likely have been impacted by a data breach in the past.\n","content":"Data breaches are a concern for anyone trying to live a life of relative privacy. Last month, PowerSchool informed its customers that hackers stole data of 62 million students. This may not have impacted you, but unless you have been practicing Extreme Privacy techniques for decades, you likely have been impacted by a data breach in the past.\nUnderstanding Data Breaches Data breaches occur when unauthorized individuals gain access to sensitive information (names, addresses, emails, phone numbers, among other details). If the breach is substantial enough, the raw data is likely to make it into the hands of data brokers that will collect, aggregate, and sell the information on the dark web.\nCheck if you have been impacted There are a number of services that can be used to check if you have been impacted by a data breach, including Mozilla Monitor, Google Dark Web Report, and haveibeenpwned.com. Some password managers offer features that compare your credentials against known breaches. These services can also be configured to send you notifications when a breach occurs. It is a good idea to become aware of these breaches as soon as you can, so that you can protect yourself from malicious behaviour such as phishing.\nIf you have had an email address or phone number for any length of time, there is a high probability that some of your data has been exposed. You can easily check by querying haveibeenpwned.com. Many of the tools that offer breach detection query the haveibeenpwned database. Although I believe this service is a public good, it also opens the door for anyone who may be looking to gain more information about your present and past use of various websites and services.\nOpting out If you have an identity that you\u0026rsquo;d like to protect, I\u0026rsquo;d suggest opting out of public searchability. This of course does not undo the data breach that happened, but it does make it more challenging for someone to quickly search for an impacted email address. Even after opting out, you can still subscribe to breach notifications, as long as you can validate that you have access to the email in question.\nThere are other websites that offer similar-style lookups, but many of them are either paywalled or require account registration.\nEmail aliases A more proactive method of reducing the likelihood of future exposures is to use an email aliasing service such as Firefox Relay, DuckDuckGo Email Protection, or if you use Proton Mail, hide-my-email aliases. This will allow you to sign up for services using an alias instead of revealing your email address. The service then forwards all emails to your real address that you configure when setting up the alias.\n","tags":["breach","darkweb","haveibeenpwned","hibp","passwords","privacy"]},{"title":"Amazon EBS CSI Driver with Terraform","href":"/blog/amazon-ebs-csi-driver-terraform/","summary":"I recently configured the Amazon EBS CSI driver and found the setup with Terraform to be more effort than expected. I wanted to avoid third-party modules and keep it as simple as possible, while remaining least privilege.\nUPDATE: This approach can also be used for the aws-efs-csi-driver\n","content":"I recently configured the Amazon EBS CSI driver and found the setup with Terraform to be more effort than expected. I wanted to avoid third-party modules and keep it as simple as possible, while remaining least privilege.\nUPDATE: This approach can also be used for the aws-efs-csi-driver\nThe Amazon EBS CSI driver docs mention that the following are needed:\nan existing EKS cluster IAM role (that allows communication to the EC2 API) EKS add-on (aws-ebs-csi-driver) OIDC provider This sounded simple enough but I was unable to find a \u0026ldquo;grab-and-go\u0026rdquo; Terraform example that followed the recommendations in the docs. I saw some suggestions about attaching an AmazonEBSCSIDriverPolicy policy to the node groups but did not think this was the best idea since this would allow many pods to potentially have access to the EC2 API.\nAfter a few minutes of LLM prompting, I was unimpressed with the results. I began to piece together the config myself, and after some trial and error, this is the Terraform that I came up with:\n# TLS needed for the thumbprint provider \u0026quot;tls\u0026quot; {} data \u0026quot;tls_certificate\u0026quot; \u0026quot;oidc\u0026quot; { url = aws_eks_cluster.main.identity[0].oidc[0].issuer } # EKS addon resource \u0026quot;aws_eks_addon\u0026quot; \u0026quot;ebs_csi_driver\u0026quot; { cluster_name = aws_eks_cluster.main.name addon_name = \u0026quot;aws-ebs-csi-driver\u0026quot; addon_version = \u0026quot;v1.29.1-eksbuild.1\u0026quot; service_account_role_arn = aws_iam_role.ebs_csi_driver.arn } # AWS Identity and Access Management (IAM) OpenID Connect (OIDC) provider resource \u0026quot;aws_iam_openid_connect_provider\u0026quot; \u0026quot;eks\u0026quot; { url = aws_eks_cluster.main.identity.0.oidc.0.issuer client_id_list = [\u0026quot;sts.amazonaws.com\u0026quot;] thumbprint_list = [data.tls_certificate.oidc.certificates[0].sha1_fingerprint] } # IAM resource \u0026quot;aws_iam_role\u0026quot; \u0026quot;ebs_csi_driver\u0026quot; { name = \u0026quot;ebs-csi-driver\u0026quot; assume_role_policy = data.aws_iam_policy_document.ebs_csi_driver_assume_role.json } data \u0026quot;aws_iam_policy_document\u0026quot; \u0026quot;ebs_csi_driver_assume_role\u0026quot; { statement { effect = \u0026quot;Allow\u0026quot; principals { type = \u0026quot;Federated\u0026quot; identifiers = [aws_iam_openid_connect_provider.eks.arn] } actions = [ \u0026quot;sts:AssumeRoleWithWebIdentity\u0026quot;, ] condition { test = \u0026quot;StringEquals\u0026quot; variable = \u0026quot;${aws_iam_openid_connect_provider.eks.url}:aud\u0026quot; values = [\u0026quot;sts.amazonaws.com\u0026quot;] } condition { test = \u0026quot;StringEquals\u0026quot; variable = \u0026quot;${aws_iam_openid_connect_provider.eks.url}:sub\u0026quot; values = [\u0026quot;system:serviceaccount:kube-system:ebs-csi-controller-sa\u0026quot;] } } } resource \u0026quot;aws_iam_role_policy_attachment\u0026quot; \u0026quot;AmazonEBSCSIDriverPolicy\u0026quot; { policy_arn = \u0026quot;arn:aws:iam::aws:policy/service-role/AmazonEBSCSIDriverPolicy\u0026quot; role = aws_iam_role.ebs_csi_driver.name } The above configuration follows the docs, binding an IAM role to the service account kube-system/ebs-csi-controller-sa using an OpenID connect provider.\nAfter applying the changes above, I deployed the sample application and noticed that the persistent volume claims were bound to EBS volumes.\n","tags":["aws","eks","ebs","aws-ebs-csi-driver","oidc","efs","aws-efs-csi-driver"]},{"title":"Setting Up Gitea Actions with Tailscale","href":"/blog/setting-up-gitea-actions-with-tailscale/","summary":"In this post I’ll go through the process of setting up Gitea Actions and Tailscale, unlocking a simple and secure way to automate workflows.\n","content":"In this post I\u0026rsquo;ll go through the process of setting up Gitea Actions and Tailscale, unlocking a simple and secure way to automate workflows.\nWhat is Gitea? Gitea is a lightweight and fast git server that has much of the same look and feel as github. I have been using it in my homelab to mirror repositories hosted on other platforms such as github and gitlab. These mirrors take advantage of the decentralized nature of git by serving as \u0026ldquo;backups\u0026rdquo;. One of the main reasons I hadn\u0026rsquo;t been using it more often was due to the lack of integrated CI/CD. This is no longer the case.\nGitea Actions Gitea Actions have made it into the 1.19.0 release. This feature had been in an experimental state up until 1.21.0 and is now enabled by default 🎉.\nSo what are they? If you\u0026rsquo;ve ever used GitHub Actions (and if you\u0026rsquo;re reading this, I imagine you have), these will look familiar. Gitea Actions essentially enable the ability to run GitHub workflows on Gitea. Workflows between Gitea and GitHub are not completely interoperable, but a lot of the same workflow syntax is already compatible on Gitea. You can find a documented list of unsupported workflow syntax.\nActions work by using a custom fork of nekos/act. Workflows run in a new container for every job. If you specify an action such as actions/checkout@v4, it defaults to downloading the scripts from github.com. To avoid internet egress, you could always clone the required actions to your local gitea instance.\nActions (gitea\u0026rsquo;s implementation) has me excited because it makes spinning up a network-isolated environment for workflow automation incredibly simple.\nIntegration with Tailscale 2024-02-10: I had originally written this post to include Tailscale-Traefik Proxy Integration, but have since removed it in favour of Tailscale Serve after learning from this example. This simplifies the setup and reduces the number of moving parts.\nSo how does Tailscale help here? Well, more recently I\u0026rsquo;ve been exposing my self-hosted services using Tailscale Serve. This allows for a nice looking dns name (i.e. gitea.my-tailnet-name.ts.net), automatic tls certificate management, and optionally allowing the address to be publically accessible (by using Funnel).\nDeploying Gitea, Traefik, and Tailscale In my case, the following is already set up:\ndocker-compose is installed tailscale magic dns is enabled My preferred approach to deploying code in a homelab environment is with docker compose. I have deployed this in a LXC on Proxmox. You could run this on a virtual machine or a physical host as well.\nThe docker-compose.yaml file looks like:\nversion: \u0026quot;3.7\u0026quot; services: gitea: image: gitea/gitea:1.21.1 container_name: gitea environment: - USER_UID=1000 - USER_GID=1000 - GITEA__server__DOMAIN=gitea.my-tailnet-name.ts.net - GITEA__server__ROOT_URL=https://gitea.my-tailnet-name.ts.net - GITEA__server__HTTP_ADDR=0.0.0.0 - GITEA__server__LFS_JWT_SECRET=my-secret-jwt restart: always volumes: - ./data:/data - /etc/timezone:/etc/timezone:ro - /etc/localtime:/etc/localtime:ro After adding the above configuration, running docker compose up -d should be enough to get an instance up and running.\nTo make it accessible at https://gitea.my-tailnet-name.ts.net from within the tailnet, install tailscale cli and run:\ntailscale serve -bg 3000 Something to consider is whether or not you want to use ssh with git. One method to get this to work with containers is to use ssh container passthrough. I decided to keep it simple and not use ssh, since communicating over https is perfectly fine for my use case.\nConnecting runners I installed the runner by following the docs. I opted for installing it on a separate host as recommended in the docs. I used the systemd unit file to ensure that the runner comes back online after system reboots. I installed tailscale on the gitea runner as well, so that it can be part of the same tailnet as the main instance.\nAfter registering this runner and starting the daemon, the runner appeared in /admin/actions/runners. I added two other runners to help with parallelization.\nRunning a workflow Now it\u0026rsquo;s time to start running some automation. I used the demo workflow as a starting point to verify that the runner is executing workflows.\nAfter this, I wanted to make sure that some of my existing workflows could be migrated over.\nThe following workflow uses a matrix to run a job for several of my hosts using ansible playbooks that will do various tasks such as patching os updates and updating container images.\nname: Run ansible on: push: schedule: - cron: \u0026quot;0 */12 * * *\u0026quot; jobs: run-ansible-playbook: runs-on: ubuntu-latest steps: - name: Check out repository code uses: actions/checkout@v4 - name: Install ansible run: | apt update \u0026amp;\u0026amp; apt install ansible -y - name: Run playbook uses: dawidd6/action-ansible-playbook@v2 with: playbook: playbooks/main.yml requirements: requirements.yml options: | --inventory inventory - name: Send failure notification uses: dawidd6/action-send-mail@v3 if: always() \u0026amp;\u0026amp; failure() with: server_address: smtp.gmail.com server_port: 465 secure: true username: myuser password: ${{ secrets.MAIL_PASSWORD }} subject: ansible runbook failed to: me@davegallant.ca from: RFD Notify body: | ${{ github.server_url }}/${{ github.repository }}/actions/runs/${{ github.run_number }} And the end result:\nYou may be wondering how the gitea runner is allowed to connect to the other hosts using ansible? Well, the nodes are in the same tailnet and have tailscale ssh enabled.\nAreas for improvement One enhancement that I would like to see is the ability to send notifications on workflow failures. Currently, this doesn\u0026rsquo;t seem possible without adding logic to each workflow.\nConclusion Gitea Actions are fast and the resource footprint is minimal. My gitea instance is currently using around 250mb of memory and a small fraction of a single cpu core (and the runner is using a similar amount of resources). This is impressive since many alternatives tend to require substantially more resources. It likely helps that the codebase is largely written in go.\nBy combining gitea with tailscale, running workflows becomes simple and fun. Whether you are working on a team or working alone, this setup ensures that your workflows are securely accessible from anywhere with an internet connection.\n","tags":["gitea","gitea actions","github actions","tailscale","self-hosted"]},{"title":"Using AKS and SOCKS to Connect to a Private Azure DB","href":"/blog/using-aks-and-socks-to-connect-to-a-private-azure-db/","summary":"I ran into a roadblock recently where I wanted to conveniently connect to a managed postgres database within Azure that was not running on public subnets. And by conveniently, I mean that I’d rather not have to spin up an ephemeral virtual machine running in the same network and proxy the connection, and I’d like to use a local client (preferably with a GUI). After several web searches, it became evident that Azure does not readily provide much tooling to support this.\n","content":"I ran into a roadblock recently where I wanted to conveniently connect to a managed postgres database within Azure that was not running on public subnets. And by conveniently, I mean that I\u0026rsquo;d rather not have to spin up an ephemeral virtual machine running in the same network and proxy the connection, and I\u0026rsquo;d like to use a local client (preferably with a GUI). After several web searches, it became evident that Azure does not readily provide much tooling to support this.\nGo Public? Should the database be migrated to public subnets? Ideally not, since it is good practice to host internal infrastructure in restricted subnets.\nHow do others handle this? With GCP, connecting to a private db instance from any machine can be achieved with cloud-sql-proxy. This works by proxying requests from your machine to the SQL database instance in the cloud, while the authentication is handled by GCP\u0026rsquo;s IAM.\nSo what about Azure? Is there any solution that is as elegant as cloud-sql-proxy?\nA Bastion Similar to what AWS has recommended, perhaps a bastion is the way forward?\nAzure has a fully-managed service called Azure Bastion that provides secure access to virtual machines that do not have public IPs. This looks interesting, but unfortunately it costs money and requires an additional virtual machine.\nBecause this adds cost (and complexity), it does not seem like a desirable option in its current state. If it provided a more seamless connection to the database, it would be more appealing.\nSOCKS 2023-12-13: An alternative to using a socks proxy is socat. This would allow you to relay tcp connections to a pod running in k8s, and then port-forward them to your localhost. If this sounds more appealing, install krew-net-forward and then run \u0026ldquo;kubectl net-forward -i mydb.postgres.database.azure.com -p 5432 -l 5432\u0026rdquo; to access the database through \u0026ldquo;localhost:5432\u0026rdquo;\nSOCKS is a protocol that enables a way to proxy connections by exchanging network packets between the client and the server. There are many implementations and many readily available container images that can run a SOCKS server.\nIt\u0026rsquo;s possible to use this sort of proxy to connect to a private DB, but is it any simpler than using a virtual machine as a jumphost? It wasn\u0026rsquo;t until I stumbled upon kubectl-plugin-socks5-proxy that I was convinced that using SOCKS could be made simple.\nSo how does it work? By installing the kubectl plugin and then running kubectl socks5-proxy, a SOCKS proxy server is spun up in a pod and then opens up port-forwarding session using kubectl.\nAs you can see below, this k8s plugin is wrapped up nicely:\n$ kubectl socks5-proxy using: namespace=default using: port=1080 using: name=davegallant-proxy using: image=serjs/go-socks5-proxy Creating SOCKS5 Proxy (Pod)... pod/davegallant-proxy created With the above proxy connection open, it is possible to access both the DNS and private IPs accessible within the k8s cluster. In this case, I am able to access the private database, since there is network connectivity between the k8s cluster and the database.\nCaveats and Conclusion The above outlined solution makes some assumptions:\nthere is a k8s cluster the k8s cluster has network connectivity to the desired private database If these stars align, then this solution might work as a stopgap for accessing a private Azure DB (and I\u0026rsquo;m assuming this could work similarly on AWS).\nIt would be nice if Azure provided tooling similar to cloud-sql-proxy, so that using private databases would be more of a convenient experience.\nOne other thing to note is that some clients (such as dbeaver) do not provide DNS resolution over SOCKS. So in this case, you won\u0026rsquo;t be able to use DNS as if you were inside the cluster, but instead have to rely on knowing private ip addresses.\n2025-01-16:: DNS over SOCKS now works with the latest dbeaver client.\n","tags":["aks","aws","azure","bastion","cloud-sql-proxy","database","eks","k8s","kubectl-plugin-socks5-proxy","proxy","socat","socks"]},{"title":"Watching YouTube in Private","href":"/blog/watching-youtube-in-private/","summary":"I recently stumbled upon yewtu.be and found it intriguing. It not only allows you to watch YouTube without being on YouTube, but it also allows you to create an account and subscribe to channels without a Google account. What sort of wizardry is going on under the hood? It turns out that it’s a hosted instance of invidious.\n","content":"I recently stumbled upon yewtu.be and found it intriguing. It not only allows you to watch YouTube without being on YouTube, but it also allows you to create an account and subscribe to channels without a Google account. What sort of wizardry is going on under the hood? It turns out that it\u0026rsquo;s a hosted instance of invidious.\nThe layout is simple, and JavaScript is not required.\nI started using yewtu.be as my primary client for watching videos. I subscribe to several YouTube channels and I prefer the interface Invidious provides due to its simplicity. It\u0026rsquo;s also nice to be in control of my search and watch history.\nA few days ago, yewtu.be went down briefly, and that motivated me enough to self-host invidious. There are several other hosted instances listed here, but being able to easily backup my own instance (including subscriptions and watch history) is more compelling in my case.\nHosting invidious The quickest way to get invidious up is with docker-compose as mentioned in the docs.\nI made a few modifications, and ended up with:\nversion: \u0026quot;3\u0026quot; services: invidious: image: quay.io/invidious/invidious restart: unless-stopped ports: - \u0026quot;0.0.0.0:3000:3000\u0026quot; environment: INVIDIOUS_CONFIG: | db: dbname: invidious user: kemal password: kemal host: invidious-db port: 5432 check_tables: true healthcheck: test: wget -nv --tries=1 --spider http://127.0.0.1:3000/api/v1/comments/jNQXAC9IVRw || exit 1 interval: 30s timeout: 5s retries: 2 depends_on: - invidious-db invidious-db: image: docker.io/library/postgres:14 restart: unless-stopped volumes: - postgresdata:/var/lib/postgresql/data - ./config/sql:/config/sql - ./docker/init-invidious-db.sh:/docker-entrypoint-initdb.d/init-invidious-db.sh environment: POSTGRES_DB: invidious POSTGRES_USER: kemal POSTGRES_PASSWORD: kemal healthcheck: test: [\u0026quot;CMD-SHELL\u0026quot;, \u0026quot;pg_isready -U $$POSTGRES_USER -d $$POSTGRES_DB\u0026quot;] volumes: postgresdata: After invidious was up and running, I installed Tailscale on it to leverage its MagicDNS, and I\u0026rsquo;m now able to access this instance from anywhere at http://invidious:3000/feed/subscriptions.\nRedirecting YouTube links I figured it would be nice to redirect existing YouTube links that others send me, so that I could seamlessly watch the videos using invidious.\nI went looking for a way to redirect paths at the browser level. I found Redirector, which can be used to modify http requests in the browser. I created the following redirect (exported as json):\n{ \u0026quot;redirects\u0026quot;: [ { \u0026quot;description\u0026quot;: \u0026quot;youtube to invidious\u0026quot;, \u0026quot;exampleUrl\u0026quot;: \u0026quot;https://www.youtube.com/watch?v=-lz30by8-sU\u0026quot;, \u0026quot;exampleResult\u0026quot;: \u0026quot;http://invidious:3000/watch?v=-lz30by8-sU\u0026quot;, \u0026quot;error\u0026quot;: null, \u0026quot;includePattern\u0026quot;: \u0026quot;https://*youtube.com/*\u0026quot;, \u0026quot;excludePattern\u0026quot;: \u0026quot;\u0026quot;, \u0026quot;patternDesc\u0026quot;: \u0026quot;Any youtube video should redirect to invidious\u0026quot;, \u0026quot;redirectUrl\u0026quot;: \u0026quot;http://invidious:3000/$2\u0026quot;, \u0026quot;patternType\u0026quot;: \u0026quot;W\u0026quot;, \u0026quot;processMatches\u0026quot;: \u0026quot;noProcessing\u0026quot;, \u0026quot;disabled\u0026quot;: false, \u0026quot;grouped\u0026quot;: false, \u0026quot;appliesTo\u0026quot;: [ \u0026quot;main_frame\u0026quot; ] } ] } Now the link https://www.youtube.com/watch?v=-lz30by8-sU will redirect to http://invidious:3000/watch?v=-lz30by8-sU\nI\u0026rsquo;m still looking for ways to improve this invidious setup. There doesn\u0026rsquo;t appear to be a way to stream in 4K yet.\n","tags":["invidious","youtube","yewtu.be","tailscale","privacy","self-hosted"]},{"title":"Virtualizing My Router with pfSense","href":"/blog/virtualizing-a-router-with-pfsense/","summary":"My aging router has been running OpenWrt for years and for the most part has been quite reliable. OpenWrt is an open-source project used on embedded devices to route network traffic. It supports many different configurations and there exists a large index of packages. Ever since I’ve connected some standalone wireless access points, I’ve had less of a need for an off-the-shelf all-in-one wireless router combo. I’ve also recently been experiencing instability with my router (likely the result of a combination of configuration tweaking and firmware updating). OpenWrt has served me well, but it is time to move on!\n","content":"My aging router has been running OpenWrt for years and for the most part has been quite reliable. OpenWrt is an open-source project used on embedded devices to route network traffic. It supports many different configurations and there exists a large index of packages. Ever since I\u0026rsquo;ve connected some standalone wireless access points, I\u0026rsquo;ve had less of a need for an off-the-shelf all-in-one wireless router combo. I\u0026rsquo;ve also recently been experiencing instability with my router (likely the result of a combination of configuration tweaking and firmware updating). OpenWrt has served me well, but it is time to move on!\npfSense I figured this would be a good opportunity to try pfSense. I\u0026rsquo;ve heard nothing but positive things about pfSense and the fact it\u0026rsquo;s been around since 2004, based on FreeBSD, and written in PHP gave me the impression that it would be relatively stable (and I\u0026rsquo;d expect nothing less because it has an important job to do!). pfSense can be run on many different machines, and there are even some officially supported appliances. Since I already have a machine running Proxmox, why not just run it in a VM? It\u0026rsquo;d allow for automatic snapshotting of the machine. There is a good video on this by Techno Tim. Tim has a lot of good videos, and this one is about virtualizing pfSense.\nRouter on a stick I had initially made the assumption that in order to build a router, you would need more than a single NIC (or a dual-port NIC) in order to support both WAN and LAN. This is simply not the case, because VLANs are awesome! In order to create a router, all you need is a single port NIC and a network switch that supports VLANs (also marketed as a managed switch). I picked up the Netgear GS308E because it has both a sufficient amount of ports for my needs, and it supports VLANs. It also has a nice sturdy metal frame which was a pleasant surprise.\nAfter setting up this Netgear switch, it should be possible to access the web interface at http://192.168.0.239. It may be at a different address. To find the address, try checking your DHCP leases in your router interface (if you plugged it into an existing router). I realized I was unable to access this interface because I was on a different subnet, so I set my machine\u0026rsquo;s address to 192.168.0.22 in order to temporarily set up this switch. I assigned a static IP address to the switch (in System \u0026gt; Switch Information) so that it was in the same subnet as the rest of my network.\nThe web interface is nothing spectacular, but it allows for managing VLANs.\nThe following configuration will:\nassign port 1 to be the LAN (connected to the Proxmox machine) assign port 8 to be the WAN (connected to my ISP\u0026rsquo;s modem) In the switch\u0026rsquo;s web interface, I went to VLAN and then 802.1Q, and then clicked on VLAN Configuration. I configured the ports to look like this:\nNote that the VLAN Identifier Setting has been setup already with two VLANs (1 and 10). More VLANs can be created (i.e. to isolate IoT devices), but 2 VLANs is all we need for the initial setup of a router.\nTo replicate the above configuration, add a new VLAN ID 10 (1 should exist by default).\nNext, go into VLAN Membership and configure VLAN 1\u0026rsquo;s port membership to be the following:\nand then configure VLAN 10\u0026rsquo;s port membership to be the following:\nNow, go into Port PVID and ensure that port 8 is set to PVID 10.\nThis above configuration will dedicate two of the eight ports to WAN and LAN. This will allow the internet to flow into the pfSense from the modem.\nSetting up pfSense pfSense is fairly easy to setup. Just download the latest ISO and boot up the virtual machine. When setting up the machine, I mostly went with all of the defaults. Configuration can be changed later in the web interface, which is quite a bit simpler.\nSince VLANs are going to be leveraged, when you go to Assign Interfaces, VLANs should be setup now like the following:\nWAN should be vtnet0.10 LAN should be vtnet0 After going through the rest of the installation, if everything is connected correctly it should display both WAN and LAN addresses.\nIf all goes well, the web interface should be running at https://192.168.1.1.\nAnd this is where the fun begins. There are many tutorials and blogs about how to setup pfSense and various services and packages that can be installed. I\u0026rsquo;ve already installed pfBlocker-NG.\nSummary It is fairly simple to setup a router with pfSense from within a virtual machine. A physical dedicated routing machine is not necessary and often does not perform as well as software running on faster and more reliable hardware. So far, pfSense has been running for over a week without a single hiccup. pfSense is a mature piece of software that is incredibly powerful and flexible. To avoid some of the instability I had experienced with OpenWrt, I enabled AutoConfigBackup, which is capable of automatically backing up configuration upon every change. I plan to explore and experiment with more services and configuration in the future, so the ability to track all of these changes gives me the peace of mind that experimentation is safe.\n","tags":["pfsense","router","openwrt","router-on-a-stick","proxmox","vlan","self-hosted"]},{"title":"Backing Up Gmail with Synology","href":"/blog/backing-up-gmail-with-synology/","summary":"I’ve used Gmail since the beta launched touting a whopping 1GB of storage. I thought this was a massive leap in email technology at the time. I was lucky enough to get an invite fairly quickly. Not surprisingly, I have many years of emails, attachments, and photos. I certainly do not want to lose the content of many of these emails. Despite the redundancy of the data that Google secures, I still feel better retaining a copy of this data on my own physical machines.\n","content":"I\u0026rsquo;ve used Gmail since the beta launched touting a whopping 1GB of storage. I thought this was a massive leap in email technology at the time. I was lucky enough to get an invite fairly quickly. Not surprisingly, I have many years of emails, attachments, and photos. I certainly do not want to lose the content of many of these emails. Despite the redundancy of the data that Google secures, I still feel better retaining a copy of this data on my own physical machines.\nThe thought of completely de-Googling has crossed my mind on occasion. Convenience, coupled with my admiration for Google engineering, has prevented me from doing so thus far. Though, I may end up doing so at some point in the future.\nSynology MailPlus Server Synology products are reasonably priced for what you get (essentially a cloud-in-a-box) and there is very little maintenance required. I\u0026rsquo;ve recently been interested in syncing and snapshotting my personal data. I\u0026rsquo;ve set up Synology\u0026rsquo;s Cloud Sync and keep copies of most of my cloud data.\nI\u0026rsquo;ve used tools such as gmvault with success in the past. Setting this up on a cron seems like a viable option. However, I don\u0026rsquo;t really need a lot of the features it offers and do not plan to restore this data to another account.\nSynology\u0026rsquo;s MailPlus seems to be a good candidate for backing up this data. By enabling POP3 fetching, it\u0026rsquo;s possible to fetch all existing emails, as well as periodically fetch all new emails. If a disaster ever did occur, having these emails would be beneficial, as they are an extension of my memory bank.\nInstalling MailPlus can be done from the Package Center:\nNext, I went into Synology MailPlus Server and on the left, clicked on Account and ensured my user was marked as active.\nAfterward, I followed these instructions in order to start backing up emails.\nWhen entering the POP3 credentials, I created an app password solely for authenticating to POP3 from the Synology device. This is required because I have 2-Step verification enabled on my account. There doesn\u0026rsquo;t seem to be a more secure way to access POP3 at the moment. It does seem like app password access is limited in scope (when MFA is enabled). These app passwords can\u0026rsquo;t be used to log in to the main Google account.\nI made sure to set the Fetch Range to All in order to get all emails from the beginning of time.\nAfter this, mail started coming in.\nAfter fetching 19 years worth of emails, I tried searching for some emails. It only took a few seconds to search through ~50K emails, which is a relief if I ever did have to search for something important.\nSecuring Synology Since Synology devices are not hermetically sealed, it\u0026rsquo;s best to secure them by enabling MFA to help prevent being the victim of ransomware. It is also wise to backup your system settings and volumes to the cloud using a tool such as Hyper Backup. Encrypting your shared volumes should also be done, since unfortunately DSM does not support full disk encryption.\nSummary Having backups of various forms of cloud data is a good investment, especially in times of war. I certainly feel more at ease for having backed up my emails.\n","tags":["synology","gmail","backup","ransomware"]},{"title":"Running K3s in LXC on Proxmox","href":"/blog/running-k3s-in-lxc-on-proxmox/","summary":"It has been a while since I’ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I’ve had success with deploying it on resource-constrained Raspberry Pis in the past. I thought that this time it’d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine’s resources.\n","content":"It has been a while since I\u0026rsquo;ve actively used Kubernetes and wanted to explore the evolution of tools such as Helm and Tekton. I decided to deploy K3s, since I\u0026rsquo;ve had success with deploying it on resource-constrained Raspberry Pis in the past. I thought that this time it\u0026rsquo;d be convenient to have K3s running in a LXC container on Proxmox. This would allow for easy snapshotting of the entire Kubernetes deployment. LXC containers also provide an efficient way to use a machine\u0026rsquo;s resources.\nWhat is K3s? K3s is a Kubernetes distro that advertises itself as a lightweight binary with a much smaller memory-footprint than traditional k8s. K3s is not a fork of k8s as it seeks to remain as close to upstream as it possibly can.\nConfigure Proxmox This gist contains snippets and discussion on how to deploy K3s in LXC on Proxmox. It mentions that bridge-nf-call-iptables should be loaded, but I did not understand the benefit of doing this.\nDisable swap There is an issue on Kubernetes regarding swap here. There are claims of support for swap in 1.22, but for now let\u0026rsquo;s disable it:\nsudo sysctl vm.swappiness=0 sudo swapoff -a It might be worth experimenting with swap enabled in the future to see how that might affect performance.\nEnable IP Forwarding To avoid IP Forwarding issues with Traefik, run the following on the host:\nsudo sysctl net.ipv4.ip_forward=1 sudo sysctl net.ipv6.conf.all.forwarding=1 sudo sed -i 's/#net.ipv4.ip_forward=1/net.ipv4.ip_forward=1/g' /etc/sysctl.conf sudo sed -i 's/#net.ipv6.conf.all.forwarding=1/net.ipv6.conf.all.forwarding=1/g' /etc/sysctl.conf Create LXC container Create an LXC container in the Proxmox interface as you normally would. Remember to:\nUncheck unprivileged container Use a LXC template (I chose a debian 11 template downloaded with pveam) In memory, set swap to 0 Create and start the container Modify container config Now back on the host run pct list to determine what VMID it was given.\nOpen /etc/pve/lxc/$VMID.conf and append:\nlxc.apparmor.profile: unconfined lxc.cap.drop: lxc.mount.auto: \u0026quot;proc:rw sys:rw\u0026quot; lxc.cgroup2.devices.allow: c 10:200 rwm All of the above configurations are described in the manpages. Notice that cgroup2 is used since Proxmox VE 7.0 has switched to a pure cgroupv2 environment.\nThankfully cgroup v2 support has been supported in k3s with these contributions:\nhttps://github.com/k3s-io/k3s/pull/2584 https://github.com/k3s-io/k3s/pull/2844 Enable shared host mounts From within the container, run:\necho '#!/bin/sh -e ln -s /dev/console /dev/kmsg mount --make-rshared /' \u0026gt; /etc/rc.local chmod +x /etc/rc.local reboot Install K3s One of the simplest ways to install K3s on a remote host is to use k3sup. Ensure that you supply a valid CONTAINER_IP and choose the k3s-version you prefer. As of 2021/11, it is still defaulting to the 1.19 channel, so I overrode it to 1.22 for cgroup v2 support. See the published releases here.\nssh-copy-id root@$CONTAINER_IP k3sup install --ip $CONTAINER_IP --user root --k3s-version v1.22.3+k3s1 If all goes well, you should see a path to the kubeconfig generated. I moved this into ~/.kube/config so that kubectl would read this by default.\nWrapping up Installing K3s in LXC on Proxmox works with a few tweaks to the default configuration. I later followed the Tekton Getting Started guide and was able to deploy it in a few commands.\n$ kubectl get all --namespace tekton-pipelines NAME READY STATUS RESTARTS AGE pod/tekton-pipelines-webhook-8566ff9b6b-6rnh8 1/1 Running 1 (50m ago) 12h pod/tekton-dashboard-6bf858f977-qt4hr 1/1 Running 1 (50m ago) 11h pod/tekton-pipelines-controller-69fd7498d8-f57m4 1/1 Running 1 (50m ago) 12h NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/tekton-pipelines-controller ClusterIP 10.43.44.245 \u0026lt;none\u0026gt; 9090/TCP,8080/TCP 12h service/tekton-pipelines-webhook ClusterIP 10.43.183.242 \u0026lt;none\u0026gt; 9090/TCP,8008/TCP,443/TCP,8080/TCP 12h service/tekton-dashboard ClusterIP 10.43.87.97 \u0026lt;none\u0026gt; 9097/TCP 11h NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/tekton-pipelines-webhook 1/1 1 1 12h deployment.apps/tekton-dashboard 1/1 1 1 11h deployment.apps/tekton-pipelines-controller 1/1 1 1 12h NAME DESIRED CURRENT READY AGE replicaset.apps/tekton-pipelines-webhook-8566ff9b6b 1 1 1 12h replicaset.apps/tekton-dashboard-6bf858f977 1 1 1 11h replicaset.apps/tekton-pipelines-controller-69fd7498d8 1 1 1 12h NAME REFERENCE TARGETS MINPODS MAXPODS REPLICAS AGE horizontalpodautoscaler.autoscaling/tekton-pipelines-webhook Deployment/tekton-pipelines-webhook 9%/100% 1 5 1 12h I made sure to install Tailscale in the container so that I can easily access K3s from anywhere.\nIf I\u0026rsquo;m feeling adventurous, I might experiment with K3s rootless.\n","tags":["k3s","proxmox","lxc","self-hosted"]},{"title":"Replacing Docker with Podman on macOS (and Linux)","href":"/blog/replacing-docker-with-podman-on-macos/","summary":"There are a number of reasons why you might want to replace Docker, especially on macOS. The following feature bundled in Docker Desktop might have motivated you enough to consider replacing Docker:\n","content":"There are a number of reasons why you might want to replace Docker, especially on macOS. The following feature bundled in Docker Desktop might have motivated you enough to consider replacing Docker:\n...ignoring Docker updates is a paid feature now?? pic.twitter.com/ZxKW3b9LQM\n\u0026mdash; Brendan Dolan-Gavitt (@moyix) May 1, 2021 Docker has been one of the larger influencers in the container world, helping to standardize the OCI Image Format Specification. For many developers, containers have become synonymous with terms like docker and Dockerfile (a file containing build instructions for a container image). Docker has certainly made it very convenient to build and run containers, but it is not the only solution for doing so.\nThis post briefly describes my experience swapping out Docker for Podman on macOS.\nWhat is a container? A container is a standard unit of software that packages up all application dependencies within it. Multiple containers can be run on a host machine all sharing the same kernel as the host. Linux namespaces help provide an isolated view of the system, including mnt, pid, net, ipc, uid, cgroup, and time. There is an in-depth video that discusses what containers are made from, and near the end there is a demonstration on how to build your own containers from the command line.\nBy easily allowing the necessary dependencies to live alongside the application code, containers make the \u0026ldquo;works on my machine\u0026rdquo; problem less of a problem.\nBenefits of Podman One of the most interesting features of Podman is that it is daemonless. There isn\u0026rsquo;t a process running on your system managing your containers. In contrast, the Docker client is reliant upon the Docker daemon (often running as root) to be able to build and run containers.\nPodman is rootless by default. It is now possible to run the Docker daemon rootless as well, but it\u0026rsquo;s still not the default behaviour.\nI\u0026rsquo;ve also observed that so far my 2019 16\u0026quot; MacBook Pro hasn\u0026rsquo;t sounded like a jet engine, although I haven\u0026rsquo;t performed any disk-intensive operations yet.\nInstalling Podman Running Podman on macOS is more involved than on Linux, because the podman-machine must run Linux inside of a virtual machine. Fortunately, the installation is made simple with brew (read this if you\u0026rsquo;re installing Podman on Linux):\nbrew install podman The podman-machine must be started:\n# This is not necessary on Linux podman machine init podman machine start Running a container Let\u0026rsquo;s try to pull an image:\n$ podman pull alpine Trying to pull docker.io/library/alpine:latest... Getting image source signatures Copying blob sha256:a0d0a0d46f8b52473982a3c466318f479767577551a53ffc9074c9fa7035982e Copying config sha256:14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab Writing manifest to image destination Storing signatures 14119a10abf4669e8cdbdff324a9f9605d99697215a0d21c360fe8dfa8471bab If you\u0026rsquo;re having an issue pulling images, you may need to remove ~/.docker/config.json or remove the set of auths in the configuration as mentioned here.\nand then run and exec into the container:\n$ podman run --rm -ti alpine Error: error preparing container 99ace1ef8a78118e178372d91fd182e8166c399fbebe0f676af59fbf32ce205b for attach: error configuring network namespace for container 99ace1ef8a78118e178372d91fd182e8166c399fbebe0f676af59fbf32ce205b: error adding pod unruffled_bohr_unruffled_bohr to CNI network \u0026quot;podman\u0026quot;: unexpected end of JSON input What does this error mean? A bit of searching led to this GitHub issue.\nUntil the fix is released, a workaround is to just specify a port (even when it\u0026rsquo;s not needed):\npodman run -p 4242 --rm -ti alpine If you\u0026rsquo;re reading this from the future, there is a good chance specifying a port won\u0026rsquo;t be needed.\nAnother example of running a container with Podman can be found in the Jellyfin Documentation.\nAliasing docker with podman Force of habit (or other scripts) may have you calling docker. To work around this:\nalias docker=podman podman-compose You may be wondering: what about docker-compose? Well, it claims to be a drop-in replacement for it: podman-compose.\npip3 install --user podman-compose Now let\u0026rsquo;s create a docker-compose.yml file to test:\ncat \u0026lt;\u0026lt; EOF \u0026gt;\u0026gt; docker-compose.yml version: '2' services: hello_world: image: ubuntu command: [/bin/echo, 'Hello world'] EOF Now run:\n$ podman-compose up podman pod create --name=davegallant.github.io --share net 40d61dc6e95216c07d2b21cea6dcb30205bfcaf1260501fe652f05bddf7e595e 0 podman create --name=davegallant.github.io_hello_world_1 --pod=davegallant.github.io -l io.podman.compose.config-hash=123 -l io.podman.compose.project=davegallant.github.io -l io.podman.compose.version=0.0.1 -l com.docker.compose.container-number=1 -l com.docker.compose.service=hello_world --add-host hello_world:127.0.0.1 --add-host davegallant.github.io_hello_world_1:127.0.0.1 ubuntu /bin/echo Hello world Resolved \u0026quot;ubuntu\u0026quot; as an alias (/etc/containers/registries.conf.d/000-shortnames.conf) Trying to pull docker.io/library/ubuntu:latest... Getting image source signatures Copying blob sha256:f3ef4ff62e0da0ef761ec1c8a578f3035bef51043e53ae1b13a20b3e03726d17 Copying blob sha256:f3ef4ff62e0da0ef761ec1c8a578f3035bef51043e53ae1b13a20b3e03726d17 Copying config sha256:597ce1600cf4ac5f449b66e75e840657bb53864434d6bd82f00b172544c32ee2 Writing manifest to image destination Storing signatures 1a68b2fed3fdf2037b7aef16d770f22929eec1d799219ce30541df7876918576 0 podman start -a davegallant.github.io_hello_world_1 Hello world This should more or less provide the same results you would come to expect with Docker. The README does clearly state that podman-compose is under development.\nSummary Installing Podman on macOS was not seamless, but it was manageable within 30 minutes. I would recommend giving Podman a try to anyone who is unhappy with experiencing forced Docker updates, or who is interested in using a more modern technology for running containers.\nOne caveat to mention is that there isn\u0026rsquo;t an official graphical user interface for Podman, but there is an open issue considering one. If you rely heavily on Docker Desktop\u0026rsquo;s UI, you may not be as interested in using podman yet.\nUpdate: After further usage, bind mounts do not seem to work out of the box when the client and host are on different machines. A rather involved solution using sshfs was shared here.\nI had been experimenting with Podman on Linux before writing this, but after listening to this podcast episode, I was inspired to give Podman a try on macOS.\n","tags":["docker","podman","containers"]},{"title":"Automatically Rotating AWS Access Keys","href":"/blog/automatically-rotating-aws-keys/","summary":"Rotating credentials is a security best practice. This morning, I read a question about automatically rotating AWS Access Keys without having to go through the hassle of navigating the AWS console. There are some existing solutions already, but I decided to write a script since it was incredibly simple. The script could be packed up as a systemd/launchd service to continually rotate access keys in the background.\nIn the longer term, migrating my local workflows to aws-vault seems like a more secure solution. This would mean that credentials (even temporary session credentials) never have to be written in plaintext to disk (i.e. where AWS suggests). Any existing applications, such as Terraform, could have their credentials passed to them from aws-vault, which retrieves them from the OS’s secure keystore. There is even a rotate command included.\n","content":"Rotating credentials is a security best practice. This morning, I read a question about automatically rotating AWS Access Keys without having to go through the hassle of navigating the AWS console. There are some existing solutions already, but I decided to write a script since it was incredibly simple. The script could be packed up as a systemd/launchd service to continually rotate access keys in the background.\nIn the longer term, migrating my local workflows to aws-vault seems like a more secure solution. This would mean that credentials (even temporary session credentials) never have to be written in plaintext to disk (i.e. where AWS suggests). Any existing applications, such as Terraform, could have their credentials passed to them from aws-vault, which retrieves them from the OS\u0026rsquo;s secure keystore. There is even a rotate command included.\n","tags":["aws","python","security","aws-vault"]},{"title":"Why I Threw Out My Dotfiles","href":"/blog/why-i-threw-out-my-dotfiles/","summary":"Over the years I have collected a number of dotfiles that I have shared across both Linux and macOS machines (~/.zshrc, ~/.config/git/config, ~/.config/tmux/tmux.conf, etc). I have tried several different ways to manage them, including bare git repos and utilities such as GNU Stow. These solutions work well enough, but I have since found what I would consider a much better solution for organizing user configuration: home-manager.\n","content":"Over the years I have collected a number of dotfiles that I have shared across both Linux and macOS machines (~/.zshrc, ~/.config/git/config, ~/.config/tmux/tmux.conf, etc). I have tried several different ways to manage them, including bare git repos and utilities such as GNU Stow. These solutions work well enough, but I have since found what I would consider a much better solution for organizing user configuration: home-manager.\nWhat is home-manager? Before understanding home-manager, it is worth briefly discussing what nix is. nix is a package manager that originally spawned from a PhD thesis. Unlike other package managers, it uses symbolic links to keep track of the currently installed packages, keeping around the old ones in case you may want to rollback.\nFor example, I have used nix to install the package bind which includes dig. You can see that it is available on multiple platforms. The absolute path of dig can be found by running:\n$ ls -lh $(which dig) lrwxr-xr-x 73 root 31 Dec 1969 /run/current-system/sw/bin/dig -\u0026gt; /nix/store/0r4qdyprljd3dki57jn6c6a8dh2rbg9g-bind-9.16.16-dnsutils/bin/dig Notice that there is a hash included in the file path? This is a nix store path and is computed by the nix package manager. This nix pill does a good job explaining how this hash is computed. All of the nix pills are worth a read, if you are interested in learning more about nix itself. However, using home-manager does not require extensive knowledge of nix.\nPart of the nix ecosystem includes nixpkgs. Many popular tools can be found already packaged in this repository. As you can see with these stats, there is a large number of existing packages that are being maintained by the community. Contributing a new package is easy, and anyone can do it!\nhome-manager leverages the nix package manager (and nixpkgs), as well the nix language so that you can declaratively define your system configuration. I store my nix-config in git so that I can keep track of my packages and configurations, and retain a clean and informative git commit history so that I can understand what changed and why.\nSetting up home-manager ⚠️ If you run this on your main machine, make sure you backup your configuration files first. home-manager is pretty good about not overwriting existing configuration, but it is better to have a backup! Alternatively, you could test this out on a VM or cloud instance.\nThe first thing you should do is install nix:\ncurl -L https://nixos.org/nix/install | sh It\u0026rsquo;s generally not a good idea to curl and execute files from the internet (without verifying integrity), so you might want to download the install script first and take a look before executing it!\nOpen up a new shell in your terminal and running nix should work. If not, run . ~/.nix-profile/etc/profile.d/nix.sh\nNow, install home-manager:\nnix-channel --add https://github.com/nix-community/home-manager/archive/master.tar.gz home-manager nix-channel --update nix-shell '\u0026lt;home-manager\u0026gt;' -A install You should see a wave of /nix/store/* paths being displayed on your screen.\nNow, to start off with a basic configuration, open up ~/.config/nixpkgs/home.nix in the editor of your choice and paste this in (you will want to change userName and homeDirectory):\n{ config, pkgs, ... }: { programs.home-manager.enable = true; home = { username = \u0026quot;dave\u0026quot;; homeDirectory = \u0026quot;/home/dave\u0026quot;; stateVersion = \u0026quot;21.11\u0026quot;; packages = with pkgs; [ bind exa fd ripgrep ]; }; programs = { git = { enable = true; aliases = { aa = \u0026quot;add -A .\u0026quot;; br = \u0026quot;branch\u0026quot;; c = \u0026quot;commit -S\u0026quot;; ca = \u0026quot;commit -S --amend\u0026quot;; cb = \u0026quot;checkout -b\u0026quot;; co = \u0026quot;checkout\u0026quot;; d = \u0026quot;diff\u0026quot;; l = \u0026quot;log --graph --pretty=format:'%Cred%h%Creset -%C(yellow)%d%Creset %s %Cgreen(%cr) %C(bold blue)\u0026lt;%an\u0026gt;%Creset' --abbrev-commit\u0026quot;; }; delta = { enable = true; options = { features = \u0026quot;line-numbers decorations\u0026quot;; whitespace-error-style = \u0026quot;22 reverse\u0026quot;; plus-style = \u0026quot;green bold ul '#198214'\u0026quot;; decorations = { commit-decoration-style = \u0026quot;bold yellow box ul\u0026quot;; file-style = \u0026quot;bold yellow ul\u0026quot;; file-decoration-style = \u0026quot;none\u0026quot;; }; }; }; extraConfig = { push = { default = \u0026quot;current\u0026quot;; }; pull = { rebase = true; }; }; }; starship = { enable = true; enableZshIntegration = true; settings = { add_newline = false; scan_timeout = 10; }; }; zsh = { enable = true; enableAutosuggestions = true; enableSyntaxHighlighting = true; history.size = 1000000; localVariables = { CASE_SENSITIVE = \u0026quot;true\u0026quot;; DISABLE_UNTRACKED_FILES_DIRTY = \u0026quot;true\u0026quot;; RPROMPT = \u0026quot;\u0026quot;; # override because macOS defaults to filepath ZSH_AUTOSUGGEST_HIGHLIGHT_STYLE = \u0026quot;fg=#838383,underline\u0026quot;; ZSH_DISABLE_COMPFIX = \u0026quot;true\u0026quot;; }; initExtra = '' export PAGER=less ''; shellAliases = { \u0026quot;..\u0026quot; = \u0026quot;cd ..\u0026quot;; grep = \u0026quot;rg --smart-case\u0026quot;; ls = \u0026quot;exa -la --git\u0026quot;; }; \u0026quot;oh-my-zsh\u0026quot; = { enable = true; plugins = [ \u0026quot;gitfast\u0026quot; \u0026quot;last-working-dir\u0026quot; ]; }; }; }; } Save the file and run:\nhome-manager switch You should see another wave of /nix/store/* paths. The new configuration should now be active.\nIf you run zsh, you should see that you have starship and access to several other utils such as rg, fd, and exa.\nThis basic configuration above is also defining your ~/.config/git/config and .zshrc. If you already have either of these files, home-manager will complain about them already existing.\nIf you run cat ~/.zshrc, you will see the way these configuration files are generated.\nYou can extend this configuration for programs such as (neo)vim, emacs, alacritty, ssh, etc. To see other programs, take a look at home-manager/modules/programs.\nGateway To Nix In ways, home-manager can be seen as a gateway to the nix ecosystem. If you have enjoyed the way you can declare user configuration with home-manager, you may be interested in expanding your configuration to include other system dependencies and configuration. For example, in Linux you can define your entire system\u0026rsquo;s configuration (including the kernel, kernel modules, networking, filesystems, etc) in nix. For macOS, there is nix-darwin that includes nix modules for configuring launchd, dock, and other preferences and services. You may also want to check out Nix Flakes: a more recent feature that allows you declare dependencies, and have them automatically pinned and hashed in flake.lock, similar to that of many modern package managers.\nWrapping up The title of this post is slightly misleading, since it\u0026rsquo;s possible to retain some of your dotfiles and have them intermingle with home-manager by including them alongside nix. The idea of defining user configuration using nix can provide a clean way to maintain your configuration, and allow it to be portable across platforms. Is it worth the effort to migrate away from shell scripts and dotfiles? I\u0026rsquo;d say so.\nYou can find my nix config here.\n","tags":["nix","dotfiles","home-manager"]},{"title":"What to Do with a Homelab","href":"/blog/what-to-do-with-a-homelab/","summary":"A homelab can be an inexpensive way to host a multitude of internal/external services and learn a lot in the process.\n","content":"A homelab can be an inexpensive way to host a multitude of internal/external services and learn a lot in the process.\nDo you want to host your own media server? Ad blocker? Reverse proxy? Are you interested in learning more about Linux? Virtualization? Networking? Security? A homelab can be a playground to enhance your computer skills, without worrying about breaking anything important.\nOne of the best parts about building a homelab is that it doesn\u0026rsquo;t have to be a large investment in terms of hardware. One of the simplest ways to build a homelab is out of a refurbished computer. Having multiple machines/nodes provides the advantage of increased redundancy, but starting out with a single node is enough to reap many of the benefits of having a homelab.\nVirtualization Virtualizing your hardware is an organized way of dividing up your machine\u0026rsquo;s resources. This can be done with something such as a Virtual Machine or something lighter like a container using LXC or runC. Containers have much less overhead in terms of boot time and storage allocation. This Stack Overflow answer sums it up nicely.\nA hypervisor such as Proxmox can be installed in minutes on a new machine. It provides a web interface and a straight-forward way to spin up new VMs and containers. Even if your plan is to run mostly docker containers, Proxmox can be a useful abstraction for managing VMs, disks and running scheduled backups. You can even run docker within an LXC container by enabling nested virtualization. You\u0026rsquo;ll want to ensure that VT-d and VT-x are enabled in the BIOS if you decide to install a hypervisor to manage your virtualization.\nServices Here is a list of some useful services to consider:\nJellyfin or Plex - a common gateway to self-hosting that enables a \u0026ldquo;self-hosted Netflix\u0026rdquo; experience that puts you in control of the content (guaranteed to make your partner and kids happy) changedetection - is a self-hosted equivalent to something like visualping.io that can notify you when a webpage changes and keep track of the diffs Adguard or Pihole - can block a list of known trackers for all clients on your local network with the added benefit of speeding up web page load times gitea - A lightweight git server that can be used to mirror git repos and host private content miniflux - a minimalist RSS reader gethomepage - A customizable landing page for quick access to services with many supported widgets that can query APIs and display information Uptime Kuma - A tool for monitoring the uptime of services, with notification support Speedtest Tracker - a way to monitor the performance of your internet connection and/or vpn connection Stirling-PDF - a self-hosted PDF manipulation tool that will keep your data private There is a large number of services you can self-host, including your own applications that you might be developing. Homelabbing allows you to have control over your data and services, and gives you the opportunity to be a software, network, and infrastructure engineer all at once.\nVPN Tailscale is a quick way to create a flat network for all of your services. With its MagicDNS, you can reference the names of machines like changedetection rather than using an IP address or managing DNS yourself. By using this mesh-like VPN, you can easily create a secure tunnel to your homelab from anywhere.\nMonitoring Monitoring can become an important aspect of your homelab after it starts to become something that is relied upon. One of the simplest ways to setup some monitoring is using netdata. It can be installed on individual containers, VMs, and also a hypervisor (such as Proxmox). All of the monitoring works out of the box by detecting disks, memory, network interfaces, etc.\nAdditionally, agents installed on different machines can all be centrally viewed in netdata, and it can alert you when some of your infrastructure is down or in a degraded state. Adding additional nodes to netdata is as simple as a 1-line shell command.\nAs mentioned above, Uptime Kuma is a convenient way to track uptime and monitor the availability of your services.\nIn Summary Building out a homelab can be a rewarding experience and it doesn\u0026rsquo;t require buying a rack full of expensive servers to get a significant amount of utility. There are many services that you can run that require very minimal setup, making it possible to get a server up and running in a short period of time, with monitoring, and that can be securely connected to remotely.\nIf you\u0026rsquo;re looking for a steady stream of ideas for your homelab, check out selfhosted.show.\n","tags":["self-hosted","proxmox","tailscale"]},{"title":"AppGate SDP on Arch Linux","href":"/blog/appgate-sdp-on-arch-linux/","summary":"AppGate SDP provides a Zero Trust network. This post describes how to get AppGate SDP 4.3.2 working on Arch Linux.\n","content":"AppGate SDP provides a Zero Trust network. This post describes how to get AppGate SDP 4.3.2 working on Arch Linux.\nDepending on the AppGate SDP Server that is running, you may require a client that is more recent than the latest package on AUR. As of right now, the latest AUR is 4.2.2-1.\nThese steps highlight how to get it working with Python3.8 by making a one-line modification to AppGate source code.\nPackaging We already know the community package is currently out of date, so let\u0026rsquo;s clone it:\ngit clone https://aur.archlinux.org/appgate-sdp.git cd appgate-sdp You\u0026rsquo;ll likely notice that the version is not what we want, so let\u0026rsquo;s modify the PKGBUILD to the following:\n# Maintainer: Pawel Mosakowski \u0026lt;pawel at mosakowski dot net\u0026gt; pkgname=appgate-sdp conflicts=('appgate-sdp-headless') pkgver=4.3.2 _download_pkgver=4.3 pkgrel=1 epoch= pkgdesc=\u0026quot;Software Defined Perimeter - GUI client\u0026quot; arch=('x86_64') url=\u0026quot;https://www.cyxtera.com/essential-defense/appgate-sdp/support\u0026quot; license=('custom') # dependencies calculated by namcap depends=('gconf' 'libsecret' 'gtk3' 'python' 'nss' 'libxss' 'nodejs' 'dnsmasq') source=(\u0026quot;https://sdpdownloads.cyxtera.com/AppGate-SDP-${_download_pkgver}/clients/${pkgname}_${pkgver}_amd64.deb\u0026quot; \u0026quot;appgatedriver.service\u0026quot;) options=(staticlibs) prepare() { tar -xf data.tar.xz } package() { cp -dpr \u0026quot;${srcdir}\u0026quot;/{etc,lib,opt,usr} \u0026quot;${pkgdir}\u0026quot; mv -v \u0026quot;$pkgdir/lib/systemd/system\u0026quot; \u0026quot;$pkgdir/usr/lib/systemd/\u0026quot; rm -vrf \u0026quot;$pkgdir/lib\u0026quot; cp -v \u0026quot;$srcdir/appgatedriver.service\u0026quot; \u0026quot;$pkgdir/usr/lib/systemd/system/appgatedriver.service\u0026quot; mkdir -vp \u0026quot;$pkgdir/usr/share/licenses/appgate-sdp\u0026quot; cp -v \u0026quot;$pkgdir/usr/share/doc/appgate/copyright\u0026quot; \u0026quot;$pkgdir/usr/share/licenses/appgate-sdp\u0026quot; cp -v \u0026quot;$pkgdir/usr/share/doc/appgate/LICENSE.github\u0026quot; \u0026quot;$pkgdir/usr/share/licenses/appgate-sdp\u0026quot; cp -v \u0026quot;$pkgdir/usr/share/doc/appgate/LICENSES.chromium.html.bz2\u0026quot; \u0026quot;$pkgdir/usr/share/licenses/appgate-sdp\u0026quot; } md5sums=('17101aac7623c06d5fbb95f50cf3dbdc' '002644116e20b2d79fdb36b7677ab4cf') Let\u0026rsquo;s first make sure we have some dependencies. If you do not have yay, check it out.\nyay -S dnsmasq gconf Now, let\u0026rsquo;s install it:\nmakepkg -si Running the client Ok, let\u0026rsquo;s run the client by executing appgate.\nIt complains about not being able to connect.\nEasy fix:\nsudo systemctl start appgatedriver.service Now we should be connected\u0026hellip; but DNS is not working?\nFixing the DNS Running resolvectl should display that something is not right.\nWhy is the DNS not being set by appgate?\n$ head -3 /opt/appgate/linux/set_dns #!/usr/bin/env python3 ''' This is used to set and unset the DNS. It seems like python3 is required for the DNS setting to happen. Let\u0026rsquo;s try to run it.\n$ sudo /opt/appgate/linux/set_dns /opt/appgate/linux/set_dns:88: SyntaxWarning: \u0026quot;is\u0026quot; with a literal. Did you mean \u0026quot;==\u0026quot;? servers = [( socket.AF_INET if x.version is 4 else socket.AF_INET6, map(int, x.packed)) for x in servers] Traceback (most recent call last): File \u0026quot;/opt/appgate/linux/set_dns\u0026quot;, line 30, in \u0026lt;module\u0026gt; import dbus ModuleNotFoundError: No module named 'dbus' Ok, let\u0026rsquo;s install it:\n$ sudo python3.8 -m pip install dbus-python Will it work now? Not yet. There\u0026rsquo;s another issue:\n$ sudo /opt/appgate/linux/set_dns /opt/appgate/linux/set_dns:88: SyntaxWarning: \u0026quot;is\u0026quot; with a literal. Did you mean \u0026quot;==\u0026quot;? servers = [( socket.AF_INET if x.version is 4 else socket.AF_INET6, map(int, x.packed)) for x in servers] module 'platform' has no attribute 'linux_distribution' This is a breaking change in Python3.8.\nSo what is calling platform.linux_distribution?\nLet\u0026rsquo;s search for it:\n$ sudo grep -r 'linux_distribution' /opt/appgate/linux/ /opt/appgate/linux/nm.py: if platform.linux_distribution()[0] != 'Fedora': Aha! So this is in the local AppGate source code. This should be an easy fix. Let\u0026rsquo;s just replace this line with:\nif True: # Since we are not using Fedora :) Wrapping up It turns out there are breaking changes in Python3.8.\nThe docs say Deprecated since version 3.5, will be removed in version 3.8: See alternative like the distro package.\nI suppose this highlights one of the caveats of relying upon the system\u0026rsquo;s Python, rather than having an isolated, dedicated environment for all dependencies.\n","tags":["linux","vpn","python"]}]