Latest posts
- Testing Linux memory limits is a bit of a painFeb 13, 2026cks
For reasons outside of the scope of this entry, I want to test how various systemd memory resource limits work and interact with each other (which means that I'm really digging into cgroup v2 memory controls). When I started trying to do this, it turned out that I had no good test program (or programs), although I had some ones that gave me partial answers. There are two complexities in memory usa
- Undo in Vi and its successors, and my views on the messFeb 12, 2026cks
The original Bill Joy vi famously only had a single level of undo (which is part of what makes it a product of its time). The 'u' command either undid your latest change or it redid the change, undo'ing your undo. When POSIX and the Single Unix Specification wrote vi into the standard, they required this behavior; the vi specification requires 'u' to work the same as it does in ex, where it is spe
- Moving to make many of my SSH logins not report things on loginFeb 11, 2026cks
I've been logging in to Unix machines for what is now quite a long time. When I started, it was traditional for your login process to be noisy. The login process itself would tell you last login details and the 'message of the day' ('motd'), and people often made their shell .profile or .login report more things, so you could see things like: Last login: Tue Feb 10 22:16:14 2026 from 128.100.X.Y
- A fun Python puzzle with circular importsFeb 10, 2026cks
Baptiste Mispelon asked an interesting Python quiz (via, via @glyph): Can someone explain this #Python import behavior? a.py contains `A = 1; from b import *` Can you guess and explain what happens when you run `python c.py`? I encourage you to guess which of the options in the original post is the actual behavior before you read the rest of this entry. There are two things going on here. The fi
- Systemd and blocking connections to localhost, including via 'any'Feb 09, 2026cks
I recently discovered a surprising path to accessing localhost URLs and services, where instead of connecting to 127.0.0.1 or the IPv6 equivalent, you connected to 0.0.0.0 (or the IPv6 equivalent). In that entry I mentioned that I didn't know if systemd's IPAddressDeny would block this. I've now tested this, and the answer is that systemd's restrictions do block this. If you set 'IPAddressDeny=loc
- The original vi is a product of its time (and its time has passed)Feb 08, 2026cks
Recently I saw another discussion of how some people are very attached to the original, classical vi and its behaviors (cf). I'm quite sympathetic to this view, since I too am very attached to the idiosyncratic behavior of various programs I've gotten used to (such as xterm's very specific behavior in various areas), but at the same time I had a hot take over on the Fediverse: Hot take: basic vim
- How we failed to notice a power failureFeb 07, 2026cks
Over on the Fediverse, I mentioned that we once missed noticing that there had been a power failure. Naturally there is a story there (and this is the expanded version of what I said in the Fediverse thread). A necessary disclaimer is that this was all some time ago and I may be mangling or mis-remembering some of the details. My department is spread across multiple buildings, one of which has my
- A surprising path to accessing localhost URLs and HTTP servicesFeb 06, 2026cks
One of the classic challenges in web security is DNS rebinding. The simple version is that you put some web service on localhost in order to keep outside people from accessing it, and then some joker out in the world makes 'evil.example.org' resolve to 127.0.0.1 and arranges to get you to make requests to it. Sometimes this is through JavaScript in a browser, and sometimes this is by getting you t
- The meaning of connecting to INADDR_ANY in TCP and UDPFeb 05, 2026cks
An interesting change to IP behavior landed in FreeBSD 15, as I discovered by accident. To quote from the general networking section of the FreeBSD 15 release notes: Making a connection to INADDR_ANY, i.e., using it as an alias for localhost, is now disabled by default. This functionality can be re-enabled by setting the net.inet.ip.connect_inaddr_wild sysctl to 1. cd240957d7ba The change's commit
- I prefer to pass secrets between programs through standard inputFeb 04, 2026cks
There are a variety of ways to pass secrets from one program to another on Unix, and many of them may expose your secrets under some circumstances. A secret passed on the command line is visible in process listings; a secret passed in the environment can be found in the process's environment (which can usually be inspected by outside parties). When I've had to deal with this in administrative prog
- The consoles of UEFI, serial and otherwise, and their discontentsFeb 03, 2026cks
UEFI is the modern firmware standard for x86 PCs and other systems; sometimes the actual implementation is called a UEFI BIOS, but the whole area is a bit confusing. I recently wrote about getting FreeBSD to use a serial console on a UEFI system and mentioned that some UEFI BIOSes could echo console output to a serial port, which caused Greg A. Woods to ask a good question in a comment: So, how do
- Estimating where your Prometheus Blackbox TCP query-response check failedFeb 02, 2026cks
As covered recently, the normal way to check simple services from outside in a Prometheus environment is with Prometheus Blackbox, which is somewhat complicated to understand. One of its abstractions is a prober, a generic way of checking some service using HTTP, DNS queries, a TCP connection, and so on. The TCP prober supports conducting a query-response dialog once you connect, but currently (as
- Early Linux package manager history and patching upstream source releasesFeb 01, 2026cks
One of the important roles of Linux system package managers like dpkg and RPM is providing a single interface to building programs from source even though the programs may use a wide assortment of build processes. One of the source building features that both dpkg and RPM included (I believe from the start) is patching the upstream source code, as well as providing additional files along with it.
- Making a FreeBSD system have a serial console on its second serial portJan 31, 2026cks
Over on the Fediverse I said: Today's other work achievement: getting a UEFI booted FreeBSD 15 machine to use a serial console on its second serial port, not its first one. Why? Because the BMC's Serial over Lan stuff appears to be hardwired to the second serial port, and life is too short to wire up physical serial cables to test servers. The basics of serial console support for your FreeBSD mach
- Why I'm ignoring pretty much all new Python packaging toolsJan 30, 2026cks
One of the things going on right now is that Python is doing a Python developer survey. On the Fediverse, I follow a number of people who do Python stuff, and they've been posting about various aspects of the survey, including a section on what tools people use for what. This gave me an interesting although very brief look into a world that I'm deliberately ignoring, and I'm doing that because I f
- Why Linux wound up with system package managersJan 29, 2026cks
Yesterday I discussed the two sorts of program package managers, system package managers that manage the whole system and application package managers that mostly or entirely manage third party programs. Commercial Unix got application package managers in the very early 1990s, but Linux's first program managers were system package managers, in dpkg and RPM (or at least those seem to be the first L
- The two subtypes of one sort of package managers, the "program manager"Jan 28, 2026cks
I've written before that one of the complications of talking about package managers and package management is that there are two common types of package managers, program managers (which manage installed programs on a system level) and module managers (which manage package dependencies for your project within a language ecosystem or maybe a broader ecosystem). Today I realized that there is a furt
- Forcing a Go generic type to be a pointer type (and some challenges)Jan 27, 2026cks
Recently I saw a Go example that made me scratch my head and decode what was going on (you can see it here). Here's what I understand about what's going on. Suppose that you want to create a general interface for a generic type that requires any concrete implementation to be a pointer type. We can do this by literally requiring a pointer: type Pointer[P any] interface { *P } That this is al
- Scraping the FreeBSD 'mpd5' daemon to obtain L2TP VPN usage dataJan 26, 2026cks
We have a collection of VPN servers, some OpenVPN based and some L2TP based. They used to be based on OpenBSD, but we're moving from OpenBSD to FreeBSD and the VPN servers recently moved too. We also have a system for collecting Prometheus metrics on VPN usage, which worked by parsing the output of things. For OpenVPN, our scripts just kept working when we switched to FreeBSD because the two OSes
- Printing things in colour is not simpleJan 25, 2026cks
Recently, Verisimilitude left a comment on my entry on X11's DirectColor visual type, where they mentioned that L Peter Deutsch, the author of Ghostscript, lamented using twenty-four bit colour for Ghostscript rather than a more flexible approach, which you may need in printing things with colour. As it happens, I know a bit about this area for two or three reasons, which come at it from different
- Understanding <code>query_response</code> in Prometheus Blackbox's tcp proberJan 24, 2026cks
Prometheus Blackbox is somewhat complicated to understand. One of its fundamental abstractions is a 'prober', a generic way of probing some service (such as making HTTP requests or DNS requests). One prober is the 'tcp' prober, which makes a TCP connection and then potentially conducts a conversation with the service to verify its health. For example, here's a ClamAV daemon health check, which con
- Pitfalls in using Prometheus Blackbox to monitor external SMTPJan 23, 2026cks
The news of the day is that Microsoft had a significant outage inside their Microsoft 365 infrastructure. We noticed when we stopped being able to deliver email to the university's institutional email system, which was a bit mysterious in the usual way of today's Internet: The joys of modern email: "Has Microsoft decided to put all of our email on hold or are they having a global M365 inbound SMTP
- What ZFS people usually mean when they talk about "ZFS metadata"Jan 22, 2026cks
Recently I read Understanding ZFS Scrubs and Data Integrity (via), which is a perfectly good article and completely accurate, bearing in mind some qualifications which I'm about to get into. One of the things this article says in the preface is: In this article, we will walk through what scrubs do, how the Merkle tree layout lets ZFS validate metadata and data from end to end, [...] This is both c
- The long painful history of (re)using <code>login</code> to log people inJan 21, 2026cks
The news of the time interval is that Linux's usual telnetd has had a giant security vulnerability for a decade. As people on the Fediverse observed, we've been here before; Solaris apparently had a similar bug 20 or so years ago (which was CVE-2007-0882, cf, via), and AIX in the mid 1990s (CVE-1999-0113, source, also)), and also apparently SGI Irix, and no doubt many others (eg). It's not necessa
- TCP, UDP, and listening only on a specific IP addressJan 20, 2026cks
One of the surprises of TCP and UDP is that when your program listens for incoming TCP connections or UDP packets, you can chose to listen only on a specific IP address instead of all of the IP addresses that the current system has. This behavior started as a de-facto standard but is now explicitly required for TCP in RFC 9293 section 3.9.1.1. There are at least two uses of this feature; to restri
- Single sign on systems versus X.509 certificates for the webJan 19, 2026cks
Modern single sign on specifications such as OIDC and SAML and systems built on top of them are fairly complex things with a lot of moving parts. It's possible to have a somewhat simple surface appearance for using them in web servers, but the actual behind the scenes implementation is typically complicated, and of course you need an identity provider server and its supporting environment as well
- People cannot "just pay attention" to (boring, routine) thingsJan 18, 2026cks
Sometimes, people in technology believe that we can solve problems by getting people to pay attention. This comes up in security, anti-virus efforts, anti-phish efforts, monitoring and alert handling, warning messages emitted by programs, warning messages emitted by compilers and interpreters, and many other specific contexts. We are basically always wrong. One of the core, foundational results fr
- Systemd-networkd and giving your virtual devices alternate namesJan 17, 2026cks
Recently I wrote about how Linux network interface names have a length limit, of 15 characters. You can work around this limit by giving network interfaces an 'altname' property, as exposed in (for example) 'ip link'. While you can't work around this at all in Canonical's Netplan, it looks like you can have this for your VLANs in systemd-networkd, since there's AlternativeName= in the systemd.link
- TCP and UDP and implicit "standard" elements of thingsJan 16, 2026cks
Recently, Verisimilitude left a comment on this entry of mine about binding TCP and UDP ports to a specific address. That got me thinking about features that have become standard elements of things despite not being officially specified and required. TCP and UDP are more or less officially specified in various RFCs and are implicitly specified by what happens on the wire. As far as I know, nowhere
- Linux network interface names have a length limit, and NetplanJan 15, 2026cks
Over on the Fediverse, I shared a discovery: This is my (sad) face that Linux interfaces have a maximum name length. What do you mean I can't call this VLAN interface 'vlan22-matterlab'? Also, this is my annoyed face that Canonical Netplan doesn't check or report this problem/restriction. Instead your VLAN interface just doesn't get created, and you have to go look at system logs to find systemd-n
- Safely querying Spamhaus DNSBLs in EximJan 14, 2026cks
When querying Spamhaus DNS blocklists, either their public mirrors or through a DQS account, the DNS blocklists can potentially return error codes in 127.255.255.0/24 (also). Although Exim has a variety of DNS blocklist features, it doesn't yet let you match return codes based on CIDR netblocks. However, it does have a magic way of doing this. The magic way is to stick '!&0.255.255.0' on the end o
- Something you don't want to do when using Spamhaus's DQS with EximJan 13, 2026cks
For reasons outside the scope of this entry, we recently switched from Spamhaus's traditional public DNS (what is now called the 'public mirrors') to an account with their Data Query Service. The DQS data can still be queried via DNS, which presents a problem: DNS queries have no way to carry any sort of access key with them. Spamhaus has solved this problem by embedding your unique access key in
- An annoyance in how Netplan requires you to specify VLANsJan 12, 2026cks
Netplan is Canonical's more or less mandatory method of specifying networking on Ubuntu. Netplan has a collection of limitations and irritations, and recently I ran into a new one, which is how VLANs can and can't be specified. To explain this, I can start with the YAML configuration language. To quote the top level version, it looks like: network: version: NUMBER renderer: STRING [...] e
- Early experience with using Linux tc to fight bufferbloat latencyJan 11, 2026cks
Over on the Fediverse I mentioned something recently: Current status: doing extremely "I don't know what I'm really doing, I'm copying from a website¹" things with Linux tc to see if I can improve my home Internet latency under load without doing too much damage to bandwidth or breaking my firewall rules. So far, it seems to work and things² claim to like the result. ¹ <documentation link> https:
- Distribution source packages and whether or not to embed in the source codeJan 10, 2026cks
When I described my current ideal Linux source package format, I said that it should be embedded in the source code of the software being packaged. In a comment, bitprophet had a perfectly reasonable and good preference the other way: Re: other points: all else equal I think I vaguely prefer the Arch "repo contains just the extras/instructions + a reference to the upstream source" approach as it's
- The Amanda backup system and "dump promotion"Jan 09, 2026cks
The Amanda backup system is what we use to handle our backups. One of Amanda's core concepts is a 'dump cycle', the amount of time between normally scheduled full backups for filesystems. If you have a dumpcycle of 7 days and Amanda does a full backup of a filesystem on Monday, its normal schedule for the next full backup is next Monday. However, Amanda can 'promote' a full backup ahead of schedu
- What 24 hours of traffic looks like to our main web server in January 2026Jan 08, 2026cks
One of the services we operate for the department is a traditional Apache-based shared web server, with things like people's home pages (eg), pages for various groups, and so on (we call this our departmental web server). This web server has been there for a very long time and its URLs have spread everywhere, and in the process it's become quite popular for some things. These days there are a lot
- Why we have some AC units on one of our our internal networksJan 07, 2026cks
I mentioned on the Fediverse a while back that we have air conditioners on our internal network. Well, technically what we have on the internal network is separate (and optional) controller devices that connect to the physical AC units themselves, but as they say, this is close enough. Of course there's a story here: Why do we have networked AC controllers? Well, they control portable AC units tha
- Prometheus, Let's Encrypt, and making sure all our TLS certificates are monitoredJan 06, 2026cks
I recently wrote about the complexities of getting programs to report the TLS certificates they use, where I theorized about writing a script to scrape this information out of places like the Apache configuration files, and then today I realized the obvious specific approach for our environment: Obvious realization is obvious: since we universally use Let's Encrypt with certbot and follow standard
- Some notes to myself on Super-based bindings in GNU EmacsJan 05, 2026cks
I recently had to deal with GNU Emacs lsp-mode in a context where I cared a bit about its keybindings, and in the process of that ran across mention of what one could call its leader prefix, s-l. People who use GNU Emacs a lot will know what this specific 's-' notation means, but I'm not one of them, so it took me a bit of research to work it out. This is GNU Emacs' notation for 'Super', one of th
- A small suggestion in modern Linux: take screenshots (before upgrades)Jan 04, 2026cks
Mike Hoye recently wrote Powering Up, which is in part about helping people install (desktop) Linux, and the Fediverse thread version of it reminded me of something that I don't do enough of: A related thing I've taken to doing before potential lurching changes (like Linux distribution upgrades) is to take screenshots and window images. Because comparing a now and then image is a heck of a lot eas
- The complexities of getting programs to report the TLS certificates they useJan 03, 2026cks
One of the practical reasons that TLS certificates have dangerous expiry times is that in most environments, it's up to you to remember to add monitoring for each TLS certificate that you use, either as part of general purpose monitoring of the service or specific monitoring for certificate expiry. It would be nice if programs that used TLS certificates inherently monitored their expiry, but that'
- A Go question: how do you test <code>select</code> based code?Jan 02, 2026cks
A while back I wrote an entry about understanding reading all available things from a Go channel (with a timeout), where the code used two selects to, well, let me quote myself: The goal of waitReadAll() is to either receive (read) all currently available items from a channel (possibly a buffered one) or to time out if nothing shows up in time. This requires two nested selects, with the inner one
- A little bit of complex design in phone "Level" applicationsJan 01, 2026cks
Modern smartphones have a lot of sensors; for example, they often have sensors that will report the phone's orientation and when it changes (which is used for things like 'wake up the screen when you pick up the phone'). One of the uses for these sensors is for little convenience applications, such as a "Level" app that uses the available sensors to report when the phone is level so you can use it
- My ideal Linux source package format (at the moment)Dec 31, 2025cks
I've written recently on why source packages are complicated and why packages should be declarative (in contrast to Arch style shell scripts), but I haven't said anything about what I'd like in a source package format, which will mostly be from the perspective of a system administrator who sometimes needs to modify upstream packages or package things myself. A source package format is a compromise
- Linux distribution packaging should be as declarative as possibleDec 30, 2025cks
A commentator on my entry on why Debian and RPM (source) packages are complicated suggested looking at Arch Linux packaging, where most of the information is in a single file as more or less a shell script (example). Unfortunately, I'm not a fan of this sort of shell script or shell script like format, ultimately because it's only declarative by convention (although I suspect Arch enforces some of
- Expiry times are dangerous, on "The dangers of SSL certificates"Dec 29, 2025cks
Recently I read Lorin Hochstein's The dangers of SSL certificates (via, among others), which talks about a Bazel build workflow outage caused by an expired TLS certificate. I had some direct reactions to this but after thinking about it I want to step back and say that in general, it's clear that expiry times are dangerous, often more or less regardless of where they appear. TLS certificate expiry
- Why Debian and RPM (source) packages are complicatedDec 28, 2025cks
A commentator on my early notes on dgit mentioned that they found packaging in Debian overly complicated (and I think perhaps RPMs as well) and would rather build and ship a container. On the one hand, this is in a way fair; my impression is that the process of specifying and building a container is rather easier than for source packages. On the other hand, Debian and RPM source packages are compl
- Python 2, GNU Emacs, and my LSP environment combine to shoot me in the footDec 26, 2025cks
So I had a thing happen: This is my angry face that GNU Emacs appears to have re-indented my entire Python file to a different standard without me noticing and I didn't catch it in time. And also it appears impossible in GNU Emacs to FIX this. I do not want four space no tabs, this is historical code that all files should be eight spaces with tabs (yes, Python 2). That 'Python 2' bit turns out to
- We should probably write some high level overviews of our environmentDec 26, 2025cks
Over on the Fediverse, I shared an old story that's partly about (system) documentation, and it sparked a thought, which is that we (I) should write up a brief high level overview of our overall environment. This should probably be one level higher than an end of service writeup, which are focused on a specific service (if we write them at all). The reason to do this is because our regular documen