arp242.net

I like using scrollbars

Fri, 10 Jul 2026 00:00:00 +0000

I like to use scrollbars. I move my mouse cursor to the scrollbar handle and move it about – scrolling like this is so much faster than with the wheel if you want to scroll more than a few lines.

I didn’t notice how often I did this until sites started breaking this. It seems to be fashionable to put a margin around the entire document, resulting in the scrollbar no longer being aligned to the edge of the screen.

This is an absolute bollockache because now you can no longer just move the mouse to the edge of the screen: you need to place it exactly on the scrollbar, requiring far greater precision.

You can see an example at /scrollmargin.html. You need a device with a mouse cursor to see the problem of course.

Some real examples, all of which started doing it in the last year or so:

FastMail
Outlook365
Stack Overflow “Beta Redesign”, although that was aborted.
Spotify (example)
GitLab (example)
Couchsurfing now completely hides the scrollbar (example) (aside: I am not saying that Couchsurfing has been infiltrated by AirBnB agents to run this communist woke libtard competition in to the ground, but I am saying that nothing Couchsurfing has done over the last 10 years contradicts that hypothesis.)

There is of course also the perennial “inappropriate scrollbar styling” problem, and recently ultra low contrast have also gained popularity; for example:

Substack (example)
MariaDB documentation (example)

I’m not against any scrollbar styling; in some cases it’s appropriate. Some more playful websites, some very specific and rare webapp use cases, things like that. I’m not a “system behaviour in every single last scenario” purist. The MariaDB reference docs is very much not one of those cases.

I like using scrollbars. I can’t be the only person who does. You’re frustrating users for no reason other than following some fad that makes things look ever so marginally nicer. No one pays that much attention to your design and they won’t notice. In ten years time everyone will look back at this in the same way we look back that extremely low contrast text fad from a decade ago: “yeah, that was rather silly”.

Chesterton's middle finger

Mon, 22 Jun 2026 00:00:00 +0000

The Chesterton’s fence analogy says that you need to be careful to change things if you don’t understand why things are the way they are, because there may be good reasons you haven’t considered yet. This is broadly good advice in many different contexts, including programming where it can be easy to “fix” some weird code only to discover there was a reason for the weirdness later on when things break.

This is Chesterson’s middle finger:

# List all commit body content (but not the subject line)
% git log --no-merges --format=format:'%b' | sed '/^$/d' | wc -l
295

That’s 295 lines of commit text over the last 13 years. In total. The commit subjects are usually just “fix page A” – even for huge changes – and are pointless. If I manually remove some dependabot, “revert commit”, and “fix typo” commit bodies it’s just 167 lines. That’s about one line a month.

There is no other documentation. There are barely any comments in the code.

This is Chesterson’s middle finger: “Yes, we did all these weird things and we’re not telling anyone why. Haha fuck you.”

This is not the first time I’ve seen a less-than-useful commit log, but it is the first time I’m hired after everyone else left. There is no one to ask.

In theory there was a three week handover period from the previous developer, but that was just as communicative as the commit log. Never have I sympathized more with Jack Bauer’s methods to extract information and I regret not employing some of them.

There are unfinished refactors. There are left-overs from removed features. There are features that were added but never linked/used but are still in the code. There are features no one seems to be using. Overall it seems to suffer quite badly from Chesterton’s gap (“There isn’t a fence yet? Lets build fences!” without asking if a fence is actually needed).

Everything is just a big fat middle finger.

Writing well is hard. Writing something vaguely passable is not. By and large there are three questions to answer: “What are you changing?”, “why are you changing it?”, and “why is this a good solution?”

Sometimes “Implement new feature X” is enough, although it’s rare there is nothing more to say – most of the time there is something to say about how/why it’s added as it is.

If you’re fixing a bug, refactoring or changing things, or making some other substantial change then it’s very rare there isn’t at least a paragraph or two to comment on the “what’s changing, why change it, and why is this good?” questions.

Writing this is not an optional extra; it’s part of the job. If you’re not doing it then you’re not doing your job as a software developer. It doesn’t need to be eloquent. It doesn’t need to be perfect English. It doesn’t need to be a comprehensive essay on the nature of reality. It’s okay to forget something (although better if you don’t). It just needs … something. Any half-way serious attempt will be infinitely better than nothing at all.

If you do nothing at all then you’re just giving everyone after you the finger.

Dependencies should be fetched directly from VCS

Fri, 12 Jun 2026 00:00:00 +0000

I’ve been writing Ruby at my new $dayjob in the last month. After spending most of the last decade writing Go it’s been a fun change of scenery. I did Ruby before (years ago) and I can’t really tell you which is “better” – Ruby is very different from Go in almost every respect and I find both quite effective in getting stuff done in their own way.

One aspect where I do feel Go is clearly better is dependency management; specifically the security aspect thereof. Go is not magically immune to malicious dependencies, but it is a lot more resistant to them chiefly because there is no “publish a package” step.

In Go dependencies are identified by URL, e.g. github.com/user/pkg. Go identifies which VCS is being used (git in this case) and fetches the tag or commit you specified in your go.mod file. The go.mod file serves as both a dependency specification and “lock file”. It lists exact versions; there is no ~>1.1. It includes both direct and indirect dependencies and lists your full dependency tree (the go command writes to go.mod).

A hash of all files is checked against known hashes on sum.golang.org to prevent tags from being replaced, and it uses a proxy to prevent repos from being left-pad’d.

There are more aspects to Go Modules, including security features, but I will skip over them for the purpose of this article. The relevant bit is “dependencies are identified by URL, the code is fetched directly from the VCS, and it does this for both direct and indirect dependencies”.

Auditing and updating dependencies is easy: I do git log -p old..new (usually via a forge web UI), read all the commits, and update the go.mod file. I don’t have many dependencies and those I do have don’t change much. It’s usually pretty fast. I don’t need to do careful in-depth reviews here; just look for suspicious stuff. Something like exec.Command(..) or http.Post(..) in a globbing library would stand out. It’s hard to really hide stuff.

I’ve been doing this for years for every dependency. As a solo developer. It’s easy. Some projects have much larger dependency trees and this becomes more time-consuming, but not hard or confusing. It’s still easy, just takes a bit of time.

For Ruby things are different as it has a “publish a package” step: you create a .gem archive and upload that to rubygems.org. You can put anything in there – no guarantee the .gem contents correspond to the source repo. To audit it I need to do something like:

curl -s https://rubygems.org/downloads/example-2.7.5.gem >old.gem
curl -s https://rubygems.org/downloads/example-2.8.2.gem >new.gem

mkdir old new
tar xf old.gem -C old
tar xf new.gem -C new

(cd old && tar xf data.tar.gz)
(cd new && tar xf data.tar.gz)

diff -urN old new

It works, I guess, but is far from easy. The individual commits are lost and is generally harder to audit. In some cases the diff is small enough that it’s okay. In other cases it’s huge and not having access to the commits is a pain. I can also totally see myself get confused about what I did and didn’t audit. I guess the number of people doing this sort of auditing is very low because it’s just such a pain.

This is not unique to Ruby: this is how many (most?) package systems work.

Almost all of the “side-channel attacks” I’ve seen are perhaps more accurately described as “package publishing attacks”. They rely on injecting something in the “publish a package” step. Whether that’s RubyGems, npm, PyPI, .tar.gz FTP downloads, or something else is a relatively minor detail. It’s rare that the actual source repo gets compromised as it’s just too visible. You need to at least slightly hide your exploit for it to be effective.

The recent npm compromises all relied on gaining access to the npm account and injecting something in the published package. xz had some exploit code in the source repo but was inert, hidden in a binary test file and only activated in the modified .tar.gz release. Back in 2018 event-stream added a dependency on flatmap-stream, which had nefarious code in index.min.js only on the published npm package.

Which points to a second problem: packages containing “compiled” resources. The JavaScript that TypeScript generates is not completely unreadable, it’s certainly much less readable and auditable than the original TypeScript. To say nothing of minified files or binary blobs. This is less of a problem in Ruby, but a far bigger problem in npm.

Last week RubyGems added a cooldown option and “AI-assisted vulnerability scanning against the most critical gems”. Not a bad thing to do as a short-term move, but I feel a more appropriate solution would be to reconsider the entire “publish a package” model. It just lacks the required transparency. AI tools are not going to magically fix that.

I probably would have created something similar to RubyGems myself twenty years ago. Distributing .tar.gz files on SourceForge was by and large how things worked and many projects did not have a publicly accessible VCS repo, or did not have one at all. Generally this worked fairly well at the time. I’m not blaming any one here – I would have done the same. RubyGems is just designed in a different world.

I’m not saying RubyGems and npm need to copy what Go does exactly in every respect and I’m not saying all of this is completely perfect either. But as far as I know, it’s the best anyone has come up with thus far. Some other aspects of Go modules (such as Minimal Version Selection) are less important.

I appreciating that completely changing how this works is hard and potentially disruptive. But dealing with this endless stream of hijacked packages is also hard and disruptive. So…

In the specific case of Bundler, there is already some support for this. You can do:

gem 'rails', git: 'https://github.com/rails/rails.git', tag: 'v8.1.2'

And it will fetch the rails gem from git, but will still fetch dependencies from rubygems.org. Maybe there is some way to cajole Bundler in to using git for everything, but it’s an uphill battle and easy to accidentally use rubygems.org.

Just thinking out loud here, but something like this would probably go a long way, and won’t break existing Gemfiles:

# Do not allow fetching anything from rubygems.org;
# changes gem() behaviour to use git.
must 'use-git'

gem 'github.com/rails/rails', 'v8.1.2'

# Indirect (automatically written and updated by "bundle install" and similar)
indirect do
    gem 'github.com/rails/actionview', 'v8.1.2'
    gem 'github.com/fxn/zeitwerk',     'v2.8.2'
    # ... etc...
end

Gemfile.lock can still be used for hashes (similar to go.sum), but is otherwise not all that useful.

There’s a bunch of details to be sorted out here and I’m not pretending those are straight-forward to sort out well, but there does seem to be a reasonable path there, I think?

Or maybe something else entirely. I don’t know. The main point is: I want to reliably audit my dependencies like a responsible developer and RubyGems makes it too hard, as do other package managers (but I don’t care about them).

Elsewhere on the interwebs

Cursed Bundler: Using go get to install Ruby Gems

Comparing compression tools

Tue, 14 Apr 2026 00:00:00 +0000

I’d like to know what the “best” compression tool is for storing archives (backups, data files). So I wrote a tool to compare them: cmp-compress.

My tl;dr take-away is:

zstd -3 for fast compression (the default).
xz -7 for the best ratio use; -8 or -9 sometimes compress better but often don’t. You probably want to test this.
zstd -12 for a good trade-off that leans towards fast.
zstd -17 for a good trade-off that leans towards better ratio.
Add -T0 to zstd to use all cores if nothing else is running on the system (or use the zstdmt wrapper).

For other use-cases all of this may of course be different.

Some tools like xz and zstd offer many differs knobs and levers; I’m sure these are useful but did not bother with them and tested only the presets. I’m not looking for an incantation to eek out the absolute best – I just want an informed practical decision on which tool to use with which flags to compress common stuff.

For the same reason I’m only comparing reasonably common compression tools already in wide-spread use. There are many others that are very similar to those listed here with minor differences (e.g. zip, 7zip), not considered stable, or just not widespread for one reason or the other.

If you’re interested in other tools or sets of flags then you can run cmp-compress yourself.

Full Results as HTML table: /compress.html. Or as JSON: /compress.json.

I ran this on the following files:

vim	5.5M	Statically linked binary
qemu-system-x86_64	29M	Dynamically linked binary
dockerd	87M	Statically linked binary
dickens	9.7M	English text, no tags
enwik8	95M	English text, with XML and Wiki tags
enwik9	950M	English text, with XML and Wiki tags
yt-dlp-2026.03.17.tar	12M	Python code
coreutils-9.10.tar	63M	C code
go1.26.1.src.tar	150M	Go code
go1.26.1.linux-amd64.tar	233M	Mix of Go code and statically linked binaries

The enwik files are from the Large Text Compression Benchmark; dickens is from the Silesia compression corpus; QEMU is version 10.2.0; Docker is version 29.4.0, and vim is a private modified fork that’s not easy to exactly reproduce but a statically linked huge build of Vim 9.1 should be close.

All of this is run on Void Linux with Linux 6.19.10 on a ThinkPad x13 with AMD Ryzen 7 7840U (8 cores/16 threads) and 32G of memory. I ran all the tests twice and discarded the first run (cmp-compress compare >/dev/null && cmp-compress compare >a.json).

gzip

The -# flag has a small effect and becomes useless above -6. Even the difference between -5 and -6 is quite small, and arguably -5 would be a better default than -6. -4 is a good setting if you want to be faster as it compresses a bit more than -1, -2, and -3, but isn’t too much slower.

The -# flag makes no meaningful difference for decompression times.

In short, use only -4, -5, or -6.

pigz

The -# behaves identical to gzip: it has no meaningful difference from gzip (in most cases pigz compresses slightly better – by a negligible amount but still). It’s often faster when using multiple threads, but single-thread compression is a bit slower.

It also adds a -11 flag to use the zopfli algorithm. The manpage says it “gives a few percent better compression at a severe cost in execution time”. It’s not exaggerating about the “severe cost”: it’s always the slowest of any tool by a huge margin. I suppose that it can be useful if you’re serving things at Google-scale, but you’re probably not, and other algorithms (brotli, zstd) are now widely-implemented. I’m going to say that in 2026 it’s useless: bzip2, xz, and zstd can all achieve better compression while being much much faster.

Aside: I kind of wish /bin/gzip would be replaced by pigz, or that gzip would use zlib and incorporate multi-threaded code. pigz seems stable, reasonably coded, and generally fairly good. I can’t find any reasons to not s/gzip/pigz/ other than vague concerns about “it’s different”, which is not entirely invalid but pigz has been around for a long time and at some point things need to move forward.

bzip2

The classic “I’m okay waiting a bit longer than gzip to get better ratios”. As a rule, compression is ~1.5 times slower, and decompression is ~3 times slower. It practically always gets a better compression ratio than gzip. More so on text than on binary files.

The -# flags make a small difference in compression ratio or time. There is no reason to use anything other than -9 (the default). bzip3 did away with -# flags entirely.

Both xz and zstd compress better than bzip2, but do take longer. Only on the dickens file does bzip2 give the same compression ratio as xz, but bzip2 is an order of a magnitude faster. Overall I’m going to say that bzip2 is rarely worth it, except perhaps if your data is primary text with minimal formatting (e.g. Markdown files, or similar). I don’t know to what degree this holds true for other languages or scripts as I only tested English.

xz

The newer “I’m okay waiting a bit longer than gzip to get better ratios”. Both compression and decompression is fairly slow, but delivers the best ratios of the common compression tools (zstd comes close, but xz is still meaningfully better).

The -# flag does little above -5, albeit more than gzip and there is a small but meaningful difference between the levels. For many files -8 and -9 are much slower than -7 while having little to no effect on ratio. As a rule you want to stick with the default of -6 unless you’ve tested it makes a meaningful difference on your data.

On binary files it outperforms bzip2 even on -0, on code at around -1, and on natural language it tends to outperform bzip2 at around -4 or -5,

Decompression speed decreases with higher -# flags, although this is not quite linear and sometimes higher -# flags have faster decompression speeds. Sometimes it’s very slow (e.g. enwik9 is almost two minutes to decompress, which is by far the slowest other than pigz -11.

zstd

Unlike gzip and xz the -# flag makes a meaningful difference at any level. The progression from -1 to -19 isn’t linear, but comes closer than anything else. In particular, the gap between -12 to -13 is consistently large with -13 being two to three times slower than -12.

Decompression is fast, although does take a noticeable hit at higher -# levels, but even at -19 it’s always faster than xz -0.

It’s almost always faster than gzip -1 at lower -# levels while delivering a better compression ratio. -12 is usually faster than gzip -6 (and -13 is slower, due to aforementioned jump).

For natural language files it compresses better than bzip2 at around -16, while being much slower. For code and binary files it’s around -6, while being faster. xz still compresses ~10% better than zstd -19, but xz is slower (often a lot slower).

The --ultra flags eeks out a slightly amount of extra compression ratio at the expense of being a lot slower, although not quite as dramatic as pigz -11. The compression ratio differences are small enough that for most cases it’s rarely useful.

--fast is not that much faster than -1 and compression ratio suffers greatly, and decompression performance is not significantly faster. There’s probably some uses cases where you really want to get the maximum compression performance, but by and large, I’m going to say it’s not very useful for most cases, outside of some streaming perhaps.

Conclusion

I’m going to say that “just use zstd” is probably decent advice: it doesn’t have the absolute best compression ratio or performance, but for many use-cases the trade-of it makes are quite good, and it has very fast decompression which is nice. zstd also has a bewildering number of configuration options, making it very flexible: e.g. for smaller files the ability to pre-train a dictionary will make a big difference (not tested in this overview: based on previous experience).

If you do need absolute maximum compression ratios then xz is often better, at the expense of being much slower. In some cases that’s a good trade-off.

For natural language text bzip2 might still be worth it, but compression seems to suffer if there’s too much formatting (which is often the case in real-world files).

I want to stress once more that all of this may depend on your specific data, system you’re using, planetary alignments, etc. “Most of the time”, “often”, and “usually” do not equal “always”.

Against best practices

Sat, 16 Nov 2024 00:00:00 +0000

I have come to believe that by and large “best practices” are doing more harm than good. Not necessarily because they’re bad advice as such, but because they’re mostly pounded by either 1) various types of zealots, idiots, and assholes who abuse these kind of “best practices” as an argument from authority, or 2) inexperienced programmers who lack the ability to judge the applicability,

Everyone else just says “X is better because Y”. This is an actual argument that can be engaged with, and engaging with actual arguments is what discovers the best solution for the situation at hand.

This is not unique to programming; even the best of ideas becomes silly once you start uncritically applying it to everything – one can find many examples in politics. There, too, it’s mostly pounded by various types of zealots, idiots, and assholes.

All of this doesn’t mean that “best practice” are bad advice. Many are worth reading, and some should always be followed. But many are little more “this thing someone said” and/or “just someone’s opinion”.

Take Postel’s “law” – what if I break this “law”? Will the police come and arrest me? Will I get the Nobel prize in physics as I’ve discovered new laws of nature? It’s not a “law” in any meaningful sense: it’s just a thing this Postel guy said in 1980, which may or may not be applicable to whatever you’re doing.

Postel’s quip was probably good advice in 1980 in the context of TCP. There were tons of little incompatibilities between existing implementations and Postel was working on the first effort to actually standardize it all. Back then it was typically harder to write fully correct implementations (no internet with tons of examples, little to no testing, less access to other implementations, etc.)

There are other cases where it’s the reasonable and practical thing to do. But as a general concept applied to all protocols or all software development I find it’s mostly a bad idea. I’m hardly the first to critique it, and it’s been controversial for decades.

In spite of that, people citing Postel’s quip as if it’s somehow a “law” are still plentiful. My law is that everyone who does so is an idiot, asshole, or some combination of both.

There are many more examples:

“Don’t use globals” is obviously good, but “never use globals under any circumstances” is just silly. I’ve seen people throw a hissy fit over “globals” in simple CLI tools with a few functions. Whoop-dee-doo.
“Don’t use unstructured GOTO” has turned into “never use goto”, “don’t use continue because it’s like a hidden goto”, and “use single point of exit”, and that kind of absolute insane bollocks.
“Don’t Repeat Yourself” (DRY) is basically good advice, but sometimes just copy/pasting things is just the more pragmatic thing to do, and not really a big deal.
I think “12 factor app” has some okay ideas, some dubious ideas, and some outright bad ideas.
While the basic ideas of “SOLID” are okay, I generally find that strong adherence does not lead to particularly good code. It’s typically 80% fucking about with how to do things, and 20% actually doing the stuff you want to do (which makes things harder to understand and change, not easier).
etc. etc. etc.

You can disagree with any of the above, and that’s fine. But citing “laws” or “best practices” are just a fallacious arguments from authority.

One of the difficulties with argueing against “best practices” is that it often goes something like this:

“X is not according to best practices”
“I think X is better because Y”
“But it’s not following best practice Z! Here is a book about it! Why don’t you follow it?! Do you want to write bad code?!”

There is kind of a gaslighting effect here, because it’s very natural to assume that maybe you should be following the “best practice”, especially when declared with great arrogance.

This is the same fallacious argument people use for safety:

“I don’t think X will improve safety.”
“What?! You don’t care about safety?!”
or: “Better safe than sorry!”

Which is why safety regulation only grows and never shrinks, even when the regulation just makes no sense at all: arguing against it is hard, because the look of things is against you. I call this the Safety Fallacy.

Jia Tanning Go code

Mon, 28 Oct 2024 00:00:00 +0000

The Go compiler skips files ending with _test.go during normal compilation. They are compiled with go test command (together will all other .go files), which also inserts some chutney to run the test functions. The standard way to do testing is to have a foo.go and foo_test.go file next to each other.

If you have a file that appears to end with _test.go but doesn’t actually end with _test.go, then it will get compiled for a regular build. For example:

a_test\uFE0E.go

U+FE0E is a variation selector. These are typically very invisible. Or more traditional homoglyph trickery:

b_t\u0435\u0455t.go"

Those Cyrillic letters appear virtually identical on my machine: b_tеѕt.go / b_test.go, or as monospace: b_tеѕt.go / b_test.go.

This can also be done with zero-width space, zero-width joiner, and perhaps a few other codepoints, but variation selectors tend to be the “most invisible” of them.

This is pretty sneaky, something like this:

// user.go
var bcryptCost = bcrypt.DefaultCost

func hashPassword(pwd []byte) []byte {
    pwd, _ := bcrypt.GenerateFromPassword(pwd, bcryptCost)
    return pwd
}

// user_test.go
func init() { // The special "init" function gets run on import.
    // Lower bcrypt cost in tests, because otherwise any test will take
    // well over a second as it's so slow.
    bcryptCost = bcrypt.MinCost
}

Is perfectly valid, but with a doctored “non-test” user_test.go it will now lower the security of all passwords, and effectively inserts a backdoor.

There are many variants one can think of: code that looks innocent and would probably pass many reviews, but will backdoor the codebase by weakening the security, or adds in “test users”, “test keys”, “default passwords”, and things like that.

Who is carefully auditing tests for security in the first place? I’ve audited a bunch of packages over the years, but I’ve never paid much attention to the test files.

Most tools don’t display this: Vim, VSCode, macOS finder, Windows Explorer, /bin/ls, GitHub, GitLab, etc. – they all just display user_test.go with no indication there’s something more there. Take a look at the test repo; all seems fairly innocent. Arguably, that’s how it should be, at least for some of these programs. Variation selectors are a Unicode feature and required to display certain types of text. That said, filenames are not really “normal text”, certainly not in the context of code.

The major exception is the git CLI with core.quotePath=true (the default), which displays the byte sequences for anything that’s not ASCII. You need to enable that setting to have any non-ASCII path display correctly, and depending on locality it’s probably a somewhat common setting. And how many people use the git CLI to review files (instead of GitHub web UI, VSCode integration, etc?) I’m a pretty heavy git CLI user, but typically don’t use “git diff” or “git log” for viewing differences between releases, and even if I did, there’s a good chance I’d miss this in a “git diff”.

Another way to detect this is to carefully pay attention to the URL in e.g. GitHub when viewing files, which displays escapes for some – but not all – of the variants. But this isn’t displayed in the PR view, only when browsing files, and is very easy to miss.

I reported this to GitHub, GitLab, and BitBucket back in June. None of them considered this to be a security issue. I don’t really understand why, because it doesn’t seem too dissimilar to LTR trickery to hide code. Also GitHub will display a warning for PRs with this sort of trickery:

The head ref may contain hidden characters: "a\uFE0Eb"

Crafted branch names is an issue but crafted files are not? Well okay 🤷

Also emailed the security@golang.org, but they never got back to me, so I guess they don’t consider it an issue either.

Is it viable to do a real-world attack with this? I don’t know, I haven’t tried. I am not employed at the University of Minnesota so I don’t go around sending malicious patches just to see what would happen.

But if I were tasked to Jia Tan a Go codebase, this seems a promising method. There’s very little obfuscation and with a bit of careful design from a malicious actor everything can seem legitimate test code that’s there for valid reasons, being only malicious because it gets compiled in the regular program, and because it still gets compiled in the test program the tests will still work. Seems like a great way to hide malicious code in plain sight.

Doing a full Jia Tan and compromising ssh is still tricky, because that requires doing the sort of stuff that typically has no business being in a test file. That’s why in xz it was hidden in binary test data. Still, with the right project and a bit of creativity I could envision this as a step in a similar scheme, especially when done by the maintainer itself.

Safe terminal escape codes

Wed, 22 May 2024 00:00:00 +0000

A long time ago when the Unix greybeards were slightly less grey beards everyone was using hardware terminals to talk to some mainframe. Those terminals had wildly different feature sets and ways to implement them, which you needed to know about if you wanted your program to run on more than one type of terminal. Hence systems like terminfo and termcap.

The TeleVideo 955 used \x1b[=5l for bold text, and the IBM 3161 used \x1b4H. Both were introduced in 1985. The TeleVideo 950 from 1980 didn’t support bold text at all.

Today everyone is using software terminal emulators, and they all just implement ANSI escape codes for the common stuff. No one is sending \x1b[=5l or \x1b4H.

Do you still need terminfo? It depends. If you just want to style some text and maybe do some basic cursor operations: probably not. If you want to use more advanced operations or read key input: probably yes. If you want maximum compatibility (there’s probably someone using a hardware Wyse terminal, or SunOS 4 with CDE): absolutely.

Here is a list of terminal escape sequences that should always work on any vaguely modern terminal, where “modern” means “since the mid-90s or so”.

This was generated by looking at every terminfo file I could find with my termfo tool (specifically, termfo find-cap) and occasionally testing some things to verify it works. I don’t care what some spec says; I care about what works.

The escape character is always omitted; so [0m is \x1b[0m.

Graphics

These can be combined in a single sequence by joining with a ;. For example \x1b[1;4m for underlined bold text. These can also be combined with colours; for example \x1b[1;38;2;0;255;0;41m to set bold, foreground colour with true colour, and background colour with 16 colour.

Just using \x1b[1m\x1b[4m also works.

Code	Terminfo (short, long)	Description
[0m	sgr0, exit_attribute_mode	Turn off all attributes and colours.
[1m	bold, enter_bold_mode	Enter bold mode.
[2m	dim, enter_dim_mode	Enter half-bright mode.¹
[3m	sitm, enter_italics_mode	Enter italic mode.²
[4m	smul, enter_underline_mode	Enter underline mode.
[5m	blink, enter_blink_mode	Enter blinking mode.³
[7m	rev, enter_reverse_mode	Enter reverse video mode.
[8m	invis, enter_secure_mode	Enter blank mode.⁴
[9m	smxx	Enter strikeout mode.

Colours are set with setab / set_a_background and setaf / set_a_foreground; the format depends on which colour scheme you want to use.

16-colours

Pretty much everything supports 16 colours, unless explicitly disabled by the user. The exact shade is determined by the terminal.

Escapes as «regular» «bright»:

Foreground	Background	Colour
[30m [90m	[40m [100m	black
[31m [91m	[41m [101m	red
[32m [92m	[42m [102m	green
[33m [93m	[43m [103m	yellow
[34m [94m	[44m [104m	blue
[35m [95m	[45m [105m	magenta
[36m [96m	[46m [106m	cyan
[37m [97m	[47m [107m	white

256-colours

Choose a colour from a pre-defined table of 256 colours. This is supported on almost all current terminal emulators, unless explicitly disabled by the user.

Where «C» is a number from 0 to 255:

[38;5;«C»m   Foreground
[48;5;«C»m   Background

True colours

Use “true” RGB colours; this is supported on almost all current terminal emulators, unless explicitly disabled by the user.

Where «R», «G», «B» are the red, green and blue values as decimal, 0 to 255:

[38;2;«R»;«G»;«B»m   Foreground
[48;2;«R»;«G»;«B»m   Background

Cursor movement

Code	Terminfo (short, long)	Description
[«R»;«C»H	cup, cursor_address	Set cursor position to «R», «C».⁵
[«N»A	cuu, parm_up_cursor	Move «N» lines up.
[«N»B	cud, parm_down_cursor	Move «N» lines down.
[«N»C	cuf, parm_right_cursor	Move «N» characters to the right.
[«N»D	cub, parm_left_cursor	Move «N» characters to the left.
7	sc, save_cursor	Save current cursor position.
8	rc, restore_cursor	Restore cursor to position of last save_cursor.
[6n	u7, user7	Request cursor, returned as `\x1b[«R»;«C»R`

The «N» can be omitted in the above, in which case it will default to 1. For cup the «R» and/or «C» can be omitted, and will default to 1.

Inserting and deleting text

Code	Terminfo (short, long)	Description
[J	ed, clr_eos	Clear from cursor to end of screen.
[K	el, clr_eol	Clear from cursor to end of line.
[1K	el1, clr_bol	Clear from cursor to beginning of line.
[«N»M	dl, parm_delete_line	Delete «N» lines, moving all the lines below up.
[«N»P	dch, parm_dch	Delete «N» characters, moving all afterwards left.
[«N»L	il, parm_insert_line	Insert «N» lines, moving all lines below down.
[«N»@	ich, parm_ich	Insert «N» characters, moving all after right.

The «N» can be omitted in the above, in which case it will default to 1.

Misc

Code	Terminfo (short, long)	Description
[?25l	civis, cursor_invisible	Make cursor invisible.⁶
[?25h	cnorm, cursor_normal	Make cursor appear normal (undo civis).⁷

Keys

Key input handling is kind of a mess, and this is where I really recommend using a terminfo library if at all possible. This also avoids the whole backspace/delete key confusion (less of an issue today than it used to be, but still exists).

Codes	terminfo (short, long)	Description
[A OA	kcuu1 key_up	Arrow up
[B OB	kcud1 key_down	Arrow down
[C OC	kcuf1 key_right	Arrow right
[D OD	kcub1 key_left	Arrow left
[5~ [I	kpp key_ppage	PageUp
[6~ [G	knp key_npage	PageDown
[2~ [L	kich1 key_ic	Insert key
[1~ [H OH [7~	khome key_home	Home key
[4~ [F OF [8~	kend key_end	End key
[11~ [M [[A OP	key_f1	F1
[12~ [N [[B OQ	key_f2	F2
[13~ [O [[C OR	key_f3	F3
[14~ [P [[D OS	key_f4	F4
[15~ [Q [[E	key_f5	F5
[17~ [R	key_f6	F6
[18~ [S	key_f7	F7
[19~ [T	key_f8	F8
[20~ [U	key_f9	F9
[21~ [V	key_f10	F10
[23~ [W	key_f11	F11
[24~ [X	key_f12	F12
\x7f \b	kbs key_backspace	Backspace key
\x7f [3~	kdch1 key_dc key_delete	Delete key; usually sends [3~, but some send \x7f
\x09	-	Tab key
\x0d	-	Enter key

Elsewhere on the interwebs

github.com/termstandard/colors

Footnotes

Except FreeBSD system console where it’s [30;1m. ↩
Not super-widely supported, sometimes displays as reverse. ↩
Often doesn’t do anything (because it’s annoying). ↩
Characters can still be copy/pasted. ↩
Top-left is 1;1 ↩
mtm terminfo doesn’t include the question mark, and Linux console adds another escape (\x1b[?1c). But for both this also works. ↩
Has [?12l for some, but works without that on all; for FreeBSD terminfo has [=0C but that doesn’t work ([?25h does) ↩

s/bash/zsh/g

Wed, 20 Oct 2021 00:00:00 +0000

You would expect this to work, no?

bash% echo $(( .1 + .2 ))
bash: .1 + .2 : syntax error: operand expected (error token is ".1 + .2 ")

Well, bash says no, but zsh just works:

zsh% echo $(( .1 + .2 ))
0.30000000000000004      # Well, "works" insofar IEEE-754 works.

There is simply no way you can do calculations with fractions in bash without relying on bc, dc, or some hacks. Compared to simply being able to use a + b it’s ugly, slow, and difficult.

There are other pretty frustrating omissions in bash; NUL bytes is another fun one:

zsh% x=$(printf 'N\x00L'); printf $x | xxd -g1 -c3
00000000: 4e 00 4c  N.L

bash% x=$(printf 'N\x00L'); printf $x | xxd -g1 -c3
bash: warning: command substitution: ignored null byte in input
00000000: 4e 4c     NL

It looks like bash added a warning recently-ish (4.4-patch 2); this one bit me pretty hard a few years ago; back then it would just get silently discarded without warning; I guess a warning is an “improvement” of sorts (fixing it, however, would be an actual improvement¹).

NUL bytes aren’t that uncommon, think of e.g. find -print0, xargs -0, etc. That all works grand, right up to the point you try to assign it to a variable. You can use NUL bytes for array assignments though, if you evoke the right incantation:

bash% read -rad arr < <(find . -type f -print0)

There are all sorts of edge-cases where you need to resort to read or readarray rather than being able to just assign in. In zsh it’s:

zsh% arr=(**/*(.))

zsh% IFS='\x00' arr=($(find . -type f -print0)) # If you must use find (rarely needed)

zsh% arr=( "${(0)$(find . -type f -print0)}" )

That (.) is a “glob qualifier” to include only regular files – more on that later.

Don’t even think of doing something like:

img=$(curl https://example.com/image.png)
if [[ $cond ]]; then
    optpng <<<"$img" > out.png
else
    cat <<<"$img" > out.png
fi

Of course you can refactor this to avoid the variable (and the example is a bit contrived), but it really ought to work. I once wrote a script to import emails from the Mailgun API. It worked great, yet sometimes images were mangled and I just couldn’t figure out why. Turns out Mailgun “helpfully” decodes attachments (i.e. removes the base64) and sends binary content, which bash (at the time) would silently discard. It took me a very long time to figure out. I forgot why, but it was hard to avoid storing the response in a variable. I ended up rewriting it to Python because of this, which was just a waste of time really. It’s this, specifically, that really soured me on bash and prompted The shell scripting trap in 2017. However, many items listed there are solved by zsh, including this one.

zsh also fixes most of the quoting:

zsh% for f in *; ls -l $f
-rw-rw-r-- 1 martin martin 0 Oct 19 06:51 asd.txt
-rw-rw-r-- 1 martin martin 0 Oct 19 06:51 with space.txt

bash% for f in *; do ls -l $f; done
-rw-rw-r-- 1 martin martin 0 Oct 19 06:51 asd.txt
ls: cannot access 'with': No such file or directory
ls: cannot access 'space.txt': No such file or directory

It’s not POSIX compatible, but who cares? bash doesn’t follow POSIX in all sorts of ways by default either because it just makes more sense, and with both you can still tell them to be POSIX-compatible if you must for one reason or the other.

Also note the convenient short version of the for loop: no need for do, done and muckery with ; before the done, which is much more convenient for quick one-liners you interactively type in. You can still do word splitting, but you need to do it explicitly:

zsh% for i in *; ls -l $=i
-rw-rw-r-- 1 martin martin 0 Oct 19 06:51 asd.txt
ls: cannot access 'with': No such file or directory
ls: cannot access 'space.txt': No such file or directory

[[ is supposed to fix [, but it still has weird quoting quirks:

zsh% a=foo; b=*;
zsh% if [[ $a = $b ]]; then
       print 'Equal!'
     else
       print 'Nope!'
     fi
Nope!

bash% a=foo; b=*
bash% if [[ $a = $b ]]; then
        echo 'Equal!'
      else
        echo 'Nope!'
      fi
Equal!

This is equal because without quotes it still interprets the right-hand side as a pattern (i.e. glob). In zsh you need to use $~var to explicitly enable pattern matching, which is a much better model than remembering when you do and don’t need to quote things – sometimes you do want the pattern matching and then you don’t want quotes; it’s not always immediately obvious if an if [[ ... statement is correct if it’s lacking quotes.

“But Martin, you should always quote things, you’re being disingenuous!” Well, I could make a comfortable living if I were paid to add quotes to other people’s shell scripts. Telling people to “always quote things” is what we’ve been doing for 40 years now and irrefutable observational evidence has demonstrated that it just does not work.

Most people aren’t shell scripting wizards; they make a living writing Python or C or Go or PHP programs, or maybe they’re sysadmins or scientists, and oh, occasionally they also write a shell script. They just see something that works and assume it has sane behaviour and don’t realize the subtle differences between $@, $*, and "$@". I think that’s actually quite reasonable, because the behaviour is odd, surprising, and confusing.

It’s also a lot more complex than just “quote your variables”, especially if you use $(..) since command substitution often needs quotes too, as well as any variables inside it. Before you know it you’ve got double, triple, or more levels of nested quotes and if you forget one set of them you’re in trouble.

It’s such a fertile source of real-world bugs that it would merit entomologist examination. If there’s a system that causes this many bugs then that system is at fault, and not the people using it. Computers and software should adapt to humans, not the other way around.

And “always quote things!” isn’t even right either, because you should always quote things except when you shouldn’t:

zsh% a=foo; b=.*;
zsh% if [[ "$a" =~ "$b" ]]; then
       print 'Equal!'
     else
       print 'Nope!'
     fi
Equal!

bash% a=foo; b=.*
bash% if [[ "$a" =~ "$b" ]]; then
        echo 'Equal!'
      else
        echo 'Nope!'
      fi
Nope!

If there are quotes around a regexp then it’s treated as a literal string. I mean, it’s consistent with = pattern matching, but also confusing because I explicitly use =~ to match a regular expression.

Another famous quoting trap is $@ vs. "$@" vs. $* vs. "$*":

zsh% cat args
echo "unquoted @:"
for a in $@; do echo "  => $a"; done

echo "quoted @:"
for a in "$@"; do echo "  => $a"; done

echo "quoted *:"
for a in $*; do echo "  => $a"; done

echo "quoted *:"
for a in "$*"; do echo "  => $a"; done

bash% bash args hello world 'test space' '*'
unquoted @:
  => hello
  => world
  => test
  => space
  => Guust1129.jpg
  => IEEESTD.2019.8766229.pdf
  [.. rest of my $HOME ..]
quoted @:
  => hello
  => world
  => test space
  => *
unquoted *:
  => hello
  => world
  => test
  => space
  => Guust1129.jpg
  => IEEESTD.2019.8766229.pdf
  [.. rest of my $HOME ..]
quoted *:
  => hello world test space *

Experiences shellers will know to (almost) always use "$@", but how often do you see it being done wrong? It’s not that strange people do it wrong either; if you learned about quoting and word splitting then $@ without quotes is actually the logical thing to use because you would expect "$@" to be treated as one argument (as "$*"). You tell people to “always add quotes to prevent word splitting and treat things as a single argument”, and then you tell them “oh, except in this one special case when the addition of quotes invokes a special kind of splitting and doesn’t follow any of the rules we previously told you about”.

In zsh, $@ and $* (and $argv) are all (read-only) arrays and it all behaves identical as you would expect with no surprises:

zsh% zsh args hello world 'test space' '*'
unquoted @:
  => hello
  => world
  => test space
  => *
quoted @:
  => hello
  => world
  => test space
  => *
unquoted *:
  => hello
  => world
  => test space
  => *
quoted *:
  => hello world test space *

Actually in bash you can do argv=("$@") and then you have an array. This is really how it should work by default.

You still need to loop over it with:

bash% for a in "${argv[@]}"; do
        echo "=> $a"
      done

Rather than just for a in $argv like in zsh. Aside from the pointless [@], why would you ever want to word-split every element of an array? There is probably some use case somewhere, but it’s exceedingly rare. Better to just skip all of that by default unless explicitly enabled with = and/or ~.

Oh, here’s another interesting tidbit:

zsh% n=3; for i in {1..$n}; print $i
1
2
3

bash% n=3; for i in {1..$n}; do echo "$i"; done
{1..3}

bash% n=3; for i in {1..3}; do echo "$i"; done
1
2
3

Why does it work like that? That’s left as an exercise for the reader ;-)

Aside from all sorts caveats that are handled much better, a lot of common things are just much easier in zsh:

zsh% arr=(123 5 1 9)
zsh% echo ${(o)arr}     # Lexical order
1 123 5 9
zsh% echo ${(on)arr}    # Numeric order
1 5 9 123

bash% IFS=$'\n'; echo "$(sort <<<"${arr[*]}")"; unset IFS
1 123 5 9
bash% IFS=$'\n'; echo "$(sort -n <<<"${arr[*]}")"; unset IFS
1 5 9 123

I had to look up how to do it in bash; the Stack Overflow answer for that one starts with “you don’t really need all that much code”. lol? I guess that’s in reply to some of the other horrendously complex answers which implement “pure bash” sorting algorithms and the like. I guess compared to that this is “not that much code”. And of course the entire thing is a minefield of expansion again; and if you forget a set of nested quotes you end up in trouble.

Arrays in general are just awkward in bash:

bash% arr=(first second third fourth)

bash% echo ${arr[0]}
first
bash% echo ${arr[@]::2}
first second

I mean, it works, and it’s not that much typing, but why do I need that [@]? Probably some (historical) reason, but zsh implements it much more readable and easier:

zsh% arr=(first second third fourth)

zsh% print ${arr[1]}        # Yeah, it's 1-based. Deal with it.
first
zsh% print ${arr[1,2]}
first second

The bash array syntax was copied from ksh; so I guess we have to blame David Korn (zsh supports it too, if you must use it). But regular subscripts are just so much easier.

And then there’s the useful features:

zsh% ls *.go
format.go  format_test.go  gen.go  old.go  uni.go  uni_test.go

zsh% ls *.go~*_test.go
format.go  gen.go  old.go  uni.go

zsh% ls *.go~*_test.go~f*
gen.go  old.go  uni.go

*.go gets expanded, and filters the pattern after the ~; *_test.go in this case. Looks a bit obscure at first glance, but bash’s ksh-style extglobs are far harder:

bash% ls !(*_test).go
format.go  gen.go  old.go  uni.go

bash% ls !(*_test|f*).go
gen.go  old.go  uni.go

!(..) is “match anything except the pattern”; the * is implied here (zsh supports !(..) if you set ksh_glob). While it works, the the pattern~filter~filter model is much easier, and also more flexible since you don’t need to start with all matches.

There are many useful things you can do with globbing; you can replace many uses of find with it, and you don’t need to worry about the caveats, -print0 hacks, etc. For example to recursively list all regular files:

zsh% ls **/*(.)
LICENSE         go.sum           unidata/gen.go             wasm/make*
README.md       old.go           unidata/gen_codepoints.go  wasm/srv*
[..]

Or directories:

zsh% ls -d /etc/**/*(/)
/etc/OpenCL/                      /etc/runit/runsvdir/default/dnscrypt-proxy/log/
/etc/OpenCL/vendors/              /etc/runit/runsvdir/default/ntpd/log/
/etc/X11/                         /etc/runit/runsvdir/default/postgresql/supervise/
[..]

Or files that were changed in the last week:

zsh% ls -l /etc/***(.m-7)    # *** is a shortcut for **/*; needs GLOB_STAR_SHORT
-rw-r--r-- 1 root root 28099 Oct 13 03:47 /etc/dnscrypt-proxy.toml.new-2.1.1_1
-rw-r--r-- 1 root root    97 Oct 13 03:47 /etc/environment
-rw-r--r-- 1 root root 37109 Oct 17 10:34 /etc/ld.so.cache
-rw-r--r-- 1 root root 77941 Oct 19 01:01 /etc/public-resolvers.md
-rw-r--r-- 1 root root  6011 Oct 19 01:01 /etc/relays.md
-rw-r--r-- 1 root root   142 Oct 19 07:57 /etc/shells

You can even order them by modified date with om (***(.m-7om)), although that’s a bit pointless here as ls will reorder them again, but if you’re looping over files it’s useful.

There is no way to do any of this in bash, you’ll have to use something like:

bash% find /etc -type f -mtime 7 -exec ls -l {} +
find: ‘/etc/sv/docker/supervise’: Permission denied
find: ‘/etc/sv/docker/log/supervise’: Permission denied
find: ‘/etc/sv/bluetoothd/log/supervise’: Permission denied
find: ‘/etc/sv/postgresql/supervise’: Permission denied
find: ‘/etc/sv/runsvdir-martin/supervise’: Permission denied
find: ‘/etc/wpa_supplicant/supervise’: Permission denied
find: ‘/etc/lvm/cache’: Permission denied
-rw-rw-r-- 1 root root  167 Oct 12 22:17 /etc/default/postgresql
-rw-r--r-- 1 root root  817 Oct 12 09:11 /etc/fstab
-rw-r--r-- 1 root root 1398 Oct 12 22:19 /etc/passwd
-rw-r--r-- 1 root root 1397 Oct 12 22:19 /etc/passwd.OLD
-rw-r--r-- 1 root root  307 Oct 12 23:10 /etc/public-resolvers.md.minisig
-rw-r--r-- 1 root root  297 Oct 12 23:10 /etc/relays.md.minisig
-r-------- 1 root root  932 Oct 12 09:57 /etc/shadow
-rwxrwxr-x 1 root root  397 Oct 12 22:23 /etc/sv/postgresql/run

Not sure how to make it ignore these errors too without redirecting stderr (more typing!) And if you think adding single letters in (..) after a pattern is hard then try understanding find’s weird flag syntax. Glob qualifies are great.

csh-style parameter substitution is pretty useful:

zsh% for f in ~/photos/*.png; convert $f ${f:t:r}.jpeg

:t to get the tail, and :r to get the root (without extension). csh could do this before I was even born, but bash can’t (it can for history expansion, but not variables). According to the bash FAQ “Posix has specified a more powerful, albeit somewhat more cryptic, mechanism cribbed from ksh”, which I find a somewhat curious statement as the above in bash is:

bash% for f in ~/photos/*.png; do convert "$f" "$(basename "${f%%.*}")"; done

Technically “more powerful” in the sense that you can do other things with it, but not really “more useful for common operations” (zsh, of course, implements % and # as well).

Note you can’t nest ${..} in bash; e.g. "${${f%%.*}##*/}" is an error:

zsh% f=~/asd.png; print "${${f%%.*}##*/}"
asd

bash% f=~/a.png; echo "${${f%%.*}##*/}"
bash: ${${f%%.*}##*/}: bad substitution

While this can quickly lead to very unreadable ASCII vomit, it’s useful on occasion, when used with care and wisdom. You can click below for a more advanced example if the children are already in bed.

Click to see NSFW content. Not suitable for children under 18!

For example, this can be used to show the longest element in an array:

print ${array[(r)${(l.${#${(O@)array//?/X}[1]}..?.)}]}

Cribbed from the zsh User's Guide.

There are many more things. I’m not going to list them all here. None of this is new; much (if not all?) of this has around for 20 years, if not longer. I don’t know why bash is the de-facto default, or why people spend time on complex solutions to work around bash problems when zsh solves them. I guess because Linux used a lot of GNU stuff and bash was came with it, and GNU stuff was (and is) using bash. Not a very good reason, certainly not one 30 years later.

zsh still has plenty of limitations; the syntax isn’t always something you’d want to show your mother for starters, as well as a number of other things. Still, it’s clearly better. I genuinely can’t find a single thing bash does better beyond “it’s installed on many systems already”.

Ubiquitousness is overrated anyway; zsh has no dependencies beyond libc and curses and is 970K on my system² and is available for pretty much all systems. Compared to most other interpreters it’s tiny, with only Lua being smaller (275K). “Stick to POSIX sh for compatibility” was good advice in 1990 when you had some SunOS system with some sun-sh and that’s what you were stuck with. Those days are long gone, and while there are a few poor souls still suck on those systems (sometimes even with csh!) chances are they’re not going to try and run your Docker or Arch Linux shell script or whatnot on those systems anyway.

Contorting yourself in all sorts of strange bends to perhaps possibly maybe make it work for a tiny fraction of users who are unlikely to use your script anyway does not seem like a good trade-off, especially since these kind of limitations tend to be organisational rather than technical, which is not my problem anyway to be honest.

Using zsh is also more portable; since it allows you to avoid many shell tools and the (potential) incompatibilities, and by explicitly setting zsh as the interpreter you can rely on zsh behaviour, rather than hoping the /bin/sh on $random_system behaves the same (even dash has some extensions to POSIX sh, such as local).

What I typically do is save files as script.zsh with:

#!/usr/bin/env zsh
[ "${ZSH_VERSION:-}" = "" ] && echo >&2 "Only works with zsh" && exit 1

This makes sure it gets run by zsh when used as ./script.zsh, and gives an error in case people type sh script.zsh or bash script.zsh in case the .zsh extension isn’t enough of a clue.

So in conclusion: s/bash/zsh/g and everything will be just a little bit better.

P.S. maybe fish is even better by the way, but I could never get over the bright colouring and all these things popping in and out of my screen; it's such a "busy" chaotic experience! I'd have to spend time disabling all that stuff to make it usable for me, and I never bothered – and if you're disabling the key selling points of a program then it's probably not intended for you anyway. Maybe I'll have a crack at it at some point though.

Footnotes

Which I assume isn’t so easy, otherwise it would have been done already. The reason it doesn’t work is an artifact from C’s NUL-terminated strings, but this kind of stuff really shouldn’t be exposed in a high-level language like the shell. It’s also a bit ironic since one of Stephen Bourne’s original goals with his shell was to get rid of arbitrary size limits on strings, which were common at the time. ↩
A full install if ~8M, mostly in the optional completion functions it ships with. Bash is about 1.3M by the way. ↩

Getting started with RimWorld modding on Linux

Wed, 29 Sep 2021 00:00:00 +0000

This describes how to create RimWorld mods on Linux; this is an introduction to both RimWorld modding and developing C♯ with Mono; it’s essentially the steps I followed to get started.

This doesn’t assume any knowledge of Unity, Mono, or C♯ but some familiarity with Linux and general programming is assumed; if you’re completely new to programming then this probably isn’t a good resource. A lot of this will work on Windows or macOS too; it’s just the C♯ build steps that are really Linux-specific, as are various pathnames etc.

RimWorld mods consist of two parts:

A set of XML definitions (“Defs”) which defines everything from items, actions you can take, research projects, weather, etc. This is the “glue” that actually makes stuff appear in the game, applies effects, etc.

For (very) simple mods this may actually be enough, and no “real” coding is required. You can use XML files to both add new stuff, and RimWorld has facilities to patch existing in-game content.
C♯ code which either adds entire new stuff, or monkey-patches existing code.

As an example we’ll make a little mod that makes it rain blood. Why? It seemed easy enough to do while also exploring some of the core concepts. Also, I was playing Slayer when I started on this. The complete example mod is on GitHub, but I encourage people to modify things manually (and maybe play around with things a bit) rather than copy/paste stuff from there; it’s just a better way to learn things.

Getting started

Before we start with the C♯ stuff let’s set up a basic mod which adds a new weather type; we just need to edit some XML for this.

Mods are located in the Mods/ directory in your RimWorld installation directory; I’m using the version I bought from the RimWorld website and extracted to ~/rimworld so that’s nice and simple. GOG.com games usually store the actual game data in a game/ subdirectory. I don’t know where Steam stores things 🤷

This directory should already exist with a Mods/Place mods here.txt. A mod must have an About/About.xml file; a minimal version looks like:

<?xml version="1.0" encoding="utf-8"?>
<ModMetaData>
    <!-- Must contain a dot; usually <author>.<modname> -->
    <packageId>arp242.RainingBlood</packageId>
    <name>Raining blood</name>

    <!-- Game versions this mod supports. More on game versions later. -->
    <supportedVersions>
        <li>1.1</li>
        <li>1.2</li>
        <li>1.3</li>
    </supportedVersions>
</ModMetaData>

See ModUpdating.txt in the RimWorld installation directory for a full description of the About.xml fields. For now, this is enough.

The official content uses the same structure as a mod except that it’s in the Data/ directory; e.g. Data/Core/ contains the base game, Data/Royalty the Royalty expansion, etc. To find the weather definitions I just used:

[~/rimworld/Data/Core]% ls (#i)**/*weather*.xml
Defs/WeatherDefs/Weathers.xml

The (#i) makes things case-insensitive in zsh, FYI. zsh is nice. You can also use find -iname if you enjoy more typing.

Weathers.xml seems to define all the weather types. I copied the definition of “rain” to Mods/RainingBlood/Defs/WeatherDefs/RainingBlood.xml with some modifications:

<?xml version="1.0" encoding="utf-8" ?>
<Defs>
<WeatherDef>
    <defName>RainingBlood</defName>
    <label>raining blood</label>
    <description>It's raining blood; what the hell?!</description>

    <!-- ThoughtDefs/RainingBlood.xml -->
    <!-- <exposedThought>SoakingWet</exposedThought> -->
    <exposedThought>BloodCovered</exposedThought>

    <!-- Copied from rain -->
    <temperatureRange>0~100</temperatureRange>
    <windSpeedFactor>1.5</windSpeedFactor>
    <accuracyMultiplier>0.8</accuracyMultiplier>
    <favorability>Neutral</favorability>
    <perceivePriority>1</perceivePriority>

    <rainRate>1</rainRate>
    <moveSpeedMultiplier>0.9</moveSpeedMultiplier>
    <ambientSounds>
        <li>Ambient_Rain</li>
    </ambientSounds>
    <overlayClasses>
        <li>WeatherOverlay_Rain</li>
    </overlayClasses>
    <commonalityRainfallFactor>
        <points>
            <li>(0, 0)</li>
            <li>(1300, 1)</li>
            <li>(4000, 3.0)</li>
        </points>
    </commonalityRainfallFactor>

    <!-- Colours modified to be reddish; just a crude effect. -->
    <skyColorsDay>
        <sky>(0.8,0.2,0.2)</sky>
        <shadow>(0.92,0.2,0.2)</shadow>
        <overlay>(0.7,0.2,0.2)</overlay>
        <saturation>0.9</saturation>
    </skyColorsDay>

    <skyColorsDusk>
        <sky>(1,0,0)</sky>
        <shadow>(0.92,0.2,0.2)</shadow>
        <overlay>(0.6,0.2,0.2)</overlay>
        <saturation>0.9</saturation>
    </skyColorsDusk>

    <skyColorsNightEdge>
        <sky>(0.35,0.10,0.15)</sky>
        <shadow>(0.92,0.22,0.22)</shadow>
        <overlay>(0.5,0.1,0.1)</overlay>
        <saturation>0.9</saturation>
    </skyColorsNightEdge>

    <skyColorsNightMid>
        <sky>(0.35,0.20,0.25)</sky>
        <shadow>(0.92,0.22,0.22)</shadow>
        <overlay>(0.5,0.2,0.2)</overlay>
        <saturation>0.9</saturation>
    </skyColorsNightMid>
</WeatherDef>
</Defs>

The location where you store this doesn’t matter as long as it’s in Defs; Defs/xxx.xml will work too. Internally all XML files in Defs/ are scanned in the same data structure; it just recursively searches for *.xml files and uses <defName>RainingBlood</defName> to identify them rather than the path.

We also need a new “exposed thought”; that’s the mood modifier that shows up in the “needs” tab; “Soaking wet” doesn’t really seem applicable if you’re “soaking wet in blood” 🙃

Let’s grep for it:

[~/rimworld/Data/Core]% rg SoakingWet
Defs/TerrainDefs/Terrain_Water.xml
12:    <traversedThought>SoakingWet</traversedThought>

Defs/ThoughtDefs/Thoughts_Memory_Misc.xml
297:    <defName>SoakingWet</defName>

Defs/WeatherDefs/Weathers.xml
98:    <exposedThought>SoakingWet</exposedThought>
202:    <exposedThought>SoakingWet</exposedThought>
267:    <exposedThought>SoakingWet</exposedThought>

Thoughts_Memory_Misc.xml seems to be what we want, so make a copy of that to Mods/RainingBlood/Defs/ThoughtDefs/RainingBlood.xml:

<?xml version="1.0" encoding="utf-8" ?>
<Defs>
<ThoughtDef>
    <defName>BloodCovered</defName>
    <durationDays>0.1</durationDays>
    <stackLimit>1</stackLimit>
    <stages>
        <li>
            <label>blood covered</label>
            <description>I'm covered in blood; yuk!</description>
            <baseMoodEffect>-30</baseMoodEffect>
        </li>
    </stages>
</ThoughtDef>
</Defs>

The meaning of the fields in both XML files should be mostly self-explanatory, but if you want to know what exactly something does you’ll need to decompile the game to read the source code. We’ll cover that later.

At this point, the basic mod should be done; let’s test it.

Running the game

Start the name normally and select the mod in the Mods panel. After this you can start the game with ./RimworldLinux -quicktest, which will start the game in a new small map with the last selected mods.

You can select “Development mode” in options, which will give you a few buttons at the top, gives you a console with `, you can speed up things a wee bit more by pressing 4 (ludicrous speed!) Most of the buttons etc. should be self-explanatory; there’s some more information on the RimWorld wiki.

Click the “debug actions” button at the top, which has “Change Weather” (filter in the top-left corner; you may need to scroll down). After clicking RainingBlood it takes a few seconds for the weather to transition and the status to show up in your colonists.

Patching the biomes

It’s all very good that we can select this from our magical debug actions, but does it actually appear in a regular game? Let’s search where the RainyThunderstorm weather is referenced (as that’s a bit more unique than just “rain”):

[~/rimworld/Data/Core]% rg RainyThunderstorm
Defs/BiomeDefs/Biomes_Cold.xml
101:      <RainyThunderstorm>1</RainyThunderstorm>
234:      <RainyThunderstorm>1</RainyThunderstorm>
389:      <RainyThunderstorm>1</RainyThunderstorm>
513:      <RainyThunderstorm>0</RainyThunderstorm>
611:      <RainyThunderstorm>0</RainyThunderstorm>

Defs/BiomeDefs/Biomes_Temperate.xml
104:      <RainyThunderstorm>1</RainyThunderstorm>
262:      <RainyThunderstorm>1</RainyThunderstorm>

Defs/BiomeDefs/Biomes_Warm.xml
109:      <RainyThunderstorm>1.7</RainyThunderstorm>
277:      <RainyThunderstorm>1.7</RainyThunderstorm>

Defs/BiomeDefs/Biomes_WarmArid.xml
79:      <RainyThunderstorm>1</RainyThunderstorm>
204:      <RainyThunderstorm>1</RainyThunderstorm>
312:      <RainyThunderstorm>1</RainyThunderstorm>

Defs/WeatherDefs/Weathers.xml
193:    <defName>RainyThunderstorm</defName>

e.g. Biomes_Cold.xml has:

<baseWeatherCommonalities>
    <Clear>18</Clear>
    <Fog>1</Fog>
    <Rain>2</Rain>
    <DryThunderstorm>1</DryThunderstorm>
    <RainyThunderstorm>1</RainyThunderstorm>
    <FoggyRain>1</FoggyRain>
    <SnowGentle>4</SnowGentle>
    <SnowHard>4</SnowHard>
</baseWeatherCommonalities>

Let’s try adding our bloody rain with a high chance of spawning:

<baseWeatherCommonalities>
    <Clear>18</Clear>
    <Fog>1</Fog>
    <Rain>2</Rain>
    <DryThunderstorm>1</DryThunderstorm>
    <RainyThunderstorm>1</RainyThunderstorm>
    <FoggyRain>1</FoggyRain>
    <SnowGentle>4</SnowGentle>
    <SnowHard>4</SnowHard>

    <RainingBlood>64</RainingBlood> <!-- References the defName -->
</baseWeatherCommonalities>

Why 64? Well, the other numbers add up to 32 and if they’re relative weights then 64 means a 2/3rd chance of our raining blood weather. “Trying it and seeing what happens” is pretty much what I’m doing here. Throw enough macaroni at a wall and sooner or later some of it will stick.¹

The easiest way to override this is to copy the XML file to your Mods/[..]/Defs/ directory. Again, the path doesn’t matter, it just looks at the defName attribute; the last one overrides any previous ones.

This is pretty useful for testing, debugging, etc. as you can focus on just the XML without worrying if it’s patched correctly. The obvious downside is that you won’t include any future updates (which may break the game due to missing fields etc.), and if someone decides to make a “RainingMen” mod then one will override the other, and you can’t have both mods. You never want to do this in a published mod, but for testing it’s useful.

Testing this is a bit annoying, since you need to wait for it to take effect. Also, it seems the game always sets the initial weather for at least 10 in-game days, so you may want to load a save game instead of using -quicktest. Remember You can press 4 for if you enabled the dev console, which speeds up the game to 15× (3 is 6×). You can also make 4 speed it up to a whopping 150× by going to the “TweakValues” developer menu and enabling TickManager.UltraSpeedBoost. I am disappointed this is called UltraFast and UltraSpeedBoost instead of RidiculousSpeed and LudicrousSpeed.

After confirming that our “override it all”-method works let’s properly patch stuff. There are several ways of patching XML resources; I’ll use XPath here, which is the easiest if you just need to patch some XML. Any XML file in Patches/ is treated as patch.

Our patch will just all biomes in Patches/Biomes.xml, but you can select for [defName=..] if you only want to patch specific ones. Remember to remove the overrides if you have any.

<?xml version="1.0" encoding="utf-8" ?>
<Patch>
<!-- Class, not class! -->
<Operation Class="PatchOperationAdd">
    <xpath>/Defs/BiomeDef/baseWeatherCommonalities</xpath>
    <value>
        <RainingBlood>64</RainingBlood>
    </value>
</Operation>
</Patch>

As previously mentioned all XML files in Defs/ are in the same data structure, so don’t worry about the pathnames. There are a number of other operations you can do; see the full documentation for more details on how patching works.

You can use xmllint from libxml2 to test queries on the commandline:

% xmllint --xpath '/Defs/BiomeDef/baseWeatherCommonalities' \
    Data/Core/Defs/BiomeDefs/Biomes_Cold.xml

Writing C♯ code

Let’s expand the mod a bit by making cannibals like raining blood and give them a mood boost, rather than a mood penalty. There isn’t any way to express that in the XML defs, so we need some code for that.

Decompiling the code

Some of the game’s source code is in the installation directory (e.g. ~/rimworld/Source) but it’s not a lot; there’s Source/Verse/Defs/DefTypes/WeatherDef.cs, but it’s not all that useful. You can more or less ignore this directory.

We’ll need to decompile the C♯ code in RimWorldLinux_Data/Managed/Assembly-CSharp.dll.² There are several tools for this; I’ll use ILSpy. This doesn’t seem packaged in most distros but there are Linux binaries for the GUI available as AvaloniaILSpy. This seems to work well enough, but I prefer to extract all the code at once so I can use Vim and grep and whatnot, and the GUI doesn’t seem to do that (there is “save code”, but that doesn’t seem to do anything).

You need to build the ilspycmd binary from source, there isn’t a pre-compiled version as far as I can find. Basic instructions:

# The ".NET home".
% export DOTNET_ROOT=$HOME/dotnet
% mkdir -p $DOTNET_ROOT

# Needs .NET SDK 6 and .NET Core 3.1; binaries from:
# https://dotnet.microsoft.com/en-us/download/dotnet/6.0
# Versions may be different; this is just indicative.
% tar xf dotnet-sdk-6.0.408-linux-x64.tar.gz -C $DOTNET_ROOT

# Add the dotnet path, the binaries we compile later will be in ~/.dotnet/tools
% export PATH=$PATH:$HOME/dotnet:$HOME/.dotnet/tools

# Just the "source code" tar.gz from the GitHub release:
# https://github.com/icsharpcode/ILSpy/archive/refs/tags/v7.2.1.tar.gz
% tar xf ILSpy-7.2.1.tar.gz
% cd ILSpy-7.2.1
% dotnet tool install ilspycmd -g

# Now decompile the lot to src.
% cd ~/rimworld
% mkdir src
% ilspycmd ./RimWorldLinux_Data/Managed/Assembly-CSharp.dll -p -o src

# Hurray!
% ls src
Assembly-CSharp.csproj       FleckUtility.cs         RimWorld/
ComplexWorker_Ancient.cs     HistoryEventUtility.cs  Verse/
ComplexWorker.cs             Ionic/                  WeaponClassDef.cs
DarknessCombatUtility.cs     Properties/             WeaponClassPairDef.cs
FleckParallelizationInfo.cs  ResearchUtility.cs

You only need to do this once. Note that the DOTNET_ROOT is a runtime dependency of ilspycmd, so don’t remove it unless you’re sure you don’t need to run it again.

The decompiled source doesn’t have any comments, and some variables seem changed from the original (e.g. num1, num2, num3, etc.) but it’s mostly fairly readable. The versions in Source do have comments, but the paths don’t quite match up (it seems many subdirs are lost in the decompile?) I considered copying them over to src but I’m not sure if the code in Source matches the exact version.

Building the Assembly

“Assembly” is C♯ speak for any compiled output such as an executable (.exe) or shared library (.dll). We need to set up a “build solution” (C♯ “Makefiles”) to build them. Start by just setting up a basic example before we start actually writing code.

By convention the source code lives in Mods/.../Source/, but I don’t think this is required since the game doesn’t do anything with it directly. The resulting DLL files should be in Mods/.../Assemblies/. Note that you will use a .dll file on Linux as well – it’s just how Mono/C♯ on Linux works. They are cross-platform, an assembly built on Linux should also work on Windows and vice-versa.

The game code lives in two namespaces: Verse and RimWorld. Verse is the game engine and RimWorld is the game built on that. At least, I think that was the intention at some point as there are all sort of RimWorld-specific things in Verse (which also references the RimWorld namespace frequently) and there isn’t really a clear dividing line, but mostly: general “engine-y things” are in Verse and “RimWorld-y things” are in RimWorld, except when they’re not.

In Source/RainingBlood.cs we’ll add a simple example to log something to the developer console:

namespace RainingBlood {
    [Verse.StaticConstructorOnStartup]
    public static class RainingBlood {
        static RainingBlood() {
            Verse.Log.Message("Hello, world!");
        }
    }
}

The [Verse.StaticConstructorOnStartup] annotation makes the code run when the game starts; the game searches for all static constructors with this annotation on startup and executes them. If you really want to know how it works you can use something like rg '[^\[]StaticConstructorOnStartup'.

Another way is inheriting from the Verse.Mod class, which allows some more advanced things (most notably implementing settings), but I’m not going to cover that here.

To build this we’ll need to set up a “build solution”, which consists of a .csproj XML file and a .sln file. This is something I mostly just copied and modified from other projects; it seems that most people are auto-generating this from Visual Studio or MonoDevelop, with little instructions on how to write these things manually. There’s probably a better way of doing some things (not a huge fan of the hard-coded paths instead of using some LDPATH analogue), but I haven’t dived in to this yet.

Here’s what I ended up with in Mods/RainingBlood/RainingBlood.csproj:

<?xml version="1.0" encoding="utf-8"?>
<Project ToolsVersion="14.0" DefaultTargets="Build" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

    <Import
        Project="$(MSBuildExtensionsPath)/$(MSBuildToolsVersion)/Microsoft.Common.props"
        Condition="Exists('$(MSBuildExtensionsPath)/$(MSBuildToolsVersion)/Microsoft.Common.props')"
    />

    <PropertyGroup>
        <RootNamespace>RainingBlood</RootNamespace>
        <AssemblyName>RainingBlood</AssemblyName>
        <!-- You probably want to modify this GUID for your mod, as it's supposed to be unique.
             This is also referenced in the .sln file.
             My system has "uuidgen" to generate UUIDs. -->
        <ProjectGuid>{7196d15e-d480-441a-a2e0-87b9696dd38f}</ProjectGuid>

        <Configuration Condition=" '$(Configuration)' == '' ">Debug</Configuration>
        <Platform Condition=" '$(Platform)' == '' ">AnyCPU</Platform>
        <OutputType>Library</OutputType>
        <AppDesignerFolder>Properties</AppDesignerFolder>
        <TargetFrameworkVersion>v4.7.2</TargetFrameworkVersion>
        <FileAlignment>512</FileAlignment>
        <TargetFrameworkProfile />
    </PropertyGroup>

    <!-- Debug build -->
    <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Debug|AnyCPU' ">
        <DebugSymbols>false</DebugSymbols>
        <DebugType>none</DebugType>
        <Optimize>false</Optimize>
        <OutputPath>Assemblies/</OutputPath>
        <DefineConstants>DEBUG;TRACE</DefineConstants>
        <ErrorReport>prompt</ErrorReport>
        <WarningLevel>4</WarningLevel>
        <UseVSHostingProcess>false</UseVSHostingProcess>
        <Prefer32Bit>false</Prefer32Bit>
    </PropertyGroup>
    <!-- Release build -->
    <PropertyGroup Condition=" '$(Configuration)|$(Platform)' == 'Release|AnyCPU' ">
        <DebugType>none</DebugType>
        <Optimize>true</Optimize>
        <OutputPath>Assemblies/</OutputPath>
        <DefineConstants>TRACE</DefineConstants>
        <ErrorReport>prompt</ErrorReport>
        <WarningLevel>3</WarningLevel>
        <Prefer32Bit>false</Prefer32Bit>
    </PropertyGroup>

    <!-- Dependencies -->
    <ItemGroup>
        <!-- The main game code (RimWorld and Verse) -->
        <Reference Include="Assembly-CSharp">
            <HintPath>../../RimWorldLinux_Data/Managed/Assembly-CSharp.dll</HintPath>
            <Private>False</Private>
        </Reference>

        <!-- C#/.NET stdlib -->
        <Reference Include="System" />
        <Reference Include="System.Core" />
        <Reference Include="System.Runtime.InteropServices.RuntimeInformation" />
        <Reference Include="System.Xml.Linq" />
        <Reference Include="System.Data.DataSetExtensions" />
        <Reference Include="Microsoft.CSharp" />
        <Reference Include="System.Data" />
        <Reference Include="System.Net.Http" />
        <Reference Include="System.Xml" />
    </ItemGroup>

    <!-- File list -->
    <ItemGroup>
        <Compile Include="Source/RainingBlood.cs" />
    </ItemGroup>

    <Import Project="$(MSBuildToolsPath)/Microsoft.CSharp.targets" />
</Project>

And Mods/RainingBlood/RainingBlood.sln:

Microsoft Visual Studio Solution File, Format Version 12.00
# Visual Studio 15
VisualStudioVersion = 15.0.27703.2035
MinimumVisualStudioVersion = 10.0.40219.1
Project("{57073194-e8b4-4a20-b60c-ee0e10947af0}") = "RainingBlood", "RainingBlood.csproj", "{7196d15e-d480-441a-a2e0-87b9696dd38f}"
EndProject
Global
    GlobalSection(SolutionConfigurationPlatforms) = preSolution
        Debug|Any CPU = Debug|Any CPU
        Release|Any CPU = Release|Any CPU
    EndGlobalSection
    GlobalSection(ProjectConfigurationPlatforms) = postSolution
        {7196d15e-d480-441a-a2e0-87b9696dd38f}.Debug|Any CPU.ActiveCfg = Debug|Any CPU
        {7196d15e-d480-441a-a2e0-87b9696dd38f}.Debug|Any CPU.Build.0 = Debug|Any CPU
        {7196d15e-d480-441a-a2e0-87b9696dd38f}.Release|Any CPU.ActiveCfg = Release|Any CPU
        {7196d15e-d480-441a-a2e0-87b9696dd38f}.Release|Any CPU.Build.0 = Release|Any CPU
    EndGlobalSection
    GlobalSection(SolutionProperties) = preSolution
        HideSolutionNode = FALSE
    EndGlobalSection
    GlobalSection(ExtensibilityGlobals) = postSolution
        SolutionGuid = {31005EA7-3F04-446F-80B2-016137708540}
    EndGlobalSection
EndGlobal

To build it you’ll need msbuild, which is not included in the Standard Mono installation. It does have xbuild, but that gives a deprecated warning pointing towards msbuild. Maybe it works as well, but I didn’t try it. Luckily msbuild does seem commonly packaged, so I just installed it from there.

I put the solution files in the project root; other people prefer to put it in the Source/ directory, but you’ll need to modify some of the paths if you put it there. To build it, simply run msbuild from the directory, or use msbuild Mods/RainingBlood to specify a path. After this you should have Assemblies/RainingBlood.dll.

After starting the game now and opening the developer console you should see “Hello, world!” in there.

Writing the code

Alrighty, now that all the plumbing is working we can actually start doing some stuff. Let’s see what grepping for exposedThought gives us:

[~/rimworld/src]% rg exposedThought
Verse/AI/Pawn_MindState.cs
415: if (curWeatherLerped.exposedThought != null && !pawn.Position.Roofed(pawn.Map))
417:     pawn.needs.mood.thoughts.memories.TryGainMemoryFast(curWeatherLerped.exposedThought);

Verse/WeatherDef.cs
35: public ThoughtDef exposedThought;

Verse/AI/Pawn_MindState.cs seems to be what we want, and reading through MindStateTick() the logic seems straightforward enough:

namespace Verse.AI {
    public class Pawn_MindState : IExposable {
        // [..]

        public void MindStateTick() {
            // [..]

            if (Find.TickManager.TicksGame % 123 == 0 &&
                pawn.Spawned && pawn.RaceProps.IsFlesh && pawn.needs.mood != null
            ) {
                TerrainDef terrain = pawn.Position.GetTerrain(pawn.Map);
                if (terrain.traversedThought != null) {
                    pawn.needs.mood.thoughts.memories.TryGainMemoryFast(terrain.traversedThought);
                }

                WeatherDef curWeatherLerped = pawn.Map.weatherManager.CurWeatherLerped;
                if (curWeatherLerped.exposedThought != null && !pawn.Position.Roofed(pawn.Map)) {
                    pawn.needs.mood.thoughts.memories.TryGainMemoryFast(curWeatherLerped.exposedThought);
                }
            }

            // [..]
        }
    }
}

So every 123rd “tick” it checks the terrain and weather and applies any mood effects. Digging a bit deeper:

The Pawn class describes a person or animal (“pawn”) in the game; every Pawn has a MindState attached to it.
On every “tick” it calls the MindStateTick() method on the attached MindState instsance as long as the pawn isn’t dead (as well as a number of other things).
One “tick” corresponds to 1/60th real second, which is 1.44 minutes in-game time. This is the game’s Planck time: everything that happens will take at least 1.44 minutes in-game.
There are also “rare ticks” (= 250 ticks = 4.15 real seconds = 6 hours in-game, or 1/4th of a day) and “long ticks” (= 1000 ticks = 33.33 real seconds = 1 day in-game) that you can hook in to various places.
If you speed up the game then ticks are just emitted faster: 3× or 6×. So instead of emitting a tick once every 1/60th second it becomes once every 1/180th second or 1/360th second.

You can find a bit more about this in Verse/Tick*.cs. It’s not really needed to know this for a simple mod like this, but it’s useful to know if you want to write actual real mods.

To add our custom logic first add a new “thought” we want to apply to Defs/ThoughtDefs/RainingBlood.xml we created earlier:

<ThoughtDef>
    <defName>BloodCoveredCannibal</defName>
    <durationDays>0.1</durationDays>
    <stackLimit>1</stackLimit>
    <stages>
        <li>
            <label>blood covered</label>
            <description>Reigning in blood!</description>
            <baseMoodEffect>10</baseMoodEffect>
        </li>
    </stages>
</ThoughtDef>

And in the Defs/WeatherDefs/RainingBlood.xml let’s add some new fields next to the exposedThought we already have:

<modExtensions>
    <!-- Class, not class! -->
    <li Class="RainingBlood.WeatherDefExtension">
        <exposedThoughtCannibal>BloodCoveredCannibal</exposedThoughtCannibal>
    </li>
</modExtensions>

The way the XML maps to C♯ code is that every entry in the XML file is expected to be a field in the *Def class (inherits from Verse.Def), for example for the existing exposedThought the WeatherDef class has:

public ThoughtDef exposedThought;

If you were to just add exposedThoughtCannibal you’d get an error telling you that exposedThoughtCannibal isn’t a field in the class:

<exposedThoughtCannibal>[...] doesn't correspond to any field in in type WeatherDef

RimWorld comes with the modExtensions field to extend Defs. In this case we’re adding it to an entire new Def, but you can also patch existing Defs with XPath and PatchOperationAddModExtension.

You’ll also need to add a new class inhereting from Verse.DefModExtension:

namespace RainingBlood {
    public class WeatherDefExtension : Verse.DefModExtension {
        public RimWorld.ThoughtDef exposedThoughtCannibal;
    }
}

The Class attribute in the XML links the XML fields to this class. The name can be anything. We can get the value in C♯ with the GetModExtension<T>() method on any Def class, where T is the type (class name) you want. For example, GetModExtension<WeatherDefExtension>() in this case. By using the type system multiple mods can attach their own extensions and not conflict.

Using Harmony

To make this actually do something we need to hook in some code; RimWorld itself doesn’t really have a “mod system” for this, but we can use Harmony. Harmony is a C♯ library to patch existing code and can do a number of things, but the most useful (and least error-prone) is to run code before or after a method. In our case, we want to run code after Pawn_MindState.MindStateTick() to apply the exposedThoughtCannibal thought.

To use this we’ll need to register it as a dependency in our About/About.xml file:

<modDependencies>
    <li>
        <packageId>brrainz.harmony</packageId>
        <displayName>Harmony</displayName>
        <steamWorkshopUrl>steam://url/CommunityFilePage/2009463077</steamWorkshopUrl>
        <downloadUrl>https://github.com/pardeike/HarmonyRimWorld/releases/latest</downloadUrl>
    </li>
</modDependencies>

We’ll also have to add it to the RainingBlood.csproj file as a dependency before the Assembly-CSharp dependency:

<!-- Dependencies -->
<ItemGroup>
    <!-- Harmony must be loaded first -->
    <Reference Include="0Harmony">
        <HintPath>../HarmonyRimWorld/Current/Assemblies/0Harmony.dll</HintPath>
        <Private>False</Private>
    </Reference> 

    <!-- The main game code (RimWorld and Verse) -->
    <Reference Include="Assembly-CSharp">
        <HintPath>../../RimWorldLinux_Data/Managed/Assembly-CSharp.dll</HintPath>
        <Private>False</Private>
    </Reference>

    [..]

Now we can use it to run some code after the Pawn_MindState.MindStateTick() method:

namespace RainingBlood {
    public class WeatherDefExtension : Verse.DefModExtension {
        public RimWorld.ThoughtDef exposedThoughtCannibal;
    }

    [Verse.StaticConstructorOnStartup]
    public static class Patch {
        static Patch() {
            // Get the method we want to patch.
            var m = typeof(Verse.AI.Pawn_MindState).GetMethod("MindStateTick");

            // Get the method we want to run after the original.
            var post = typeof(RainingBlood.Patch).GetMethod("PostMindStateTick",
                       System.Reflection.BindingFlags.Static|System.Reflection.BindingFlags.Public);

            // Patch stuff! The string passed to the Harmony constructor can be
            // anything, and can be used to identify/remove patches if need be.
            new HarmonyLib.Harmony("arp242.rainingblood").Patch(m,
                postfix: new HarmonyLib.HarmonyMethod(post));
        }

        // The special __instance parameter has the original class instance
        // we're extending. This is based on the argument name.
        public static void PostMindStateTick(Verse.AI.Pawn_MindState __instance) {
            var pawn = __instance.pawn;

            // Same condition as MindStateTick, but inversed for early return.
            if (Verse.Find.TickManager.TicksGame % 123 != 0 ||
                !pawn.Spawned || !pawn.RaceProps.IsFlesh || pawn.needs.mood == null)
                return;

            // Is this pawn a cannibal? If not, then there's nothing to do. You
            // can also expand this by checking for the Ideology cannibalism
            // memes, but this just checks the "cannibalism" trait on colonists.
            if (!pawn.story.traits.HasTrait(RimWorld.TraitDefOf.Cannibal))
                return;

            // Let's see if the current weather has our new exposedThoughtCannibal.
            var w = pawn.Map.weatherManager.CurWeatherLerped;
            if (!w.HasModExtension<WeatherDefExtension>())
                return;
            var t = w.GetModExtension<WeatherDefExtension>().exposedThoughtCannibal;
            if (t == null)
                return;

            // Remove any existing thought that was applied and apply our
            // cannibalistic thoughts.
            if (w.exposedThought != null)
                pawn.needs.mood.thoughts.memories.RemoveMemoriesOfDef(w.exposedThought);
            pawn.needs.mood.thoughts.memories.TryGainMemoryFast(t);
        }
    }
}

I use the “manual method” here as that’s a bit easier to debug if you did something wrong, but you can also use the annotations. Again, see the Harmony documentation. One thing you need to watch out for is getting the BindingFlags.[..] right. If you don’t then the reflection library won’t find your method and it’ll return null. See the GetMethod() documentation. This part actually took me quite a bit to get working. Unfortunately RimWorld doesn’t have a REPL or console (AFAIK?) but you can use some printf-debugging with Verse.Log.Message($"{var}") and the like.

I’m not going to step through the rest of the code in more detail here; I think most of it should be obvious. I mostly just found this be looking through various code and some strategic grepping. You can test this by using the Debug Actions menu, which allows assigning the Cannibalism trait to a colonist.

Next steps

The above wasn’t really all that useful as such, and there are many more parts of RimWorld modding – most of which I haven’t looked at in detail yet – but this should at least give a decent base to get started with.

I have to say that I found a lot of documentation and guides on the topic to be of, ehm, less-than-stellar quality :-/ The RimWorld wiki has a whole bunch of pages, but – with a few exceptions linked in the article here – I found many are unclear, outdated, or both, and in a few cases just downright wrong. Keep that in mind if something doesn’t work: usually it’s a mistake to assume the documentation is wrong instead of you, but here it might actually be the case. I’ll see if I’ll write some more if I keep up interest in this.

Some additional reading for topics not covered:

Multi-version mods

Details how to make a mod compatible with both 1.2 and 1.3. I elided this for simplicity, and also because quite frankly I don’t really care as I’m just interested in writing some mods that work for me to fix/improve some things 🤷
RimWorld art source

The original PSD files for all art in RimWorld. Useful if you want to use a modified version in your mod.
Mod folder structure

Covers some things not used in this example, such as sounds, textures, and i18n.
TDBug adds some debug things which seem useful. Haven’t tried it yet.

Missing parts

And some things I’d still like to improve/figure out:

I would really really like a REPL, debugger, or some other way to speed up the dev cycle. RimWorld takes fairly long to start (almost a minute on my laptop) and toying around with things is kinda annoying and time-consuming.

The closest I found is How I got RimWorld debugging to work; the CLI works on Linux (run with dnSpy.Console.exe, from the .NET download) but the GUI doesn’t (and never will, as the Windows-specific GUI toolkit things aren’t implemented on Linux), but this doesn’t support the debugger (just decompilation).

I tried the generic sdb Mono debugger, but the game doesn’t load directly with Mono but rather via the 32M UnityPlayer.so, so using that seems difficult. Using gdb works, but actually doing useful stuff with it (i.e. breakpoints, calling functions, displaying variable values) seems harder, but I haven’t spent that much time with it yet.

Making the game start faster would help too, or an automatic “script” to run on startup (i.e. to apply certain debug actions).
On Linux the Assemblies are in RimWorldLinux_Data/, but on Windows and macOS this directory is RimWorldWin64_Data/ and RimWorldMac_Data/. Right now the build solution builds just on Linux, but I’d like to be able to make it build on all systems.

Hard-coding this path seems common; to get other mods to build I had to manually s/Win64/Linux/ some things, which is not ideal. I couldn’t figure out how to make it cross-platform.

Footnotes

Although I later did confirm that they’re relative weights, see Verse.TryRandomElementByWeight(), which can be examined after decompiling the source. ↩
Explicitly allowed in the EULA it turns out: “You’re allowed to ‘decompile’ our game assets and look through our code, art, sound, and other resources for learning purposes, or to use our resources as a basis or reference for a Mod. However, you’re not allowed to rip these resources out and pass them around independently.” I wish they’d just make this easier by distributing more code, but ah well. ↩

Stallman isn't great, but not the devil

Thu, 25 Mar 2021 00:00:00 +0000

So Richard Stallman is back at the FSF, on the board of directors this time rather than as President. I’m not sure how significant this position is in the day-to-day operations, but I’m not sure if that’s really important.

How anyone could have thought this was a good idea is beyond me. I’ve long considered Stallman to be a poor representative of the community, and quite frankly it baffles me that people do. I’m not sure what the politics were that led up to this decision; I had hoped that after Stallman’s departure the FSF would move forward and shed off some of the Stallmanisms. It seems this hasn’t happened.

To quickly recap why Stallman is a poor representative:

Actively turned many people off because he’s such a twat; one of the better examples I know of is from Keith Packard, explaining why X didn’t use the GPL in spite of Packard already having used it for some of his projects before:

Richard Stallman, the author of the GPL and quite an interesting individual lived at 5405 DEC square, he lived up on the sixth floor I think? Had an office up there; he did not have an apartment. And we knew him extremely well. He was a challenging individual to get along with. He would regularly come down to our offices and ask us, or kind of rail at us, for not using the GPL.

This did not make a positive impression on me; this was my first interaction with Richard directly and I remember thinking at the time, “this guy is a little, you know, I’m not interesting in talking to him because he’s so challenging to work with.”

And so, we should have listened to him then but we did not because, we know him too well, I guess, and met him as well.

He really was right, we need to remember that!
His behaviour against women in particular is creepy. This is not a crime (he has, as far as I know, never forced himself on anyone) but not a good quality in a community spokesperson, to put it mildly.
His personal behaviour in general is … odd, to put it mildly. Now, you can be as odd as you’d like as far as I’m concerned, but I also don’t think someone like that is a good choice to represent an entire community.
Caused a major and entirely avoidable fracture of the community with the Open Source movement; it’s pretty clear that Stallman, him specifically as a person, was a major reason for the OSI people to start their own organisation. Stallman still seems to harbour sour grapes over this more than 20 years later.
Sidetracking of pointless issues (“GNU/Linux”, “you should not be using hacker but cracker”, “Open Source misses the point”, etc.), as well as stubbornly insisting on the term “Free Software” which is confusing and stands in the way if communicating the ideals to the wider world. Everyone will think that an article with “Free Software” in the title will be about software free of charge. There is a general lack of priorities or pragmatism in almost anything Stallman does.
Stallman’s views in general on computing are stuck somewhere about 1990. Possibly earlier. The “GNU Operating System” (which does not exist, has never existed, and most likely will never exist¹) is not how to advance Free Software in modern times. Most people don’t give a rat’s arse which OS they’re using to access GitHub, Gmail, Slack, Spotify, Netflix, AirBnB, etc. The world has changed and the strategy needs to change – but Stallman is still stuck in 1990.
Insisting on absolute freedom to the detriment of more freedom compared to the status quo. No, people don’t want to run a “completely free GNU/Linux operating system” if their Bluetooth and webcam doesn’t work and if they can’t watch Netflix. That’s just how it is. Deal with it.

His views are quite frankly ridiculous:

If [an install fest] upholds the ideals of freedom, by installing only free software from 100%-free distros, partly-secret machines won’t become entirely functional and the users that bring them will go away disappointed. However, if the install fest installs nonfree distros and nonfree software which make machines entirely function, it will fail to teach users to say no for freedom’s sake. They may learn to like GNU/Linux, but they won’t learn what the free software movement stands for.

[..]

My new idea is that the install fest could allow the devil to hang around, off in a corner of the hall, or the next room. (Actually, a human being wearing sign saying “The Devil,” and maybe a toy mask or horns.) The devil would offer to install nonfree drivers in the user’s machine to make more parts of the computer function, explaining to the user that the cost of this is using a nonfree (unjust) program.

Aside from the huge cringe factor of having someone dressed up as a devil to install a driver, the entire premise is profoundly wrong; people can appreciate freedom while also not having absolute/maximum freedom. Almost the entire community does this, with only a handful of purist exceptions. This will accomplish nothing except turn people off.
Crippling software out of paranoia; for example Stallman refused to make gcc print the AST – useful for the Emacs completion and other tooling – because he was afraid someone might “abuse” it. He comes off as a gigantic twat in that entire thread (e.g. this).

How do you get people to use Free Software? By making great software people want to use. Not by offering some shitty crippled product where you can’t do some common things you can already do in the proprietary alternatives.

Luckily, the backlash against this has been significant, including an open letter to remove Richard M. Stallman from all leadership positions. Good. There are many things in the letter I can agree with. If there are parliamentary hearings surrounding some Free Software law then would you want Stallman to represent you? Would you want Stallman to be left alone in a room with some female lawmaker (especially an attractive one)? I sure wouldn’t; I’d be fearful he’d leave a poor impression, or outright disgrace the entire community.

But there are also a few things that bother me, as are there in the general conversation surrounding this topic. Quoting a few things from that letter:

[Stallman] has been a dangerous force in the free software community for a long time. He has shown himself to be misogynist, ableist, and transphobic, among other serious accusations of impropriety.

[..]

him and his hurtful and dangerous ideology

[..]

RMS and his brand of intolerance

Yikes! That sounds horrible. But closer examinations of the claims don’t really bear out these strong claims.

The transphobic claim seems to hinge entirely on his eclectic opinion regarding gender-neutral pronouns; he prefers some peculiar set of neologisms (“per” and “pers”) instead of the singular “they”. You can think about his pronoun suggestion what you will – I feel it’s rather silly and pointless at best – but a disagreement on how to best change the common use of language to be more inclusive does not strike me as transphobic. Indeed, it strikes me as the exact opposite: he’s willing to spend time and effort to make language more inclusive. That he doesn’t do it in the generally accepted way is not transphobia, a “harmful ideology”, or “dangerous”. It’s really not.

Stallman is well known for his excessive pedantry surrounding language; he’s not singularly focused on the issue of pronouns and has consistently posted in favour of trans rights.

Stallman’s penchant to make people feel unconformable has long been known; and should hardly come as a surprise to anyone. Many who met him in person did not leave with an especially good impression of him for one reason or the other. His behaviour towards women in particular is pretty bad; many anecdotes have been published and they’re pretty 😬

But … I don’t have the impression that Stallman dislikes or distrusts women, or sees them as subservient to men. Basically, he’s just creepy. That’s not good, but is it misogyny? His lack of social skills seem to be broad and not uniquely directed towards women. He’s just a socially awkward guy in general. I mean, this is a guy who will, when giving a presentation, will take off his shoes and socks – which is already a rather weird thing to do – will then proceed to rub his bare foot – even weirder – only to proceed to appear to eat something from his foot – wtf wtf wtf?!

If he can’t understand that this is just … wtf, then how can you expect him to understand that some comment towards a woman is wtf?

Does all of this excuse bad behaviour? No. But it shines a rather different light on things than phrases such as “misogynist”, “hurtful and dangerous ideology”, and “his brand of intolerance” do. He hasn’t forced himself on anyone, as far as I know, and most complaints are about him being creepy.

I don’t think it’s especially controversial to claim that Stallman would have been diagnosed with some form of autism if he had been born several decades later. This is not intended as an insult or some such, just to establish him as a neurodivergent² individual. Someone like that is absolutely a poor choice for a leadership position, but at the same time doesn’t diversity also mean diversity of neurodivergent people, or at the very least some empathy and understanding when people’s exhibit a lack of social skills and behaviour considered creepy?

At what point is there a limit if someone’s neurodiversity drives people away? I don’t know; there isn’t an easy answer to his. Stallman is clearly unsuitable for a leadership role; but “misogynist”? I’m not really seeing it in Stallman.

The ableist claim seems to mostly boil down to a comment he posted on his website regarding abortion of fetuses with Down’s syndrome:

A new noninvasive test for Down’s syndrome will eliminate the small risk of the current test.

This mind lead more women to get tested, and abort fetuses that have Down’s syndrome. Let’s hope so!

If you’d like to love and care for a pet that doesn’t have normal human mental capacity, don’t create a handicapped human being to be your pet. Get a dog or a parrot. It will appreciate your love, and it will never feel bad for being less capable than normal humans.

It was later edited to its current version:

A noninvasive test for Down’s syndrome eliminates the small risk of the old test. This might lead more women to get tested, and abort fetuses that have Down’s syndrome.

According to Wikipedia, Down’s syndrome is a combination of many kinds of medical misfortune. Thus, when carrying a fetus that is likely to have Down’s syndrome, I think the right course of action for the woman is to terminate the pregnancy.

That choice does right by the potential children that would otherwise likely be born with grave medical problems and disabilities. As humans, they are entitled to the capacity that is normal for human beings. I don’t advocate making rules about the matter, but I think that doing right by your children includes not intentionally starting them out with less than that.

When children with Down’s syndrome are born, that’s a different situation. They are human beings and I think they deserve the best possible care.

He also made a few other comments to the effect of “you should abort if you’re pregnant with a fetus who has Down’s syndrome”.

That last paragraph of the original version was … not great, but the new version seems okay to me. It is a women’s right to choose to have an abortion, for any reason, including not wanting to raise a child with Down’s syndrome. This is already commonplace in practice, with many women choosing to do so.

Labelling an entire person as ableist based only on this – and this is really the only citation of ableism I’ve been able to find – seems like a stretch, at best. It was a shitty comment, but he did correct it which is saying a lot in Stallman terms, as I haven’t seen him do that very often.

Phrases like “a dangerous force”, “dangerous ideology”, and “brand of intolerance” make it sound like he’s crusading on these kind of issues. Most of these are just short notes on his personal site which few people seem to read.

Most of the issues surrounding Stallman seem to be about him thinking out loud, not realizing when it is or is not appropriate to do so, being excessively pedantic over minor details, or just severally lacking in social skills. This can be inappropriate, offensive, or creepy – depending on the scenario – but that’s just something different than being actively transphobic or dangerous. If someone had read only this letter without any prior knowledge of Stallman they would be left with the impression that Stallman is some sort of alt-right troll writing for Breitbart or the like. This is hardly the case.

I think Stallman should resign of newly appointed post, and from GNU as well, over his personal behaviour in particular. Stallman isn’t some random programmer working on GNU jizamabob making the occasional awkward comment, he’s the face of the entire movement. Appointing “a challenging individual to get along with” – to quote Packard – is not the right person for the job. I feel the rest of the FSF board has shown spectacular poor judgement in allowing Stallman to come back.³

But I can not, in good conscience, sign the letter as phrased currently. It vastly exaggerates things to such a degree that I feel it does a gross injustice to Stallman. It’s grasping at straws to portray Stallman as the most horrible human being possible, and I don’t think he is that. He seems clueless on some topics and social interactions, and find him a bit of a twat in general, but that doesn’t make you a horrible and dangerous person. I find the letter lacking in empathy and deeply unkind.

In short, I feel Stallman’s aptitudes do not apply well for any sort of leadership position and I would rather not have him represent the community I’m a part of, even if he did start it and made many valuable contributions to it. Just starting something does not give you perpetual ownership over it, and in spite of all his hard work I feel he’s been very detrimental to the movement and has been a net-negative contributor for a while. A wiser version of Stallman would have realized his shortcomings and stepped down some time in the late 80s to let someone else be the public face.

Overall I feel he’s not exactly a shining example of the human species, but then again I’m probably not either. He is not the devil and the horrible person that the letter makes him out to be. None of these exaggerations are even needed to make the case that he should be removed, which makes it even worse.

It’s a shame, because instead of moving forward with Free Software we’re debating this. Arguably I should just let this go as Stallman isn’t really worth defending IMO, but on the other hand being unfair is being unfair, no matter who the target may be.

Footnotes

A set of commandline utilities, libc, and a compiler are not an operating system. Linux (the kernel) is not the “last missing piece of the GNU operating system”. ↩
Neurodivergency is, in a nutshell, the idea that “normal” is a wide range, and that not everyone who doesn’t fits with the majority should be labelled as “there is something wrong with them” such as autism. While some some people take this a bit too far (not every autist is high-functioning; for some it really is debilitating) I think there’s something to this. ↩
I guess this shouldn’t come as that much of a surprise, as the only people willing and able to hang around Stallman’s FSF were probably similar-ish people. It’s probably time to just give up on the FSF and move forward with some new initiative (OSI is crap too, for different reasons). I swear we’ve got to be the most dysfunctional community ever. ↩