Writing Software You Don't Run

The annoying lesson is this: some software escapes your runtime.

You write it, release it, and then it runs on machines you do not own, inside environments you cannot inspect, at versions you cannot force-upgrade, with local state you cannot safely delete because somebody might actually care about it.

That sounds like blockchain infrastructure, because it is. Random people run your node software in their homes, labs, validators, cloud instances, and cursed Hetzner boxes. You do not control when they upgrade. You do not control what data they have persisted. You do not control which peers they talk to. The best thing you get is logs, bug reports, and occasionally a Discord message that says it broke with no additional context. Beautiful.

But the same shape appears outside blockchains too.

It appears in CLIs. It appears in local agent runtimes. It appears in plugins, browser caches, mobile apps, self-hosted deployments, desktop apps, SDKs, file formats, sync engines, and anything that stores state on a user’s machine.

You are not just writing code.

You are writing code for future machines you do not control.

The rote problem that felt like a node problem

The thing that triggered this thought was reviewing some code around fingerprints in rote CLI.

At first glance, rote and blockchain infrastructure have very little in common. One is a CLI for running workflows and adapters. The other is a distributed pile of cryptography, consensus, economic incentives, peer-to-peer networking, and extremely online people arguing about finality.

So, you know, basically the same thing.

The shared constraint is not consensus. The shared constraint is ownership.

With rote, we are building software that runs on users’ machines. We do not run it ourselves. We do not control the version installed on their laptop. We do not control their ~/.rote folder. We do not control which adapter manifests they have cached, which flows they have downloaded, or what weird experimental version they ran three weeks ago before going back to stable.

That means every persisted JSON blob is a tiny protocol.

Every adapter manifest fetched from the hub is a wire format.

Every flow front matter block is a compatibility boundary.

And once you see that, a bunch of blockchain engineering patterns suddenly stop looking like blockchain-specific paranoia and start looking like normal adult software engineering. Horrible news, frankly.

Version everything that crosses time

The main rule is simple:

If data touches the wire or gets persisted on the user’s machine, it should have an explicit version.

Not an implied version. Not “we can infer it because this field exists.” Not “well, the optional field is missing, so it must be the old one.” I mean a boring version tag at the top.

Something like this, in Rust-shaped pseudocode:

#[derive(Serialize, Deserialize)]
#[serde(tag = "version")]
enum AdapterManifest {
    V1(AdapterManifestV1),
    V2(AdapterManifestV2),
}

struct AdapterManifestV1 {
    name: String,
    command: String,
}

struct AdapterManifestV2 {
    name: String,
    command: String,
    base_url: Option<String>,
    capabilities: Vec<String>,
}

The important part is not the exact Rust syntax. The important part is that V1 means V1 forever.

Once AdapterManifestV1 has been written to a user’s disk, or sent over the wire, or included in a release, you do not keep editing it until it becomes three raccoons in a trench coat pretending to be a schema.

You add V2.

Then V3.

This is one of the blockchain habits I wish more normal software copied: the old implementation stays old.

If block version 1 was valid in 2024, then the code that handles block version 1 should still handle that exact shape in 2034. Not “mostly the same parser, except with branches for v1, v2, and v3 hidden inside it.” Not “the generic implementation learned some new behavior and we hope it still means the old thing.” A real version boundary.

Code is cheap. User trust is not.

So yes, copy-paste the v1 handler if you have to. Freeze it. Put the old behavior behind the V1 match arm and make people feel slightly embarrassed about touching it.

match manifest {
    AdapterManifest::V1(v1) => load_v1_manifest_exactly_like_v1_always_did(v1),
    AdapterManifest::V2(v2) => load_v2_manifest(v2),
    AdapterManifest::V3(v3) => load_v3_manifest(v3),
}

That duplication is not laziness. It is the compatibility boundary made visible. V2 is allowed to behave differently because it gets its own branch. V1 is allowed to keep working because nobody is “cleaning up” its behavior to fit the current model.

This is how you get real backwards compatibility instead of the cursed version where a maintainer says, “oh, that worked two versions ago, but not anymore,” about data your own software wrote.

Then V4.

Then, if you live long enough to see V9, congratulations, your product has users and your past self has become your enemy.

This was already visible in rote. An adapter manifest changed shape. A base URL was added. Other fields appeared. The Rust definition accumulated optional fields and comments like “kept for backwards compatibility” or “do not break old manifests.”

That works for a while. Everything works for a while. That is how the trap gets you.

The problem is that nobody can tell which fields are mutually exclusive, which combinations are valid, and which ones only exist because an old release accidentally wrote them once. Humans get confused. Agents get confused. Tests get vague. Eventually some unlucky person changes a field that looked harmless and breaks an old user state they did not know existed.

I have met this bug before. It usually wears a blockchain hat.

Upgrade paths should be boring

Once your persisted data is explicitly versioned, upgrades become a normal function instead of a séance.

impl AdapterManifestV1 {
    fn upgrade(self) -> AdapterManifestV2 {
        AdapterManifestV2 {
            name: self.name,
            command: self.command,
            base_url: None,
            capabilities: vec![],
        }
    }
}

impl AdapterManifestV2 {
    fn upgrade(self) -> AdapterManifestV3 {
        // boring, explicit, testable
    }
}

Do not jump directly from V1 to whatever the current version happens to be today.

Go one version at a time:

1. Load old data.
2. Identify its version.
3. Upgrade V1 -> V2.
4. Upgrade V2 -> V3.
5. Continue until current.
6. Persist current only after the upgrade actually succeeded.

This is not glamorous engineering. Nobody is going to put “wrote a really boring migration function” on a conference slide unless they have become emotionally damaged by production systems.

So, naturally, here I am.

The reason one-step migrations matter is that every version boundary gets a concrete meaning. You can test it. You can review it. You can delete support intentionally later. You can explain to the user why their data is too old instead of pretending the software can just vibe its way through history.

In blockchain clients, this shows up as hardfork logic, database migrations, state snapshots, protocol upgrades, transaction formats, peer capability changes, and all the little compatibility shims that let old nodes and new nodes avoid immediately setting the network on fire.

In CLIs and agent runtimes, it is the same idea with less dramatic branding.

Unsupported versions are a feature

A nice side effect of explicit versions is that your software can say no. Imagine a user is running an older rote binary and downloads an adapter manifest produced for a newer version. Without a version boundary, the old binary can only parse best-effort or crash somewhere unrelated.

With explicit versions, it can fail before doing damage:

This manifest is version 3. This rote binary supports up to version 2. Please upgrade rote.

That is a product feature. The user learns what to do, support learns what happened, and the system does not silently execute something it does not understand.

This is one of the blockchain lessons that generalizes very cleanly. Old nodes see things they do not understand; old CLIs do too. Clients need to distinguish “old but supported” from “too old” and “too new.” If you do not define that boundary, your parser defines it for you, and parsers are not known for their product taste.

Golden data tests

The next pattern is committing examples of old data into the codebase and testing them forever.

This is another thing that sounds painfully obvious until you look at how many projects do not do it.

If your adapter manifest format matters, commit example manifests.

If your flow front matter matters, commit example flows.

If your local cache format matters, commit cache fixtures.

If your wire message matters, commit encoded bytes.

Then write tests that load them.

#[test]
fn loads_v1_manifest_fixture() {
    let bytes = include_bytes!("fixtures/adapter-manifest-v1.json");
    let manifest = AdapterManifest::from_slice(bytes).unwrap();
    let current = manifest.upgrade_to_current().unwrap();

    assert_eq!(current.name(), "example");
}

The fixture becomes a contract. If a future change breaks it, the test tells you that you broke something real, not just some abstract property of the current code.

Blockchain clients do this everywhere because they have to. Genesis files. Blocks. Receipts. Transactions. Fork transition cases. Encoded payloads. State tests. Consensus test vectors. The boring pile of fixtures is what keeps everyone honest when the implementation changes.

For user-hosted software, golden data tests are the same safety net.

They answer one question:

Can the version we are about to ship still understand what the versions we already shipped wrote down?

That is the question. Everything else is commentary.

Test old binaries too

Golden data tests catch one class of failure: current code can read old data.

They do not catch the other direction:

What happens when an old binary exists in the world and sees new data?

This is where blockchain infrastructure gets properly annoying in a useful way.

In a network, you rarely get to assume everyone upgraded. Old binaries keep running. New binaries gossip messages. Some nodes are behind. Some operators are cautious. Some forgot they run a node at all. Some have a systemd service that will outlive us all.

So serious protocol projects test across versions.

The practical version looks like this:

1. Fetch an older release binary.
2. Create a realistic home directory or data directory.
3. Run the old binary and let it write state.
4. Upgrade to the new binary.
5. Verify the migration.
6. Sometimes do the reverse scenario: make sure old binaries fail clearly on new data.

For rote, this would mean grabbing old release artifacts, creating a fake ~/.rote, installing adapters, writing flows, then upgrading through the real CLI path instead of only calling migration functions in isolation.

That sounds heavier than unit tests because it is.

It is also how you find the bugs that unit tests politely ignore.

This matters more once auto-upgrade exists. If users manually upgrade and something goes horribly wrong, you can maybe tell a few early adopters to delete a folder and pretend nobody saw anything. Not ideal, but survivable.

If your software auto-upgrades itself and corrupts local state, you just distributed the footgun at scale.

Stonks.

Determinism is not only for consensus

Blockchain engineers become weird about deterministic encoding because non-determinism is how you create expensive disagreement.

If two nodes serialize the same logical object differently, hashes change. Signatures fail. Merkle roots diverge. Consensus explodes. People on Twitter become experts in your bug within 45 minutes.

Outside blockchains, the blast radius is usually smaller, but the shape is the same.

If your CLI fingerprints an adapter, the fingerprint must be based on deterministic bytes. If your agent caches tool definitions, the cache key must not change because object field order changed. If your hub signs manifests, the signed bytes need to be canonical. If your migration says two manifests are equivalent, it needs a stable definition of equivalent.

Otherwise you get bugs that look like ghosts:

• “It works on my machine, but my coworker gets a different fingerprint.”
• “The cache misses every time after upgrade.”
• “The signature fails only for manifests generated by the other tool.”
• “The agent thinks the adapter changed, but the diff is empty.”

These bugs are not mystical. They are just bytes again.

Bytes, as usual, are where the vibes end.

Crash-safe upgrades

There is one more pattern that deserves more respect: upgrades should survive being interrupted.

A migration that works only if the process stays alive, the laptop stays awake, the disk does not fill, and the user does not press Ctrl-C is not really a migration. It is a trust exercise.

If the software owns the deployment environment, you can sometimes choreograph things. Maintenance window. Database transaction. Rollback plan. On-call person with coffee and regret.

On a user’s laptop, good luck.

They close the lid. The battery dies. The process gets killed. The terminal gets closed. Their antivirus gets creative. The filesystem does filesystem things.

So upgrades need boring mechanics:

• write new files next to old files, then atomically rename;
• keep backups until the new version is verified;
• make migrations idempotent where possible;
• record migration progress;
• never delete the only copy before the replacement is durable;
• make restart after partial migration boring.

Again, blockchain clients have this scar tissue because local databases are enormous, upgrades can take time, and node operators are not NPCs that exist to follow your happy path.

But the same scar tissue belongs in CLIs once they store meaningful local state.

Especially agent CLIs, because agents will happily automate the discovery of every edge case you forgot to test. Ask me how I know.

The pattern underneath the patterns

The concrete checklist is useful:

1. Version persisted and wire data.
2. Keep old structs immutable.
3. Upgrade one version at a time.
4. Commit golden fixtures.
5. Test old binaries against new releases.
6. Use deterministic encoding for fingerprints and signed data.
7. Make migrations crash-safe.
8. Fail clearly when versions are unsupported.

But the deeper pattern is ownership.

When you run the service, you can sometimes fix your mistakes centrally. You deploy a patch. You run a migration. You inspect the database. You roll back. You SSH into the box and do whatever morally questionable thing needs doing at 02:00.

When users run the software, you lose that privilege.

The data is not yours anymore.

The runtime is not yours anymore.

The version schedule is not yours anymore.

You can ship new code, but the world gets a vote on when it arrives.

That changes the design posture. Compatibility stops being a cleanup task and becomes part of the product. Schemas stop being internal implementation details and become public-ish contracts. Migrations stop being chores and become user safety mechanisms. Error messages stop being afterthoughts and become the difference between “upgrade rote” and “delete your home folder, lol.”

This is not blockchain advice

Well, it is blockchain advice.

But it is not only blockchain advice.

Blockchain just makes the lesson impossible to avoid. It punishes implicit versioning, ambiguous bytes, optimistic compatibility, and sloppy migrations with the kind of enthusiasm normally reserved for compiler errors in Rust.

The same lesson applies anywhere software escapes your control:

• CLIs with local config and caches;
• agents running on customer machines;
• plugin ecosystems;
• mobile apps with old versions in the wild;
• desktop apps with local project files;
• APIs consumed by generated clients;
• browser extensions;
• edge deployments;
• anything that signs, hashes, caches, persists, or syncs data.

The uncomfortable truth is that once your software leaves your deployment, your users become part of the distributed system.

They have old versions. They have old data. They have partial upgrades. They have broken environments. They have network failures. They have caches from before you were enlightened.

You can either design for that, or you can discover it through support tickets.

I recommend the first one. The second one builds character, but I have enough character now, thanks.

i Key concepts ›

Wire format

The shape of data exchanged between independently deployed programs. If one side can upgrade without the other, the wire format is a compatibility contract, not an internal detail.

Golden data test

A test that keeps real encoded examples in the repository and verifies that current code can still read them. Basically a tiny museum of promises you already made.

Anyway

The funny part is that none of this is advanced.

Version your data. Test your old data. Make upgrades explicit. Do not corrupt user state. Tell the user when their binary is too old.

Basic stuff.

And yet basic stuff becomes non-basic the moment your software is running on a machine you cannot touch.

That is the whole trick. Blockchain taught me this by being unusually hostile to disagreement. rote is teaching the same lesson in a calmer outfit: local state, remote manifests, adapter compatibility, auto-upgrades, and agents that will run the workflow whether or not the schema designer had a good breakfast.

So if you are building software that persists data, speaks a protocol, or gets copied into somebody else’s runtime, treat it like it will outlive your current assumptions.

Because if the product works, it will.

And future you deserves at least one boring migration path instead of another heroic debugging story.