Patterns From Five Years in Blockchain

After five years in blockchain, the projects blur less by token, chain, or product category than by the boundary they forced me to reason about.

The NFT game, the rental protocol, and Hypersea were application-level systems on EVM chains. Axelar-Solana was a cross-chain port where the security model stayed recognizable, but the transaction shape had to change completely. Irys was a protocol client problem: storage, consensus, execution, pricing, and data movement all had to agree under fork choice.

The visible work changed from NFT reveals to Solana packets to Programmable Data. The recurring question did not: which system is responsible for this fact, and how does every other system learn it without inventing a slightly different version?

What proficiency actually means

Being good at blockchain engineering is not a separate species of engineering. It starts with ordinary backend competence: designing APIs people can use, maintaining services that survive production traffic, writing SDKs that hide the right complexity, and building systems that other engineers can integrate without becoming protocol experts by accident.

Then the constraint stack gets taller.

You need to be comfortable in complex build systems where the same codebase may target a server, a browser, a smart-contract VM, a zkVM guest, or an embedded runtime with different assumptions. You need enough distributed-systems taste to reason about propagation, retries, fork choice, mempools, availability, and partial failure. You need enough networking intuition to know when bytes, latency, and peer behavior stop being performance details and start affecting consensus health.

You also need to understand the machines your code runs on. Sometimes that means EVM opcodes, gas accounting, ABI encoding, and precompile behavior. Sometimes it means Solana account layouts, eBPF constraints, and CPI rules. Sometimes it means proving execution inside a RISC-V zkVM and accepting that the guest will not give you the same observability as the host.

Cryptography matters, but usually in the applied engineering sense: signatures, hash functions, Merkle commitments, proof verification, domain separation, and the exact bytes being signed or proven. Financial incentives matter too. A protocol can be technically valid and still fragile if fees, rewards, MEV, validator behavior, or user incentives push actors toward the wrong equilibrium.

That combination is what makes the work hard. Blockchain engineering is backend engineering under adversarial replication, scarce execution budgets, awkward VMs, cryptographic boundaries, financial incentives, and unusually expensive disagreement.

Bytes are protocol

I keep coming back to bytes because the systems kept punishing every attempt to treat encoding as an implementation detail.

In the first post, byte-level agreement showed up in small but sharp ways. EIP-712 signatures only verify if the wallet, Rust signer, and Solidity contract reconstruct the same typed-data hash. Merkle proofs only work if the leaf encoding, ordering, hashing, and proof path match exactly. NFT metadata hashes only mean anything if the bytes pinned to IPFS are the bytes the token points at.

In the Axelar-Solana work, bytes became the architecture. Twenty-seven 65-byte secp256k1 signatures were already 1,755 bytes. Solana’s transaction ceiling was 1,232 bytes for the entire serialized transaction. The proof was not “large” in a vague performance sense. It was impossible to submit in the reference shape. That changed the Gateway from a monolithic verifier into a Merkle-streaming state machine.

In Irys, bytes became time. Gossiping a block is not free. Fetching Programmable Data chunks is not free. Moving extra proof data across peers affects validation latency, and validation latency affects fork choice. A few bytes shaved from a hot message can become less propagation delay across every hop.

That does not mean every protocol should optimize every struct until nobody can read it. The point is to know when bytes are part of the consensus or trust boundary. If they affect a signature digest, a Merkle root, a transaction ceiling, a gas cost, a compute budget, a gossip message, or a proof size, they are not merely plumbing.

They are part of the protocol surface.

Off-chain is still real

Blockchains often describe themselves by what is trustless. Production applications are usually defined by the parts that are not.

Indexers, relayers, signers, pinning services, RPC nodes, price oracles, generated clients, block explorers, wallets, monitoring, and test harnesses are not side characters. They decide what users can see, submit, recover from, and debug. The protocol may remain correct without them, but the product often does not remain usable.

The important distinction is not “off-chain bad.” The important distinction is whether the off-chain system can change truth or only carry it.

This framing helped more than the usual purity arguments. A relayer that cannot forge approvals is still operationally load-bearing. An indexer that cannot change contract state can still mislead every user interface. A generated client that has no consensus role can still determine whether anyone can integrate the program without hand-assembling account metas.

The work is to make those roles explicit. What can this service lie about? What can it delay? What can it make unavailable? What does the chain verify anyway? What do users see if it falls behind?

Those questions are more useful than asking whether the system is decentralized in the abstract.

Side quest: SVM inside RISC-0

The stack looked strange when said out loud: a Solana VM running inside a RISC-0 VM running on a normal host machine.

The host was the laptop or server process orchestrating the proof. RISC-0 was the guest environment: a RISC-V machine whose execution could be proven. Inside that guest, we needed enough of the Solana runtime to execute real Solana programs. So the work was not only “compile some Rust to RISC-V.” It was adapting the SVM’s memory model, syscalls, account handling, and program loading so that Solana programs still saw the world they expected.

That extra VM boundary came with a tax. The host can use normal logs, debuggers, IDE support, and operating-system assumptions. The RISC-0 guest has a narrower execution environment, proof-size pressure, guest memory constraints, and cross-target Rust compilation issues. The SVM adds another ABI on top: eBPF programs, Solana syscalls, account layouts, and program-derived assumptions that existing programs already rely on.

Each layer made observability worse. A value that is obvious in the host may need to be serialized into the guest. A failure inside the SVM may surface as a guest execution failure, not as a friendly Solana runtime error. IDEs understand the host target best, understand the RISC-V guest less well, and do not magically understand “Solana eBPF semantics inside a zkVM guest.” Every extra VM makes the system more verifiable in one sense and harder to inspect in another.

The compatibility test was whether real programs worked without redefining them. SPL token transfers, NFT mints, and program creation all had to run under the same rules Solana programs expect. Not toy programs. The proof only mattered if the thing being proven was still meaningfully Solana.

That is what compatibility really means. The promise is not that your system looks familiar in a diagram. The promise is that existing tools, contracts, wallets, libraries, and mental models continue to produce correct results at the boundary where they touch you.

Tooling is part of the workload

The tooling is worse than mainstream backend development.

That is not a complaint so much as an input to planning.

In normal backend work, you usually have mature SDKs, debuggers, observability defaults, stable protocol docs, libraries that agree with their own examples, and a long trail of production incidents someone else already wrote about. Blockchain work often has some of that, but not evenly and not at the layer where you are stuck.

The first post had Rust crates that did not yet cover EVM-adjacent encoding well enough, so pieces like EIP-712 encoding, Solidity-style packed encoding, and Merkle proof generation had to be written or checked directly. The Axelar-Solana work had an optimization detour through rkyv because we initially treated the problem as serialization overhead before the final requirements made it clear that the proof shape itself had to change.

Irys needed a multi-node test harness because the wrong-fork bug was not visible from unit tests or logs alone. The SVM-inside-RISC-0 work had the same flavor: every nested execution environment made ordinary debugging less ordinary. Cross-target Rust builds, guest/host serialization, proof-time failures, and runtime errors all had to be treated as first-class work, not incidental annoyance.

There were smaller examples too. I worked on IBC relaying for a large enterprise’s permissioned chain when the spec, implementation, and GitHub issues still felt like three partially overlapping sources of truth. I also touched viem and wagmi integrations on the EVM side, where the protocol may be correct but the developer experience still depends on typed clients, hooks, generated ABIs, and libraries making the right thing easy.

The lesson was not “avoid young ecosystems.” Young ecosystems are where the interesting work often is. The lesson was to budget for tool-building as part of protocol work. If your test harness cannot create the failure, the failure may keep looking mystical. If your generated clients are missing, every integrator becomes a protocol engineer by accident. If your docs do not say where the boundary is, somebody will guess, and their guess will become production traffic.

Tooling is not separate from engineering velocity. In protocol work, it is often the only way to make the state space small enough to reason about.

Five years later

Crypto has a bad reputation among software engineers. Scams, extraction, pointless tokens, Discord theatrics, and financial nihilism are not imaginary criticisms. I was around for NFT summer. There is no honest version of this retrospective where that context disappears.

But the infrastructure side remains the hardest engineering I have worked on.

Consensus algorithms, adversarial networks, cryptographic commitments, custom runtimes, storage economics, proof systems, fee markets, VM compatibility, propagation speed, fork choice, and operational liveness all compose with each other. A change in one layer often becomes a rule another layer must validate.

I am not a researcher. I am an engineer who likes building systems where the constraints are real enough to push back.

Five years ago I went looking for hard problems. I found them. The pattern I kept seeing was not that blockchains remove trust, infrastructure, or complexity. They move those things into sharper boundaries.

The good work is making those boundaries explicit enough that the system can survive them.