The YAML parser that got out of hand

In 2018, I wrote a YAML parser in Bash. It was funny, useful, and—at the centre—a bad parser design.

Frank Vumbaca helped turn the first experiment into something people could actually use. He added blocks, lists, stdin support, documentation, and years of patient fixes. Then the repository mostly sat still. It kept doing its small job while I went on to other things.

I came back eight years later intending to do a full audit, close the old issues, replace Travis, rename the default branch, and ship a respectable release. Then I asked Codex a more dangerous question: if we were willing to break the CLI, how much better could this become?

A useful stunt

The original YAML.sh translated YAML into flattened paths and values. That was a sensible shape for shell scripts: easy to print, easy to grep, and just enough structure to query a configuration file.

It was also the ceiling. Once a mapping, sequence, alias, tag, or empty collection had been flattened, important relationships were gone. Every new feature had to reconstruct meaning the parser had already thrown away. Editing YAML without mangling it was almost impossible.

I asked whether there was a better parsing design I had missed. There was. The missing idea was not a smarter regular expression. It was an intermediate representation that respected YAML.

The answer was a node graph. Mappings remain mappings. Sequences remain ordered. Empty collections survive. Tags and source positions stay attached. Anchors and aliases can share identity. Queries and updates operate on nodes instead of guessing their way back from strings.

The new shape

YAML stream → node graph → expression stream → values / YAML

Permission to break it

That change broke the old CLI. I decided that was fine. Compatibility with a clever limitation was less valuable than making the premise useful.

We also stopped requiring Bash. The released program is one POSIX shell file with its AWK parser and evaluator embedded. The .sh name still fits: the shell owns the executable, arguments, files, and process behaviour; AWK does the work it is unusually good at. Calling it yaml.awk would describe an implementation detail while hiding the way people actually use it.

The goal became precise: yq-shaped configuration work with no runtime dependencies beyond /bin/sh and AWK. Not a miniature clone of every yq feature. Not a standards-compliance trophy. A useful tool for bootstrap scripts, tiny containers, old machines, and the mildly cursed recovery shell where installing the better tool is the problem you are trying to solve.

Once that premise was clear, breaking the interface was not reckless. It was the cost of making the product coherent.

“The point is to have fun.”

The tool took shape

Once the node graph existed, each capability made the next one possible. Traversal became an expression stream. Expressions became writable transformations. A source layer learned to preserve comments, whitespace, and style. File handling grew into preflighted multi-file transactions with rollback.

The releases trace that architecture becoming useful. Each one made a specific part of the premise real while keeping the runtime to one POSIX shell file and AWK.

v0.3Audit the old tool and make its limits explicit.
v1.0Replace flattened paths with a proper YAML node graph.
v1.2Turn queries into writable transformations.
v1.4Measure compatibility instead of describing it vaguely.
v1.6Add practical language depth without losing the one-file runtime.
v1.9Make multi-file edits explainable, rollback-safe, and transactional.
v1.10Make the repository the unit of work: one query, one process, one preflight.

Make it real

The hard part was not producing more code. The hard part was deciding what evidence would let us believe it.

We compared YAML.sh with the official YAML Test Suite and with yq itself. We added real configuration workflows from Kubernetes, Docker Compose, GitHub Actions, GitLab CI, and deployment overlays. We generated expressions, hostile inputs, large documents, deep collections, and presentation-preserving mutations. A supported behaviour needed a test; nearby unsupported behaviour needed an explicit error instead of a confident misparse.

That distinction matters. Matching 2,628 categorized yq programs does not mean YAML.sh contains all of yq. Passing 282 expected YAML Test Suite cases does not mean every corner of the YAML specification is implemented. The numbers describe the tested contract. They do not dissolve the boundary around it.

By v1.10, that contract was large enough to be useful and specific enough to be trusted. One command can check a repository without writing, bind data across files, prepare every mutation in one process, skip files that are already correct, roll back a broken commit, and leave a JSON audit trail containing paths but not values.

yq programs: 2,628 / 2,628; in the categorized release corpus
YAML outcomes: 282 / 282; expected YAML Test Suite cases
property coverage: 672 cells; layout × size × state × behavior
extra dependencies: 0; the runtime is POSIX sh + AWK

Building with Codex

Codex made the feedback loop unusually short. A product question could become an implementation, meet the YAML Test Suite and yq, expose the next missing idea, and return to the design while the context was still fresh.

That mattered because YAML.sh is a project made of boundaries: portability against convenience, fidelity against size, and ambition against a runtime people can still read. Moving quickly through code and evidence let us explore those boundaries without making the premise vague.

I came away less interested in how much code an agent can produce than in how much more thoroughly a small project can test its ideas. The constraint still shaped the tool. Codex made it possible to ask harder questions of it.

Where it fits

YAML.sh is not yq in disguise. yq remains the right answer when you want its complete language, codecs, ecosystem, and mature performance envelope. YAML.sh deliberately leaves out dynamic evaluation, file-loading operators, dates, many codecs, and other capabilities that would weaken its small and inspectable runtime.

It is better at a narrower thing: arriving as one readable text file in an environment that already has a shell and AWK. There is no package manager, language runtime, YAML library, or mystery binary. The security model can also be smaller because several powerful classes of behaviour simply do not exist. Its repository mode can check without writing, update several files from shared data, skip no-ops, roll back a partial commit, and explain changed paths without recording values.

That is enough of a difference to be a product, not just a stunt. Sometimes yq is exactly right. Sometimes you are writing the script that would install yq.

The constraint is the fun part

YAML in shell still sounds like a bad idea. I like that. Good constraints create a shape you would not reach by chasing generality, and a little absurdity can provide enough energy to stay with the boring work that makes an idea dependable.

The premise survived the rewrite: one file, no dependencies, real YAML work. Everything else was allowed to change.

The point was to have fun. The trick was refusing to use fun as an excuse to be sloppy.