How I Fed My Filesystem's Brain to a Ramdisk (and Got Everything Back)

I almost lost a terabyte of data last week, including a plaintext file with my crypto wallet seed phrases in it. The way I broke it was so dumb, and the way it got fixed was so interesting, that I figured I should write the whole thing down. Partly as a warning, partly because the recovery ended up being genuinely cool.

Short version: don’t ever btrfs device add anything that lives in RAM. Not even for a minute. Not even “just for a balance.”

The setup#

My desktop ran Linux on a 1TB NVMe drive, LUKS encrypted, btrfs on top with zstd compression and a bunch of subvolumes (home, a few distro roots, snapshots, the usual hoarder layout). The filesystem was old. Like, generation counter at 856,022 old. That number matters later.

One day it started throwing this at me:

No space left on device

Meanwhile df claimed I had tens of gigabytes free. If you’ve run btrfs long enough, you know this moment, and you know it’s technically not a bug. btrfs carves the disk into chunks (about 1GB each) and each chunk is dedicated to either data or metadata. My disk had free space inside the data chunks, but every last byte of the drive was already allocated to some chunk. So when metadata needed to grow, there was no unallocated space left to carve a new metadata chunk from. ENOSPC, with “free” space visible. Great UX, I know.

The textbook fix is btrfs balance, which compacts half-empty chunks and gives space back to the unallocated pool. Except balance needs some scratch space to work with, and I had literally none (I checked later: about 65MB unallocated on a 1TB drive). So balance also said ENOSPC.

The mistake#

The internet has a trick for this situation. I’d seen it in forums a dozen times. Make a temporary block device in RAM, add it to the filesystem, run the balance with all that lovely extra room, then remove it. Easy.

So I did this:

truncate -s 3G /dev/shm/scratch
losetup /dev/loop9 /dev/shm/scratch
btrfs device add /dev/loop9 /mnt
btrfs balance start -dusage=10 /mnt
btrfs device remove /dev/loop9 /mnt   # spoiler: this failed

A 3GB file on tmpfs, wrapped in a loop device. Looks harmless. btrfs filesystem show just lists a friendly /dev/loop9. Nothing about the name screams “this is literally RAM and will cease to exist.”

Here’s what I didn’t appreciate at the time: btrfs device add doesn’t create a scratch area. It makes that device a full, first-class member of the filesystem, and the allocator is free to put anything it wants on it. Including the most important 32MB of the entire disk.

And here’s the part that turns this from “risky” into “a trap.” I did try to remove it. The remove failed with, I kid you not:

No space left on device

Think about what device remove has to do: relocate every chunk off the departing device onto the remaining ones. Relocation allocates new chunks. New chunks need unallocated space. My disk had none. That’s the whole reason I was in this mess. So the same condition that made me add the ramdisk made it mathematically impossible to remove the ramdisk. The door locked behind me the moment I walked in, and I only found out when I turned around.

At that point my filesystem had a piece of RAM as a permanent, load-bearing structural member. The correct move was to stop, think, delete some data to free up space, and coax the remove through. What I actually did, after staring at the ENOSPC from the remove command and feeling the walls close in, was the most human thing possible.

I rebooted.

I don’t have a good justification. Some lizard-brain part of me still believed the universal fix applies universally. Reboots solve problems. That’s the deal we have with computers. So I rebooted a machine whose filesystem was partially stored in RAM, and the RAM did what RAM does.

What I didn’t know yet was how bad the situation already was before I touched the power. The balance had started relocating chunks, and one of the first things it moved was the SYSTEM chunk. And because the SYSTEM chunk uses the DUP profile (two copies for safety), it moved both copies. Both to the ramdisk. DUP means two copies, not two copies on different devices. On a filesystem with a nearly full disk and a shiny empty ramdisk, guess where the allocator put them.

So, to recap the sequence: the balance had already moved the SYSTEM chunk (both copies) onto the ramdisk, the remove was mathematically impossible, and then I voluntarily power-cycled the only copy of my filesystem’s address map out of existence. Speedrun complete.

What actually died#

To understand the damage, you need one piece of btrfs internals. Everything in btrfs is addressed by “logical” offsets in a big flat address space. The thing that translates logical addresses to physical locations on disk is the chunk tree. The chunk tree itself lives in SYSTEM chunks. The superblock carries a little bootstrap array (sys_chunk_array) with just enough info to find the SYSTEM chunks, read the chunk tree, and from there find everything else.

My chunk tree was on the ramdisk. Both copies. So the mount failed like this:

BTRFS error: failed to read chunk root
bad tree block 1898188046336, bytenr mismatch, want=1898188046336, have=0

have=0 means the address resolves to nothing and reads back zeros. Every byte of my actual data was still sitting on the NVMe, perfectly intact. But the map was gone. It’s a library where all the books are fine, but someone burned the only catalog and none of the shelves are labeled.

I checked the backup roots in the superblock. All four generations of them pointed into the same dead 32MB logical range. The SYSTEM chunk hadn’t moved in ages, so every backup pointed at the same crater.

I’ll be honest about the emotional arc here. The data I was worried about wasn’t the distro roots or the git repos. It was a file literally named notes, containing seed phrases and private keys, sitting in my home subvolume. Yes, plaintext. Yes, I know. We’ll get to the lessons section.

The flailing phase (do not copy this part)#

Here’s where I’m supposed to say “the first thing I did was image the disk.” It was not. The first thing I did was spend a good while throwing every remedy the internet offered directly at the real disk, because that’s what panic does to a person.

Every rescue mount option combination. ro, rescue=usebackuproot, rescue=nologreplay, degraded, all of them, in various stacking orders like I was trying cheat codes. They all bootstrap through the chunk tree, so every attempt died on the same missing 32MB. Dead end, but at least a read-only one.

btrfs rescue subcommands. zero-log, super-recover, that family. Same wall. Worth pausing on how reckless this was: some of these write to the disk. I was gambling with the only copy of my data and didn’t even register it at the time, because tools with “rescue” in the name feel safe the way /dev/loop9 feels safe. I got lucky that none of them made anything worse.

btrfs rescue chunk-recover. The official tool for exactly this disaster: it scans the whole device, finds surviving chunk tree fragments, rebuilds the tree. It crashed with an assertion failure, BUG_ON(!ce). Turns out that’s btrfs-progs issue #393, open since 2021: chunk-recover blows up when the filesystem references a missing device. Which is more or less the main reason you’d need it. The tool built for the job crashes on the most likely version of the job.

PhotoRec. With everything else failing, I dropped to the floor: signature carving. The output was about as satisfying as you’d expect. Anonymous numbered fragments, no filenames, no directory structure, and since the filesystem used zstd compression, most files came out as garbage or didn’t carve at all. Sifting through f1284736.txt-style debris looking for a seed phrase is an experience I don’t recommend.

Somewhere in that despair I remembered an external HDD lying in a drawer. You know how it is with spinning rust: too slow to use for anything, too functional to throw away. Suddenly it was the most valuable object in the house. I finally did what should have been step zero and imaged the entire decrypted device onto it with testdisk.

From that moment on, every experiment ran against the image or against copy-on-write overlays of it, never against the original. The original NVMe became the backup. That single decision is what made everything that follows stress-free: any mistake was a deleted overlay file, not a deeper hole.

The two facts that saved everything#

At this point I roped Claude into the problem and we went digging into btrfs internals. Two properties of the filesystem turned “hopeless” into “recoverable, actually”:

Fact one: btrfs metadata blocks are self-describing. Every 16KB metadata block starts with a header containing the filesystem UUID, a checksum, the generation, which tree owns it, and, crucially, its own logical address. Sit with that for a second. If I find a metadata block at physical offset P and its header says “I am logical address L”, then I’ve just learned one entry of the destroyed map: this chunk is offset by P minus L. Blocks in the same chunk all share that delta. Find enough blocks and you’ve rebuilt the map for every metadata chunk, no chunk tree required.

Fact two: the device tree survived. btrfs keeps a second, inverse mapping. The dev tree is full of DEV_EXTENT items that say “this physical range on this device belongs to the chunk at this logical address.” It covers every chunk, including data chunks (whose contents have no headers and can’t self-identify). And the dev tree lives in normal METADATA chunks, which were never on the ramdisk. Mine was at generation 856,022. That’s the filesystem’s final transaction. The inverse map was not just intact, it was current to the last commit before the crash.

So the plan built itself:

1. Scan the disk for metadata block headers, derive the metadata chunk map from the deltas.
2. Use that partial map to read the dev tree.
3. The dev tree gives the complete map, data chunks included.
4. Teach the tools to use a map from a file instead of the dead chunk tree.

Making the scan not take ten hours#

A naive scan reads all 953GB. Over USB, that’s most of a day. But metadata chunks aren’t sprinkled around as confetti. They’re big contiguous regions, 256MB to 1GB each (the SYSTEM chunk is 32MB). So you don’t need to read everything to find them. You can probe: read 64KB every 16MB and check for blocks with the right filesystem UUID. A 16MB sampling grid physically cannot miss a region that’s 32MB or larger.

That turned the “find the metadata” phase into about 40 minutes. Then a dense scan of just the flagged regions (about 39GB) with CRC verification on every candidate block, another half hour, and I had 2,497,843 verified metadata blocks in a CSV: physical offset, claimed logical address, owner tree, generation.

One gotcha bit us during map derivation. Because btrfs is copy-on-write, old balances leave stale copies of relocated blocks lying around, and they have valid checksums. Naive clustering sees the same logical range at two physical locations and thinks “oh, a DUP mirror” when one side is actually a fossil from three years ago. The fix: when ranges conflict, trust the copy whose blocks carry newer generations. Real DUP mirrors have identical content so the ordering doesn’t matter for them, and fossils get demoted.

The patch#

Then we patched btrfs-progs so that every tool can bootstrap from a map file instead of the on-disk chunk tree:

btrfs restore --chunk-map map.txt -l /dev/mapper/thing
btrfs inspect-internal dump-tree --chunk-map map.txt -t device /dev/mapper/thing

The map file is embarrassingly simple. One chunk per line: logical address, length, type flags, then devid and physical offset pairs. On open, the mappings get inserted straight into the in-memory lookup structures, chunk root reading is skipped entirely, and everything downstream just works. No writes to the patient needed.

With the scan-derived map injected, dump-tree -t device printed the entire dev tree on the first try. I actually laughed. Its 964 dev extent records became the final map: 940 chunks. And that dump contained the single best piece of news of the whole ordeal:

Exactly one chunk had all of its stripes on the dead ramdisk: the 32MB SYSTEM chunk itself. Zero data chunks lost. Zero metadata chunks lost.

The balance had died before moving anything else. Total real loss: the catalog, and only the catalog. Which we had just finished rebuilding from the shelves.

Getting the files (and the seed phrases) out#

From there btrfs restore with the injected map could pull files without ever mounting the filesystem. First priority was obvious. We hunted down every file named notes and extracted them before anything else. It turned up in three places (my home directory plus two of my own backup copies I’d forgotten about), all three byte-identical by md5, all three decompressed cleanly from zstd. Three independently stored copies agreeing bit for bit is about as good an integrity signal as you can ask for. I could breathe again.

Assorted practical notes from the extraction phase:

• The --path-regex syntax of restore is a puzzle in itself. The regex gets matched against every parent directory during descent, so “give me /a/b/c” has to be written as ^/(|a(|/b(|/c)))$. Nested empty alternations. Beautiful. Horrible.
• Tree-walking over a USB hard drive is agony (random 16KB reads, effectively). The fix: copy the ~39GB of known metadata regions into a sparse file on the laptop’s SSD at identical offsets, then stitch a device-mapper linear table that serves metadata regions from the SSD copy and data regions from the USB image. Listing all 13.3 million files across my subvolumes took minutes instead of hours after that.
• 10.4 million of those files were node_modules, caches, and package manager junk, so restore also grew an --exclude option. My actual home subvolume came out complete: about 119GB, every file, verified.

Fixing the filesystem for real#

Extracting files is recovery. But the elegant ending is making the original filesystem mountable again, and since the only destroyed object was the chunk tree, and the surviving trees describe everything perfectly, that was actually possible: write a brand new chunk tree.

The catch is consistency. At mount, the kernel cross-checks the chunk tree against the dev tree, the block group items against the chunks, the stripe counts, the superblock bootstrap array, the device items, the byte totals. Change one structure and you must change all of them, in agreement, or the mount is refused. Hand-editing that with a hex editor is technically possible and spiritually devastating.

Instead this became a new btrfs-progs subcommand, btrfs rescue inject-chunk-tree, built on top of the map injection (which conveniently lets progs open the broken filesystem read-write). What it does:

1. Reads the truth from the surviving trees: stripe placement from dev extents, chunk type flags from block group items. No external inputs beyond the map used to open the fs.
2. Allocates a fresh SYSTEM chunk in unallocated space. In my case the drive being 100% allocated (remember, the original sin) left exactly one 64MB gap. Precisely enough for a DUP pair of 32MB stripes. The margin on this recovery was zero.
3. Writes a new chunk tree there through progs’ normal transaction machinery, so extent bookkeeping, block group accounting and free space info all update themselves.
4. Erases the ghost: deletes the dead device’s dev extents and stats, the dead SYSTEM block group, and the stale extent records of the old chunk tree blocks. Sets num_devices back to 1. Fixes the byte totals.
5. Rebuilds sys_chunk_array and invalidates the free space tree so the kernel regenerates it on the next writable mount.
6. Commits. One transaction. The filesystem comes out the other side as a clean single-device btrfs, as if the ramdisk had never existed.

We got it wrong twice before getting it right, which is exactly why every attempt ran against a dm-snapshot overlay of the image instead of anything real. First failure: the modern tree checker refuses to let you insert into an empty chunk tree root, because a chunk tree “must never be empty.” Chicken, meet egg. (Bypassable per-path, once you find the flag.) Second failure was a proper C classic: struct btrfs_chunk embeds its first stripe as the last struct member, my code appended stripes after the full struct, and stripe zero went out as stack garbage. The fun part: metadata still read fine afterwards, because DUP’s second stripe was correct and the mirror fallback silently saved every read. The failure mode that killed my filesystem was also quietly papering over my bug. btrfs giveth.

Third attempt, on the overlay: btrfs check passed, and then the actual Linux kernel mounted the thing. Read-only first, then read-write. All subvolumes present, no warnings, and it read my notes file through the normal VFS path with data checksums verified, matching the extracted copies exactly.

Then I ran the same command on the real desktop drive. It mounts. It just… mounts, like nothing ever happened.

What I’m taking away from this#

1. Never add a volatile device to btrfs. Not a file on /dev/shm, not zram, not brd. The allocator will put the only copy of your most critical metadata there without a second thought, because you told it the device was trustworthy. If you’re stuck at ENOSPC: try btrfs balance start -dusage=0 then -dusage=5 first (they need almost no scratch space). If you truly need a temporary device, use a loop file on another physical disk.
2. Check that the exit exists before you walk in. device remove needs enough free space on the remaining devices to relocate everything off the departing one. On a full disk (the only situation where you’d reach for this trick) that space may not exist, and then the “temporary” device is not temporary anymore. And when you find yourself trapped like that, the one thing you must not do is the reflexive thing. A reboot is not a fix when part of your filesystem is made of RAM. Ask me how I know.
3. DUP is not redundancy against device loss. Two copies on one device is zero copies when that device goes away.
4. Image first, then experiment. I did this backwards: I spent the panic phase running rescue commands (some of which write!) against the only copy of my data, and only imaged the disk after PhotoRec had thoroughly demoralized me. Nothing I ran happened to make things worse, but that was luck, not judgment. Once the image existed, every step of the real recovery was reversible, including two buggy chunk tree rebuilds that would have been catastrophic on the original. The slow dusty HDD in your drawer is worth more than every rescue command combined. Image first.
5. Read the superblock before despairing. btrfs inspect-internal dump-super -fFa works with no chunk tree at all, and it told us the whole story in ten minutes: the second device, its suspiciously RAM-sized 3GB, both SYSTEM stripes on it, every backup root dead.
6. btrfs is remarkably recoverable if you go low enough. Self-describing metadata blocks and a surviving inverse mapping meant the “unrecoverable” chunk tree was rebuildable from first principles. I have a hard time imagining pulling this off on most other filesystems.
7. Seed phrases do not belong in a file called notes. I got outrageously lucky, three times over, thanks to backup copies I didn’t even remember making. Hardware wallet, steel plate, encrypted vault. I’m fixing this before I do literally anything else.

The tooling from this whole adventure (the scanner, the map derivation scripts, the --chunk-map injection patches, the inject-chunk-tree subcommand, docs and all) lives on the recovery-devel branch of my btrfs-progs fork, rebased on upstream devel, in case it ever saves someone else’s terabyte. Start with recovery-tools/README.md, which walks the whole workflow. The official tool for this failure mode has been crashing on it since 2021. Now there’s at least one path through that doesn’t.

Total data lost: 32MB, containing nothing but the map we rebuilt. Total data recovered: everything else. Lessons: several, learned the way that actually sticks.