The Curse of Soldered RAM

Oct 15 23:33:35 razor-crest kernel: BTRFS warning (device dm-0): csum failed root 257 ino 166754 off 5859786752 csum 0x31019647 expected csum 0x6d009929 mirror 1
Oct 15 23:33:35 razor-crest kernel: BTRFS error (device dm-0): bdev /dev/mapper/luks-571c7357-a343-49f3-993c-167c8c5dcec6 errs: wr 0, rd 0, flush 0, corrupt 3856936704, gen 0

These failure made me wonder if there were RAM errors, so I added a memtest86+ boot image to my Ventoy USB stick and booted from it.

Memtest86+

This tool boots an absolute minimal OS that then proceeds to write a bunch of patterns to every physical memory address possible and reads them back. If the value read back doesn’t match what was written then there is a problem.

If left to run it will do multiple passes, because some errors only show up intermittently. But in this case it showed a bunch of errors almost immediately.

In the past this would have been an easy fix, the output would have been enough to narrow down which of the DIMMs was faulty, or at worst you would pull one out and run the test again to do a binary search. Once identified you could just order a replacement DIMM (or take the opportunity to upgrade) and move forward.

But in the search for even thinner laptops (because a DIMM socket adds ~6mm of depth) and maybe cynically ensure that any failures result in needing to buy a whole new laptop, RAM is now soldered directly to the motherboard of most laptops (things like the FrameWork are the exception) so unless you are skilled at replacing ball grid chips it’s not really an option to replace.

Depreciation over 5 years and advances means it is actually probably time to find a new laptop, but I’d like to see if there is any way to keep the Lenovo X1 Carbon running as it’s still a very capable machine and could replace the even older Dell XPS 13 I use as my personal laptop.

I’ll write about the replacement later.

Memmap

Whlie I wait for new laptop to arrive, I thought I’d see if I could make it more stable.

A little research turned up 2 options that could be passed to Linux kernel to have it avoid using the bad sectors

• badram
• memmap

badram turns out to be an out of tree patch that is only usually shipped in Debian/Ubuntu based distros. I run Fedora on this machine so that wasn’t going to work.

memmap is a core kernel feature, so should be available on Fedora. You need to supply a list of memory addresses and lengths in the format length$offset on the kernel command line.

Memtest86+ can output the correct format (for both badram and memmap) and even save it to a file, but I couldn’t get it work with the iso image I’d put on the Ventoy USB stick, so I had to resort to taking a photo of the screen.

I then had to manually transcribe this, which as “fun”.

All the documentation for adding kernel command line arguments say to edit the /etc/default/grub file and add the values to the GRUB_CMDLINE_LINUX entry and then run grub2-mkconfig -o /boot/grub2/grub.cfg

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-571c7357-a343-49f3-993c-167c8c5dcec6 rhgb quiet memmap=\$memmap_val memmap=0x30$0x3fe90140,0x48$0x3fe90b08,0x48$0x3fe90ba8,0x48$0x3fe916a8,0xf0$0x3fe91c00,0x90$0x3fe92560,0xf0$0x3fe92f00,0x50$0x3fe93200,0x90$0x3fe93260,0xf0$0x3fe93800,0xf0$0x3fed4300,0x70$0x3fed4900,0x70$0x3fed4980,0xf0$0x3fed5400,0xf0$0x3fed5e00,0xf0$0x3fed6700,0xf0$0x3fed6d00,0xf0$0x3fed7000,0x30$0x3fed7a00,0xb0$0x3fed7a40"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

I tried this but when I restarted the machine it failed to boot.

I managed to get to grub screen and show the kernel command line it was trying to boot and it looked like the $0 in the memmap entry had been replaced by grub2-mkconfig. This smacks of classic environment variable substitution ($0 is usually the name of the script that is being run).

I found this RedHat bug describing a similar problem and it had a solution to add the value as variable.

# grub2-editenv /boot/grub2/grubenv set memmap_val='0x30\$0x3fe90140,0x48\$0x3fe90b08,0x48\$0x3fe90ba8,0x48\$0x3fe916a8,0xf0\$0x3fe91c00,0x90\$0x3fe92560,0xf0\$0x3fe92f00,0x50\$0x3fe93200,0x90\$0x3fe93260,0xf0\$0x3fe93800,0xf0\$0x3fed4300,0x70\$0x3fed4900,0x70\$0x3fed4980,0xf0\$0x3fed5400,0xf0\$0x3fed5e00,0xf0\$0x3fed6700,0xf0\$0x3fed6d00,0xf0\$0x3fed7000,0x30\$0x3fed7a00,0xb0\$0x3fed7a40'

Change the /etc/default/grub to

GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
GRUB_DISABLE_SUBMENU=true
GRUB_TERMINAL_OUTPUT="console"
GRUB_CMDLINE_LINUX="rd.luks.uuid=luks-571c7357-a343-49f3-993c-167c8c5dcec6 rhgb quiet memmap=\$memmap_val"
GRUB_DISABLE_RECOVERY="true"
GRUB_ENABLE_BLSCFG=true

And run grub2-mkconfig -o /boot/grub2/grub.cfg again.

This time things booted properly and the correct values can be seen in /proc/cmdline

BOOT_IMAGE=(hd0,gpt2)/vmlinuz-7.2.11-200.fc44.x86_64 root=UUID=869b183b-b858-40e5-8da7-e923dd367c2a ro rootflags=subvol=root rd.luks.uuid=luks-571c7357-a343-49f3-993c-167c8c5dcec6 rhgb quiet memmap=0x30$0x3fe90140,0x48$0x3fe90b08,0x48$0x3fe90ba8,0x48$0x3fe916a8,0xf0$0x3fe91c00,0x90$0x3fe92560,0xf0$0x3fe92f00,0x50$0x3fe93200,0x90$0x3fe93260,0xf0$0x3fe93800,0xf0$0x3fed4300,0x70$0x3fed4900,0x70$0x3fed4980,0xf0$0x3fed5400,0xf0$0x3fed5e00,0xf0$0x3fed6700,0xf0$0x3fed6d00,0xf0$0x3fed7000,0x30$0x3fed7a00,0xb0$0x3fed7a40

I’ve created a text file on a USB stick with the memmap values so I can re-apply it if I end up doing a full reinstall. This should keep things running until the new laptop arrives and hopefully for a bit longer.

Next task is to make sure I’ve backed everything I need up and then run btrfs check --fix to see if I can sort out the meta data in the file system.