All posts by roderick

Verify Error After Writing Linux Image onto SD Card

software, , , , , , , , , , , , ,

First of all, Hakuna Matata. The summary is that a mismatch on readback after burning a Linux image to an SD card is not only generally harmless – it’s expected. If that’s all you need, feel free to skip the rest of this article. Root cause here. What follows details how I arrived at that conclusion.

I recently got a Le Potato arm-based computer, something very similar to a Raspberry Pi. Libre Computer’s website had images available for burning the operating system onto the micro SD card.

Here was my process for writing the image onto the SD card in Linux. Before inserting the SD card into the computer, look for any other disks:

$ ls /dev/sd*
/dev/sda  /dev/sda1
$

Now plug in the SD card, and look again to see which devices appeared. Unmount any file systems that might be mounted, which in this case, are /dev/sdb1 and /dev/sdb2. Note that we do not unmount the root device, /dev/sdb. Type

$ ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sdb  /dev/sdb1  /dev/sdb2
$ sudo umount /dev/sdb1 /dev/sdb2
umount: /dev/sdb1: not mounted.
$ ls /dev/sd*
/dev/sda  /dev/sda1  /dev/sdb  /dev/sdb1  /dev/sdb2
$

Turns out that /dev/sdb1 wasn’t mounted, anyway. We will now write the raw image onto the SD card, which is the /dev/sdb device:

$ sudo dd if=raspbian.img bs=1M of=/dev/sdb status=progress
2388656128 bytes (2.4 GB, 2.2 GiB) copied, 55 s, 43.4 MB/s
2288+0 records in
2288+0 records out
2399141888 bytes (2.4 GB, 2.2 GiB) copied, 202.214 s, 11.9 MB/s
$

The bs=1M parameter above is not strictly necessary, but may help performance a bit. Neither is status=progress necessary, but I like to have an indication of how things are going. A return to the prompt indicates success. But is is success, really? How do we know that the image hasn’t had any errors in writing? I attempt to read it back, as follows:

$ sudo dd if=/dev/sdb bs=1M count=2288 of=readback.img status=progress
2380267520 bytes (2.4 GB, 2.2 GiB) copied, 106 s, 22.4 MB/s
2288+0 records in
2288+0 records out
2399141888 bytes (2.4 GB, 2.2 GiB) copied, 107.044 s, 22.4 MB/s
$

And now compare the readback to the original. The word count shows that differences.log is empty – meaning, no differences.

$ cmp -l raspbian.img readback.img > differences.log
$ wc differences.log 
0 0 0 differences.log
$

But then, I got too smart for myself, and too dumb for myself, at the same time. (Forgive me, I come from the early era of burning CD’s, when it was notoriously unreliable…) I wondered if that readback had just come from cache, not the actual SD card. I unplugged and re-plugged the SD card, unmounted the file systems, and read back the image again, and compared again:

$ sudo umount /dev/sdb1 /dev/sdb2
$ sudo dd if=/dev/sdb bs=1M count=2288 of=readbackagain.log status=progress
2380267520 bytes (2.4 GB, 2.2 GiB) copied, 106 s, 22.4 MB/s
2288+0 records in
2288+0 records out
2399141888 bytes (2.4 GB, 2.2 GiB) copied, 107.064 s, 22.4 MB/s
$ cmp -l raspbian.img readbackagain.log > differences2.log
$ wc differences2.log 
  751091  2253273 14270729 differences2.log
$

Whoa! 751,091 lines in the file, meaning that many different bytes. I noticed that the miscompares were clustered in addresses around 272M, 339M, 606M, 963M, and 1.93G . Bad reader? I repeated the experiment with different readers:

I tried various readers, 3 of which were USB 2.0, and one USB 3.0. Three was also the embedded reader in the computer. 3 of the readers were rated for SD-HC, while one was rated for SD-XC. There were differences in writing and reading speed, but the failure was always around the same addresses.
The troubleshooting guide on Libre Computer’s site suggested that there might be bad cards out there.
I tried various other cards, one original SD, three SD-HC, and two SD-XC. The error was the same on all of these, even the brand new SanDisk Extreme, purchased directly from Amazon, Shipped by Amazon. The failures were always at the same addresses. If these were hardware errors on the cards, surely, they couldn’t all have bad cells at exactly the same regions! Or could they? The one exception was the 2 GB card. It was too small to hold the entire image, so I burned only what would fit, and read that back. Perfect match. Explanation at the end why that was.

I found a Linux program called badblocks, and used it to test each card. They were all good.

I tried other burning software such as Raspberry Pi Imager, and Win32 Disk Imager (under Windows 10). There were still mismatches in Windows, although fewer of them than with Linux. Below is the output of Win32 Disk Imager:

The write operation reports success.
However, a subsequent Verify Only fails.

As a further test, I tried writing something to the SD card that was NOT a filesystem image. I just took a large ZIP file, wrote it, unplugged and replugged the SD card, then read back the contents. Perfect match! Only then did it dawn on me what was going on.

Root Cause

The operating system on my computer was mounting the file system on the newly-written SD card, and in doing so, updated various tables, such as last access time. Windows does this automatically when a new mass storage device appears, and so did Linux when I unplugged and reseated the SD card before reading. It’s normal operation, and expected.

Answers to Questions

Why did the readback work when the SD card was not unplugged and replugged? Because Linux did not mount the newly-written file systems.

Why did Windows have less miscompares? Because two file systems were burned onto the card: an EFI partition, which was FAT32, and an EXT4 linux partition. Windows couldn’t read the Linux partition, so only put its signature and stuff onto the EFI partition.

Why did the 2GB SD card work? Because it was too small to hold the entire 2.4 GB image, I was only able to write a partial image. Linux saw that as an invalid file system, and never tried to write anything to it.