Hint needed: EMMC iommu and /bootefi etc

Hi @elly and all the others at chrultrabook. I met you and your team at 37c3 :wink: It was a pleasure. (@steeler@bildung.social)

I bought an Picasso/Dali ThinkPad C13 Yoga Chromebook (Morphius) right after your talk and I´m now somehow stuck.

I wonder where I can find information how to make it boot properly under linux. There are two Linux notes in the support devices chart but there are no recommended tutorials how to do the recommended step ? You also mentioned in antother thread that a team is working on sorting out the boot issues. Can you tell me where to find more information on this topic ?

—snip
Needs to add “iommu=pt” to cmdline

eMMC models need to put /boot/efi and /boot on USB
—snap

I can boot a fedora 39 from USB Stick right now but the OS Installer does not see the HD / ssd controller.

So how can I add iommu=PT or pit /boot/efi and /boot on usb ? Could you please hint to a tutorial how to do this?

Hi!

It is a bit strange, because eMMC detection issue only affects EDK2, and iommu is needed in case of framebuffer corruption.

Could you post an output from lsblk from TTY?

Wow… Now it´s getting a bit strange.

The output doesnt show me any drive.

So I was curious and opened the shell again:

The emmc slot of the board is empty. Just openend it to find out that the adapter is completely missing :wink: Nevertheless chromeOS was booting fine until I flashed coreboot. Probably I should reflash the stock firmware ? (how to do this, btw?)

The specs says It should have a 64gig emmc (full specs for 20UX-000EGE)

I wonder now where the 64gig emmc is located ? :wink:

Now that is strange. Mind posting dmesg as well? It would help identifying the root cause.
I see four options in this case:

  1. You can buy and solder an M.2 slot (other people have done that and it worked) and install any NVME drive, then proceed as normal
  2. Restore stock firmware and use submarine instead of EDK2
  3. Restore stock and use EDK2 to load GRUB/kernel from mSD/USB, then use eMMC as rootfs
  4. Keep current Coreboot build and help us find out why eMMC isn’t being detected :wink:

As for restoring, it’s the same as initial flashing - connect to WiFi and run Matt’s script. If you can’t (for one reason or another), you can simply flash it back using flashrom flashrom -p internal -w /mnt/usb/<MORPHIUS-STOCK>.rom although I’m not sure if it keeps VPD (I think it should since it’s in SI_BIOS, but not 100% sure).

eMMC will be located close to SoC, most likely under the black insulating “tape” :stuck_out_tongue:

1 Like

Yeah! :wink:

I´ll go with 4) \O/ - Happy to help somehow…

Here we go with dmesg

Forum didn´t allow me to put much txt into the editor. Uploaded:

—snip

LINK

—snap

Thx for looking into this…

Nevertheless I would love 1) - nevertheless guess that´s way too timeconsuming and fiddly for my shaking hands…

I could do that for you, as I happen to live in Strasbourg (right next to German border) - just a matter of getting M.2 socket quickly as ordering from China would take at least a week :wink:

[    8.069969] mmc0: ADMA error: 0x02000000
[    8.069983] mmc0: sdhci: ============ SDHCI REGISTER DUMP ===========
[    8.069988] mmc0: sdhci: Sys addr:  0x00000000 | Version:  0x00001002
[    8.069994] mmc0: sdhci: Blk size:  0x00007200 | Blk cnt:  0x00000001
[    8.070000] mmc0: sdhci: Argument:  0x00000000 | Trn mode: 0x00000013
[    8.070007] mmc0: sdhci: Present:   0xf1ff0000 | Host ctl: 0x00000011
[    8.070013] mmc0: sdhci: Power:     0x0000000f | Blk gap:  0x00000000
[    8.070019] mmc0: sdhci: Wake-up:   0x00000000 | Clock:    0x0000e8c7
[    8.070025] mmc0: sdhci: Timeout:   0x0000000a | Int stat: 0x00000000
[    8.070031] mmc0: sdhci: Int enab:  0x03ff000b | Sig enab: 0x03ff000b
[    8.070037] mmc0: sdhci: ACmd stat: 0x00000000 | Slot int: 0x00000000
[    8.070044] mmc0: sdhci: Caps:      0x61fec8b2 | Caps_1:   0x00000577
[    8.070050] mmc0: sdhci: Cmd:       0x0000083a | Max curr: 0x00c80064
[    8.070056] mmc0: sdhci: Resp[0]:   0x00000700 | Resp[1]:  0xffffffff
[    8.070062] mmc0: sdhci: Resp[2]:   0x320f5903 | Resp[3]:  0x00d02701
[    8.070060] AMD-Vi: Event logged [IO_PAGE_FAULT device=0000:00:13.1 domain=0x0000 address=0xcc620200 flags=0x0050]
[    8.070067] mmc0: sdhci: Host ctl2: 0x00000000
[    8.070073] mmc0: sdhci: ADMA Err:  0x00000001 | ADMA Ptr: 0xcc620200
[    8.070075] mmc0: sdhci: ============================================
[    8.070077] mmc0: sdhci: cc620200: DMA 0xcb5d6800, LEN 0x0200, Attr=0x21
[    8.070080] mmc0: sdhci: cc620208: DMA 0x00000000, LEN 0x0000, Attr=0x03
[    8.070142] mmc0: error -5 whilst initialising MMC card
[    8.072279] mmc0: Failed to initialize a non-removable card

Interesting, I’ve seen someone else reporting issues with eMMC on a different platform. Maybe there’s a regression in Linux kernel (wouldn’t be first… or 30th)?

Those picasso machines with eMMC are a bit special, since they’re the only x86 platforms (that I know of) using MMIO storage

I happen to have DIRINBOZ here (also Google/Zork with eMMC) so I should be able to reproduce the issue. Will report back :slight_smile:

1 Like

Nice! :wink: Nevertheless I guess a 2 core 4gig system with a “slow” cpu is not really worth the effort soldering it to hell :wink:

I really love the formfactor of the Thinkpad C13 - but I guess I want a ryz 5-7 and 8/16 gig version soon as I seem to lick blood in the chomebook universe right now :wink: Nevertheless theses machines seem rather rare these days… at least on ebay and “Kleinanzeigen” - a german “small ads”-system - right after your presentation at 37C3 :wink:

Looking forward…

Hi!

Sorry to keep you waiting, I’m back with my findings.
First of all, thanks for reporting the issue!

I found that was caused by @MrChromebox’s commit:

After reverting it and building 4.22, I attempted booting Fedora installer:

[   19.890245] sdhci: Secure Digital Host Controller Interface driver
[   19.896529] sdhci: Copyright(c) Pierre Ossman
[   19.942718] mmc0: SDHCI controller on ACPI [AMDI0040:00] using ADMA
[   19.967128] usbcore: registered new interface driver usb-storage
[   19.981542] ccp 0000:01:00.2: ccp enabled
[   20.004917] ccp 0000:01:00.2: tee enabled
[   20.018805] ccp 0000:01:00.2: psp enabled
[   20.052307] AMD-Vi: AMD IOMMUv2 loaded and initialized
[   20.239264] mmc0: new HS400 MMC card at address 0001
[   20.259838] mmcblk0: mmc0:0001 DA4128 116 GiB
[   20.280929]  mmcblk0: p1
[   20.285535] mmcblk0boot0: mmc0:0001 DA4128 4.00 MiB
[   20.292948] mmcblk0boot1: mmc0:0001 DA4128 4.00 MiB
[   20.300288] mmcblk0rpmb: mmc0:0001 DA4128 16.0 MiB, chardev (236:0)

So regression is in our Coreboot tree, not Linux (phew). I will test our bootloader (called submarine), which should help you with getting up and running on stock w/ eMMC :slight_smile:

1 Like

Reverting that will break S3 resume

1 Like

Yes, I know :stuck_out_tongue:

Maybe we should tell people with eMMC ZORK to stay on stock firmware until we’ll have x86 MMIO driver for EDK2?

Something like “if /dev/mmcblk* exists, then exit 1, printf ‘your device uses eMMC and getting it to work right now is complicated’”.

It’s not like it provides much value since you need external media which payload can detect, and you cannot boot Windows either.

Using submarine is a good stop-gap in my opinion for Linux users that just want their machines to work. I know you haven’t been following this project so quick rundown for you - it’s basically LinuxBoot:

  1. We reserve 16MB of eMMC, where we create ChromeOS kernel partition.
  2. We embed u-root (minimal roots) inside slimmed-down Linux kernel without any networking and with basic framebuffer, no external modules.
  3. We package that into ChromeOS KPART that depthcharge expects and flash it onto eMMC.
  4. Users can change GBB flags to disable beep and reduce waiting time while running stock builds (highly recommended).
  5. Coreboot loads depthcharge, which then finds Linux kernel on eMMC within that 16MB partition we reserved and jumps to it.
  6. LinuxBoot starts, initializes basic devices (FB, ATKBD, SDHCI) and loads u-root, which looks for Linux kernel images on all attached storage devices (eMMC, NVME, USB - you name it) on FAT32, EXT4 or ISO.
  7. User is presented with list of available options (including dropping to shell), they can modify cmdline on the fly and/or boot from selected image.
  8. Image is being loaded into memory and executed using Linux’s kexec call.

There are downsides to this approach of course. No UEFI services so you cannot boot Windows, and PSP modeswitch will be in “Developer” instead of “Production”, which will disable HDCP for instance.

Thankfully we can probably patch ACPI tables (if needed) using ACPI_TABLE_UPGRADE. If my hunch is right, target Linux kernel should use ACPI patched by previous stage (LinuxBoot/Submarine).
If you’re interested, look at the following docs for details:
Documentation/admin-guide/acpi/initrd_table_override.rst

1 Like

No way to sorry - :wink: Its all our free time, eh ? :wink:

You should then set up 2 checks. Second case is that that system has a secondary media storage like a m2 adapter which should work but runs the emmc useless after installation.

How high are chances right now that there will be a fix for s3 and mmio at the same time ? Are there any plans to fix the regression ? :woozy_face:

1 Like

I’ve already reverted that revert locally and implemented a proper fix, just need to release some new builds

1 Like

Matt just answered that, haha :stuck_out_tongue:

So, if you would like to proceed as before, you can use MORPHIUS’s mSD slot to store /boot/efi and /boot and use internal eMMC for rootfs.

It’s janky, but should work. Just need to update the build, so you can run his script (just like before) and update the FW.

Or as I mentioned, using LinuxBoot is also an option.

Important note about Fedora: Anaconda selects btrfs by default, which is a terrible idea on eMMC. According to our testing, good 'ol EXT4 performs best (as F2FS support in mainline is frankly terrible).

We will have an EDK2 driver for it eventually, but it wasn’t a priority since there are a lot of other things to work on. Not to mention soldering an M.2 slot is also an option, will give you faster storage and more capacity :wink:

1 Like

Is there a way to find out on your github when you pushed a new release ? Can´t find a list in the main repository where to look. Neither way: Thanks A LOT for all your efforts you put in here, Matt - highly appreciated!

Looks like the NVMe slot is unpopulated- is this machine still under warranty?

You can check date format in sources (and commit that changes it) here:

just updated again, let’s see if this fixes the issue

Allright. Updated to @MrChromebox Coreboot v20240107

I tried to install Fedora again - and now it sees the partition right.

Nevertheless the 64gb size of the mmcblk0 is too small ;( So I decided to skip fedroa to install Mint 21.2 which also recognizes the MMC and does not install so much bloat.

When I got this right now I need to put the bootloader on an USB device as Coreboot doesn´t recognize the MMC as a boot device, right ? Or should it work after the fixes of @MrChromebox ?

How can I manage to wirte the right boot loader onto the device to make the mmc recognizable at boot or right after boot time ? :wink: I guess it´s the sentence from the manual:

eMMC models need to put /boot/efi and /boot on USB

Should lead to a explanation link in Supported Devices and Platforms | Chrultrabook Docs

p.s. copying the /boot/efi and /boot folders from emmc to a fresh formated usb stick in fat32 didn´t do the trick. ;(

You forgot to do manual partitioning, which is very much needed in this case:

  1. Plug another USB or mSD stick/card that will store your bootloader and the kernel.
  2. Click on mmcblk0 (eMMC), manual, clear current partition table.
  3. Pick USB/mSD, create /boot/efi partition (FAT32 ~200 - 600MB) and /boot (EXT4 ~1GB)
  4. Create rootfs on eMMC (/ ext4 100% free, or however you like, please remember about encrypting it with luks2).
  5. Continue with the installation.

You can also use dd to wipe the partition table:
dd if=/dev/zero of=/dev/mmcblk0 bs=512k count=1

External storage media will need to be present while:
a) Booting the system up
b) Installing updates
Otherwise it can be safely unplugged while the system is running :slight_smile:

2 Likes