Firmware + Driver issues galore

I thought I bought a couple Chromebooks that had been tested and would be well-supported on Linux, but for the last few months it’s been one constant issue after another.

Did I just choose the 2 worst devices or are others in the community simply okay using chromebooks with limited functionality?

Here are some of the issues that I’ve battled and failed to address:

  1. UEFI settings won’t save changes to boot order
  2. EC settings don’t persist for live/dual boot (backlight, charge control, etc)
  3. Whole bunch of audio errors, mono-only audio, and random crackling from speakers
  4. CPU fan randomly spins fast (acpi-fan errors in syslog)
  5. Bluetooth works sporadically (bunch of errors in syslog)

To me it defeats the point of using a different OS on this hardware if the vast majority of the features/functionality don’t work, or work so unreliably.

What am I missing here?

EDIT: Forgot to mention another BIG one:
6. Sleep doesn’t work reliably.

Um you didn’t mention which Chromebooks you had.

Asus CX9 and Dell 7410.

@sadlerm Which one(s) do you have? What’s your experience been like?

I have a Lenovo Chromebook S340 (laser14). It doesn’t have a fan, so I haven’t had to deal with any fan-related issues. I also haven’t had any audio or Bluetooth issues either.

Are you using RW_LEGACY or full ROM? It does seem like Tiger Lake Chromebooks as a whole have quite a lot of firmware issues.

One of my devices is cometlake, and from looking at Supported Devices and Platforms | Chrultrabook Docs, it seems the Jasperlake devices may have camera issues, most Alderlake devices have touchpad/camera/audio issues, and Stoneyridge devices need some sort of special firmware (“built-in”, whatever that means).

The generations older than Cometlake and the low-power models wouldn’t sufficiently meet my needs in the coming years.

I’m using the full ROM, from what I understand hardware support with the legacy ROM would only have been worse, no?

I don’t know if Drallion is special in some way, but I do know a few people using Comet Lake Chromebooks running Linux with no real issues.

The only advice I can give is consider installing Windows on Drallion, it may fix your audio and Bluetooth instability.

  1. It does just fine, we have working SMMSTORE (NVRAM) on Intel platforms.
  2. Yes, that’s normal. When you reboot the machine, EC resets stored values when driver binds to it. All x86 machines do this, so it’s not specific to Chromebooks.
  3. We don’t have many JSL/ADL devices to test, so point taken - there are some rough edges.
  4. You most likely haven’t added systemd service we mention in our documentation. That’s a known issue if EC is in failsafe mode.
  5. We don’t touch bluetooth at all. Bluetooth on all x86 machines is connected over internal USB bus. If something isn’t working, it’s likely an upstream bug affecting your WiFi chipset.
  6. It’s been fixed in recent release.

First of all @elly I’m sorry if this thread offended you in some way, that was certainly not my intention. Your post references a “we” as not being responsible, so I’m assuming you’re a substantial part of this Chrultrabook community.

I do appreciate the substantial efforts that devs have spent on various aspects of the ecosystem (firmware, drivers, testing, etc). I would hope my sharing my honest experience (and asking others for theirs) does not take away from that effort in any way!

No worries, no offense taken :stuck_out_tongue:

I do realize that project’s been under-maintained for a while. There are two people maintaining Linux support - me being one of them, and I’ve been struggling with health issues and burnout for the past 6 months.

Maintaining Linux is essentially a full-time job with amount of regressions upstream is introducing. You update kernel let’s say - 6.6 to 6.7 and you find out that audio is now broken, suspend, machines start randomly freezing and so on. You need to catch those bugs and fix them before they hit your users.

StoneyRidge is especially cursed, because ACP (Audio Co-Processor) doesn’t expose DAI links without GPU firmware embedded in the kernel, and someone broke Designware I2S in upstream while adding some stuff for RISC-V.

Ideally we would have machines from all generations to test, but it’s not really feasible. I do have some plans to pick up the pace, but it will come after I leave my current job by the end of August.

If you would like to help us fix issues you’ve mentioned, please run the following on affected machines:

My deepest gratitude to you and all the other stalwarts here for contributing all the legwork to build this community! Thank you also for providing that context just now – it is very illuminating! I had never realized that even basic support can be such an ongoing project in the face of distro and kernel updates.

I’m going to need to give it some more thought, but this info has mostly confirmed my suspicion that more stability may only be possible with better manufacturer support. I may need to start looking for a made-for-Linux laptop. I’d have bought one of Nova Custom’s Dasharo-based laptops a long time ago if they were thinner/lighter, or came in convertible/touchscreen variants, or were just slightly cheaper.

I would certainly love to help contribute to the effort, but at this point I’m beyond overwhelmed just with all the continuous troubleshooting and upkeep. And like you, I do have a few other projects/causes that I am already responsible for.

You’re welcome!

I had never realized that even basic support can be such an ongoing project in the face of distro and kernel updates.

That’s nothing new unfortunately, many long-term Linux developers are quite frustrated with it (i.e Rene Rebe). Many patches submitted to upstream never get merged, some are being merged straight away, and many people submit patches that fix their issue… but don’t test if it breaks things for others.

this info has mostly confirmed my suspicion that more stability may only be possible with better manufacturer support

We’re trying our best, but Google definitely isn’t making it any easier. Much like Apple and Asahi project - Google doesn’t mind we do this (and their engineers genuinely think what we do is “cool”), but doesn’t help at all.
We had good relations with ChromeOS team in the past, but many of them got laid off in recent wave of layoffs. I personally spoke to Simon Glass (who’s an great, passionate person who geniunely cares), but… you know how corporations tend to be.

Those devices, despite being x86_64, are closer to ARM than x86 in many ways. Sensors are provided by Embedded Controller, DSP and I2C/I2S audio codecs instead of HDA, MIPI cameras (JasperLake, AlderLake) and we’ve heard in MeteorLake there’s also a different way of how sensors are connected to the platform - making it more similar to ARM.

Google is maintaining their own kernels and doesn’t care about upstream much - in fact, many patches are outright breaking functionality for us.
Makes you wonder about project’s future with their recent announcement:

Nova Custom’s Dasharo-based laptops

The funny story is that I nearly ended up working on Dasharo. I spoke to 3mdeb folks at FOSDEM and they really wanted me onboard, but I received a better offer elsewhere. Such a small world, eh?

You probably would be really happy with the purchase, very passionate team of great developers with friendly approach. Really happy to see them partnering with ODM to provide such machines in European market :slight_smile:

I would certainly love to help contribute to the effort, but at this point I’m beyond overwhelmed just with all the continuous troubleshooting and upkeep. And like you, I do have a few other projects/causes that I am already responsible for.

That’s perfectly understandable. Maybe using an LTS kernel would work better for you though?
I’m certainly annoyed by constant bugs in amdgpu and SOF on my personal machines - hence why I’m in the process of writing a proposal to Fedora Project. Having up-to-date userspace with LTS kernel would solve many of those issues in my opinion.

Circling back to your initial issues, it does sound like you’re experiencing a regression with bluetooth in upstream (and I bet it’s Intel’s Bluetooth - iwlwifi has been a disaster since kernel 5.7 days).

Please re-test sleep on coreboot 2405 we released few days ago, MrChromebox had merged a patch from me fixing sleep on TigerLake, Alderlake, MeteorLake (and most likely JasperLake).

As for audio, debug script should tell us what’s wrong. Not many people showed up with JasperLake machines, so it’s likely we simply don’t have UCM for your machine yet (or it’s incorrect).

Indeed, everything in your post is confirming what I suspected all along. It’s exciting to take Google’s open source stuff and open up some new usability options with it, but there are numerous limitations to what can be accomplished with that approach. And these limitations certainly won’t help the reputation of Linux/open-source, still seen by most as being experimental projects, only to be used by coders and sysadmins.

Dasharo’s future is quite bright, indeed! Too bad they missed having you on their team. But then again, who knows what the future holds in this small world of ours, right?:wink:

It’s also too bad Heads is so limited in its applications (and, I suspect, in its dev team). Nova Custom already told me they have no intentions of working on an “Ultrabook” model in the immediate future.

Framework’s offerings look interesting, but at this point they are just ridiculously overpriced. Being as financially focused as they are, it’s possible that their product range may never be completely painless to use Linux on – plus their support ecosystem is still incomplete.

This is what leaves me stuck with my Chrultrabooks for the moment…

Does Fedora come with an LTS kernel? Or are you suggesting that I could simply continue using the older kernels when I see an issue pop up with the newer ones?

These issues with “cutting edge” kernels is the main reason I used Debian in the past. It was indeed stable as a rock, but it seems not as many folks use it on personal devices & laptops. Drivers, GUI issues, and issues with newer software were all harder to troubleshoot and I found the documentation/self help tools to be quite lacking. Then again, Arch seems like the only distro with sufficient documentation — their wiki is amazing for looking up a quick issue without needing to spend a whole day researching obscure web resources! But that’s not the distro for me for other reasons.

I’m currently on Ultramarine 40 with KDE Plasma. Very open to suggestions if something else will make my life easier with a similar setup.

Now that you mention it, I do see “deep” appear for the first time in /sys/power/mem_sleep ! Will need further testing to confirm if S3 is indeed working now, but it’s looking promising.

I spent several hours playing with the audio configs, playing with Wireplumber, switching between ALSA/PulseAudio and Pipewire, etc. Then I gave up and created an issue on WeirdTreeThing’s github, alongwith the file from the debug script. There was no response at all, and I eventually figured out several days later that Github had shadow-banned my account for absolutely no reason. I’ve never spammed or trolled anyone, and had been active on many other repos with the same account for at least a few months before this happened. Many more days later, and their support team still hasn’t bothered responding to the ticket I created asking for an explanation. I expect they will want me to provide some sort of identity or other verification, which I will absolutely refuse to do. “Fluck” Microsoft and their evil empire…

All of that to say that I’m really quite burnt out chasing these issues, it’s turned into the equivalent of a second job for many weeks now. Assuming I don’t pour gasoline on the device and light a match when the next new problem crops up, I will still endeavour to post that audio debug script here on this forum. But I know that chances are it’s going to take at least some effort and diligence to resolve, and this has already been an awful enough experience that I know I need to find a better device ASAP.

Unfortunately no, not at the moment. You can use older kernel if you notice a regression since Fedora keeps two previous kernels installed, and I’m in the process of writing a proposal for LTS kernel in fedora.

Many people are frustrated with regressions introduced by upstream - not just us. For instance, David Revoy - French artist and FOSS advocate recently returned to Debian because he was frustrated with kernel regressions… and I don’t blame him one bit. I’m currently running EOL 6.8.11 on my personal workstation because kernel 6.9 broke AMDGPU (again) and I get random lockups…

Arch seems like the only distro with sufficient documentation — their wiki is amazing for looking up a quick issue without needing to spend a whole day researching obscure web resources! But that’s not the distro for me for other reasons.

Thankfully a lot of their documentation applies to other distributions :stuck_out_tongue:

Now that you mention it, I do see “deep” appear for the first time in /sys/power/mem_sleep ! Will need further testing to confirm if S3 is indeed working now, but it’s looking promising.

Well, Intel deprecated S3 state in TigerLake. There’s no more S3 state, only s0ix. I fixed it few months ago, but for some reason my ELDRID (TGL) still wakes up if i move it too quickly (i.e running to catch a tram with laptop in my backpack). Haven’t had the time to look into why EC decides to wake up AP by gyro events though (yet).

Github had shadow-banned my account for absolutely no reason

Yikes, sorry to hear that. I’m hoping you’ll find time/energy to help us fix this bug though - I’m still surprised to hear about it because I have CML and TGL devices here (KOHAKU, ELDRID) and both of them have been working fine for over a year (with exception of regressions I mentioned here and there).

FYI: Sleep is still broken even with the latest firmware release. Also, I’m not seeing any mention of sleep improvements in any of the recent releases (or commits):

https://github.com/MrChromebox/chrome-ec/branches/all

I’m not sure if the issue is the device not sleeping at all, or waking up at some point with the screen still off. I am unable to perform any further troubleshooting so I’m mentioning this for information only.

At this point, I am only hoping that others don’t end up buying either the drobit or the drallion like I did, expecting full hardware support from reading Supported Devices and Platforms | Chrultrabook Docs