Summary #

I’ll talk about:

  • Why I invested in an eGPU
  • The Razer Core X Chroma, and why I chose it
  • How to get eGPUs working with Linux
  • Lessons Learned

Introduction #

eGPUs is short for external GPUs: With the advent of Thunderbolt 3 and it’s ability to expose PCI express buses via a USB-C cable, it is now possible to connect a GPU to any device that has a thunderbolt 3 controller. Laptops with a USB-C thunderbolt port has become extremely common, and both my personal and work laptops expose such a port.

Motivation #

Typically, one would buy a separate GPU either to play video games or to run some machine learning models. However, I purchased an eGPU for my day-to-day work and for coding.

Why? Because:

  • my day-to-day work environment utilizes web applications heavily, which in turn run much smoother when there is a powerful GPU to accelerate rendering.
  • I use two 4k Displays at home, and the current GPUs that are embedded into ultrabooks (for Intel laptops, typically Intel UHD Graphics 6xx) struggle with that resolution (effectively 2 x 4K).

I originally tried to just use a laptop with a thunderbolt 3 dock, but I found that the stuttering of the screen and the very slow rendering experience when I had 4 browser windows opened was putting a hamper on my productivity. I had a desktop with a GTX 1060 at home for comparison, so I knew the experience that a high performance GPU can enable.

Choosing an eGPU #

So what eGPU should I choose? There’s typically two varieties:

eGPU Enclosure #

Effectively bring-your-own-graphics card. An eGPU enclosure provides:

  • a thunderbolt 3 controller and port to connect your device to the GPU
  • a power supply to power whatever GPU you want to use
  • a PCI-E slot to insert your GPU

Embedded eGPU #

An embedded GPU provides everything mentioned above, but also includes the GPU, hard-wired. I don’t believe there are a lot of advantages to this setup, aside from the smaller size due to the ability to have a custom board that wires all the components together.

My Choice: Razer Core X Chroma #

UPDATE 20200905: I’d recommend checking out the Mantiz Saturn Pro. I haven’t tried it, but it seems to work better than the Chroma on Linux due to different USB boards.

Ideally, I wanted to go for something that reduced the amount of cables that I had to mess with. The Razer Core X Chroma not not only provided an eGPU, but also provided a 4 USB 3.0 ports and a 1GBps Ethernet port. With those ports, I wouldn’t need to connect a dock on top of my eGPU to connect my keyboard, mouse, headset, and video camera. Theoretically, I should just be able to connect a single USB-c port to get all of that.

I chose an external GPU because I wanted to be able to upgrade my graphics card and not have to buy a new encoslure (as the Core X Chroma is already $400 USD!).

note: The reality turned out to be a bit different, so I wouldn’t recommend a Core X Chroma (or any eGPU I’ve seen) for the additional USB ports and Linux. But read on for details.

Tangent: with thunderbolt3, GPU performance is bottlenecked #

As part of my research, I learned that thunderbolt3 will not be able to provide as much bandwidth as a PCI-E port. The reason is that thunderbolt3 actually exposes 4 lanes of a PCI-E v3.0 bus (aka PCIEX4).

Most motherboards provide at least 16 lanes of PCIE and versions much higher than 3.0, and provide similar ports on their motherboard. In other words, thunderbolt3 will be limited by the PCIEx4 v3.0 bandwidth (32Gbps), while desktop GPUs will be able to leverage 128Gbps (PCIEx16 v5.0) by 2021.

For my purposes (mostly development, heavy web browsing), I never noticed the difference. But really heavy gaming could see a hit. This forum post indicates a 20% loss in FPS and thus effective performance.

Tangent 2: Peripheral performance is also bottlenecked #

The issues that I mentioned above are not specific to GPUs. The reality is that the data that can transfer through the Thunderbolt3 is capped, regardless of the number of devices communicating through that bus.

That means, if your eGPU also comes with additional ports (e.g. USB or Ethernet), the data transferring through those peripherals are also consuming of the bandwidth of the port.

So if you have a lot of data running through your eGPU, Ethernet, and USB ports, you’ll see some throttling due to the fact that Thunderbolt3 cannot handle all the throughput of all devices at the same time. Most likely, this will affect the GPU the most.

In my anecdotal experience, this hasn’t been a problem, but it is a likely theoretical one.

Tangent 3: The Core X Chroma USB ports + ethernet don’t work well #

With the Core X Chroma, there were a few Linux-specific limitations I ran into:

  1. All USB 3.0 splitters I tried didn’t work.
  2. The Ethernet works inconsistently.

I think 1 isn’t a huge problem for most people: I had a setup that required me to use at least one USB 3.0 splitter. 4 ports are generally enough for any docks.

However, the ethernet not working at all was a big issue for me. It would work fine for a few seconds, but then completely cut out. Using an ethernet-to-usb3.0 adapter doesn’t work either, on the couple I’ve tried.

It turns out these are both due to the fact that they are leveraging ASM1142 USB 3.1 Controllers, and seem to require a firmware update to reconcile:

  • https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1749961
  • https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1667750

Unfortunately these bugs have been around for a while.

eGPU on Linux #

In July 2020, there are still some rough edges around eGPUs on Linux. In Windows and OSX I’ve heard stories of seamless hotplugging: the OS detecting and utilizing the eGPU as soon as it’s plugged in. On Windows, some games are even sophisticated enough to leverage the eGPU when it’s connected, and seamlessly switch pack to the internal GPU on eGPU removal.

With Linux, there’s a constraint today with Xorg. Although the Linux kernel does detect the GPU as soon as it’s connected, Xorg must be restarted with configuration specifying the eGPU before it’s displays and device are available to Xorg.

Two ways to eGPU: prime and separate device #

There’s a second way of leveraging an eGPU: Using something like Nvidia’s optimus where rendering is offloaded to the eGPU, but the result is sent back to the dedicated GPU for rendering.

This still requires a restart of Xorg to expose the eGPU as something that can be leveraged, and also comes with the added overhead of sending the rendered result back to the host machine via thunderbolt3 (really the PCI-E lanes exposed under the hood).

As a result, I didn’t explore this option.

The step by step #

Ultimately, the process to enable and disable eGPUs are:

  1. plug in the eGPU
  2. authorize the thunderbolt3 eGPU (if that hasn’t been done already)
  3. modify Xorg configuration (typically /etc/x11/xorg.conf) to be set to discover the newly plugged in eGPU.
  4. restart Xorg

There’s a few scripts which help make this process a one-line CLI call, which can also be integrated into systemd or whatever init system.

Here’s an example of the xorg.conf configuration I used:

# /etx/x11/xorg.conf
Section "Module"
    Load           "modesetting"
EndSection

Section "Device"
    Identifier     "Device0"
    Driver         "nvidia"
    VendorName     "NVIDIA Corporation"
    BusID          "PCI:8:0:0"
    Option         "AllowEmptyInitialConfiguration"
    Option         "AllowExternalGpus" "True"
EndSection

The BusID will need to be replaced with your device’s BusID as it shows up in lspci.

Ultimately I used a script called egpu-switcher which does all of this automatically (besides restart Xorg). This has been working great.

Final Thoughts #

There is still a lot of work that needs to be done for eGPUs to be as seamless on Linux as on Windows and OSX. It also seems like there’s some hardware logistics that needs to be reconciled as well, and some level of acceptance that no Thunderbolt protocol will be as fast as the cutting-edge PCI standard.

I needed an eGPU to have a seamless development experience, and my setup works really well for that. But if you’re looking for a bleeding-edge gaming experience, you’ll have to take concessions with eGPUs.

Side note: why I still have hope for the Core X Chroma #

As mentioned above, my main issue with the Core X Chroma is the non-functioning Ethernet port. There are a lot of anecdotal stories around the Core X Chroma working great for Windows and OSX, without any firmware updates on the Chroma side. As such, I hope that either a workaround comes out in the Linux kernel for the ASM1142 controller, or Razer releases something to address this on the controller side.