My Linux Gaming Experience - Trying out Steam Proton and GPU Pass-Through
1. Introduction
1.1 What is this?
This is, I guess, a retelling of my experiences with setting up Linux for the purpose of gaming, how I went about setting things up, problems I came across and how I overcame them. I’m not a writer mind you, this may not be an entertaining read but the next heading explains why I’m writing this.
1.2. Why are you writing this?
I’m bored.... Also I think I have some pieces of information that could make life easier for others trying to do the same thing as I have, that I personally had trouble finding through my own research and thus want to share. Also, it’s fun to share experiences.
Also, I want to rant. So.. Yeah.
1.3. But why are you writing this in English?
Basically, my hope is that if someone is searching for some of the information that I provide in this post, they might end up here and find some answers. Because of that I don’t want to limit the potential readers to those that only understand Swedish.
1.4. Which distribution are you using?
I’m using Manjaro KDE and that’s the distribution that I’m going to write about in this text, meaning most things are going to be arch focused and may not translate to other distributions of Linux.
1.5. What hardware are you using?
CPU: Ryzen 5 1600 – Overclocked to 3.8GHz
Motherboard: Asus Prime B350 Plus – More on this later.
Host GPU: Nvidia GTX 560 – More on this later.
Guest GPU: Nvidia GTX 780
RAM: Corsair 16GB 3000MHz CL16
1.6. So... Are you like an expert on this?
AHAHAHAH- no. No I am not, in fact I am far from it. I started using Linux like a week ago. I doubt my own knowledge in many areas I take up in this text, as you’ll see if you continue reading. In other words, when reading, be critical of any claims I make and don’t just blindly follow my instructions, try to understand them, some instructions might screw things up otherwise.
2. Installation and distributions
First of all, I didn’t start out with Manjaro KDE, I first tried Manjaro Cinnamon. I liked the style and look of Cinnamon very much, but sadly it did not live up to my expectations when it comes to performance. Moving windows had so much stutter that it felt horrible using it for everyday usage. I didn’t really find any way to work around this... So I gave up and tried installing KDE instead, a failed attempt, which led me to completely reinstall Manjaro, this time with Manjaro KDE from the start.
Installation of Manjaro is pretty straight forward and it handles Nvidia drivers very well assuming you choose non-free drivers during the setup, it’s possible you might be able to change to non-free after the fact as well, not something I’ve looked into.
Setup of everyday applications went pretty well. I’ve got Steam, Discord and Spotify running pretty well, the only issue I’m experiencing is startup issues with Spotify, something I can’t be bothered to look into. I’ve also got Kodi working, I’m currently writing this in LibreOffice Writer so got that working... Got KeePassXC working.. And my browser of choice, Vivaldi. Basically what I’m getting is that the setup for my everyday applications went pretty well and I haven’t felt that I’m really missing anything that is exclusive to Windows, a good sign I would argue.
3. First attempt at gaming – Steam Proton
So let’s get to the part you’re here for – gaming.
The first thing I tried was simply starting up The Binding of Isaac: Rebirth... and it ran just fine, not like I expected it not to considering the type of game it is. Next I tried Hellblade: Senua’s Sacrifice, the purpose was to test out Steam Proton which was the main thing I was interested in trying out. According to the ProtonDB at the time, it seemed like it would work fine with a small performance hit – “Sweet!” I thought. The reality was somewhat more complicated. My first attempt to start the game.... left it crashing to desktop with literally no message whatsoever – after that it wanted to update – and then it failed to update because corrupted update files(?!) ... Anyway, a lot of uninstalling and reinstalling later I got it to launch at the very least. My experience with that game in Windows was a predominantly smooth experience with around 80-100 FPS, my experience with that game in Linux was a predominantly stuttering experience with around 40-70 FPS (mostly between 50 and 60 FPS) – I’m not sure why my experience was so filled with stuttering – My monitor IS running at 120Hz, but my experience using windows and scrolling is also not very smooth – smoother than in Cinnamon mind you, but still not smooth. (Later I went through some guides to get KDE Plasma on Nvidia more smooth – I haven’t tested if this also affected games).
I kind of gave up on it right about here, the stuttering was bad, sure, but the huge dip in FPS was enough for me to not consider this approach good enough in the long run. Instead I started looking into alternative methods of gaming on Linux and came across Looking Glass which sounded like just the right thing.
4. Second attempt at gaming – Running Windows virtualized
4.1. Looking Glass
So when I first heard about Looking Glass it was described to me as a way to virtualize Windows and pass-through a dedicated GPU to that virtual machine in order to get near native performance in games. Let’s just say that that’s not what Looking Glass is, I can’t say for certain what Looking Glass is since I stopped reading about it when I realized it wasn’t what I was looking for – But basically it is taking the above, which is already supposed to be set up, and then makes it so that the output of the GPU can be transferred from the guest (Windows) which otherwise needs a dedicated monitor, and displays that output in a Window on the host machine – Basically making it easier to switch back and forth I guess. I didn’t find a need for this myself and the limitations of doing it this way outweighed the benefits in my personal opinion, I’d rather just use a dedicated monitor input on my monitor.
4.2. KVM – qemu, libvirt, ovmf and virt-manager
That first description I heard of Looking Glass was actually the description of using setting up and running KVM, in this case qemu with libvirt, ovmf and virt-manager. Now, I’m going to be completely honest, I have no idea what these individual parts actually do in isolation... I couldn’t tell you what qemu does, I couldn’t tell you what libvirt does, I could tell you ovmf had something to do with uefi firmware maybe? Virt-manager seems to be a way to manage libvirt virtual machines? I don’t know, who cares, what I do know is that these pieces together lets us easily run and manage virtual machines, I guess... Do I know that? Embarrassing if I’m wrong.
The best resource I’ve found for setting this up is this arch wiki post which was also my go-to knowledge base when setting things up: Arch Wiki - PCI passthrough via OVMF
4.2.1. Prerequisites
This kind of setup has requirements. First of all you’re going to need two GPUs, unless you’re using a graphics card that supports SR-IOV – which most if not all consumer grade graphics cards does not. Why two GPUs? Well you need one for your Linux host, and one for your Windows guest. The second thing you need is a CPU that supports hardware virtualization for KVM but also IOMMU for passing through devices (like your guest GPU) to your virtualized guest. You’re also going to need a motherboard that supports both of these technologies. While not a prerequisite, you might want to look into how your motherboard handles PCIeX16 when two slots are being used, my motherboard goes into x16 and x4 mode which is not ideal – more on this later. Oh right, the guest GPU also needs to support UEFI.
4.2.2. The setup
The setup was actually pretty straight forward if following the arch wiki page, it did however require some page-hopping to different wiki pages, most notably the page for kernel parameters: https://wiki.archlinux.org/index.php/Kernel_parameters and mkinitcpio: https://wiki.archlinux.org/index.php/Mkinitcpio#Image_creation_and_activation (for the latter resource, I ended up using “sudo mkinitcpio -P” which was somewhat hard to reach due to all the information there.)
Since I was going to use my GTX 560 as my host GPU I had to downgrade the Nvidia drivers since the GTX 560 doesn’t support the newest ones, or rather the newest ones doesn’t support the GTX 560. This was a very easy operation in Manjaro through the System Settings > Hardware Configuration window, just an uninstall of the old and reinstall of the new.
Getting my GTX 780 to be properly isolated was a bit of a chore though, what I’ve learned however is that if it’s actually set up correctly but just doesn’t want to be isolated (set to vfio-pci) then rebooting and disabling IOMMU in the BIOS/UEFI > starting up into Linux > rebooting and enabling IOMMU again in the BIOS/UEFI... seems to fix it... at least it has so far (no, just the act of rebooting didn’t do it for me, I had to do the IOMMU enable/disable cycle.)
Since I was passing through an Nvidia card which are notorious for being problematic (error 43 caused by Nvidia drivers because they noticed it’s running in a virtual machine and Nvidia won’t allow us to have nice things) I actually went through the necessary Nvidia workarounds before I even set up the pass-through of the GPU to the virtual machine. Instructions for that is on the arch wiki. (btw, I have no clue how the hell to use vi – every time I try I just break absolutely everything, so I personally use “sudo EDITOR=nano virsh edit [name of virtual machine]” instead – just a tip if you, like me, prefer to use nano.)
And then.... nothing... I passed through my GPU to the virtual machine, but it wouldn’t output anything. If I used Spice to get a window for the virtual machine, I could see in Device Manager that I received the dreaded error 43 and I spent the next two days trying to get it to work, searching far and wide for anyone with the Nvidia drivers as tenacious as mine – Except it had nothing to do with the Nvidia drivers, it was the fact that I was trying to pass through my boot GPU.
4.2.3. The problems with passing through a boot GPU
First of all, what is a boot GPU? The boot GPU is the GPU that is used to boot your system, it’s the GPU that is used to output your POST during startup for example. In order to do this the GPU needs to be initialized – which for some reason screws up something something making it near impossible to pass it through to your guest VM – because it has already been initialized... Whatever that actually means.
Solution? Well... As far as I understand, it should be possible to dump your VBIOS from the GPU in a good state – and then pass that VBIOS to the guest VM during boot. This is something I never got working due to not being able to dump the VBIOS because of permission errors even when trying to do the things with sudo – I don’t know how sudo doesn’t have the permissions... but it doesn’t for me. But hey, you can just download the VBIOS from the Internet! Yes, and I did, and it didn’t work. I essentially gave up on trying to pass through the boot GPU.
Actual solution? Change my boot GPU... which is a huge problem. As I mentioned earlier, my motherboard runs the two PCIeX16 slots in x16 and x4 mode when two GPUs are connected, the configuration here is very static, it’s x16 and x4, not x4 and x16. This is a problem mostly because the x16 slot is ALWAYS the boot GPU apparently, I haven’t found a way to change this in the BIOS. This means that I have to run the gaming GPU in x4 mode, which isn’t ideal due to the limited bandwidth. Other motherboards may or may not support switching boot GPU, I think it’s easier if your CPU has integrated graphics though but I have no actual experience with that.
So I switched the GTX 780 to the x4 slot and the GTX 560 to the x16 slot and I felt dirty doing it... But hey, at least it ... didn’t work... Read on to understand why.
4.2.4. The problems with passing through a GPU in a shared IOMMU group
So IOMMU groups are... ... I don’t know. Basically IOMMU is used to... isolate? virtualize? Devices, but the thing is that IOMMU doesn’t work with devices, it works with groups, and it’s basically down to your motherboard how these IOMMU groups are set up. The x16 PCI slot on my motherboard is in its own group – but that was the boot GPU so that one is unusable because of that, my x4 slot however is in a group with a shit ton of other things... So if I wanted to pass that through to the gust, I’d have to pass through literally everything in that group... I don’ think I’d be able to do that to be honest, considering the things in that group seemed rather important for the host to have... So... Time to give up, right? You’d think so, but no! Apparently there is a way to bypass this by making... something... think that each device is in its own group... Look, I don’t know how these things work and I don’t really have the mindset after troubleshooting for hours on end to do a deep dive in every single thing that comes up. Anyway, so there’s a patch... a kernel patch... that you can use to fix the groups. Great, how do I apply it? Ohoho, wouldn’t you like to know! Well... that was a pain to figure out, since I couldn’t find many previous guides on this.
4.2.4.1. Building a custom kernel
So, if I just used straight up Arch, then there would be a package on the AUR with the patch applied to the kernel and I could just use that... But I’m using Manjaro, so I had to apply the patch to the Manjaro kernel instead. Now, this was a right pain in the ass because I couldn’t find any guides to do this, so I had to basically figure it out by myself with bits and pieces from different sources. Since I couldn’t find any information on how to do this, I’ve decided to make a complete guide on how to do this right here.
First of all you’re going to need the Manjaro kernel source. That can be found here: https://gitlab.manjaro.org/packages/core – I would personally recommend using whichever kernel series you are currently on.
Note: Some sources I’ve found say that installing such a kernel can cause network and video drivers to stop working, apparently you need to also build the relevant extras which can be found here: https://gitlab.manjaro.org/packages/extra – However I did not do this and it seems to work fine – but I think that is simply because the kernel I downloaded is the exact same version I was already using beforehand which meant I already had the proper version of the extras installed – but if you build a different version of the kernel than you are using or have extras for then you’re probably going to have to build those as well, not having done it I’m not able to tell you how to do it, but I assume it’s the generally same procedure as for the kernel, minus the ACS patch.
I’m using the 4.19.something kernel and so I did this in the home directory in the terminal:
## The numbers represents the order of execution and a continuation in the same terminal window unless the sequence is broken.
1. mkdir Builds
2. cd Builds
3. git clone https://gitlab.manjaro.org/packages/core/linux419.git
You also need the actual ACS patch which I got from: https://aur.archlinux.org/linux-vfio.git
4. git clone https://aur.archlinux.org/linux-vfio.git
5. cp /linux-vfio/add-acs-overrides.patch /linux419/add-acs-overrides.patch
Now you need to add the acs override patch to the PKGBUILD file in the /linux419 directory:
6. cd linux419
7. nano PKGBUILD
Here you want to navigate to the large list that starts with source=(“something” in my case this is on line 25. You then want to add ‘add-acs-overrides.patch’ to this list. For example:
[...]
source=(“https://www.kernel.org/pub/linux/kernel/v4.x/linux-${_basekernel}.tar.xz”
[...]
‘0012-bootsplash.patch’
‘0013-bootsplash.patch’
# ACS
‘add-acs-overrides.patch’)
[...]
You now also need to add the the sha256 hash to the sha256sums list which should be right below the source list – I’ll leave finding the sha256 hash to you since it’s pretty easy on Linux. I think you need to place it at the same place in the list as you placed the acs patch in the source list, I’m not sure about this though but I wouldn’t take any chances and just put them both at the end of the respective lists.
If you go down a bit in the file you’ll see lines starting with patch -Np1 -i [...], we need to add a line like this for the ACS override. For example:
# add BFQ scheduler
patch -Np1 -i "${srcdir}/0001-BFQ-${_bfq}-${_bfqdate}.patch"
# add ACS overrides
patch -Np1 -i "${srcdir}/add-acs-overrides.patch"
That’s all we have to change... I think... Oh right – I’m not an expert on this as is probably very clear by this point, so don’t rely on my instructions being 100% correct, I won’t take responsibility for any damages you cause by trying to install a borked kernel.
We now actually have to build the kernel:
8. makepkg -s
I should probably warn you that it could take hours to complete the whole build process... You can shorten it by configuring it to use multiple cores/threads when building it, but I’ll leave it up to you to find out how to do that.
Once it has finished building, assuming it all went okay, then the next step is to install the kernel:
## Replace the actual filenames below with the ones that are relevant for you.
9. sudo pacman -U ./linux419-4.19.33-1-x86_64.pkg.tar.xz ./linux419-headers-4.19.33-1-x86_64.pkg.tar.xz
After this you just have to add a kernel parameter, I’ll leave how to do that to the arch wiki but what you need to add is this: pcie_acs_override=downstream,multifunction
The multifunction part may or may not be needed, it was for me. Also please note that this apparently is a security issue(?) - I’d recommend reading more on the arch wiki and whichever resource that points to if you’re worried about it.
Remember to reboot.
After that you check your IOMMU groups to see if the acs patch worked, if it worked you should see your devices divided into a ton of more groups. At this point you should be able to find the rest on the arch wiki.
4.2.5. Setup continued
After that ordeal I finally got the guest GPU to output to the monitor. I didn’t have any issues with the Nvidia drivers, the previous config to prevent the error 43 seemed to have worked... So.. On to the next parts!
I set up mouse and keyboard pass-through using evdev (description in arch wiki) and it worked nicely. I also set up my CPU topology as the arch wiki explains, taking care to follow the Ryzen part, it was a bit confusing to be honest but after reading it carefully a few times it became more logical. I set it up to use 4 cores (8 threads) of my 6 cores (12 threads) – I don’t predict needing more CPU cores for the games I play and I’d prefer to save the CPU cores for other things I do on the host while playing (I use a secondary monitor which shows the host at all times, I’d prefer it to work relatively well when watching videos on that monitor while playing for example.)
I actually skipped the instructions for huge pages, according to the wiki Qemu will already use huge pages by default – the problem is that it might not be able to get enough of them something something – so it suggests statically setting huge pages... which permanently disables that memory from the host... No thank you. Basically I’ll continue using it with the defaults until I feel a need to change this. (If I had more memory, say 32GB, I would have considered it.)
I followed the instructions to get audio from the guest to the host via PulseAudio and it worked fine, no crackling or such yet. I also followed the instructions on how to pass-through a physical disk/partition in order to pass-through my Games SSD. (If I understand correctly, this drive should ideally be dismounted from the host system while it is in use by the guest system.)
4.2.6. Back to gaming
Time to play some games! I load up Hellblade: Senua’s Sacrifice.... and... The mouse input is extremely inconsistent and there’s lagging... hmm.. *starts digging deeper into the wiki*
4.2.6.1. The problem with CPU frequency governor and scaling
So basically it seems like the issue is that while the guest is trying to use a butt-ton of CPU powas, the KVM doesn’t seem to relay this to the CPU, meaning that even though the game might want the CPU to go full frequency, it might be idling on low frequencies which was the case for me.
Solution? Use cpupower command to set the frequency governor to performance mode (or schedutil to go back (at least in my case, yours may vary)). Now, this is probably a pain in the ass to do manually every time you want to play a game, so I looked into it and libvirt/qemu uses hooks to send start and stopped events to scripts... I don’t think I explained that very well... Eh, let me show you instead.
Terminal:
1. cd /etc/libvirt/hooks
2. sudo nano qemu
Enter this:
#!/bin/bash
if [[ $1 == "win10" ]] && [[ $2 == "start" ]] || [[ $2 == "stopped" ]]
then
if [[ $2 == "start" ]]
then
cpupower frequency-set -g performance
else
cpupower frequency-set -g schedutil
fi
fi
But replace win10 to whatever the name is of your virtual machine. What does this do? When the VM starts, the script will be called with the argument “start” which then executes cpupower command to set the governor to performance (always max frequency). If the VM shuts down then the script is called with the argument “stopped” and the cpupower command is executed to set the governor to schedutil. Remember to set execute rights on the file. I would also restart the libvirtd service just in case.
That should be it for that I think, you can use the command watch lscpu to see if the frequency changes correctly as you start and stop the VM.
4.2.7. Back to gaming – finally – right?
So now we have all of that set aaaaaaand... It actually works surprisingly well. I’m getting near identical performance as I do in Windows (non-virtualized). There is however one issue, remember how I said I moved the GPU to the PCIe slot running in x4 mode? Yeah, in Hellblade I get around 50-70 FPS (even in non-virtualized Windows – since the GPU is still in the x4 slot) which is still far below what I had in Windows at the start of my adventure. Time to give up and just accept the life of dual-booting, right? NOPE! After some research it seems that the Asus Prime X370 Pro has very nice IOMMU groups as well as PCIe x8 – x8 with dual GPUs, so naturally I’ve ordered one of those, it should probably arrive at the end of the week or next week. I’ll post an update once I’ve gotten it set up and share any performance differences.
I tried Killing Floor 2 too by the way, it ran just fine but the lack of my custom input settings made me put playing off until I’ve migrated my save/config files from my non-virtualized Windows.
4.2.8. That’s cool and all, but manually changing my monitor input between the two sources is a pain in the ass, the OSD for my monitor sucks!
I AGREE! And this is where I start to see a point in Looking Glass after all. Buuuut – What if you could change monitor input through Linux using a keyboard shortcut? Well, I looked into it, and it’s possible! Although there are some requirements. You need a monitor that supports DDC/CI which my BenQ XL2420T (I think that’s the right model) does. You also need to load a necessary module. I’ll go through the setup here:
First of all you want to check if your monitor supports DDC/CI, I’ll leave how to do that up to you, I just checked my monitor OSD and found a setting for it and it was set to enabled.
After that I’d just follow this guide for the general setup, it worked fine for me for the initial setup: https://passthroughpo.st/switching-monitor-inputs-software-ddc/
The guide above goes through how to set up hooks to run it automatically at start and stop of the VM, I do a lot of switching between the host and guest without shutting down though, so I wanted a keyboard shortcut to do this. Since the command requires the use of sudo we can’t just add a shortcut right away, instead it was a pain in the ass to set up but I’ll go through it.
First of all we want to make a script:
1. sudo nano /usr/local/bin/change-monitor-input
Now enter the script, my script looks like this:
#!/bin/bash
if test "$(id -u)" -ne 0 ; then
sudo "$0" "$1"
exit $?
fi
if [ $1 == "DisplayPort" ]; then
ddcutil -d 1 setvcp 60 0x0f
fi
if [ $1 == "DVI" ]; then
ddcutil -d 1 setvcp 60 0x03
fi
Now, I don’t know exactly what the first part does, but I think it tries to run itself as sudo in case it wasn’t run as sudo, that’s what we want at least. The second and third part however check if the argument passed to the script are either DisplayPort or DVI (since those are the inputs I use) and if either is correct then it calls ddcutil with what is for me the correct arguments and values, yours would probably vary from mine.
After that we want to make sure root is the owner of this file:
2. sudo chown root:root /usr/local/bin/change-monitor-input
And then we want to make sure everyone can execute it but only root can write to it (so no other less-privileged user can change the script and then execute it as root):
3. sudo chmod 755 /usr/local/bin/change-monitor-input
At this point I’d test the script just to make sure it works, for example by running sudo change-monitor-input DisplayPort in my case, you’ll need to enter a password of course.
Warning – the following instructions could really screw things up if done wrong or if my instructions are wrong – Strongly recommended to back things up and have a live cd ready in case you need it – Yes I screwed it up when I did it and it was a PITA to fix it.
What you want to do now is update the sudoers file to allow your user to run that script in particular as root but without a password. Now, on Manjaro at least, there is a file called 10-installers in /etc/sudoers.d/ that will cause problems for us so we’re going to have to deal with that, but it’s important that you follow these instructions in ORDER:
1. sudo EDITOR=nano visudo
What we want to do here is find the line:
#%wheel ALL=(ALL) ALL
and edit it to
%wheel ALL=(ALL) ALL
This is the same line that is otherwise located in the 10-installers file, you can double-check this to make sure.
Now BELOW the %wheel line, you want to add this line:
user ALL=(root) NOPASSWD: /usr/local/bin/change-monitor-input
Make sure to replace "user" with your actual user, for example I wrote sanya in mine – Save the file and close it.
Now to delete the 10-installer file that will otherwise cause issues, at least it did for me:
2. sudo rm /etc/sudoers.d/10-installer
After all that, set up some keyboard shortcuts using whatever tools you have with your DE to execute the command change-monitor-input DisplayPort etc.
I should probably mention that trying to use ddcutil to change to an input with no signal causes my BenQ monitor to freeze up and I’ll have to shut it off and on in order to fix it... So.. beware.
5. How about you?
What are your experiences with gaming on Linux? Had any luck with Steam Proton or Wine? Tried GPU pass-through yourself? Feel free to share any tips you might have. Also feel free to share any thoughts or feedback you might have on this post.
I'm personally looking forward to trying out the new motherboard, my only fear is that PCI x8 mode won't be enough for future more powerful graphics cards, so I'm not sure how long this specific motherboard will last for my purposes, but we'll see.