Friday, March 30, 2012

VGA passthrough with KVM on Fedora 16

I've succesfully managed to pass my Radeon HD6770 graphics card to a virtual machine using KVM on Fedora 16.

UPDATE: Please check newer post for updated informations. (See KVM page ->)

http://www.youtube.com/watch?v=E4rF9JRVZuY


I've been trying to do this for a long while now and this is the first time it worked well enough to start doing interesting stuff (eg Heaven Benchmark, Age of Empire, Halo, I'll try Battlefield 3 in the next few days). I've tryed both with kvm and xen, but xen is in my oppinion too difficult to setup, al least for me and I haven't managed to get it working reliable (al least not yet).

My "compute" PC:

  • Intel i7-2600 non-K processor, this one has VT-x and VT-d - any processor with VT-d Intel / IOMMU AMD should work though. This is very important!
  • ASRock H67M-GE motherboard with VT-d support, I went with ASRock because it was written in black and white on their website, in the motherboard bios manual that it explicity supported VT-d , not a generic Intel virtualization option, or VT-x as other motherboars. Again this is very important!
  • 8GB of DDR3 RAM, the more the better
  • VTX3D Radeon HD6770 1GB DDR5 128-bit
  • 120GB A-DATA S510 SSD
  • Sirtec 500W power supply

This is how I made it work (I used Fedora 16, so I don't guarante this will work on other distros too) (Please read a lot more before you try this, and read this post until the last . before you try anything, I won't be responsible if you're system breaks, BACK UP, if you're doing this on a machine with valuable data):

  1. I installed Fedora 16, the 64 bit kind, from  DVD, minimal setup, no desktop, as this will also serve as my "compute" (eg programming, testing, dev, server, all in one torture)  pc, as well as my gaming rig. I went with no swap partition and a LVM setup where I reserved a 50GB for the vm and the rest to /. Notice I didn't create any user accounts so I used root.
  2. I updated Fedora to have the latest packeges and kernel and rebooted.
  3. I setup ssh acces and installed the virtualization related packeges:
    1. # yum install @virtualization
    2. # service libvirtd start
  4. For Intel cpu's you have to specifiy a boot parameter;
    1.  Append "intel_iommu=on" to /etc/defaut/grub GRUB_CMDLINE_LINUX.
    2. Updated grub 
      1. # grub2-mkconfig -o /boot/grub2/grub.cfg
    3. Reboot
  5. Follow these instructions, for pci passthrough troubleshooting: https://fedoraproject.org/wiki/How_to_debug_Virtualization_problems#PCI_device_assignment 
  6. I also blacklisted the radeon driver - requiers reboot or 
    1. # modeprobe -r radeon
  7. Installing the vm
    1. I used virt-manager new virtual machine dialog, and used the 50GB lvm as raw storage and pointed the way to the Windows 7 install DVD. Do NOT try to add/modify any other options, unless you really know what you are doing. ( in my case adding USB devices eg keyboard and mice would result in system frezes -  you should try to add keyboard and mouse support to the VM if possible, just keep in mind that if windows is loading slow it might be from that; or when I activated APIC or ACPI,  after I instaled graphics card driver, Windows wouldn't start because of BSOD)
    2. I installed Windows 7 x64
    3. I added the graphics card and HDMI audio of the card using virt-manager to my vm, I let Windows install the driver, rebooted and well benchmarked it.


I've played some games using the vm, and well it was ok, I couldn't say that it wasn't a physical machine, but it's stil not perfect. What I'd like to do is to remote acces the vm and play the games, a bit like OnLive, but sofar SPICE/VNC from virt-manager go dark when I attach the graphic card, I cand stil send keyboard and mouse input to the vm but the image frezes - and I think that this is just how they were supposed to work; Windows RDP no go, AOE3 won't even start; standard VNC and TeamViewer aren't fast enough for high quality, skipped frames and things like that. If I won't find something suitable, I'll code it myself, this is probably be my 2012 big summer project.

I've experieced some bugs, like the fact that sometimes, when I'd shut down the vm, the amount of ram allocated to it wouldn't be freed, i had either to restart libvirtd, kill some programs, or reboot. Sometimes virt-manager would crash.

And i'm a bit dissapointed by the networking capabilities of KVM, I did try the latest virtIo drivers for Windows ( http://alt.fedoraproject.org/pub/alt/virtio-win/latest/images/bin/ ) but they weren't as fast at NAT (x10-100 slower!!!), and I'd like/need to make the vm a part of my home network, this is something i'll have to investigate more.

THERE IS NOT A UNIFIED UPTODATE KVM USER MANUAL. The best I could find was a 2010 German wikibooks website, this is where I really miss Virtual Box, it has the best documentation out there (at least comparing with KVM and XEN) for virtualization programs.


27 comments:

  1. Very cool! Xen seems to offer near-native performance in a few examples I've seen. The trickiest part is finding information on it, but it is straightforward to set up if you're used to some digging around and problem solving. It would be interesting to see you do a comparison of the two on the same machine, to help better understand which areas KVM needs work in.

    ReplyDelete
  2. This is beyond cool :)

    I have been looking for a sign --any sign-- that VGA passthrough works under KVM.

    Now, if someone translated what you did to Debian-derivative --I guess the steps are similar but I am not that good, yet.

    I have just bookmarked this page and will come back periodically to check if you've made any further progress (or even did an install on a Debian-derivative).

    Thank you for sharing this --that there's hope :)

    ReplyDelete
  3. Hello,
    Great post! Thanks very much for providing this information. I've setup a similar config, except with an AMD Phenom X6 and ASUS M5A99X EVO.

    Unfortunately I have a performance problem as of the two games I've tried testing on:
    * 'killing floor' is far slower and jerky compared to non virtualised.
    * L4D2 crashes before getting to the game menu.

    IOMMU is definitely working, as the passed through card gives me DVI out, and I'm able to play killing floor at very low settings, but passing through either my HD6850 or HD6450 gives no difference in performance.

    I also get a bunch of IRQ MSI messages in the /var/log/messages. Do you get anything similar?

    Cheers, Evan.

    ReplyDelete
  4. Hi, I have been quite busy with school (i'm a 1st year undergraduate Computer Science) and as souch I haven't had enough time to try a lot of games, but in my case old games like AOE III, Halo CE, they had no problem running in the vm, I did try to play Battlefield 3, although it started fine, it got stucked in a loop at the beginig of single player.

    Performance wise, the Heaven benchmark fps from the vm with ATI HD6770 are a bit higher than what I got when I first tested Windows 7 native, but that was marginal and I think is due to newer drivers.

    I do plan to do a lot more testing and a proper benchmark, native Win7 gaming vs KVM and/or Xen VM VGA Passthrough gaming with more games on Fedora 17 and/or Ubuntu 12.04 host in about 4-5 weeks from now.

    ReplyDelete
  5. Great job!

    I have an Evga SR-2 board with dual CPUs (24 cores), 2 Radeon 6990 (4 GPUs), 3 Dell U3011 displays, obscene amounts of RAM and disk...

    I have been waiting for an opportunity to turn this into a Linux box (Ubuntu) with KVM + VGA Passthrough so that I can do development in near-enough native speeds.

    I'll be among the first ones to try out your configs and benchmarks.

    ReplyDelete
  6. Nice tuto. Gonna try this one in the coming days with an nvidia. Always dreamed to get better perfs in a VM.

    Btw, if you wanna do remote gaming, give a try to http://www.splashtop.com/home Using it for months to play from my android tablet and this remote control application is really really awesome for gaming or video streaming

    ReplyDelete
  7. Thanks for sharing this info!

    I'm really interested in how it works for you on Ubuntu 12.04. I've seen other Fedora reports with Xen but never with KVM. I thought the KVM team dropped VGA passthrough in favor of Spice, which is something altogether different.

    Good luck!

    ReplyDelete
  8. Cool, You did it.
    Some time ago I've tried with HD 4890 and failed..
    Any advices?
    http://toomeek.waw.pl/blog/?p=764

    I see You've also redirected video to external monitor.. ?
    For best networking performance You need something like PC-over-IP protocol, or very good video codec for VNC.

    ps. Google redirected me to Your YT video :D

    ReplyDelete
  9. TooMeeK, I'm not familiar with Scientific Linux, it's a RHEL/CentOS clone right? I'd guess first of all you should try something newer if you can, 2.6.32 is old, for me switching from Fedora 16 (kernel 3.3.0) to Fedora 17 (kernel 3.3.4 + updated programs) solved the last few bugs I had. Also make sure there isn't anything that uses or owns the video card before you try to attach it to the VM, especially the video card driver.

    ReplyDelete
  10. Yep, CERN is CentOS clone, but it's not exacly Scientific Linux, it's another distro ;)
    So, I'm gonna try with Fedora, I have additional GF 210 card, so maybe will work as primary video and will try to assign HD4890 to VM..
    Just tried that (unbind VGA card), however it didn't worked. My keyboard LEDs was all blinking together.

    ReplyDelete
  11. One more question: Did You succeed with ONE VGA in host system? Or used two of them? Radeon and something else, like on-board VGA?
    Your screen should goes to black if it was single VGA.

    ReplyDelete
  12. I have an Intel HD3000 integrated video from the cpu, and an AMD HD6770 that I pass to the VM. This machine for me is a headless VM server so when I connect a keyboard, mouse and monitor to it, I play with the VGA passthrough VM, for me each of the video cards displays something when the VM is on, the HD3000 shows the login console for Fedora, and the HD6770 shows the Windows VM desktop, games, applications. Also in the UEFI I've selected the onboard VGA (Intel HD3000) as primary VGA.

    ReplyDelete
    Replies
    1. I did not succeed with Fedora. Will try later with other OS/BIOS combination.
      Maybe my VGA card is too old, so will try with my SAN HBA also.

      Delete
  13. Teradici PCoIP would probably support the 3d graphics remotely

    ReplyDelete
  14. Hey,
    I was able to set it all up to a point on linux mint 13. Thing is - after VM with attached vga card is booted - monitor goes black (power saving mode) and i cant see a thing. Any idea what could be causing it? I attached two pci devices:
    01:00.0 VGA compatible controller: NVIDIA Corporation GF116 [GeForce GTX 550 Ti] (rev a1)
    01:00.1 Audio device: NVIDIA Corporation GF116 High Definition Audio Controller (rev a1)
    Attaching only vga caused error - complaining about 01:00.1 being in-use.
    I also have on-board intel graphics card. Tried to boot using that - vm could not start either. Too bad i did not copy error message.

    ReplyDelete
  15. When you tried to boot the vm with the Nvidia card, was the monitor connected to it, to the VGA/HDMI whatever of the Nvidia card?
    Unfortunately I don't know if this will work with KVM as easy as the AMD card I use. I don't have a proper Nvidia card to test. But let's see if we can make it work ;).
    If you could make sure IOMMU and Passthrough are fully enabled first of all (what processor do you have, Intel or Amd? ). Then if you could post the log of the vm after you try to boot it with the Nvidia card. Also in the BIOS/UEFI what's the primary graphic card? onboard or pci express?

    ReplyDelete
  16. Yes, my monitor was connected to nvidia video card when i tried to boot VM with those pci devices passed to it. I use nvidia GPU as primary. There also is some lame intel onboard vga card. Setting that as primary in UEFI, booting OS and trying to launch VM gave me error

    My environment:
    CPU intel core i5 x64 (quad-core) (supports vt-x and vt-d),
    Motherboard supports those too (since UEFI has settings to turn them on)

    rndbit-desktop ~ # kvm-ok
    INFO: /dev/kvm exists
    KVM acceleration can be used

    rndbit-desktop ~ # dmesg | grep IOMMU
    [ 0.000000] Intel-IOMMU: enabled
    [ 22.230870] vboxpci: IOMMU not found (not registered)

    VirtualBox (i have it installed) one must not be important i guess..

    VM log file (assuming it is /var/log/libvirt/qemu/WinServer2008.log, WinServer2008 is VM name ofc ) is completely empty.

    I tried to boot using on-board vga card and launch VM with attached pci devices. This is result:
    Error starting domain: Unable to read from monitor: Connection reset by peer
    Traceback (most recent call last):
    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 45, in cb_wrapper
    callback(asyncjob, *args, **kwargs)
    File "/usr/share/virt-manager/virtManager/asyncjob.py", line 66, in tmpcb
    callback(*args, **kwargs)
    File "/usr/share/virt-manager/virtManager/domain.py", line 1120, in startup
    self._backend.create()
    File "/usr/lib/python2.7/dist-packages/libvirt.py", line 551, in create
    if ret == -1: raise libvirtError ('virDomainCreate() failed', dom=self)
    libvirtError: Unable to read from monitor: Connection reset by peer

    lspci still lists those pci devices tho.


    What i should expect here? Can attached pci video card to VM still provide services to host? It would be rather unconvenient if attached pci device gets to be exclusive to VM, which then would mean i need two separate video cards, for vm and for host.

    ReplyDelete
  17. What about dmesg | grep -e DMAR -e IOMMU ? It has to show both IOMMU and DMAR stuff. What OS do you use? Grub2 or Grub legacy?

    Can you try the following things:
    - put Intel Integrated video as primary in the UEFI
    - uninstall or blacklist and unload any driver for the nvidia card
    - now try to boot the vm with the nvidia card attached

    It's very strange that the log is empty, it should always record something, at least the startup command for the vm. If you boot the vm without any video card attached, does the log contain something?

    The easy way to passthrough the video card to a vm is to have a secondary video card just for the vm, and a primary (like the intel integrated one that should be more than enough for day to day activities) for the base OS. But you could also try to use the same video card for the base OS and the vm, that's a little more difficult to setup because you have to unbind it from the OS first and then pass it to the vm.

    XEN might work better with nvidia cards than KVM
    http://wiki.xen.org/wiki/Xen_VGA_Passthrough

    ReplyDelete
  18. Could you expand on point 6 please - how do you blacklist the video card?

    ReplyDelete
    Replies
    1. Append blacklist radeon to /etc/modprobe.d/blacklist.conf

      Delete
  19. hello, i also have been experimenting with kvm and vga passthrough, i was able to get my 7870 sapphire card working http://thread.gmane.org/gmane.comp.emulators.kvm.devel/94524

    but i suffer from bluescreens on the reboot of the vm, do you also have this issue or could this be my videocard?

    ReplyDelete
  20. omg. Okay I think I'll do a real write up on this shortly.

    Long short, KVM works with most ATI cards pretty well, out of the box with Ubuntu/Debian/Mint/whatever (At the time of this writing, I'm running Mint13 - Cinnamon).

    Login as root:
    sudo su -
    #You'll definitely need this.
    apt-get install virt-manager

    #May need this. I needed it for one card but not for another.
    echo "options kvm allow_unsafe_assigned_interrupts=1" > /etc/modprobe.d/kvm.conf

    #Most definitely need this.
    echo "pci-stub" >> /etc/modules
    #And this.
    usermod -a -G libvirtd yourusername #Replace with your username.

    Then, really, all you have to do is startup virt-manager, create a new guest (I've used Win7/8 both successfully), shut down the guest and add your PCI device via the virt-manager interface.
    There are a TONNE of old instructions on the Internets and most of them are massively outdated. There really isn't any more to this than what I've just mentioned.

    There are of course some caveats. Some systems may require you to append "iommu=pt iommu=1" to your grub command line like so:

    nano /etc/default/grub
    #find the line that says GRUB_CMDLINE_LINUX="" and make it look like this:
    GRUB_CMDLINE_LINUX="iommu=pt iommu=1"

    I do not need that line with my system (Sabertooth 990fx, AMD 8120fx 8core)

    My Radeon 67xx worked perfect. My RadeonHD 9800 works, but when the guest reboots, the host crashes.

    Long short, older ATI cards seem to work very well with very little configuration.

    ReplyDelete
  21. Nvidia is unsupported except for Quadro cards. I've never got one to work with KVM though with a tonne of work and many angry all-nighters, I've got an 8800gts working poorly under Xen.
    I gave up for now and bought an ATI card for this purpose. Any ATI card 6xxx or lower will work with KVM very easily with no special configuration. KVM has an issue with rebooting guests attached to 7xxx series cards (host crashes with kernel bug at drivers/pci/ats.c:62, but I'm working on this).

    7xxx series cards work perfect with Xen 4.1 and later, Xen's just a little scattered right now so it takes some work to setup. All you REALLY need to know is that with Xen, you must use the "xen-pciback" driver to bind your card to Xen guest. This can be done by adding "xen-pciback.passthrough=1 xen-pciback.hide=(01:00.0)(01:00.1)" to the XEN kernel at boot. Note, this is not the same as adding to your regular kernel command line. Xen has a special Dom0 kernel that is also added to grub.

    I'm doing some experimenting right now but once I'm done I'll have writeups for several cards on both Xen and KVM, and possibly some .deb packages to automate things.

    ReplyDelete
  22. Also, in regards to the OP's mention of Network speed, try just using a bridged connection. I'm getting full duplex GB without issue.
    Just setup a bridge to wherever and add the device as "shared device" to your guest.

    ReplyDelete
  23. Network speed is not a problem anymore, I've done just that ;)

    ReplyDelete
  24. Question - does the vga passthrough only allow one active vm? So if I wanted to run win7 and linux vms at the same time I could not do this?

    ReplyDelete
    Replies
    1. You can and run as many VMs as you want as long as you have enough system resources for all of them + physical machine and you don't have 2 VMs that try to access th same passthrugh hardware.

      Delete

I reserve the right to delete at any time any comment I deem inappropiate.