Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

vgpu_unlock not working with 1050Ti as 2 x Tesla P40 #75

Open
nworbneb opened this issue Oct 5, 2021 · 8 comments
Open

vgpu_unlock not working with 1050Ti as 2 x Tesla P40 #75

nworbneb opened this issue Oct 5, 2021 · 8 comments

Comments

@nworbneb
Copy link

nworbneb commented Oct 5, 2021 •
edited
Loading

Many thanks for developing this script. I'm trying to use a 4Gb 1050Ti and passthrough as 2 x 2Gb Tesla P40's. I'm using Proxmox 7.0-11. I've previously passed through the whole card to a Windows 10 VM successfully.

NVIDIA drivers seem to be installed OK:

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.32.04    Driver Version: 460.32.04    CUDA Version: N/A      |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 105...  On   | 00000000:04:00.0 Off |                  N/A |
|  0%   36C    P8    N/A /  72W |   2046MiB /  4095MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

I've defined two vGPU:

ed49b5c8-8a6c-4621-98e5-fac5204b9c99 0000:04:00.0 nvidia-47 (defined)
a4cdce3c-5631-47bf-b9c0-89fa14f02fe3 0000:04:00.0 nvidia-47 (defined)

Proxmox QEMU VM is configured:

agent: 1
args: -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/ed49b5c8-8a6c-4621-98e5-fac5204b9c99,display=off,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-pci-vendor-id=0x10de,x-pci-device-id=0x1b38,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x11a0' -uuid ed49b5c8-8a6c-4621-98e5-fac5204b9c99
bios: ovmf
boot: order=virtio0;net0;ide0;ide2
cores: 8
cpu: host
ide2: local:iso/virtio-win-0.1.196.iso,media=cdrom,size=486642K
machine: pc-q35-6.0
memory: 16384
name: retroemu.pve.xarin.com
net0: virtio=F6:08:58:72:87:9B,bridge=vmbr0,firewall=1
numa: 0
ostype: win10
scsihw: virtio-scsi-pci
smbios1: uuid=a091ae9f-ba27-4f3b-be09-84af7950be40
sockets: 1
vga: none
virtio0: local-zfs:base-150-disk-0/vm-153-disk-0,cache=writeback,discard=on,size=100G
vmgenid: ed49b5c8-8a6c-4621-98e5-fac5204b9c99

It looks like the NVIDIA driver is ok?

Oct 05 17:08:59 pve nvidia-vgpu-mgr[389029]: vgpu_unlock loaded.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: vgpu_unlock loaded.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_env_log: vmiop-env: guest_max_gpfn:0x0
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_env_log: (0x0): Received start call from nvidia-vgpu-vfio module: mdev uuid ed49b5c8-8a6c-4621-98e5-fac5204b9c99 GPU PCI id 00:04:00.0 config params vgpu_type_id=47
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_env_log: (0x0): pluginconfig: vgpu_type_id=47
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_env_log: Successfully updated env symbols!
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: op_type: 0x20801322 failed.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: op_type: 0x2080014b failed.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: op_type: 0xa0810115 failed.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): gpu-pci-id : 0x400
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): vgpu_type : Quadro
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): Framebuffer: 0x74000000
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): Virtual Device Id: 0x1b38:0x11e9
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): FRL Value: 60 FPS
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: ######## vGPU Manager Information: ########
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: Driver Version: 460.32.04
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: op_type: 0x2080012f failed.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): Cannot query ECC status. vGPU ECC support will be disabled.
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): Init frame copy engine: syncing...
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: (0x0): vGPU migration disabled
Oct 05 17:08:59 pve nvidia-vgpu-mgr[389041]: notice: vmiop_log: display_init inst: 0 successful

But when I start the VM I get an errror and in the VM there is no NVIDIA device visible.

root@pve:~# qm start 153
no efidisk configured! Using temporary efivars disk.
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/ed49b5c8-8a6c-4621-98e5-fac5204b9c99,display=off,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-pci-vendor-id=0x10de,x-pci-device-id=0x1b38,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x11a0: warning: vfio ed49b5c8-8a6c-4621-98e5-fac5204b9c99: Could not enable error recovery for the device

Any suggestions would be appreciated.
@DualCoder
Copy link
Owner

For some reason your VM's UUID equals the vGPU's UUID. This should not happen and might be the cause of the issue. You should pass the UUID of the vGPU as the -device argument and the UUID of the VM (system) to the -uuid argument.

Example from https://docs.nvidia.com/grid/12.0/grid-vgpu-user-guide/index.html#adding-vgpu-to-red-hat-el-kvm-vm-qemu-cli

-device vfio-pci,sysfsdev=/sys/bus/mdev/devices/aa618089-8b16-4d01-a136-25a0f3c73123 \
-uuid ebb10a6e-7ac9-49aa-af92-f56bb8c65893

@nworbneb
Copy link
Author

nworbneb commented Oct 6, 2021

Thank you for your assistance. I tried using a different UUID (ebb10a6e-7ac9-49aa-af92-f56bb8c65893 from the example) and the UUID from the PVE vmgenid (ed49b5c8-8a6c-4621-98e5-fac5204b9c99) but neither made any difference.

I still get the "Could not enable error recovery for the device" error when starting the VM and I don't have any NVIDIA devices in the device manager in Win10. I can only connect to the VM with RDP, Parsec shows a black screen (to be expected with no GPU)

I tried removing the UUID argument entirely and i get:

root@pve:~# qm start 153
no efidisk configured! Using temporary efivars disk.
kvm: -device vfio-pci,sysfsdev=/sys/bus/mdev/devices/ed49b5c8-8a6c-4621-98e5-fac5204b9c99,display=off,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0,x-pci-vendor-id=0x10de,x-pci-device-id=0x1b38,x-pci-sub-vendor-id=0x10de,x-pci-sub-device-id=0x11a0: vfio ed49b5c8-8a6c-4621-98e5-fac5204b9c99: error getting device from group 61: Connection timed out
Verify all devices in group 61 are bound to vfio-<bus> or pci-stub and not already in use
start failed: QEMU exited with code 1

Group 61 is the device assigned to the vGPU:

/sys/kernel/iommu_groups/60/devices/a4cdce3c-5631-47bf-b9c0-89fa14f02fe3
/sys/kernel/iommu_groups/61/devices/ed49b5c8-8a6c-4621-98e5-fac5204b9c99

I also tried removing the PCI spoofing:
args: -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/ed49b5c8-8a6c-4621-98e5-fac5204b9c99' -uuid 00000000-0000-0000-0000-000000000153
but same status. VM boots into Win10 but does not show any NVIDIA hardware in device manager.

@mcdull
Copy link

mcdull commented Oct 12, 2021 •
edited
Loading

I am also having the same issue. And my config and observation below.
1 GPU nvidia 1080 for passthrough (to a single VM)
1 GPU nvidia 1050 Ti 4GB and passing through as 2 x P40 (P6000 driver being used)

I can see the device in windows guest, and parsec can ocassionally see one or two frames and hang.
If I uses RDP to connect, the GUI will freeze every few seconds and system is not responsive.

I see no vgpu unlock message in dmesg

@mcdull
Copy link

mcdull commented Oct 12, 2021

Oh yes, make sure you use the 5.12 patches.

@mcdull
Copy link

mcdull commented Nov 14, 2021

It seems to work fine on my system. Make sure it is using old version supported drivers. Not the most updated one. For me i am using 460

@jiangcuo
Copy link

jiangcuo commented Jan 20, 2022 •
edited
Loading

try to use proxmox 6.4.

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 460.73.01    Driver Version: 460.73.01    CUDA Version: 11.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  GeForce GTX 1050    On   | 00000000:01:00.0 Off |                  N/A |
|  0%   46C    P8    N/A /  70W |   2039MiB /  2045MiB |      3%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      6661    C+G   vgpu                             1014MiB |
|    0   N/A  N/A      6735    C+G   vgpu                             1014MiB |
+-----------------------------------------------------------------------------+

gtx1050 is ok

agent: 1
args: -device 'vfio-pci,sysfsdev=/sys/bus/mdev/devices/d1bdac94-7044-11ec-90d6-0242ac120003,display=off,id=hostpci0.0,bus=ich9-pcie-port-1,addr=0x0.0' -uuid d1bdac94-7044-11ec-90d6-0242ac120003
audio0: device=ich9-intel-hda,driver=none
boot: order=scsi0;ide2;net0
cores: 8
cpu: host
ide2: none,media=cdrom
machine: pc-q35-5.2
memory: 12288
name: Windows2
net0: virtio=AC:BD:EF:9D:38:14,bridge=vmbr0
numa: 0
ostype: win10
scsi0: Nvme:9997/base-9997-disk-0.qcow2/102/vm-102-disk-0.qcow2,discard=on,size=256G,ssd=1
scsihw: virtio-scsi-single
smbios1: uuid=0ecd7afc-f508-4958-9e4b-cfbedabf4f25
sockets: 1
vga: none
vmgenid: 76952324-df1a-4cd9-ae15-6fe51289c686

pls make sure two uuid is same one.
and add efi disk when you use ovmf.

@dpawsbear
Copy link

hello, can you tell me which version of you use vgpu driver and you guest virtual machine(win10) driver version?

@mcdull
Copy link

mcdull commented Feb 24, 2022

460.73.01 merged driver
Same 460 series driver for windows client.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mcdull @DualCoder @dpawsbear @nworbneb @jiangcuo