Support libkrun (mostly for Virtio-GPU Venus), maybe via krunkit

[AkihiroSuda]

libkrun seems to have merged the PR for supporting Virtio-GPU Venus (https://docs.mesa3d.org/drivers/venus.html)

• Extend virtio-gpu to support Venus on macOS containers/libkrun#174

QEMU still hasn't merged the patches for supporting Venus: https://patchew.org/Xen/20231219075320.165227-1-ray.huang@amd.com/

[AkihiroSuda]

lima should probably use https://github.com/containers/krunkit for implementing the libkrun driver.
Not sure if it should be a built-in driver, or we have to implement the plug-in system.

[balajiv113]

We can explore and finalise the approach for plug-in model. As i already see several experimental PR's for supporting different drivers.

This will help us in providing different options based on performance and usecases.

[nirs]

See minikube krunkit driver:
kubernetes/minikube#20826

[afbjorklund]

Would make it more sense to use the libkrun API instead of the "krunkit" CLI, like with the VZ API instead of the "vfkit" CLI?

https://github.com/containers/libkrun

[AkihiroSuda]

Would make it more sense to use the libkrun API instead of the "krunkit" CLI, like with the VZ API instead of the "vfkit" CLI?

https://github.com/containers/libkrun

That should be also fine after finalizing the driver plugin gRPC, as libkrun uses Rust, not Go.

[afbjorklund]

The library is in C (i.e. the ABI is), so the Rust CLI wraps it in "unsafe"

https://github.com/containers/libkrun/blob/main/include/libkrun.h

Something like this code, which would work equally fine in Go too:

https://github.com/containers/krunkit/blob/main/src/context.rs

[afbjorklund]

One benefit of using libkrun, is that it also supports Linux (not only macOS)

This is good, so that it can be tested also on developer machines (unlike Vz)

---

Made a sample Go wrapper, found out that cwrap doesn't support C99 bool...

package main

import (
	"h12.io/cwrap"
)

var krun = &cwrap.Package{
	PacName: "krun",
	PacPath: "go-krun",
	From: cwrap.Header{
		Dir:           "/usr/local/include",
		File:          "libkrun.h",
		NamePattern:   `\Akrun(.*)`,
		CgoDirectives: []string{"pkg-config: libkrun"},
	},
}

func main() {
	err := krun.Wrap()
	if err != nil {
		panic(err)
	}
}

Then again, it was not impossible to fix. I guess real C programmers use int?

package krun // import "github.com/containers/libkrun/go-krun"

func AddDisk(ctxId uint32, blockId string, diskPath string, readOnly bool) (ret int32)
func AddDisk2(ctxId uint32, blockId string, diskPath string, diskFormat uint32, ...) (ret int32)
func AddVirtiofs(ctxId uint32, cTag string, cPath string) (ret int32)
func AddVirtiofs2(ctxId uint32, cTag string, cPath string, shmSize int) (ret int32)
func AddVsockPort(ctxId uint32, port uint32, cFilepath string) (ret int32)
func AddVsockPort2(ctxId uint32, port uint32, cFilepath string, listen bool) (ret int32)
func CheckNestedVirt() (ret int32)
func CreateCtx() (ret int32)
func FreeCtx(ctxId uint32) (ret int32)
func GetShutdownEventfd(ctxId uint32) (ret int32)
func SetConsoleOutput(ctxId uint32, cFilepath string) (ret int32)
func SetDataDisk(ctxId uint32, diskPath string) (ret int32)
func SetEnv(ctxId uint32, envp []string) (ret int32)
func SetExec(ctxId uint32, execPath string, argv []string, envp []string) (ret int32)
func SetGpuOptions(ctxId uint32, virglFlags uint32) (ret int32)
func SetGpuOptions2(ctxId uint32, virglFlags uint32, shmSize int) (ret int32)
func SetGvproxyPath(ctxId uint32, cPath string) (ret int32)
func SetKernel(ctxId uint32, kernelPath string, kernelFormat uint32, initramfs string, ...) (ret int32)
func SetLogLevel(level uint32) (ret int32)
func SetMappedVolumes(ctxId uint32, mappedVolumes []string) (ret int32)
func SetNestedVirt(ctxId uint32, enabled bool) (ret int32)
func SetNetMac(ctxId uint32, cMac *byte) (ret int32)
func SetPasstFd(ctxId uint32, fd int32) (ret int32)
func SetPortMap(ctxId uint32, portMap []string) (ret int32)
func SetRlimits(ctxId uint32, rlimits []string) (ret int32)
func SetRoot(ctxId uint32, rootPath string) (ret int32)
func SetRootDisk(ctxId uint32, diskPath string) (ret int32)
func SetSmbiosOemStrings(ctxId uint32, oemStrings []string) (ret int32)
func SetSndDevice(ctxId uint32, enable bool) (ret int32)
func SetTeeConfigFile(ctxId uint32, filepath string) (ret int32)
func SetVmConfig(ctxId uint32, numVcpus byte, ramMib uint32) (ret int32)
func SetWorkdir(ctxId uint32, workdirPath string) (ret int32)
func Setgid(ctxId uint32, gid uint32) (ret int32)
func Setuid(ctxId uint32, uid uint32) (ret int32)
func SplitIrqchip(ctxId uint32, enable bool) (ret int32)
func StartEnter(ctxId uint32) (ret int32)

The documentation didn't make it either, so have to use libkrun.h for that.

---

EDIT: When comparing the auto-generated code, seems int was confused as well.

i.e. C uint64_t translated to Go int, but C int translated to Go int32. Oh well.

[nirs]

One benefit of using libkrun, is that it also supports Linux (not only macOS)

libkrun on Linux is not the same as libkrun-efi for macOS. On Linux it is meant to run single application with better isolation. On macOS it can be used like vfkit to run standard vms.

So I think this will be a macOS only solution like vz and socket_vmnet.

[afbjorklund]

Yeah, I guess you would have to use external EFI - like you have to with QEMU

"libkrun-efi: Variant that bundles OVMF/EDK2 for booting a distribution-provided kernel"

Not sure about the single app though, the example runs a Fedora chroot shell

./chroot_vm ./rootfs_fedora /bin/sh

[afbjorklund]

So I think this will be a macOS only solution like vz and socket_vmnet.

Makes sense, I think there are other drivers with better support for KVM

QEMU with VirtIO GPU Vulkan Support (qemu virtio-gpu virglrenderer)

https://gist.github.com/peppergrayxyz/fdc9042760273d137dddd3e97034385f

[afbjorklund]

Added the Go version here, with the basic libkrun wrapper and chroot_vm example:

https://github.com/afbjorklund/go-libkrun

https://github.com/containers/libkrun#chroot_vm

Fixed the go-cwrap program as per above, to handle stdbool and to not filter out int.