Experiencing slow network under VZ virtualization

[terev]

Description

Lima: HEAD-0da3240
Colima: version=HEAD-cf522e8,driver=vz,mounts=virtiofs
MacOS: Ventura 13.1

I'm running HEAD of lima and colima and I think I'm still seeing something similar to this issue #1285 frequently . The most often occurrence of this issue for me is when pulling many docker images while starting a docker compose project. The command hangs and sometimes times out producing a message like:

Error response from daemon: Head "********************": dial tcp: lookup gcr.io on 192.168.5.3:53: read udp 192.168.5.15:40202->192.168.5.3:53: i/o timeout
Error: exit status 18

In the host agent logs ha.stderr.log I see many of the following message:
{"level":"debug","msg":"Stopping udp proxy (read udp 192.168.5.2:58193: i/o timeout)","time":"2023-01-26T01:12:39-05:00"}

[jandubois]

Do you have repro instructions you can share, e.g. a docker-compose file using only public images?

Is the problem specific to vz, or does it also happen with qemu?

[terev]

I'm only able to reproduce on vz . It seems to successfully respond to queries for a while then the Stopping udp proxy messages start showing up. Additionally when stopping this vm the command hangs until terminated and logs:

panic: runtime/cgo: misuse of an invalid Handle

goroutine 17 [running, locked to thread]:
runtime/cgo.Handle.Value(...)
	/opt/homebrew/Cellar/go/1.19.5/libexec/src/runtime/cgo/handle.go:124
github.com/Code-Hex/vz/v3.changeStateOnObserver(0x0, 0x14000002680?)
	/Users/trevorfoster/Library/Caches/Homebrew/go_mod_cache/pkg/mod/github.com/!code-!hex/vz/v3@v3.0.4/virtualization.go:160 +0xec

The command hangs but when querying for the status of the vm it thinks it's stopped.

I'm not really able to reproduce using images available publicly. I attempted merging a couple of these https://github.com/docker/awesome-compose together to emulate a lot of images being pulled but unable to repro the super long hanging. However it does seem to be generally slower than qemu. And I'm still seeing those stopping udp proxy messages.

[jandubois]

Thanks! This looks like it is a separate issue from #1285, as the crash is in the vz code.

[balajiv113]

@terev
Is the slow network only related to DNS resolution / even the download speeds are slow ??

Here again, there are 2 different issues,

1. Actual issue with DNS
2. Panic on VM Stop (Same as Lima VM hangs and when stopped produces a cgo panic #1299). This panic has nothing to do with the slowness issue

[terev]

@balajiv113 it does seem like it might just be DNS . here's my small experiment:
VZ

$ colima start issuetest -t vz --mount-type virtiofs
$ colima ssh -p issuetest
$ for i in $(seq 1 10); do curl -s -w 'lookup: %{time_namelookup} total: %{time_total}\n' -o /dev/null https://www.google.com ; done
lookup: 1.009936 total: 1.180874
lookup: 1.014016 total: 1.200459
lookup: 1.012715 total: 1.210918
lookup: 1.014798 total: 1.194656
lookup: 1.010661 total: 1.188050
lookup: 1.014198 total: 1.194364
lookup: 1.012562 total: 1.179951
lookup: 1.010642 total: 1.193922
lookup: 1.006803 total: 1.212509
lookup: 1.013336 total: 1.188702

QEMU

$ colima start issuetest -t qemu
$ colima ssh -p issuetest
$ for i in $(seq 1 10); do curl -s -w 'lookup: %{time_namelookup} total: %{time_total}\n' -o /dev/null https://www.google.com ; done
lookup: 0.011018 total: 0.202526
lookup: 0.003772 total: 0.165436
lookup: 0.003735 total: 0.200770
lookup: 0.003029 total: 0.190208
lookup: 0.005203 total: 0.189349
lookup: 0.001524 total: 0.169970
lookup: 0.001482 total: 0.173584
lookup: 0.005046 total: 0.185870
lookup: 0.002242 total: 0.181922
lookup: 0.003841 total: 0.194537

Let me know if there's anything you'd like from me.

[balajiv113]

Thanks this helps.

Possible Reason
Looks like gvisor-tap-vsock is doing a forwarding of connections for the guest -> host access. In this DNS case, udp-proxy is being used to forward packets and that might be the reason for this slowness

@AkihiroSuda / @jandubois
Should we use in-build DNS for the vz driver ?? In this case, the requests will be replied by the gateway itself. So no forwarding will happen

VZ with inbuild DNS

lookup: 0.034468 total: 0.503908
lookup: 0.002234 total: 0.505414
lookup: 0.002497 total: 0.451307
lookup: 0.002519 total: 0.448397
lookup: 0.002834 total: 0.503139
lookup: 0.002498 total: 0.501404
lookup: 0.002626 total: 0.502992
lookup: 0.003732 total: 0.434553
lookup: 0.002362 total: 0.472442
lookup: 0.002220 total: 0.500188

[jandubois]

Should we use in-build DNS for the vz driver ?? In this case, the requests will be replied by the gateway itself.

I think replacing our host resolver with the gvisor implementation is at least the medium term goal. Can we still define static names like host.lima.internal though? The user can define additional names, either as IP addresses or as (CNAME) aliases in the lima.yaml file.

However, is this slowness of the UDP proxy expected? DNS does not cause a lot of traffic, so where does all this latency come from?

[terev]

I'm very new to networking in linux and the networking architecture implemented by this project but I have a few questions that may be dumb (feel free to ignore). That change appears to bring lookups very close to QEMU performance which seems great. But I'm curious why the http roundtrip is also so much slower than in QEMU? Does this have to do with differences between VZ and QEMU's network setup that facilitates connections from the guest or something to do with the different virtualizations? I would have expected VZ to be the same or better.

[balajiv113]

slowness of the UDP proxy expected

Not sure,. Currently investigating this. Will update if i find something

Does this have to do with differences between VZ and QEMU's network setup

@terev Exactly this. The way network is developed/configured in thr framework used is entirely different just the working is more/less the same

[balajiv113]

@terev
Just did a dig on both vz and QEMU. DNS is working as expected actually :)

Difference ?
The actual difference is coming from colima not from lima.
With lima both QEMU and VZ are taking same time (around lookup: 1.009936 total: 1.180874)

Difference in colima
Colima has their own version of gvproxy as network driver (Same framework used in lima vz driver). This is by default used for QEMU profiles but not supported for vz drivers.

When colima uses gvproxy, they are making use of gvproxy in-build DNS (Same one i have mentioned above with improved performance, the requests will be replied by the gateway itself. So no forwarding will happen). That's why colima QEMU is replying faster in comparison to colima VZ

[balajiv113]

is this slowness of the UDP proxy expected?

@jandubois - i did a performance of guest -> host for UDP, both slirp and gvisor-tap-vsock are performing same. So i would take this as expected slowness

[terev]

@balajiv113 Hmm so are you saying Colima uses slirp for VZ networking then? And this is the difference because lima uses gvproxy for VZ by default?

[balajiv113]

@terev Not exactly.

Working in lima
The below is the request flow for both VZ and QEMU

Guest -> google.com -> slirp (for QEMU) / gvisor-tap-vsock (for VZ) -> forwards udp -> lima hostagent resolves DNS

Working in colima

Guest -> google.com -> slirp (for QEMU --network-driver slirp) / gvisor-tap-vsock (for VZ) -> forwards udp -> lima hostagent resolves DNS

QEMU (--network-driver gvproxy)
Guest -> google.com -> slirp / gvisor-tap-vsock -> resolves DNS

So only in colima QEMU with --network-driver gvproxy the performance will be faster as there is no UDP forwarding happening.

[terev]

@balajiv113 Ohhh ok I think I understand. It seems like forwarding traffic is the ideal choice for DNS but there's a way to respond from within the guest which may help. But for other tcp and udp connections they have a large amount of added latency due to the virtual network forwarding.

Weird that there's so much latency added for forwarded traffic. I assume that's a huge rabbit hole to investigate. Should an issue be opened with the gvisor-tap-vsock project?