Prefer additional network over user network

[nirs]

If using additional network (e.g. lima:shared), override the dhcp rote metric so the additional metric is preferred.

This fixes issues with k8s components (like submariner) that use the first default route and break since the user network has not connectivity between vms. With this change they always use the additional network.

Example routes with this change when using socket_vmnet:

$ ip route
default via 192.168.105.1 dev lima0 proto dhcp src 192.168.105.7 metric 100
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200
192.168.105.0/24 dev lima0 proto kernel scope link src 192.168.105.7 metric 100
192.168.105.1 dev lima0 proto dhcp scope link src 192.168.105.7 metric 100

Example routes without additional network:

$ ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200

Another way to solve this issue is to fixup the metric in the provisioning script as done in RamenDR:
https://github.com/RamenDR/ramen/blob/c02119785e734e15511236edd935c04ff71b6646/test/drenv/providers/lima/k8s.yaml#L37

But I think it is better to fix this in lima, since the current network configuration is very problematic.

[nirs]

@jandubois can you review this?

[jandubois]

I think this change is fine: why add additional networks if they should not have routing priority over the builtin Slirp network.

But I would like to see approval from @balajiv113 before merging.

[balajiv113]

Am good with the changes in context with Slirp network.

My only concern is that, once user-v2 is made default i believe this will not be needed right ?
As user-v2 does already support vm <-> vm communication. Just to ensure user-v2 takes priority

[nirs]

My only concern is that, once user-v2 is made default i believe this will not be needed right ? As user-v2 does already support vm <-> vm communication. Just to ensure user-v2 takes priority

It will be need, since user-v2 is not providing an ip address that can be accessed from the host. If it was providing both host to vm and vm to vm access, I could use it instead of socket_vmnet, and I would not have a problem with 2 networks in the same vm.

I think the simple rule is, if you added an additional network, it is has priority. Otherwise like @jandubois says, why add additional network?

If we need more, we can add a metric: N parameter to the network object, so users get what they need.

I think the important goal is to make it easy for user to configure the system in the best way, not try to predict the the best way and enforce it.

I also tested using both lima: user-v2 and socket: /var/run/socket_vmnet and it works, but it does not solve the issue with 2 networks. I can test again with this change.

Ideally lima would let me choose the only network - when using:

networks:
  - socket: /var/run/socket_vmnet

There will be only eth0 using the vmnet network.

[jandubois]

Am good with the changes in context with Slirp network.

Ok, I'm going to merge based on this agreement. 😄

My only concern is that, once user-v2 is made default i believe this will not be needed right ? As user-v2 does already support vm <-> vm communication. Just to ensure user-v2 takes priority

I don't think so. Why do you add another network in addition to user-v2 if the additional network shouldn't take priority.