Add k8s-vmnet example #2633
Closed
Add k8s-vmnet example #2633
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
If using additional network (e.g. lima:shared), override the dhcp rote
metric so the additional metric is preferred.
This fixes issues with k8s components (like submariner) that use the
first default route and break since the user network has not
connectivity between vms. With this change they always use the
additional network.
Example routes with this change when using socket_vmnet:
$ ip route
default via 192.168.105.1 dev lima0 proto dhcp src 192.168.105.7 metric 100
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200
192.168.105.0/24 dev lima0 proto kernel scope link src 192.168.105.7 metric 100
192.168.105.1 dev lima0 proto dhcp scope link src 192.168.105.7 metric 100
Example routes without additional network:
$ ip route
default via 192.168.5.2 dev eth0 proto dhcp src 192.168.5.15 metric 200
192.168.5.0/24 dev eth0 proto kernel scope link src 192.168.5.15 metric 200
192.168.5.2 dev eth0 proto dhcp scope link src 192.168.5.15 metric 200
Another way to solve this issue is to fixup the metric in the
provisioning script as done in RamenDR:
https://github.com/RamenDR/ramen/blob/c02119785e734e15511236edd935c04ff71b6646/test/drenv/providers/lima/k8s.yaml#L37
But I think it is better to fix this in lima, since the current network
configuration is very problematic.
Signed-off-by: Nir Soffer <[email protected]>
This example show how to run multiple connected kubernetes cluster using a shared network. You can access all the clusters via the shared network, clusters can access each other. This configuration was derived from k8s.yaml template, with the following changes: - Ensure that the API server and kublet listen on the shared network. This is important for pods that need to listen on the host network. An example is rook ceph test cluster. - Disable port forwarding, since we access the host via the shared network, and automatic port forwarding cannot work for multiple clusters. - Since we access the API server via the shared network, don't modify the kubeconfig copied to the host. - Enable parallel image pulls for faster provisioning of complex clusters. - Allow unprivileged pods to access block devices. Required for kubevirt virtual machines or replicating block volumes using volsync. These changes were extracted from RamenDR k8s.yaml: https://github.com/RamenDR/ramen/blob/main/test/drenv/providers/lima/k8s.yaml Signed-off-by: Nir Soffer <[email protected]>
|
I think we can add a job using this, creating 2 tiny clusters and accessing them using the ip address on the vmnet network. Such test will ensure that upstream lima change will not break our use case. |
Example usageThis can also a new integration test. |
|
Thanks, but due to the maintenance cost I'm not sure we want to expand the template list. Maybe third-party templates can be linked from https://lima-vm.io/docs/templates/ |
|
Also, I'd suggest to use user-v2 network rather than vmnet for simplicity |
We cannot, with user-v2 you cannot access the vms from the host via the ip address, and we don't want to manage port forwarding for multiple clusters, which may also not work with complex setups (e.g connecting clusters with submariner). |
Make sense, it also more complicated to test, but I would like this to be part of lima and tested with any pr, to ensure that our use case continue work. |
Probably the CI of your project should run a canary job with lima@master ? |
We use lima only on Apple silicon, and macos-14 does not support nested virtualization, so we cannot run lima vms. Maybe can run lima on x86 runners (macos-13?), I did try this yet. |
Yes, it is what we have been using. |
|
@AkihiroSuda another option is to replace the existing k8s.yaml with this version. It works in the same way, just not depending on port forwarding, and it works for multiple clusters, while the current template does not (only one is accessible via the port forwarding). |
Using nested virtualization or emulation? |
No,
Nested virt. |
If anything I would like to see k8s.yaml get smaller/simpler, not the opposite... I don't think that will happen, upstream prefers "README" over fixing get.k8s.io Even flannel is a bit overkill for the single-host scenario, bridge would be just fine. Had some thoughts on using an external "k8s.sh": |
Closed
|
Opened an issue: |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This example show how to run multiple connected kubernetes cluster using
a shared network. You can access all the clusters via the shared
network, clusters can access each other.
This configuration was derived from k8s.yaml template, with the
following changes:
Ensure that the API server and kublet listen on the shared network.
This is important for pods that need to listen on the host network. An
example is rook ceph test cluster.
Disable port forwarding, since we access the host via the shared
network, and automatic port forwarding cannot work for multiple
clusters.
Since we access the API server via the shared network, don't modify
the kubeconfig copied to the host.
Enable parallel image pulls for faster provisioning of complex
clusters.
Allow unprivileged pods to access block devices. Required for kubevirt
virtual machines or replicating block volumes using volsync.
These changes were extracted from RamenDR k8s.yaml:
https://github.com/RamenDR/ramen/blob/main/test/drenv/providers/lima/k8s.yaml
Based on #2632 since this depends on this change. Without it we need to fixup the route metric during provisioning.