Caching packages and images on the lima host

[afbjorklund]

Currently lima has support for caching VM images, and for caching built-in container runtimes (i.e. containerd/buildkitd)

INFO[0001] Attempting to download the image from "https://cloud-images.ubuntu.com/impish/current/impish-server-cloudimg-amd64.img"  digest=
INFO[0002] Using cache "/home/anders/.cache/lima/download/by-url-sha256/ac74da77a6828e35de7edaa06fdbb33d12ef97cce2726550017e3c1066c88fb1/data" 
INFO[0002] Attempting to download the nerdctl archive from "https://github.com/containerd/nerdctl/releases/download/v0.13.0/nerdctl-full-0.13.0-linux-amd64.tar.gz"  digest="sha256:ed2d3662fd2866875e107df20f7da6dd32fafa8fa3e8aa9b6b774af20af5ddaa"
INFO[0002] Using cache "/home/anders/.cache/lima/download/by-url-sha256/f298e7c4e95e0e021bab86e5ee04fa4762a7f8ee44a2ab105b46f70ab4a28cfd/data" 

What would be the best way to cache "other" binaries ? Mount a directory and use a custom script ? Run a local proxy ?

The use case was from the kubernetes example, where it does a kubeadm package installation and uses k8s.gcr.io images.

anders@lima-k8s:/var/cache/apt/archives$ du -hxcs *.deb
8.0K	apt-transport-https_2.0.6_all.deb
32K	conntrack_1%3a1.4.5-2_amd64.deb
11M	cri-tools_1.19.0-00_amd64.deb
80K	ebtables_2.0.11-3build1_amd64.deb
8.4M	kubeadm_1.22.3-00_amd64.deb
8.7M	kubectl_1.22.3-00_amd64.deb
19M	kubelet_1.22.3-00_amd64.deb
24M	kubernetes-cni_0.8.7-00_amd64.deb
316K	socat_1.7.3.3-2_amd64.deb
44K	sshfs_3.6.0+repack+really2.10-0ubuntu1_amd64.deb

anders@lima-k8s:/var/cache/kubeadm/amd64$ sudo crictl images
IMAGE                                TAG                 IMAGE ID            SIZE
k8s.gcr.io/coredns/coredns           v1.8.4              8d147537fb7d1       13.7MB
k8s.gcr.io/etcd                      3.5.0-0             0048118155842       99.9MB
k8s.gcr.io/kube-apiserver            v1.22.3             53224b502ea4d       31.2MB
k8s.gcr.io/kube-controller-manager   v1.22.3             05c905cef780c       29.8MB
k8s.gcr.io/kube-proxy                v1.22.3             6120bd723dced       35.9MB
k8s.gcr.io/kube-scheduler            v1.22.3             0aa9c7e31d307       15MB
k8s.gcr.io/pause                     3.5                 ed210e3e4a5ba       301kB
quay.io/coreos/flannel               v0.14.0             8522d622299ca       21.1MB

---

Question

Could there be some new option in the yaml, to help out with caching ?

Currently I just copy the deb packages, and load images from archive...

233M /var/cache/kubeadm/amd64/kubernetes-v1.22.3-images.tar

limactl shell k8s sudo nerdctl -n k8s.io load

There is some support for setting a http_proxy/https_proxy in the environment, maybe similar for apt and containerd ?

So that one could use a local pull through cache, and run this server on the host or in the local network somewhere.

• https://wiki.debian.org/AptCacherNg

/etc/apt-cacher-ng/acng.conf (on host)

# Port:3142
BindAddress: localhost

/etc/apt/apt.conf.d/proxy.conf (in lima vm)

Acquire::http::Proxy "http://localhost:3142/";

/etc/apt/sources.list.d/kubernetes.list (in lima vm, as a workaround for not being able to cache https urls)

deb [signed-by=/usr/share/keyrings/kubernetes-archive-keyring.gpg] http://HTTPS///apt.kubernetes.io/ kubernetes-xenial main

That is: s|https://|http://HTTPS///|

• https://docs.docker.com/registry/recipes/mirror/

/etc/docker/registry/config.yml (on host)

version: 0.1
http:
  addr: localhost:5000
proxy:
  remoteurl: https://k8s.gcr.io

/etc/containerd/config.toml (in lima vm)

version = 2
[plugins."io.containerd.grpc.v1.cri".registry.mirrors]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."k8s.gcr.io"]
    endpoint = ["http://localhost:5000"]

Assuming a rewrite of "localhost" (127.0.0.1) to gateway (192.168.5.2)

Or something more generic, like: https://github.com/fgrehm/vagrant-cachier

sharing a common package cache among similiar VM instances

[afbjorklund]

Something that would be really nice was if the local image cache was stargazer-aware. Not sure how hard that is.

https://github.com/containerd/stargz-snapshotter. Then the images could be "preprocessed", in the host cache ?

[jandubois]

The following doesn't really work for your package/image cache, and I think configuring a mirror via a provisioning script may be the right thing to do. But if you packages/images are available in a single downloadable tarball, then it might work:

What would be the best way to cache "other" binaries ?

My idea for this has been to extend the mounts setting to specify different source location and mount points:

mounts:
- location: /mnt/cache
  from: "{{.Home}}/Library/Cache"
  writable: false

And then extend source to also allow for URLs, which would be downloaded by Lima using the regular caching mechanism, and the mount would point into our cache directory (only read-only mounts allowed).

One complication is that mounts only supports directories and not files.

---

Once you can specify source and destination separately, location (and from) might not be the best names, but for now location provides backwards compatibility with what we have (and from defaults to location if not specified).

[afbjorklund]

It might actually be better to provide an upstream VM image that has all the packages and images in that case, because there are no such tarballs (unless you create them yourself). Upstream is relying on apt and crictl to pull the individual files:

apt-get install -y kubelet kubeadm kubectl

kubeadm config images pull

The proxy servers would be neat, but I'm not sure how much hassle it would be to configure them. I don't mean so much starting them on the host, but translating and applying them to configuration from the cidata or similar startup scripts ?

[jandubois]

there are no such tarballs

It depends on the use case. E.g. for k3s there are "airgap" tarballs with all required images for each release, so caching that on the host would make some sense.

The proxy servers would be neat, but I'm not sure how much hassle it would be to configure them. I don't mean so much starting them on the host, but translating and applying them to configuration from the cidata or similar startup scripts ?

I think this is outside the scope of lima, and should be provided by a project wrapping lima for a particular use case/scenario (like Rancher Desktop or colima do).

[afbjorklund]

I think the docker format also allowed for better compression (xz), since it was not individually compressed (gzip)

Example:
230M images.tar
161M images.txz

But I don't know how you disable compression with nerdctl, only with podman. And it is besides the main topic.

[afbjorklund]

I think this is outside the scope of lima, and should be provided by a project wrapping lima for a particular use case/scenario

I guess most normal people don't set up and tear down the same configuration over and over... Just missed the "cachier".

[bpinto]

I'm wondering if snapshots could be a solution to this that is maybe inside the scope of lima?

[afbjorklund]

Snapshots could be used as a workaround, or one could use custom VM images (like lima-alpine does).