Migrating a library Godeps.json with transitive deps

[sttts]

In the process of adding dep support to k8s.io/client-go and friends I hit the following problem:

k8s.io/client-go depends on many packages that do not support golang/dep (= do not ship a Gopkg.toml). Some of them are not direct dependencies of k8s.io/client-go itself, i.e. client-go's Gopkg.toml cannot use constraints. At the same time, client-go is not the root package, but users depend on it using golang/dep. Hence, client-go's overrides have no effect either.

In addition, client-go's dependencies are complex enough that we want to avoid putting the burden on the users to declare dependencies themselves.

1. How can we provide those dependencies to the users while the packages we depend on do not support golang/dep?
2. Even if all dependencies support golang/dep some day, I claim that there will be situations when we cannot trust them or just have to override them for technical reasons. Imagine client-go depends on foo v1 and foo depends on bar v2. Now bar v2.1.2 is released, but – while source code compatible – brakes a feature of foo. In client-go we would have no way to restrict to <v2.1.2.

While (1) suggests that we might want to follow overrides in dependencies until the direct dependencies ship Gopkg.toml, (2) suggests that the problem won't go away in the future with full golang/dep support throughout all packages.

[sttts]

This seems to be a blocker for any further work towards supporting golang/dep in k8s.io/client-go. Ideas how to work around it, are very welcome.

/cc @carolynvs @sdboyer

[sttts]

I wonder whether we need something like primary and secondary overrides:

• the user defines primary overrides for his/her root project,
• imports may define secondary overrides.

Primary always trump over secondary ones. If secondary overrides conflict, the resolution is failed and the root project has to deconflict by defining primary overrides.

In other words, we have only two layers, not a complete hierarchy.

[deads2k]

I think I just hit this vendoring k8s.io/client-go and apimachinery. I got the correct level of client-go (4.0) and the wrong level of apimachinery (master) that it depends on. The write up in the description here explains the problem well overall.

@lavalamp this is going to keep dep from being a reasonable contender for client-go for a while.

[sttts]

@deads2k if we add a native Gopkg.toml we can actually restrict the direct imports of client-go correctly (using a revision constraint). The issue here is one step further: there are (transitive) imports from our dependencies which we don't control:

E.g. client-go -> apimachinery -> docker/docker -> docker/runc. We control the first two and can create a Gopkg.toml for each. Then everything up to docker/docker is specified. But we cannot specify docker/runc, but have to trust Docker adding Gopkg.toml for their projects. If they do that one day, great.

But if we find an issue in docker/runc we cannot override that version either, but depend again on Docker doing this. If the issue is not directly relevant for docker/docker, but for our usecase of it, we are screwed as Docker will refuse (for good reason) to follow our special requirement.

[sdboyer]

oooook, buncha things here!

In addition, client-go's dependencies are complex enough that we want to avoid putting the burden on the users to declare dependencies themselves.

this is the second time this has come up as a significant point of frustration, with little feasible recourse - shurcooL/github_flavored_markdown#12 /cc @shurcooL @rtfb. i also believe that complexity-gatekeeping in this way is an important value that projects can and should be able to provide, at least in some way. so yeah, i'm bumping this up my priority list of things that really need fixing.

I wonder whether we need something like primary and secondary overrides:

i think the property you're really hunting for here is the transitive power that we give to overrides, but not constraints. this is something we could do - we could even extend the power to constraints. or have an extra value within a [[constraint]] that allows it to say, "this should apply transitively." but at least one problem is always going to be that...

hmm. actually. i've not thought this all the way through and am just spitballing, but if we make the transitivity flag explicit, then it could effectively mitigate the harms here without many of the unintended side effects described in https://gist.github.com/sdboyer/b0813bf2b9dba58a335a85092085472f. those side effects occur if we make constraints always transitive, because we can't distinguish between old, decrepit constraints and ones that are actually meaningful, which could end up creating an ecosystem with lots of "ghost" conflicts - conflicts between old constraints declared by projects that they never cleaned up. maybe those constraints are on a project that doesn't even end up in the depgraph. (i realize that sounds extra-absurd, but it could be damaging to solving efficiency if we try to "defer" mutual constraint agreement checks until a package actually shows up. not sure.)

but if a project can specify a flag that a constraint is transitive...yeah, this needs more whiteboarding, etc., but unless i'm missing something, this could work well.

this, though:

Primary always trump over secondary ones. If secondary overrides conflict, the resolution is failed and the root project has to deconflict by defining primary overrides.

at first glance, this basically just seems like an arms race to me: "they made a constraint, so we fired back with a secondary override, and now y'all need a primary override to settle the dispute." i don't think that direction is sustainable.

in any case, right now:

How can we provide those dependencies to the users while the packages we depend on do not support golang/dep?

the simplest thing you can do, for now, is to hoist up these packages as your own direct dependencies, so that you can constrain them as normal. ordinarily that means a required field in Gopkg.toml, but required is a power only afforded to the root project, so if your goal is to make things better for your dependers, then you'll need to make a dummy import somewhere:

package whatever

import _ "github.com/docker/runc"

the limitation there is that your constraint will only be activated if your dependers import the package in which that declaration exists. that may or may not be obnoxious in the k8s.io/client-go case.

[sttts]

but required is a power only afforded to the root project

I was excited about the described workaround until I read this sentence.

For the other idea with the imports: we have many packages in client-go (and apimachinery, and apiserver, ...) and we would have to add those artifical dummy imports everywhere. While technically feasible, this also means you get the whole dependency chain for the smallest program (go build times will shoot to the sky I fear). On the other hand, I am happy that dep ignores constraints for packages in not-imported packages.

[sttts]

i think the property you're really hunting for here is the transitive power that we give to overrides, but not constraints. this is something we could do - we could even extend the power to constraints. or have an extra value within a [[constraint]] that allows it to say, "this should apply transitively."

Yes, this is what we look for.

I wonder what you have in mind with this flagging of transitive constraints? This sounds like my primary-vs-secondary idea that you want to change the logic for them in case of conflicts?

[sttts]

One thought, not to forget: transitive constraints inherited from a dependency on packages that your root package does not import, must still be ignored. Example: docker/runc above if my root package does not depend on docker/docker.

In other words, this behaviour must be different to overrides (which are unconditional by definition). So I agree, what we need something which is more like a transitive constraint than an override that is inherited from a dependency.

[lavalamp]

@deads2k I filed https://github.com/kubernetes/apimachinery/issues/23

[redbaron]

Maybe I am missing something, but wouldn't [[constraint]] + [[override]] work even if your project doesn't use that dependency directly?

[[override]] pulls right dependencies when you are building project
[[constraint]] make any parent project to choose transitive dependencies within your constraint

EDIT: should have tried before replying :) [[constraint]] have no effect if your project doesn't use it. why is that?

[carolynvs]

[[constraint]] has no effect if your project doesn't use it. why is that?

Two options:

1. Use [required]=["github.com/foo"] but you can't specify the version or anything.
2. Use a dummy import, e.g. imports _ "github.com/foo" and then your constraint will work.

[redbaron]

What is downside of respecting [constraint] of transient dependencies without using import _ hack? Would you accept patch?

BTW, import trick doesn't work reliably, seems that you'd need to add it to every subpackage, which can possibly imported by parent project,

[carolynvs]

@redbaron Care to open a new issue? These questions are about how to use dep, and aren't related to the original post.