I know we talked about custom URLs already on slack a few weeks ago.
We have so many topics that seems waiting on each other, the URLs seems like one that is actually actionnable.
Here is a recap:
##Why ?
Having control over our own URLs is future proofing.
Given the limitations of go get (always git clones all the history), it is desirable to have a smart git proxy in the near future decoupling package URLs from repository URLs
Analytics (useful to know if our pkgs are actually used or not by external projects)
##How ?
@tiborvass and I are proposing the use of go.mobyproject.org
I donāt think itās too long (only 3 more chars than github.com/moby) and itās mostly automatically imported anyways.
And at first we propose 3 sections:
/pkg/<..> used for all our external pkgs
/cmd/<...> used for the command line tools like moby or linuxkit
/x/engine used for the current monolith
WDYT ? leaving aside the domain name decision, Iām starting a PoC to move things forward.
I would define /x/ as a general temporary location for packages though.
I donāt really have a preference regarding the domain name.
But we need to define how/if the url will support different version of the packages (e.g. by adding a version number at the end of the URL . Something like.vx.x or ?vx.x)
Each section will have a bunch of different packages, is that right?
I think we should use /x/ for everything until we have some way of doing semantic versioning, and a release process that allows us follow the semver guidelines. As long as these packages are part of some larger repo that is difficult.
An option might be to maintain branches for each package. That would allow us to selectively cherry-pick and merge the changes into the correct version.
What is an external package? Are we planning on exposing what we have today, or should we wait until the reorganization of pkg is done, then expose the new package?
I believe there are things in containerd that might be considered āexternal packagesā as well. Will we create urls for those as well?
I like the single url that lets things easily move as packages get pushed out to the community to own. The smart git proxy is mentioned as a near future goal, does that mean weāll have something different like a simple url redirect before then?
Iām also trying to wrap my head around how the paths apply to existing projects. Letās take an example with moby/tool. That has directories currently for cmd, examples, pkg, test, and vendor.
@dnephin@mlaventure Iām afraid if we use /x/ for everything we might never move away from it.
Thatās why in my proposal, only engine is behind /x/ since will will be heavily modified (componentized)
I feel like pkg are somehow stable and I donāt see the need to put them behind /x/
Same for the moby tool, I was thinking about using /cmd/ for go gettable binaries, so I donāt see why to put it behind /x/
@mlaventure@dnephin regarding the versioning, maybe itās an unpopular opinion, but I donāt think it should be part of the url. many of our pkgs are so small, I donāt want to have to manage versions on each and everyone of them, including when to bump the version.
I donāt see the custom URLs replace vendoring, so ppl will still vendor our pkgs and pin the commit then need.
@dnephin by external pkgs, I think we only need āItem 2 - Split packages with poor cohesionā from the reorganization of pkg. once this is done, we route the custom urls to the pkg in moby/moby and then gradually when we move the pkg to another repo, update the custom URLs, it wonāt impact users.
@chanezon every time we brought up mo.by concerns were raised about .by and since itās not something users would time often, I think something longer is fine.
@bmitch3020 I was thinking of /cmd for the actual binaries, no matter where they are hosted
go get go.mobyproject.org/cmd/moby would get me github.com/moby/tool/cmd/moby
go get go.mobyproject.org/cmd/linuxkit would get me github.com/linuxkit/linuxkit/src/cmd/linuxkit
For the rest we could use /x/ like go.mobyproject.org/x/tool/pkg, go.mobyproject.org/x/tool/tests
I would urge caution in how creative we get with import path mapping. For the most part, we need to ensure that each set of path corresponds to a project. If we have multiple projects, we need to ensure that they all have the same number of path segments. For example, if we have mo.by/foo and mo.by/bar, there should be a git repository corresponding to each one. This will cause some repetition but will avoid future headaches. The main concern now would be multiple checkouts of the same codebase. Please donāt do this:
go get go.mobyproject.org/cmd/linuxkit would get me github.com/linuxkit/linuxkit/src/cmd/linuxkit
In the long run, this will confuse tooling because it will cause a double checkout of linuxkit if there are go packages under other imports. It will also complicate Go tooling that makes assumptions, such as vim-go.
Also, I am not sure about the use of the /x/ prefix, unless we lose the subdomain. The Go project uses golang.org/x/ to denote that everything under /x/ is a go project. This would make sense if we use mo.by as a shortener for other things besides Go package. For example, a nice scheme would be mo.by/x/<project>. That would map well to what is known and used and avoid problems.
We discussed this domain, but the .by TLD (which is for Belarus) was discouraged to use; itās fine for non-critical things, but probably not the best thing to use for our packages.
My own goal is for go programs that import moby components can do so with the new url and not break on a move. So a stable proxy to the top level of each repo (e.g. github.com/moby/tool) would be needed for me. Iāll leave the discussion of whether thatās /x/tool or somewhere else up to the experts.
Whatās the scope of projects that would be included in this proxy? Is it just github.com/moby projects, including pieces from linuxkit that will be moved over later? Or would this include anything that docker has released to the community like containerd and runc?
I think we should use this opportunity to create a fresh new namespace for our stable packages, with the following properties:
Strong interface stability guarantees. Maybe we promise semver and a 1 year interface deprecation notice?
Flat namespace. Only one thing can be called āfooā, ever.
Location-independent. Different packages may live in different repositories. We reserve the right to move the source of each any package to any repository at any time. and change repositories independently.
Only stable packages. For packages that donāt meet the interface stability guarantee, the stable URL is not available. Users should be clearly warned that stability is not guaranteed and the package interface may change at any time.
Only libraries. I donāt think we should encourage installing end-user binaries with āgo getā. Developers can still āgo installā from their source checkout. People who like living dangerously can still āgo getā from the unsupported github url.
No special case for engine. I donāt think we need to create a special case for importing from engine. Letās just choose a name for the engine, move it out as weāre discussing in the āfind a new homeā thread, and just expose that as a package name. OR, letās flatten the stable packages from the engine repo into the flat namespace described above. Either way, no special case or ā/xā needed.
The topic of .by was discussed on community slack a while back, sadly it has scrolled off into the ether and dockercommunity.slackarchive.io seems to be down. I think this and this are in the vicinity of those discussions, should it come back (I opened them yesterday but didnāt have a chance to write here until now).
Personally my biggest concern was that we needed to go out of our way to find someone who could read the language the T&Cs were written in. Given that go.mobyproject.org is not that long and is not something that is typed all that frequently (if nothing else you copy and paste it from the line aboveā¦) and mo.by is only moderately cute (sorry!) we should go with the subdomain of the main domain IMHO.
@ijc@thaJeztah Well, whatever we do superficially, we do need to be ensure that we heed the warnings Iāve outlined above.
Specifically, the requirement for location independence is going to be at odds with tooling and what is possible for vanity import paths:
Effectively, the namespace of the import must point to a logical project. Routing to individual packages in that project will cause problems with tooling.
Could a logical project be as simple as a special publishing repo, say in github.com/moby/go.mobyproject.org, where we vendor the versions we want
to publish?
That would probably give us the best control over versioning and namespace. I think it checks off most your requirements and avoids the usability problems.
The only issue with this approach is that it may make the contribution workflows complex, but changes should be low volume, given the stability requirements.
