Cutoms golang URLs

Hi,

I know we talked about custom URLs already on slack a few weeks ago.
We have so many topics that seems waiting on each other, the URLs seems like one that is actually actionnable.

Here is a recap:

##Why ?

  • Having control over our own URLs is future proofing.
  • Given the limitations of go get (always git clones all the history), it is desirable to have a smart git proxy in the near future decoupling package URLs from repository URLs
  • Analytics (useful to know if our pkgs are actually used or not by external projects)

##How ?

@tiborvass and I are proposing the use of go.mobyproject.org
I donā€™t think itā€™s too long (only 3 more chars than github.com/moby) and itā€™s mostly automatically imported anyways.

And at first we propose 3 sections:

  • /pkg/<..> used for all our external pkgs
  • /cmd/<...> used for the command line tools like moby or linuxkit
  • /x/engine used for the current monolith

WDYT ? leaving aside the domain name decision, Iā€™m starting a PoC to move things forward.

LGTM,

I would define /x/ as a general temporary location for packages though.

I donā€™t really have a preference regarding the domain name.

But we need to define how/if the url will support different version of the packages (e.g. by adding a version number at the end of the URL . Something like.vx.x or ?vx.x)

1 Like

+1

Each section will have a bunch of different packages, is that right?

I think we should use /x/ for everything until we have some way of doing semantic versioning, and a release process that allows us follow the semver guidelines. As long as these packages are part of some larger repo that is difficult.

An option might be to maintain branches for each package. That would allow us to selectively cherry-pick and merge the changes into the correct version.

What is an external package? Are we planning on exposing what we have today, or should we wait until the reorganization of pkg is done, then expose the new package?

I believe there are things in containerd that might be considered ā€œexternal packagesā€ as well. Will we create urls for those as well?

lgtm
if you want something shorter, we also have mo.by

P@

I like the single url that lets things easily move as packages get pushed out to the community to own. The smart git proxy is mentioned as a near future goal, does that mean weā€™ll have something different like a simple url redirect before then?

Iā€™m also trying to wrap my head around how the paths apply to existing projects. Letā€™s take an example with moby/tool. That has directories currently for cmd, examples, pkg, test, and vendor.

Is it simply:

Or will the proxy be leveraging those sub directories somehow:

@dnephin @mlaventure Iā€™m afraid if we use /x/ for everything we might never move away from it.

Thatā€™s why in my proposal, only engine is behind /x/ since will will be heavily modified (componentized)
I feel like pkg are somehow stable and I donā€™t see the need to put them behind /x/
Same for the moby tool, I was thinking about using /cmd/ for go gettable binaries, so I donā€™t see why to put it behind /x/

@mlaventure @dnephin regarding the versioning, maybe itā€™s an unpopular opinion, but I donā€™t think it should be part of the url. many of our pkgs are so small, I donā€™t want to have to manage versions on each and everyone of them, including when to bump the version.
I donā€™t see the custom URLs replace vendoring, so ppl will still vendor our pkgs and pin the commit then need.

@dnephin by external pkgs, I think we only need ā€œItem 2 - Split packages with poor cohesionā€ from the reorganization of pkg. once this is done, we route the custom urls to the pkg in moby/moby and then gradually when we move the pkg to another repo, update the custom URLs, it wonā€™t impact users.

@chanezon every time we brought up mo.by concerns were raised about .by and since itā€™s not something users would time often, I think something longer is fine.

@bmitch3020 I was thinking of /cmd for the actual binaries, no matter where they are hosted

  • go get go.mobyproject.org/cmd/moby would get me github.com/moby/tool/cmd/moby
  • go get go.mobyproject.org/cmd/linuxkit would get me github.com/linuxkit/linuxkit/src/cmd/linuxkit

For the rest we could use /x/ like go.mobyproject.org/x/tool/pkg, go.mobyproject.org/x/tool/tests

I would urge caution in how creative we get with import path mapping. For the most part, we need to ensure that each set of path corresponds to a project. If we have multiple projects, we need to ensure that they all have the same number of path segments. For example, if we have mo.by/foo and mo.by/bar, there should be a git repository corresponding to each one. This will cause some repetition but will avoid future headaches. The main concern now would be multiple checkouts of the same codebase. Please donā€™t do this:

go get go.mobyproject.org/cmd/linuxkit would get me github.com/linuxkit/linuxkit/src/cmd/linuxkit

In the long run, this will confuse tooling because it will cause a double checkout of linuxkit if there are go packages under other imports. It will also complicate Go tooling that makes assumptions, such as vim-go.

Also, I am not sure about the use of the /x/ prefix, unless we lose the subdomain. The Go project uses golang.org/x/ to denote that everything under /x/ is a go project. This would make sense if we use mo.by as a shortener for other things besides Go package. For example, a nice scheme would be mo.by/x/<project>. That would map well to what is known and used and avoid problems.

1 Like

We discussed this domain, but the .by TLD (which is for Belarus) was discouraged to use; itā€™s fine for non-critical things, but probably not the best thing to use for our packages.

Do we have any measure of the stability of Belarusā€™ registrar? I havenā€™t seen them on any lists.

ā€¦

My own goal is for go programs that import moby components can do so with the new url and not break on a move. So a stable proxy to the top level of each repo (e.g. github.com/moby/tool) would be needed for me. Iā€™ll leave the discussion of whether thatā€™s /x/tool or somewhere else up to the experts.

Whatā€™s the scope of projects that would be included in this proxy? Is it just github.com/moby projects, including pieces from linuxkit that will be moved over later? Or would this include anything that docker has released to the community like containerd and runc?

I think we should use this opportunity to create a fresh new namespace for our stable packages, with the following properties:

  • Strong interface stability guarantees. Maybe we promise semver and a 1 year interface deprecation notice?
  • Flat namespace. Only one thing can be called ā€œfooā€, ever.
  • Location-independent. Different packages may live in different repositories. We reserve the right to move the source of each any package to any repository at any time. and change repositories independently.
  • Only stable packages. For packages that donā€™t meet the interface stability guarantee, the stable URL is not available. Users should be clearly warned that stability is not guaranteed and the package interface may change at any time.
  • Only libraries. I donā€™t think we should encourage installing end-user binaries with ā€˜go getā€™. Developers can still ā€˜go installā€™ from their source checkout. People who like living dangerously can still ā€˜go getā€™ from the unsupported github url.
  • No special case for engine. I donā€™t think we need to create a special case for importing from engine. Letā€™s just choose a name for the engine, move it out as weā€™re discussing in the ā€œfind a new homeā€ thread, and just expose that as a package name. OR, letā€™s flatten the stable packages from the engine repo into the flat namespace described above. Either way, no special case or ā€œ/xā€ needed.

The topic of .by was discussed on community slack a while back, sadly it has scrolled off into the ether and dockercommunity.slackarchive.io seems to be down. I think this and this are in the vicinity of those discussions, should it come back (I opened them yesterday but didnā€™t have a chance to write here until now).

Personally my biggest concern was that we needed to go out of our way to find someone who could read the language the T&Cs were written in. Given that go.mobyproject.org is not that long and is not something that is typed all that frequently (if nothing else you copy and paste it from the line aboveā€¦) and mo.by is only moderately cute (sorry!) we should go with the subdomain of the main domain IMHO.

I agree; besides, comparing the current (GitHub) and go.mobyproject.org, itā€™s still shorter

github.com/moby/moby/foo
go.mobyproject.org/foo

@ijc @thaJeztah Well, whatever we do superficially, we do need to be ensure that we heed the warnings Iā€™ve outlined above.

Specifically, the requirement for location independence is going to be at odds with tooling and what is possible for vanity import paths:

Effectively, the namespace of the import must point to a logical project. Routing to individual packages in that project will cause problems with tooling.

Could a logical project be as simple as a special publishing repo, say in github.com/moby/go.mobyproject.org, where we vendor the versions we want
to publish?

More of this please

:slight_smile:

That would probably give us the best control over versioning and namespace. I think it checks off most your requirements and avoids the usability problems.

The only issue with this approach is that it may make the contribution workflows complex, but changes should be low volume, given the stability requirements.