The application formerly known as the Docker Engine is going to be split into components. Each component will provide an API. These APIs will likely use gRPC.
Problem
There are many consumers of the existing Docker Engine REST API. We need to support consumers of this API while components are being extracted so that all consumers have adequate time to transition to the new APIs.
Proposal
We can support all of the current API users by creating a component which provides the existing REST API. This component will use the routes in api/server/router/ with a backend implementation which performs APIs calls to the gRPC API.
Plan
When the first component with a gRPC interface is extracted implement a new Backend (the interface required by the router) which performs gRPC calls to the component.
Repeat step 1 as each component is removed
After all the components have been removed, move api/server/ and the new backend implementations into a new docker/rest-api-v1 repo.
Formally deprecate the legacy API by setting a date on which the legacy REST API component will no longer be included in the Docker CE/EE releases.
Alternative Proposals
The following proposals were considered, but are not recommended.
Extract the rest-api-v1 component before other components
Similar to the proposal above, except step 3 is done before step 1. This option is not recommended because it re-introduces the problems we had with the docker/engine-api repo, which will significantly increase the work to extract components. Since the primary goal is to support the extraction of components this works against us.
The problem occurs when a component is removed. In-process calls must be replaced by gRPC calls, but those calls would be in a vendored repo. So a component can’t be removed from the monolith until the rest-api has been updated and vendored back into the monolith. But the rest-api can’t be tested without the monolith because during the transition a significant part of the backend implementation will still be provided by the monolith. With an extracted rest-api component extracting components requires coordination between three different repos. By removing it at the end we don’t need to coordinate any changes, they can be performed sequentially.
Client Library or SDK
A client library can be created for all major languages. As new gRPC endpoints are created the library can be updated to make API calls to the new endpoints, and remove the old REST API endpoints.
This option is not recommended for the following reasons:
Users who have applications in languages not covered by the library will have no option for transiting to the new APIs
The work required to implement the library in one language is comparable to the work required to implement the resp-api component, however a library must be created for multiple languages, which multiplies the required work by the number of supported languages. The total work required for this solution is significantly higher.
There are currently only two officially supported languages. Creating libraries in any other language is wasted effort because it builds against an API that is being deprecated and removed.
All the libraries will need to provide an interface that resembles the legacy APIs instead of the new APIs. Once the transition is complete users will be stuck with a non-optimal interface and will require another transition to a new version of the library.
Each component implements a legacy REST API
Instead of a single REST component, each new component could implement part of the old REST API.
This option is not recommended for the following reasons:
It increases the work required to create a new component, which is the critical path
The boundaries between components would be influenced by the existing API
The monolith would still be required to act as a proxy to the API in each component, which violates one of the goals of the new architecture.
I like the first proposal. Once migration to gRPC is complete, the docker/rest-api-v1 repository can live its live as its own component, and used as a compatibility layer for those that need it.
Extract the rest-api-v1 component before other components
Similar to the proposal above, except step 3 is done before step 1. This option is not recommended because it re-introduces the problems we had with the docker/engine-api repo, which will significantly increase the work to extract components. Since the primary goal is to support the extraction of components this works against us.
I would’ve loved this alternative proposal (move the API out now, vendor it in, and gradually add gRPC endpoints/API’s, but discussing with @dnephin brought up the downsides, so because of that, I agree it may not be a good choice
I’d just have this to add regarding a possible SDK:
IMHO such a SDK will inevitably be opinionated in regard of which options/endpoints it exposes and what operations it allows. Hence, I think it only makes sense if such a SDK is to be part of a product (e.g. Docker), not this project. The GRPC interface in themselves shall be enough of a client IMHO.
As such, I wouldn’t have included it as a possible solution for the mobyproject, but I expect downstream projects to want to go this way to offer some convenience/added value to their user and hide away unneeded details/complexity (e.g. in most common scenarios noone cares about most of the fields that can be changed in the REST API).
After all the components have been removed, move api/server/ and the new backend implementations into a new docker/rest-api-v1 repo.
Formally deprecate the legacy API by setting a date on which the legacy REST API component will no longer be included in the Docker CE/EE releases.
Can you please consider placing the repo under moby/ rather than docker/, so that 3rd parties can easily corporate on the compatibility issue, even after the Docker product. drop the support for the legacy API?
Nice write up and the recommended solution is much better than the other options.
Does #4 mean that any code using the REST API directly will need a rewrite to gRPC? If so, is there a downside of leaving a permanent REST to gRPC gateway in place other than needing to maintain an extra piece of code (I’m thinking about whether it could be moved to contrib instead of completely removed)?
Does #4 mean that any code using the REST API directly will need a rewrite to gRPC?
Yes, any code would eventually need to be rewritten to use the new APIs.
is there a downside of leaving a permanent REST to gRPC gateway in place
The place it exists now isn’t going to exist in the future, so leaving it in place isn’t really an option. The downside is that it’s maintenance overhead. I expect it will continue to live on as a community project.
How long of a deprecation window are we planning?
I think it’s too early to say. It will depend on when other APIs available. I would personally say that we would need to wait at least a year from the time when the last API is converted, so probably ~2 years.
I agree with the overall approach, but I disagree with #4 - I think not
supporting a REST API going forward is a mistake. You’re basically chopping
off all vanilla-HTTP/curl users. As mentioned, not everything has a grpc
library, nor will everyone want to use one - sometimes just a simple
curl/HTTP-GET is all that’s needed.
I think not supporting a REST API going forward is a mistake. You’re basically chopping
off all vanilla-HTTP/curl users.
I agree that there is a lot of benefit in having a vanilla HTTP+JSON API. However, I think that’s a separate issue. This thread is only about the transition path for the V1 API. We can’t support the legacy API forever. I think the best option is a long deprecation period with a well communicated deadline. After that deadline the component can be maintained by anyone who is still interested in the V1 API.
I think not supporting a REST API going forward is a mistake. You’re
basically chopping
off all vanilla-HTTP/curl users.
I agree that there is a lot of benefit in having a vanilla HTTP+JSON API.
However, I think that’s a separate issue. This thread is only about the
transition path for the V1 API. We can’t support the legacy API forever. I
think the best option is a long deprecation period with a well communicated
deadline. After that deadline the component can be maintained by anyone who
is still interested in the V1 API.
Maybe there is a middle ground here which would address the 2 outstanding
issues in this thread:
I agree with Sebastiaan that it would be preferable to split out the
REST API in its own repository right away, but we haven’t found a
convenient way to do that.
I agree with Doug that some users will always be interested in a simple
curl-friendly compat layer.
I would really like grpc to be a first-class citizen for all components
by the end of June. With the current plan that probably wouldn’t happen
until the end of the year. Until we have full grpc support, including in
the monolith, we can’t develop common tooling to automate the generation of
client libraries, command-line clients, documentation, service discovery
between components, etc. That tooling would bring lots of benefits to Moby.
Without it, the project is basically on hold.
What if we adapted Daniel’s proposal like so:
Create a new engine/GRPC API which directly maps to engine/REST,
call-for-call. No need to get fancy or idiomatic: just map REST operations
to corresponding grpc calls in a basic, almost naive to make proxying very
easy.
Implement a engine/REST to engine/GRPC proxy. Because the engine/GRPC
interface is tailor-made for this problem, the implementation can be made
production-ready on a relatively short timeframe: let’s say arbitrarily by
end of June
Make the new proxy the official engine/REST implementation, and move it
to its own repository. No need to deprecate it! It can continue its life as
a simple curl-friendly interface for container management. We will continue
to support it in Docker, and perhaps make it an optional component in the
future once everyone switches to grpc (but there’s no rush).
Over time we spin out the components. The GRPC interfaces for those
smaller components will gradually offer good substitutes to the more
generic and slower-moving engine/RPC. One day we can deprecate engine/GRPC
and make it an optional component. But until we do, we can benefit from a
full-grpc environment, and use that to build kickass tools
Implement a engine/REST to engine/GRPC proxy. Because the engine/GRPC
interface is tailor-made for this problem, the implementation can be made
production-ready on a relatively short timeframe: let’s say arbitrarily
by
end of June [image removed]
Make the new proxy the official engine/REST implementation, and move it
to its own repository. No need to deprecate it! It can continue its life
as
a simple curl-friendly interface for container management. We will
continue
to support it in Docker, and perhaps make it an optional component in the
future once everyone switches to grpc (but there’s no rush).
To me the support statement is the critical piece of this. I wouldn’t want
it
to become stale over time. While I expect the GRPC interfaces to be richer
I’m still hoping that the REST APIs continue to grow as moby adds features.
So REST might be for 80-90% of the usecases but GRPC gives the full feature
set.
Or another way I view it… as the opinionated Docker CLI grows it would be
nice to see the REST APIs grow too. Then today’s users who might view the
CLI and REST APIs as 2 alternatives of the same set of features can
continue
to have this choice going forward.
Meanwhile the monolith now exposes engine/GRPC instead of engine/REST by
end of June. We move it to its own repository (see https://forums.mobyproject.org/t/topic-find-a-good-an-non-confusing-
home-for-the-remaining-monolith/).
Now we have a functional full-GRPC platform, and we can start building
cooler tooling as early as July.
Over time we spin out the components. The GRPC interfaces for those
smaller components will gradually offer good substitutes to the more
generic and slower-moving engine/RPC. One day we can deprecate
engine/GRPC
and make it an optional component. But until we do, we can benefit from a
full-grpc environment, and use that to build kickass tools [image
removed]
Thoughts?
I don’t see a whole lot of benefit in having an interim grpc API that’s just the HTTP/REST-ish API as GRPC.
The current API is a kludge of REST and non-REST RPC semantics and rubbing GRPC on it doesn’t really fix anything, with the exception of automatic client generation… but there’s already many clients out there.
I also see sustaining the existing API by adding new functionality to it the as the same exact problem that introducing an interim GRPC API.
The problems with the API isn’t that it’s not GRPC but rather just that some decisions turned out to be not great and we continually have to work around them to make stuff happen.
Introducing an interim GRPC API is also another API we’ll have to support or break compatibility (again).
I don’t see a whole lot of benefit in having an interim grpc API
that’s just the HTTP/REST-ish API as GRPC.
The current API is a kludge of REST and non-REST RPC semantics and
rubbing GRPC on it doesn’t really fix anything, with the exception
of automatic client generation… but there’s already many clients out
there.
I also see sustaining the existing API by adding new functionality
to it the as the same exact problem that introducing an interim GRPC API.
The problems with the API isn’t that it’s not GRPC but rather just
that some decisions turned out to be not great and we continually
have to work around them to make stuff happen.
Introducing an interim GRPC API is also another API we’ll have to
support or break compatibility (again).
To be clear, to me its not the shape of the API that important, its
the presence of a REST API at all. If we end up changing to a new
REST API that’s different from v1 then that’s fine, as long as it
exists and aligns nicely with the CLI.
I’m sure backwards compat will matter to some folks, but right now
I’m just worried about no path forward vs a migration path.
having an intermediate gRPC API sounds like it’s only to make sure things get set in motion, but likely results in more issues down the line (support/compatibility), so I share the concerns mentioned above.
If we set clear goals, and a timeframe, I assume we can do without an intermediate thing, and still make sure things are moving
Create a new engine/GRPC API which directly maps to engine/REST, call-for-call.
Ok, so I think it would work if we regard it as an “internal” API that’s really used as
a step to allow moving the API while we work on the definitive API. However, the moment
that API is considered a “public” API, it will mean we’ll have to maintain the gRPC API as
well (gRPC has options for deprecating etc, but I’m not sure that’s a good thing to use
from the start - if we can avoid it)
What would be the advantage of the intermediate gRPC API if it has the same flaws
as the REST API (only having a different format over the wire).
Create a new engine/GRPC API which directly maps to engine/REST,
call-for-call.
Ok, so I think it would work if we regard it as an “internal” API
that’s really used as
a step to allow moving the API while we work on the definitive API.
However, the moment
that API is considered a “public” API, it will mean we’ll have to maintain
the gRPC API as
well (gRPC has options for deprecating etc, but I’m not sure that’s a good
thing to use
from the start - if we can avoid it)
What would be the advantage of the intermediate gRPC API if it has the
same flaws
as the REST API (only having a different format over the wire).
Move out the API component to a separate repository faster
Reach our goal of going “full gRPC” faster
At the cost of using a temporary gRPC API
Is that better than moving the API and vendoring the component in?
Big difference is that in Daniel’s proposal, we would only use a gRPC API
for a component once the design for that component’s API is ready. This
would cause less disruptions (someone implementing the temporary gRPC API
only to have to re-implement again because it was just temporary)
We should take into account that the client must be able to talk to older
daemons, which only talk REST, so still has to support the REST API besides
doing gRPC.
Maybe it’s my interpretation, so I’ll leave others to provide their input
as well.
Generation of client libraries is nice, but we also don’t really have this problem right now because the existing API is already ubiquitous.
CLI clients to do what? The ported API is going to be hamstrung to what the existing API can do.
Finishing up the open-api spec that we currently have would likely be less work and provide these benefits as well.
Documentation is already there on the existing API spec, certainly it’s more fragile than generation from proto specs, but it does already exist.
Can you explain what service discovery between components would mean in this case and why the ported API would benefit from it any more than having the existing API talking to the grpc backend?
