# Proposal: Engine V1 REST API Transition
**URL:** https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49
**Category:** Architecture
**Created:** [May 18, 2017, 4:06pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49 "2017-05-18T16:06:14Z")
**Posts on this page:** 20
**Page:** 1
### Author:  [@dnephin](https://forums.mobyproject.org/u/dnephin)
#### Post date: [May 18, 2017, 4:06pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/1 "2017-05-18T16:06:14Z")
This is a proposal for [https://github.com/moby/moby/issues/32873](https://github.com/moby/moby/issues/32873)
## Background
The application formerly known as the Docker Engine is going to be split into components. Each component will provide an API. These APIs will likely use gRPC.
## Problem
There are many consumers of the existing Docker Engine REST API. We need to support consumers of this API while components are being extracted so that all consumers have adequate time to transition to the new APIs.
## Proposal
We can support all of the current API users by creating a component which provides the existing REST API. This component will use the routes in `api/server/router/` with a backend implementation which performs APIs calls to the gRPC API.
### Plan
1. When the first component with a gRPC interface is extracted implement a new Backend (the interface required by the router) which performs gRPC calls to the component.
2. Repeat step 1 as each component is removed
3. After all the components have been removed, move `api/server/` and the new backend implementations into a new `docker/rest-api-v1` repo.
4. Formally deprecate the legacy API by setting a date on which the legacy REST API component will no longer be included in the Docker CE/EE releases.
## Alternative Proposals
The following proposals were considered, but are not recommended.
### Extract the rest-api-v1 component before other components
Similar to the proposal above, except step 3 is done before step 1. This option is not recommended because it re-introduces the problems we had with the `docker/engine-api` repo, which will **significantly** increase the work to extract components. Since the primary goal is to support the extraction of components this works against us.
The problem occurs when a component is removed. In-process calls must be replaced by gRPC calls, but those calls would be in a vendored repo. So a component can’t be removed from the monolith until the rest-api has been updated and vendored back into the monolith. But the rest-api can’t be tested without the monolith because during the transition a significant part of the backend implementation will still be provided by the monolith. With an extracted rest-api component extracting components requires coordination between three different repos. By removing it at the end we don’t need to coordinate any changes, they can be performed sequentially.
### Client Library or SDK
A client library can be created for all major languages. As new gRPC endpoints are created the library can be updated to make API calls to the new endpoints, and remove the old REST API endpoints.
This option is not recommended for the following reasons:
- Users who have applications in languages not covered by the library will have no option for transiting to the new APIs
- The work required to implement the library in one language is comparable to the work required to implement the resp-api component, however a library must be created for multiple languages, which multiplies the required work by the number of supported languages. The total work required for this solution is significantly higher.
- There are currently only two officially supported languages. Creating libraries in any other language is wasted effort because it builds against an API that is being deprecated and removed.
- All the libraries will need to provide an interface that resembles the legacy APIs instead of the new APIs. Once the transition is complete users will be stuck with a non-optimal interface and will require another transition to a new version of the library.
### Each component implements a legacy REST API
Instead of a single REST component, each new component could implement part of the old REST API.
This option is not recommended for the following reasons:
- It increases the work required to create a new component, which is the critical path
- The boundaries between components would be influenced by the existing API
- The monolith would still be required to act as a proxy to the API in each component, which violates one of the goals of the new architecture.
---
### Author:  [@thaJeztah](https://forums.mobyproject.org/u/thaJeztah)
#### Post date: [May 18, 2017, 4:51pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/2 "2017-05-18T16:51:00Z")
I like the first proposal. Once migration to gRPC is complete, the `docker/rest-api-v1` repository can live its live as its own component, and used as a compatibility layer for those that need it.
> Extract the rest-api-v1 component before other components
> Similar to the proposal above, except step 3 is done before step 1. This option is not recommended because it re-introduces the problems we had with the docker/engine-api repo, which will significantly increase the work to extract components. Since the primary goal is to support the extraction of components this works against us.
I would’ve loved this alternative proposal (move the API out now, vendor it in, and gradually add gRPC endpoints/API’s, but discussing with @dnephin brought up the downsides, so because of that, I agree it may not be a good choice
---
### Author:  [@mlaventure](https://forums.mobyproject.org/u/mlaventure)
#### Post date: [May 18, 2017, 4:51pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/3 "2017-05-18T16:51:08Z")
I agree with the recommended solution.
I’d just have this to add regarding a possible SDK:
IMHO such a SDK will inevitably be opinionated in regard of which options/endpoints it exposes and what operations it allows. Hence, I think it only makes sense if such a SDK is to be part of a product (e.g. Docker), not this project. The GRPC interface in themselves shall be enough of a client IMHO.
As such, I wouldn’t have included it as a possible solution for the mobyproject, but I expect downstream projects to want to go this way to offer some convenience/added value to their user and hide away unneeded details/complexity (e.g. in most common scenarios noone cares about most of the fields that can be changed in the REST API).
---
### Author:  [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)
#### Post date: [May 19, 2017, 2:21am UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/4 "2017-05-19T02:21:42Z")
+1 to the proposal (and not the alts)
---
### Author:  [@AkihiroSuda](https://forums.mobyproject.org/u/AkihiroSuda)
#### Post date: [May 19, 2017, 2:29am UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/5 "2017-05-19T02:29:33Z")
> After all the components have been removed, move api/server/ and the new backend implementations into a new docker/rest-api-v1 repo.
> Formally deprecate the legacy API by setting a date on which the legacy REST API component will no longer be included in the Docker CE/EE releases.
Can you please consider placing the repo under `moby/` rather than `docker/`, so that 3rd parties can easily corporate on the compatibility issue, even after the Docker product. drop the support for the legacy API?
---
### Author:  [@bmitch3020](https://forums.mobyproject.org/u/bmitch3020)
#### Post date: [May 22, 2017, 2:45pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/6 "2017-05-22T14:45:20Z")
Nice write up and the recommended solution is much better than the other options.
> [@dnephin](#):
>
> Plan
>
> 1. When the first component with a gRPC interface is extracted implement a new Backend (the interface required by the router) which performs gRPC calls to the component.
> 2. Repeat step 1 as each component is removed
> 3. After all the components have been removed, move api/server/ and the new backend implementations into a new docker/rest-api-v1 repo.
> 4. Formally deprecate the legacy API by setting a date on which the legacy REST API component will no longer be included in the Docker CE/EE releases.
Does #4 mean that any code using the REST API directly will need a rewrite to gRPC? If so, is there a downside of leaving a permanent REST to gRPC gateway in place other than needing to maintain an extra piece of code (I’m thinking about whether it could be moved to contrib instead of completely removed)?
How long of a deprecation window are we planning?
---
### Author:  [@dnephin](https://forums.mobyproject.org/u/dnephin)
#### Post date: [May 23, 2017, 3:24pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/7 "2017-05-23T15:24:06Z")
> Does #4 mean that any code using the REST API directly will need a rewrite to gRPC?
Yes, any code would eventually need to be rewritten to use the new APIs.
> is there a downside of leaving a permanent REST to gRPC gateway in place
The place it exists now isn’t going to exist in the future, so leaving it in place isn’t really an option. The downside is that it’s maintenance overhead. I expect it will continue to live on as a community project.
> How long of a deprecation window are we planning?
I think it’s too early to say. It will depend on when other APIs available. I would personally say that we would need to wait at least a year from the time when the last API is converted, so probably ~2 years.
---
### Author:  [@duglin](https://forums.mobyproject.org/u/duglin)
#### Post date: [May 23, 2017, 3:54pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/8 "2017-05-23T15:54:10Z")
I agree with the overall approach, but I disagree with #4 - I think not
supporting a REST API going forward is a mistake. You’re basically chopping
off all vanilla-HTTP/curl users. As mentioned, not everything has a grpc
library, nor will everyone want to use one - sometimes just a simple
curl/HTTP-GET is all that’s needed.
thanks
-Doug
---
### Author:  [@dnephin](https://forums.mobyproject.org/u/dnephin)
#### Post date: [May 23, 2017, 4:21pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/9 "2017-05-23T16:21:50Z")
> I think not supporting a REST API going forward is a mistake. You’re basically chopping
> off all vanilla-HTTP/curl users.
I agree that there is a lot of benefit in having a vanilla HTTP+JSON API. However, I think that’s a separate issue. This thread is only about the transition path for the V1 API. We can’t support the legacy API forever. I think the best option is a long deprecation period with a well communicated deadline. After that deadline the component can be maintained by anyone who is still interested in the V1 API.
I’ve tried to keep this proposal separate from discussions about the V2 API. I think if the V2 API is gRPC, there may be ways to support those APIs with a separate component that uses something like [GitHub - grpc-ecosystem/grpc-gateway: gRPC to JSON proxy generator following the gRPC HTTP spec](https://github.com/grpc-ecosystem/grpc-gateway).
---
### Author:  [@shykes2](https://forums.mobyproject.org/u/shykes2)
#### Post date: [May 24, 2017, 4:27am UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/10 "2017-05-24T04:27:07Z")
> dnephin [https://forums.mobyproject.org/u/dnephin](https://forums.mobyproject.org/u/dnephin)
> May 23
>
> I think not supporting a REST API going forward is a mistake. You’re
> basically chopping
> off all vanilla-HTTP/curl users.
>
> I agree that there is a lot of benefit in having a vanilla HTTP+JSON API.
> However, I think that’s a separate issue. This thread is only about the
> transition path for the V1 API. We can’t support the legacy API forever. I
> think the best option is a long deprecation period with a well communicated
> deadline. After that deadline the component can be maintained by anyone who
> is still interested in the V1 API.
>
> I’ve tried to keep this proposal separate from discussions about the V2
> API. I think if the V2 API is gRPC, there may be ways to support those APIs
> with a separate component that uses something like
> [GitHub - grpc-ecosystem/grpc-gateway: gRPC to JSON proxy generator following the gRPC HTTP spec](https://github.com/grpc-ecosystem/grpc-gateway).
Maybe there is a middle ground here which would address the 2 outstanding
issues in this thread:
1. I agree with Sebastiaan that it would be preferable to split out the
REST API in its own repository right away, but we haven’t found a
convenient way to do that.
2. I agree with Doug that some users will always be interested in a simple
curl-friendly compat layer.
3. I would really like grpc to be a first-class citizen for all components
by the end of June. With the current plan that probably wouldn’t happen
until the end of the year. Until we have full grpc support, including in
the monolith, we can’t develop common tooling to automate the generation of
client libraries, command-line clients, documentation, service discovery
between components, etc. That tooling would bring lots of benefits to Moby.
Without it, the project is basically on hold.
What if we adapted Daniel’s proposal like so:
- Create a new engine/GRPC API which directly maps to engine/REST,
call-for-call. No need to get fancy or idiomatic: just map REST operations
to corresponding grpc calls in a basic, almost naive to make proxying very
easy.
- Implement a engine/REST to engine/GRPC proxy. Because the engine/GRPC
interface is tailor-made for this problem, the implementation can be made
production-ready on a relatively short timeframe: let’s say arbitrarily by
end of June 🙂
- Make the new proxy the official engine/REST implementation, and move it
to its own repository. No need to deprecate it! It can continue its life as
a simple curl-friendly interface for container management. We will continue
to support it in Docker, and perhaps make it an optional component in the
future once everyone switches to grpc (but there’s no rush).
- Meanwhile the monolith now exposes engine/GRPC instead of engine/REST by
end of June. We move it to its own repository (see
[https://forums.mobyproject.org/t/topic-find-a-good-an-non-confusing-home-for-the-remaining-monolith/](https://forums.mobyproject.org/t/topic-find-a-good-an-non-confusing-home-for-the-remaining-monolith/)).
Now we have a functional full-GRPC platform, and we can start building
cooler tooling as early as July.
- Over time we spin out the components. The GRPC interfaces for those
smaller components will gradually offer good substitutes to the more
generic and slower-moving engine/RPC. One day we can deprecate engine/GRPC
and make it an optional component. But until we do, we can benefit from a
full-grpc environment, and use that to build kickass tools 🙂
Thoughts?
---
### Author:  [@duglin](https://forums.mobyproject.org/u/duglin)
#### Post date: [May 24, 2017, 2:41pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/11 "2017-05-24T14:41:36Z")
AM:
> Implement a engine/REST to engine/GRPC proxy. Because the engine/GRPC
> interface is tailor-made for this problem, the implementation can be made
> production-ready on a relatively short timeframe: let’s say arbitrarily
> by
> end of June [image removed]
>
> Make the new proxy the official engine/REST implementation, and move it
> to its own repository. No need to deprecate it! It can continue its life
> as
> a simple curl-friendly interface for container management. We will
> continue
> to support it in Docker, and perhaps make it an optional component in the
> future once everyone switches to grpc (but there’s no rush).
To me the support statement is the critical piece of this. I wouldn’t want
it
to become stale over time. While I expect the GRPC interfaces to be richer
I’m still hoping that the REST APIs continue to grow as moby adds features.
So REST might be for 80-90% of the usecases but GRPC gives the full feature
set.
Or another way I view it… as the opinionated Docker CLI grows it would be
nice to see the REST APIs grow too. Then today’s users who might view the
CLI and REST APIs as 2 alternatives of the same set of features can
continue
to have this choice going forward.
> Meanwhile the monolith now exposes engine/GRPC instead of engine/REST by
> end of June. We move it to its own repository (see
> [https://forums.mobyproject.org/t/topic-find-a-good-an-non-confusing-](https://forums.mobyproject.org/t/topic-find-a-good-an-non-confusing-)
> home-for-the-remaining-monolith/).
> Now we have a functional full-GRPC platform, and we can start building
> cooler tooling as early as July.
>
> Over time we spin out the components. The GRPC interfaces for those
> smaller components will gradually offer good substitutes to the more
> generic and slower-moving engine/RPC. One day we can deprecate
> engine/GRPC
> and make it an optional component. But until we do, we can benefit from a
> full-grpc environment, and use that to build kickass tools [image
> removed]
> Thoughts?
Overall, it sounds good to me.
thanks!
-Doug
---
### Author:  [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)
#### Post date: [May 24, 2017, 3:26pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/12 "2017-05-24T15:26:06Z")
I don’t see a whole lot of benefit in having an interim grpc API that’s just the HTTP/REST-ish API as GRPC.
The current API is a kludge of REST and non-REST RPC semantics and rubbing GRPC on it doesn’t really fix anything, with the exception of automatic client generation… but there’s already many clients out there.
I also see sustaining the existing API by adding new functionality to it the as the same exact problem that introducing an interim GRPC API.
The problems with the API isn’t that it’s not GRPC but rather just that some decisions turned out to be not great and we continually have to work around them to make stuff happen.
Introducing an interim GRPC API is also another API we’ll have to support or break compatibility (again).
---
### Author:  [@duglin](https://forums.mobyproject.org/u/duglin)
#### Post date: [May 24, 2017, 3:43pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/13 "2017-05-24T15:43:11Z")
> I don’t see a whole lot of benefit in having an interim grpc API
> that’s just the HTTP/REST-ish API as GRPC.
> The current API is a kludge of REST and non-REST RPC semantics and
> rubbing GRPC on it doesn’t really fix anything, with the exception
> of automatic client generation… but there’s already many clients out
> there.
> I also see sustaining the existing API by adding new functionality
> to it the as the same exact problem that introducing an interim GRPC API.
> The problems with the API isn’t that it’s not GRPC but rather just
> that some decisions turned out to be not great and we continually
> have to work around them to make stuff happen.
> Introducing an interim GRPC API is also another API we’ll have to
> support or break compatibility (again).
To be clear, to me its not the shape of the API that important, its
the presence of a REST API at all. If we end up changing to a new
REST API that’s different from v1 then that’s fine, as long as it
exists and aligns nicely with the CLI.
I’m sure backwards compat will matter to some folks, but right now
I’m just worried about no path forward vs a migration path.
-Doug
---
### Author:  [@thaJeztah](https://forums.mobyproject.org/u/thaJeztah)
#### Post date: [May 24, 2017, 4:09pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/14 "2017-05-24T16:09:15Z")
having an intermediate gRPC API sounds like it’s only to make sure things get set in motion, but likely results in more issues down the line (support/compatibility), so I share the concerns mentioned above.
If we set clear goals, and a timeframe, I assume we can do without an intermediate thing, and still make sure things are moving
---
### Author:  [@shykes2](https://forums.mobyproject.org/u/shykes2)
#### Post date: [May 24, 2017, 4:28pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/15 "2017-05-24T16:28:38Z")
Sebastiaan: what issues specifically do you see?
Brian: I say you can’t think of benefits of switching to all-grpc quickly,
but I lisged several. Can you address them?
Thanks both
---
### Author:  [@thaJeztah](https://forums.mobyproject.org/u/thaJeztah)
#### Post date: [May 24, 2017, 4:36pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/16 "2017-05-24T16:36:40Z")
@shykes this part;
> Create a new engine/GRPC API which directly maps to engine/REST, call-for-call.
Ok, so I think it would _work_ if we regard it as an “internal” API that’s really used as
a step to allow moving the API while we work on the definitive API. However, the moment
that API is considered a “public” API, it will mean we’ll have to maintain the gRPC API as
well (gRPC has options for deprecating etc, but I’m not sure that’s a good thing to use
from the start - if we can avoid it)
What would be the advantage of the intermediate gRPC API if it has the same flaws
as the REST API (only having a different format over the wire).
(just trying to understand)
---
### Author:  [@shykes2](https://forums.mobyproject.org/u/shykes2)
#### Post date: [May 24, 2017, 4:42pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/17 "2017-05-24T16:42:23Z")
> thaJeztah [https://forums.mobyproject.org/u/thajeztah](https://forums.mobyproject.org/u/thajeztah)
> May 24
>
> @shykes [https://forums.mobyproject.org/u/shykes](https://forums.mobyproject.org/u/shykes) this part;
>
> Create a new engine/GRPC API which directly maps to engine/REST,
> call-for-call.
>
> Ok, so I think it would _work_ if we regard it as an “internal” API
> that’s really used as
> a step to allow moving the API while we work on the definitive API.
> However, the moment
> that API is considered a “public” API, it will mean we’ll have to maintain
> the gRPC API as
> well (gRPC has options for deprecating etc, but I’m not sure that’s a good
> thing to use
> from the start - if we can avoid it)
>
> What would be the advantage of the intermediate gRPC API if it has the
> same flaws
> as the REST API (only having a different format over the wire).
>
> (just trying to understand)
I listed several advantages.
---
### Author:  [@thaJeztah](https://forums.mobyproject.org/u/thaJeztah)
#### Post date: [May 24, 2017, 5:06pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/18 "2017-05-24T17:06:00Z")
The advantage I see is that we’re able to
- Move out the API component to a separate repository faster
- Reach our goal of going “full gRPC” faster
- At the cost of using a _temporary_ gRPC API
- Is that better than moving the API and vendoring the component in?
Big difference is that in Daniel’s proposal, we would only use a gRPC API
for a component once the design _for_ that component’s API is ready. This
would cause less disruptions (someone implementing the temporary gRPC API
only to have to re-implement again because it was just temporary)
We should take into account that the client must be able to talk to older
daemons, which only talk REST, so still has to support the REST API besides
doing gRPC.
Maybe it’s my interpretation, so I’ll leave others to provide their input
as well.
---
### Author:  [@cpuguy83](https://forums.mobyproject.org/u/cpuguy83)
#### Post date: [May 24, 2017, 5:16pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/19 "2017-05-24T17:16:50Z")
> [@Solomon\_Hykes](#):
>
> we can’t develop common tooling to automate the generation of
> client libraries, command-line clients, documentation, service discovery
> between components, etc.
Generation of client libraries is nice, but we also don’t really have this problem right now because the existing API is already ubiquitous.
CLI clients to do what? The ported API is going to be hamstrung to what the existing API can do.
Finishing up the open-api spec that we currently have would likely be less work and provide these benefits as well.
Documentation is already there on the existing API spec, certainly it’s more fragile than generation from proto specs, but it does already exist.
Can you explain what service discovery between components would mean in this case and why the ported API would benefit from it any more than having the existing API talking to the grpc backend?
---
### Author:  [@shykes2](https://forums.mobyproject.org/u/shykes2)
#### Post date: [May 24, 2017, 5:28pm UTC](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49/20 "2017-05-24T17:28:21Z")
I’m sorry, I don’t have the energy for this. Forget I said anything.
[Next page](https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition/49.md?page=2)