One of the main goals of moby-core is to define a component architecture such that moby-core can be broken up into smaller pieces but still be able to be combined together to work as a unit.
This componentization work requires a redesign of the current moby-core architecture.
The purpose of this topic is to get the discussions around such an architecture started.
Some questions to pose for those interested:
What is a component?
a. Can a component live in process?
b. Can a component live out-of-process?
What are the (if any) existing natural separations in moby-core today that would be low-hanging fruit for componentization?
How are components registered and found?
How do components communicate?
Should moby-core be a hard-coded set of components or should it be able to swap out components?
a. If swap out, is this build-time or runtime?
b. How does this relate to the moby tool, which is designed to combine various components into a system?
Just a note for this discussion, there is some pre-existing (POC level) work to decouple various moby-core subsystems:
1.a. No. That is not a component.
b. Yes, it must. It must run in a container.
2.
Build, swarm and logging are the big ones (other than of course containerd). The REST API. Plugins. The built in stuff that plugins are equivalent to (eg libnetwork).
3.
At startup time (not during runtime, that was a mistake in plugin design IMHO). Via sockets in filesystem, or maybe sockets that are passed as file descriptors.
4.
grpc over unix sockets/named pipes.
5. moby-core is a fixed base set plus what are now plugins. If you want to replace the base set thats a another moby project, not core.
a. start up time. Build time is too inflexible, run time is too racy and dynamic.
b. the tool can assemble these to run them. You can also do it by hand, but it might be fiddly.
containerd, swarmkit, networking (this is further componentized into SD, LB, Gossip, Overlay, …), moby-core api,
In a static way possible. Declared and consumed at the bootup.
grpc over UDS
As per earlier discussions on this topic, the assembled components in the moby-core repo is a reference implementation of a container platform that satisfies 80% of the use-case/requirements and are swappable. With a proper GRPC API definitions, moby-core should be able to run with different component implementation.
@cpuguy83 on the networking side I already started separating the gossip DB from the rest of libnetwork and the next in my todo list will be Service Discovery. maybe we can have a chat together on this and exchange ideas.
It seems easy to say a component must be a container, but what about “the thing that runs containers” (for the purposes of this discussion I will assume that is containerd). containerd can’t itself be run in a container because it is the thing that runs containers.
So either containerd is not a moby component or we need to expand our definition.
The process isolation of a component is just one of many properties. I think questions 3 (discovery), and 4 (communication) are also part of this definition. We also need to answer:
how are components distributed/installed?
how are component lifecycles managed ? (ex: started, stopped, reloaded on failure, etc)
how are components configured and reconfigured?
As mentioned, while containerd seems obvious, it doesn’t match our earlier definition of a component.
Each of these is going to be a massive effort, so let’s not worry about trying to identify a bunch of them. Let’s try to identify 3 or so options, and pick just a single one of those options as the “first component”.
As mentioned, these seem like good options: build, swarmkit, volumes, and plugins.
Both responses so far (and also afaik Solomon’s recommendation) is to use unix domain sockets. The question is how does each component receive the path to a socket?
Does each “component interface” (ex: “the plugin interface”) have a predefined name, and the path is hardcoded based on the name?
Does a plugin have to request the paths for each component it depends on at startup?
Does moby-core (aka “the framework”) pass a full list of paths for all components to each component?
Isn’t “build-time swap out” effectively equivalent to “hardcoded” ? If not, how are they different?
There’s already enough work to do, we should start with the easiest option, which I think is build-time. Run-time swap out introduces the problem of how to distribute and install components that aren’t built in. Let’s not take on that extra work when there is already plenty of other higher priority things to work on.
What do you mean by logging? container logging? or logging from each component? Wouldn’t container logging just be part of containerd?
There were 2 mentions of this, but I don’t think the REST API qualifies as “low-hanging fruit for componentization”. To provide the REST API we need a gRPC interface for every other component. That means we have to complete all other component work before we can actually provide the REST API as a component. This was discussed in https://forums.mobyproject.org/t/proposal-engine-v1-rest-api-transition
Another option is “events”. Events will be generated from every component, and they will most likely need to be aggregated and provided from a single API endpoint.
An “events” component could be implemented using kafka (or an existing message queue implementation), so that the client can consumer events as a stream.
Only slightly pedantic:
If all components must be in a container, and containerd is a component, does this imply that containerd must be in a container?
Are components aware of each other or should they communicate through something central (e.g. moby-core component)?
does this imply that containerd must be in a container
I think it should.
While containerd 1.0 supports namespacing clients and moby-core could technically piggy-back off of the host’s containerd if one was there, as I see it right now there are only two benefits to this:
Easier for user to get a full view of the system, metrics, etc. (not something to downplay)
One less process (meh).
Meanwhile we lose some isolation here, namely mount isolation where mounts are frequently leaked into other process’ namespaces which causes all kinds of issues. Being able to control this a bit better than we do today would be exceptionally nice.
We also lose control of version in this case.
What I’d like to do to get this started is get moby-core building via moby/tool.
To begin with we’ll just start with moby-core as a single component.
Beyond that there are a couple of pieces in moby-core today that are “almost components” that could pretty easily be extracted out, I’d like to pick 2 to work on: containerd and libnetwork.
They idea here is the yaml spec for moby/tool will get larger (and the main moby-core daemon smaller) as we add these components.
WDYT? Is anyone interested in working on either of these?
“Discourse” merged one of my draft responses for the previous thread and the reply to this thread
I withdrew my previous reply and posting a new one here…
Good idea. @flavio.crisciani is currently working on componentizing the gossip and Service Dicovery layers and I think they are perfect candidates for this approach.
In addition, I think we can show multiple flavors of moby with different UX components such as portainer, swarmpit, etc…
This topic seems to have lost track of the original question. What does the moby/tool have to do with architecture? I believe that tool is only for packaging and distribution (build time not run time) so it should have very little to no impact on the architecture.
I don’t see how this is possible. containerd should be the “thing” that runs containers, so if a component must be a conatiner, how can it run itself? It must already be running before any component can be run, because components would run as containerd containers.
I agree moby-core should not assume a containerd already exists on the host. moby-core should start it’s own containerd. So I think the question still remains. Do we relax the definition of a component and say that some components (containerd) are special and can run outside a container?
This sounds good, but I’m not understanding what it has to do with run-time architecture.
I don’t understand how we can start this work without first resolving these architecture questions.
I think this raises the question of what granularity do we want for components.
If we’re talking about very fine grained components then we’ll probably want some config file that allows us to define which the dependencies of a component. Otherwise it’s going to be extremely difficult to assemble a system.
I think it might be better to start with larger components so that we have fewer interfaces to manage. Each component can structure itself as necessary. Once we have the larger pieces split out we can look at splitting out smaller components from there.
This sounds good, but I’m not understanding what it has to do with run-time architecture.
Because there is up-front work to be done to actually package/build this.
I also think with a static component config there is a lot less to do up front in terms of breaking things apart and changing the whole architecture.
I don’t see how this is possible. containerd should be the “thing” that runs containers
Indeed. But we can also use runc to do this.
Right now linuxkit uses https://github.com/linuxkit/linuxkit/tree/master/pkg/containerd/cmd/service to fire up a containerd instance, and then start services defined from the moby build output.
We can just use this as the base at least up front. But still I think moby-core’s containerd should run in a container (note, this doesn’t mean a docker-style container)
Planning to have a quick sync on this topic for those who want to join: https://docker.zoom.us/j/579405180 @ 10:30AM PDT/ 5:30 PM UTC today (sorry for the last minute notice).
We shouldn’t define a component as a container. This couples implementation with behavior. As was already discussed, containerd would be a particularly problematic case. There are other components required to implement components that may be hard to bootstrap if we make containers a requirement. It is possible to make this model work if we have some sort of “bootstrap” layer. There are also cases where it might make sense to implement several components in a single container or implement a component as the construction of several containers.
In general, we should define components by their behavior. Let’s say we have generic model where components register themselves with a discovery service. They might announce that they implement a specific interface, over UDS or other means. They might also declare that they require certain resources to implement their component, creating a dependency graph. We may also define lifecycle as part of this. This functionality would fall into the “behavior” of the component.
That said, we should make it really easy to implement a component as a container and most components should be containers.
I will qualify this with none of it is gospel, only proposals.
We intend to open a PR to moby/moby detailing these items further after discussion here. @dnephin@mavenugo@flavio.crisciani Please amend or add to things I’ve missed from the call (or misinterpreted)
What is a component?
All components must be in a container
i. A “container” here does not mean a “docker run” style container
ii. Component containers can be fine-tuned (via moby-tool, for example) to what it needs
Components are statically defined and cannot be changed at runtime
Components interface over node-local sockets (e.g. unix sockets, named pipes) with GRPC
Components are opaque to moby-core
i. Example, moby-core doesn’t care if a component consists of multiple processes, or even sub-containers as long as there is one API to talk to.
ii. Not moby-core’s responsibility to monitor or interact with other things that the component spins up (e.g. maybe the component creates its own set of containers).
Init/Monitoring/Discovery
We want to have a moby-init/supervisor component that is responsible for init and health checking
For day 0 (i.e. initial testing, POC work) we can hard-code the discovery aspect (e.g. this is containerd, it’s socket is at a special path that will always be the same)
Overall
There is quite some work to do here to be in an ideal architecture. We do not want to short-change this ideal architecture (ever).
Breaking this work up into smaller, actionable pieces is neccessary to make progress. Seeing how this will work with some of the more well defined pieces, such as containerd and libnetwork, will help define what other components are, how discovery occurs, etc.
Future meetings
We want to setup a regular meeting cadence to sync up on components. I will be sending out an invite for next week.
I expect that these will be weekly for now.
DM me your email address if you want to be added to the calendar invite, we’ll also post a topic on the forum.
Just a reminder, we will be having a call today (https://docker.zoom.us/j/191433912) at 1PM EDT/6PM CET
Going forward I’ll try and schedule this a bit better for folks not in the US.
