Let’s discuss the evolution of testing in the Moby Project – mainly
integration-cli tests and their future.
- Most of our integration tests (in
integration-clipackage) are using the
dockercli to execute — now that this CLI has be exctracted elsewhere we should not depend on it anymore.
- Some tests in this packages are more than just integration tests and could be seen as more end-to-end test of
docker. These should move to our
e2erepository and/or have a set of end-to-end tests for
mobyin the future.
- All the tests are in the same package, in huge files and quick a few of them are a little bit to noisy.
Here is a proposal (that I’ll try to keep up-to-date) of what should happen and in what order — based on our daily in the community slack.
integration-clipackage — i.e. not adding anymore tests in there that are using the
dockercli. Also add a check to validate that (i.e. if there is any addition in
integration-cli, the build fail).
- Create an
mobyapi integration tests
docker_api_*_test.gosuites there in small packages — i.e.
integration/image/buildfor build-related tests,
integration/container/runfor container run related tests,
integration/distributionfor push/pull related tests, etc…
- List tests that in
integration-clithat should move away, either in a end-to-end repository, as api tests or as unit tests.
apitests we should use the
clientpackage (it’s unit tested so risks of weirdiness should be limited)
There is more open-question too :
- Should we freeze the
integration-clior should we extract them in a repo ? Either way, we need a CI job that execute those on
moby/mobyfor an indeterminate time.
Should we use
client(docker go sdk) for the
integration-cli/requestpackage ? It’s probably a mix of those too but better discussing it.
- Which testing framework to use for those integration tests ?
go-check, should we do the same for
integrationapi tests or should we go back to
testifyfor example ?
If you think there is more point to discuss on that topics, feel free to talk about it, I’ll update that according to the discussions