Should we freeze the integration-cli or should we extract them in a repo ? Either way, *we need a CI job that execute those on moby/moby for an indeterminate time.
I think we want to "freeze" them in the short term, they are most of our test coverage right now.
docker/e2e repo exists, I assume based on the name, that it will test the docker product, not the moby project, so we would end up with a cyclic dependency between them. Over time components that are removed from moby/moby will have their own tests suites and we should be able to remove some tests.
I see some potential problems/complications with freezing integration-cli
- We will need to make exceptions for fixing test flakes
- We will want to remove tests occasionally, which will probably involved some
+ lines in the diff, even if it's only removing a test (because of shared code, and table tests)
- Some tests make poor assumptions and could need to be changed because of a backend/api change. Do we require contributors to rewrite the test as an API test at that time?
Which testing framework to use for those integration tests ? integration-cli uses go-check, should we do the same for integration api tests or should we go back to testing with testify for example ?
I'm partial to testify. What does go-check provide?