Note prior to the Switchover to in-tree functional tests session at Kilo Summit.
TL;DR: death to unittest and unit tests.
There are many goals and considerations for this discussion; there's no point enumerating all of them here. Instead I'd like to explore two things that I see as opportunities provided by this change:
- A chance to change the style of at least some of our tests and some of our testing from inscrutable to something more readable (and thus useful for learning by humans).
- A chance to explore additional technologies to enhance the extent of coverage tests can provide both in gate and in developers' environments.
I'm not going to be shy about it: I find our tests much harder to read than I would prefer. This seems to be because of a combination of the abhorrent unittest (the Python package) style of testing and devotion to the concept of a "unit" test (and the resulting overuse of mocks). Much of the time it is extremely challenging to figure out if the test is really testing much. More than once I've had code which I gave a positive review come back with bugs after it merged. Bugs which would have been caught by better tests but I had assumed the existing tests were okay. If the tests had been more readable and/or less mock-ridden I would have caught the problem.
Maybe I'm just crap at reading code, but really, should I have to work so hard to read tests?
Simply put: If we're going to be writing new test suites, let's write them well, using readability and usefulness as a development tool as a higher priority than their identity as unit, regression, integration or functional tests.
What I want is for it to be cheap and easy for me to write and run effective tests before ever pushing to gerrit. I want to be ashamed if my code fails in the gate because that will be a clear sign that I didn't run the tests locally before asking for review. For this to work -- without a vast array of mocked or fake services -- it needs to be easy for tests to run services, at least in a limited form, and it needs to be easy to setup and teardown those services.
Some technologies we may wish to explore, not specifically to use them, but to farm for ideas:
- my declarative http spec
- Mike Bayers transactional db fixtures spec
- dox
- py.test, for simple assertion-based testing without the OO cruft