Things to do Automated Testing Framework =========================================================================== Last revised: November 30th, 2010 This document includes the list of things that need to be done in ATF that are most requested by the users. This information used to be available in an ad-hoc bug tracker but that proved to be a bad idea. I have collected all worthy comments in here. Please note that most work these days is going into Kyua (see http://code.google.com/p/kyua/). The ideas listed here apply to the components of ATF that have *not* been migrated to the new codebase yet. For bug reports or ideas that apply to the components that already have been migrated, please use the bug tracker in the URL above. Similarly, whenever a component is migrated, the ideas in this file should be revised and migrated to the new bug tracker where appropriate. --------------------------------------------------------------------------- Add build-time checks to atf-sh The 0.7 release introduced build-time tests to atf-c and atf-c++, but not to atf-sh. Expose the functionality to the shell interface. This will probably require writing an atf-build utility that exposes the C code and can be called from the shell. --------------------------------------------------------------------------- Revisit what to do when an Atffile lists a non-existent file --------------------------------------------------------------------------- Add ATF_CHECK* versions to atf-c++ to support non-fatal tests --------------------------------------------------------------------------- Implement race-condition tests gcooper: I would think that stress/negative tests would be of more value than race condition tests (they're similar, but not exactly the same in my mind). In particular, 1. Feed through as much data as possible to determine where reporting breaks down. 2. Feed through data quickly and terminate ASAP. The data should be captured. Terminate child applications with unexpected exit codes and signals (in particular, SIGCHLD, SIGPIPE, exit codes that terminate, etc). 3. Open up a file descriptor in the test application, don't close the file descriptor. 4. fork(2) a process; don't wait(2) for the application to complete. There are other scenarios that could be exercised, but these are the ones I could think of off the topic of my head. -- jmmv: 1. The thing is: how do you express any of this in a portable/abstract interface? How do you express that a test case "receives data"? What does that exactly mean? I don't think the framework should care about this: each test should be free to decide where its data is and how to deal with it. 2. Ditto. 3. Not sure I understand your request, but testing for "unexpected exit codes" is already supported. See wiki:DesignXFail for the feature design details. 4. What's the problem with this case? The test case exits right away after terminating the execution of its body; any open file descriptors, leaked memory, etc. die with it. 5. forking and not waiting for a subprocess was a problem already addressed. I kinda have an idea of what Antti means with "race condition tests", but every time I have tried to describe my understanding of matters I seem to be wrong. Would be nice to have a clear description of what this involves; in particular, what are the expectations from the framework and how should the feature be exposed. As of now, what I understand by "race condition test" is: a test case that exercises a race condition. The test case may finish without triggering the race, in which case it just exists with a successful status. Otherwise, if the race triggers, the test case gets stuck and times out. The result should be reported as an "expected failure" different from timeout. -- pooka: Yup. Plus some atf-wide mechanism for the operator to supply some kind of guideline if the test should try to trigger the race for a second or for an hour. -- jmmv: Alright. While mocking up some code for this, I think that your two requests are complementary. On the one hand, when you are talking about a "race condition" test you really mean an "expected race condition" test. Correct? If so, we need to extend the xfail mechanism to add one more case, which is to report any failures as a race condition error and, if there is no failure, report the test as successful. On the other hand, the atf-wide mechanism to support how long the test should run for can be thought as a "stress test" mechanism. I.e. run this test for X time / iterations and report its results regularly without involving xfail at all. So, with this in mind: * For a test that triggers an unfixed race condition, you set xfail to race mode and define the test as a stress test. Any failures are reported as expected failures. * For a test that verifies a supposedly-fixed race condition, you do *not* set xfail to race mode, and only set the test to stress test. Any failures are reported as real failures. These stress test cases implement a single iteration of the test and atf-run is in charge of running the test several times, stopping on the first failure. Does that make sense? --------------------------------------------------------------------------- Implement ATF_REQUIRE_ERRNO pooka: Most of the lines in tests against system functionality are: if (syscall(args) == -1) atf_tc_fail_errno("flop") Some shorthand would be helpful, like ATF_REQUIRE_ERRNO(syscall(args)) Also, a variant which allows arbitrary return value checks (e.g. "!= 0" or "< 124" or "!= size") would be nice. -- gcooper: There's a problem with this request; not all functions fail in the same way ... in particular compare the pthread family of functions (which return errno) vs many native syscalls. Furthermore, compare some fcntl-like syscalls vs other syscalls. One size fits all solutions may not be a wise idea in this case, so I think that the problem statement needs to be better defined, because the above request is too loose. FWIW, there's also a TEST macro in LTP, which tests for non-zero status, and sets an appropriate set of global variables for errnos and return codes, respectively. It was a good idea, but has been mostly abandoned because it's too difficult to define a success and failure in a universal manner, so I think that we need to be careful with what's implemented in ATF to not repeat the mistakes that others have made. -- jmmv: I think you've got a good point. This was mostly intended to simplify the handling of the stupid errno global variable. I think this is valuable to have, but maybe the macro/function name should be different because _ERRNO can be confusing. Probably something like an ATF_CHECK_LIBC / ATF_CHECK_PTHREAD approach would be more flexible and simple. =========================================================================== vim: filetype=text:textwidth=75:expandtab:shiftwidth=2:softtabstop=2