-
Notifications
You must be signed in to change notification settings - Fork 0
LibarchiveAddingTest
How to extend the libarchive test suites.
Libarchive is now a fairly complex piece of software that runs on a number of different platforms. A thorough test suite is essential both for verifying new ports and for ensuring that future changes don't break existing functionality.
Any significant change to libarchive, including most bug fixes, should be accompanied by new tests or changes to existing tests. This article explains how the libarchive test suites work and how to extend them.
Each major component--libarchive, bsdtar, and bsdcpio--has a test program that exercises the functionality of that component. These test programs are compiled in the same way that the rest of the libarchive suite is compiled.
To run the test programs, you need to give them two pieces of information:
- The full path to the directory holding the "reference files"
- The full path to the executable program being tested (not applicable to libarchive_test since libarchive is compiled into the test program)
The bsdtar_test and bsdcpio_test programs run bsdtar or bsdcpio repeatedly; they need the full path to the appropriate executable. Although bsdtar_test and bsdcpio_test are specifically intended for testing bsdtar and bsdcpio, they should be usable for testing other tar and cpio programs. In fact, running these test programs against other tar implementations is a good way to verify that the test programs themselves are working correctly.
When run, the test programs output a progress message for each test, an error message for each failed assertion, and a final summary:
Running tests in: /tmp/bsdtar_test.2009-02-17T21.30.40-000
Reference files will be read from: /home/tim/libarchive/trunk/tar/test
Running tests on: /home/tim/libarchive/trunk/bsdtar
Exercising: bsdtar 2.6.900a - libarchive 2.6.900a
0: test_0
1: test_basic
2: test_copy
test_copy.c:171: Assertion failed: Ints not equal
0=0
lstat(name2 + 3, &st2)=-1
test_copy.c:171: Failed 264 times
3: test_getdate
4: test_help
... more output omitted ...
1 of 13 tests reported failures
Total of 105097 assertions checked.
Total of 264 assertions failed.
Total of 0 assertions skipped.The header lines here record:
- the directory that will be used for scratch files during the test. If a test fails, the scratch files will be left behind in this directory for further debugging.
- The directory from which the reference files will be read.
- For bsdtar_test and bsdcpio_test, the full path to the executable being exercised.
- Basic version information about the target.
Each test program consists of a number of "tests". Each test has a name and is implemented in a C source file with the same name as the test. Tests work by performing some series of operations and making "assertions" about the results. For example, many of the libarchive tests open and read an archive and assert that particular operations succeeded or failed. (Yes, it is often important to verify that illegal requests generate appropriate errors.)
Here is a somewhat edited excerpt from `test_compat_zip`, which verifies compatibility with various ZIP format archives:
DEFINE_TEST(test_compat_zip)
{
/* ... setup omitted ... */
assert((a = archive_read_new()) != NULL);
assertEqualInt(ARCHIVE_OK,
archive_read_support_compression_all(a));
assertEqualInt(ARCHIVE_OK,
archive_read_support_format_all(a));
extract_reference_file(name);
assertEqualInt(ARCHIVE_OK,
archive_read_open_filename(a, name, 10240));
/* Read first entry. */
assertEqualInt(ARCHIVE_OK, archive_read_next_header(a, &ae));
assertEqualString("META-INF/MANIFEST.MF", archive_entry_pathname(ae));The `assertXXXX` macros check that their arguments satisfy certain conditions. If the assertion fails--for example, if the name of the first entry is not "META-INF/MANIFEST.MF"--the macro will report the problem.
There are two important differences between the `assertXXX` macros used in these test harnesses and the ISO C standard `assert` macro: First, these assert macros don't exit on failure. By default, they report the failure and return zero (the C notion of "false"). Second, these macros include variants that perform a variety of specific tests. These specific versions (such as `assertEqualInt` and `assertEqualString` in the example above) generate detailed log messages on failure. In particular, they print the value of both arguments; this greatly simplifies diagnosing failures.
Each test resides in a C source file with the same name as the test. The test itself is a function that takes no arguments. The test is declared using the `DEFINE_TEST()` macro. This macro serves both to ensure that the test is declared correctly and as a label that can be used to locate all defined tests. (On Unix-like platforms, a simple `grep` operation is used to construct a file called `list.h` that holds the names of all of the tests. This makes it very easy to add new tests.)
The test harness determines which tests to run. It goes through the following steps whenever it runs a test:
-
Closes all file descriptors except for stdin, stdout, and stderr.(This screws up libc on some platforms so has been removed.) - Creates a temporary directory whose name matches the name of the test and switches into that directory.
- Resets the current locale.
- Calls the test function.
- If there were no assertion failures, it will remove the temporary directory. (If `-k` is specified, temporary directory are left even if the test succeeds.)
-
If there are any open file descriptors other than stdin, stdout, and stderr, it reports an error.Tests should never leave open file descriptors.
- The current directory is empty when the test starts.
- Any files created in the current directory will be removed for you.
- The current locale is the default "C" locale.
- Release all memory. The test suites are occasionally run under a memory debugger to detect leaks in the libarchive library.
- Close all opened files. This helps to catch file descriptor leaks in libarchive.
- Not read or write absolute paths.
Some tests are specific to a particular platform. Such tests should use appropriate platform-specific macros as follows:
#if __PLATFORM
/* ... various helper functions ... */
#endif
DEFINE_TEST(foo_platform)
{
#if __PLATFORM
/* ... tests as usual .... */
#else
skipping("platform-specific tests");
#endif
}In particular, note that all tests are compiled and run on all platforms.
Most tests are not platform-specific and will thus end up running on many different platforms. In order to simplify writing such tests, try to use platform-independent coding:
- Use stdio `fopen()`, `fwrite()`, `fread()`, and `fclose()` to access files whenever feasible.
- Look through the `test.h` header to see if there are assertXxx() functions that you can use. There's a list of the more popular ones below, but new ones are often added.
The following is a necessarily incomplete list of assert functions available to tests:
- Basic equality: `assertEqualInt`, `assertEqualString`, `assertEqualMem`
- File creation: `assertMakeFile`, `assertMakeSymlink`, `assertMakeHardlink`, `assertMakeDir`
- File tests: `assertIsReg`, `assertIsDir`, `assertIsSymlink`, `assertFileSize`, `assertFileNlinks`, `assertFileMtime`
- File contents: `assertFileEmpty`, `assertFileNonEmpty`, `assertFileContents`, `assertTextFileContents`
Many tests require reading a pre-constructed reference file. Such files are stored with the source code for the associated test suite. Reference files are named according to the test and must be uuencoded to be checked into source control.
For example, if you need a reference tar archive to use with `test_foo`, the file should be named `test_foo.tar` and stored in source control as `test_foo.tar.uu`.
Within the test code, you can recover the reference file with:
extract_reference_file("test_foo.tar");Look at `test_read_format_cpio_bin_be.c` for a simple example of this usage.
A few of the older tests store reference data within the source code as a hex-encoded array of characters. This was common before `extract_reference_file()` was added and is not recommended for new code.
- DO use asserts liberally. It's common to have an assert on almost every line.
- DO use assertEqualInt, assertEqualString, assertEqualMem to test equality instead of plain assert(); the specialized forms give a lot more information on a failure.
- DO test your tests; experiment by changing a piece of code and make sure your test fails. If you think you've found a bug, we recommend writing the test first, make sure the test fails, then fixing the bug.
- DO run all of the tests before submitting a change. Depending on your build environment, `make test` or `make check` will usually run all of the tests.
- DON'T rely on `HAVE_` macros from config.h. (If the tests use the same `HAVE_` macros as the code being tested then configuration problems will be covered up.)
- DO use runtime tests for platform features. For example, the ACL tests try to set an ACL on a file and read it back to determine if ACL support is available, then they exercise the libarchive ACL handling.
- DO look at existing tests. Often, a bug can be tested by adding just a couple of asserts to an existing test instead of writing a new one.
- DO improve existing tests. In particular, if you see a test failure that is hard to understand, consider adding a failure() message or comments so the next person will have an easier time.
- DO ask on the mailing lists if you have questions. Some tests are better written than others.
There are quite a few read tests that simply read a pre-built input file and verify the results. These use `extract_reference_file()` to decode a uuencoded input file, then open that file with libarchive and verify the results. Most of these are pretty straightforward.
Most of the write tests really just verify that libarchive can read what it writes. These generally use the memory interfaces. They feed archive entries into the writer to create an archive in memory and then open the memory again to read the data back and verify that it is the same.
A few write tests write an archive into memory and then inspect the actual bytes to verify that the archive was created correctly. These tests are nice to have but are tedious to build.
There are also a couple of such validators for bsdtar and bsdcpio. These are very tricky since the exact data contains values that the test cannot completely control (such as the current time or username).
Libarchive's disk I/O APIs, as well as bsdtar and bsdcpio, need to operate by verifying files on disk. To make this easier, the test framework now has a large number of assertions to verify timestamps, permissions, and other basic file data. These assertions use appropriate system calls for each platform; new tests should try to avoid using `stat()`, which is not uniformly available.
The fuzz tester is one of the few tests that does not aggressively use assertions. Its purpose is to try and crash libarchive by feeding it input that is slightly damaged. (Remember that libarchive does have some failsafe code that deliberately aborts the entire program if certain invariants are violated.)
Since a fuzz failure causes a crash, it's impossible to report the error to the console, so the fuzz tester instead generates a randomly-damaged file, saves that file to disk, then runs the file through libarchive to try to provoke a crash. If libarchive crashes the entire tester, the input that caused the crash will be available on disk for further analysis.
Although it is unusual for a test program to use random input, the fuzz tester has proven quite useful at uncovering poor error handling. Unfortunately, the need to write the data to disk before each test is a significant performance issue.
The large tar tester attempts to exercise boundary cases with very large entries, up to 1 terabyte. Of course, writing a 1 terabyte entry to an uncompressed archive on disk or memory is unacceptable, both for performance and space reasons. Even gzip or bzip2 compression won't help; although smaller, the output is still large and the time needed to compress such a large amount of data is prohibitive. So the large tar tester uses a trick.
Remember that an uncompressed tar archive consists of alternating headers and bodies. If the bodies consist entirely of zero bytes, then all that's needed to reconstruct the archive is to record the relatively small headers and store a count of each contiguous block of zeros. This simple "run-length encoding" is very effective, compressing a sample archive with a dozen entries down to just a few kilobytes, even if those entries range up to 1TB.
By writing a custom I/O layer that implements such compression, we can write very large entries through libarchive then read them back and verify that libarchive correctly handles all of the boundary cases.
Of course, even scanning output to determine zero blocks can take quite a while, so the large tar tester makes one more optimization: Libarchive is "mostly" zero-copy. When you give it a large block to write to an entry body, it will pass pointers straight through to the output routine, except in those cases where it must copy data to correctly build blocks. The large tar tester takes advantage of this to quickly detect when a block being written out by the archive writer is the same as some of the data that was given by the test as the entry body, and similarly when reading the archive back.
Taken together, these tricks allow the large tar writer to test for issues such as proper storage of very large file sizes (tar files support several different ways to store file sizes which trade off portability versus range; libarchive tries to use the most portable one it can for each entry) and integer overflow (Windows 32-bit `off_t` broke the large tar test and led libarchive to make heavier use of `int64_t` internally) in a test that runs in under 1/4 second.