Unit testing is an essential part of a good Quality Control program. For a good general discussion of unit testing, visit the Wikipedia article.
Open source projects, with multiple developers working in parallel around the world, can greatly benefit from unit testing. The main benefits are:
Software testing systems usually run through a spectrum from "pure" unit tests through to fully integrated systems tests. We've described low level unit tests above. Integrated testing typically involves some sort of script that simulates user actions and then verifies that the result matches what's expected. This sort of "end to end" test verifies that all parts of the system are working correctly.
It's unfortunate that there is no clear nomenclature to describe all the intermediate stages of testing. The next stage beyond testing a single unit of code is subsystem testing. A subsystem test verifies that two or more units of code are interacting correctly to produce the desired result. In the simplest case, a subsystem test can be created simply by replacing mock objects with real objects and running unit tests on the top level module. In practise, this tends to not work as well as expected, because the original unit test data wasn't designed for a subsystem test, or because the nature of the test cases needs to be changed in order to fully test the subsystem. After all there is little point in simply repeating the unit test cases; the objective of a subsystem test should be to test boundary conditions and special cases that would be difficult to duplicate in unit tests.
Once a subsystem has been tested, it can be integrated into a larger system, which is still a subset of the whole product. Tests can be written for larger and larger subsystems, but at each stage the complexity of the tests increases. At some point, the effort required to hand craft tests exceeds the benefits of running them. This is where integrated testing comes in.
Integrated testing involves recording a user's interaction with the system into a script that can be replayed. The testing framework then compares the system's response with the expected response and passes or fails the test. The PHPUnit testing framework that we use has the ability to work with Selenium, a browser based test automation tool.
The purpose of unit tests is to isolate a module of code. A test that tests only one thing provides better information than a test that involves several object interactions. But how do we isolate an object from its dependencies? By writing stub classes. xUnit Patterns defines a the hierarchy of dummy classes, ranging from simple to complex:
Unit testing capabilities in Joomla are still at an early stage. The intention is to define more standards for developing tests, and then to expand the scope of available tests
The SVN repository contains code under the /testing path. /testing/trunk contains code based on the SimpleTest framework. In early December 2007, the development team elected to move to the PHPUnit framework.
Work on using PHPUnit has been done in /testing/branches/2007-12-17. Some new tests have been added, many old tests from the SimpleTest days are completely broken.
At this point, PHPUnit based tests only run in a command line environment.
If you can commit to the Joomla code base, then you should consider yourself part of the unit test team!
Writing tests concurrently with code (or even before) is a good way to not only save development time, but a great tool for defending against regressions. Writing tests early in the development cycle also helps identify and resolve design issues sooner, which reduces refactoring.
If you want to get started on unit testing, get in touch with Alan Langford (instance) or Ray Tsai (mihu). Either of us will be happy to help out.
At risk of stating the obvious, in the "purest" case the purpose of a unit test is to isolate a unit of code from its environment and to test the operation of that code.
This isolation is usually achieved by writing dummy classes that emulate the code unit's environment. These dummy objects can be passive, by simply simulating the environment, or they can be more active, keeping track of how they are being used by the test unit and reporting any variations from the expected behaviour. See Mock Objects in Joomla for a detailed example.
An interesting aspect of writing tests is that they become de facto detailed technical specifications of the interfaces between units of code. The fact that these specifications can be verified in an automated way makes them a superb resource when refactoring code.
The test code has a few templates designed to kick-start a test. They are:
Test files follow the form class-sequence-type-test.php, for example JObject-0000-class-test.php. For tests that are not class based, the first element refers to the object being tested. An example of this is the e-mail cloaking plugin test, which is called emailcloak-0000-mode1-test.php.
Joomla unit tests use a customized test "runner". Every test directory should have a "runtests.php" file. Runtests has a few options, mostly to allow the selection of specific tests. The command line options are:
e.g. --class-filter /JDate/ Selects only tests that have a class part that matches the regular expression.
Turns on additional debugging output.
Prints information on options and exits.
Selects only tests that have a sequence part that matches the regular expression.
Selects only tests that have a test part that matches the regular expression.
TODO: Expand command line parsing to add other features of the PHPUnit framework, such as output formats, code metrics reports, etc.
Before you start make sure you have installed PHPUnit and of course PHP (5!) properly...
To get the unit tests to run on your Joomla! installation, perform the following steps:
The unit test will the run, and the results are rendered.
The provided configurations should work out of the box. We have seen problems with it (currently the cause is unknown). If you get an error like blow, the solution is pretty easy.
file=/var/www/unittest/runtests.php posn=17 base=runtests.php /var/www JPATH_BASE does not point to a valid Joomla! installation: JPATH_BASE = /var/www Please modify your copy of "TestConfiguration.php"
Modify the "TestConfiguration.php" file and change the definition of the JPATH_BASE so it points to the path of you Joomla! installation, in the example below the Joomla! installation is installed at "/var/www/update".