Actions

Difference between revisions of "Unit Testing"

From Joomla! Documentation

(News and Updates)
m (Reverted edits by Tom6 (talk) to last revision by Radiant tech)
(23 intermediate revisions by 11 users not shown)
Line 1: Line 1:
 
{{cookiejar}}
 
{{cookiejar}}
 +
Note that this article is badly out of date.
 
== News and Updates ==
 
== News and Updates ==
 +
2011 10 26 Use version 3.5.15 of PHPUnit. Please see reference material on github.
 +
 +
2009 10 06: Tests now depend on the SVN version of PHPUnit 3.4.1.
 +
 +
2008 06 24: Update to reflect move of PHPUnit code from branch to trunk (former trunk now in /branches/old_simpletest).
 +
 
2008 06 22: Add information on limiting tests by version.
 
2008 06 22: Add information on limiting tests by version.
  
Line 8: Line 15:
  
 
== Unit Testing ==
 
== Unit Testing ==
Unit testing is an essential part of a good Quality Control program.
+
Unit testing is not only an essential part of a good Quality Control program, it is an aid to development as well. Writing new tests before writing code helps focus the developer on the problem at hand. The practice also encourages writing smaller, more loosely coupled, more reusable, and more maintainable code units and these benefits often outweigh the benefits gained by treating unit tests solely as a QC tool. When used in this manner correctness becomes more a by-product of the process than the goal.
 
For a good general discussion of unit testing, visit the [http://en.wikipedia.org/wiki/Unit_test Wikipedia article].
 
For a good general discussion of unit testing, visit the [http://en.wikipedia.org/wiki/Unit_test Wikipedia article].
  
Line 26: Line 33:
 
Once a subsystem has been tested, it can be integrated into a larger system, which is still a subset of the whole product. Tests can be written for larger and larger subsystems, but at each stage the complexity of the tests increases. At some point, the effort required to hand craft tests exceeds the benefits of running them. This is where integrated testing comes in.
 
Once a subsystem has been tested, it can be integrated into a larger system, which is still a subset of the whole product. Tests can be written for larger and larger subsystems, but at each stage the complexity of the tests increases. At some point, the effort required to hand craft tests exceeds the benefits of running them. This is where integrated testing comes in.
  
Integrated testing involves recording a user's interaction with the system into a script that can be replayed. The testing framework then compares the system's response with the expected response and passes or fails the test. The PHPUnit testing framework that we use has the ability to work with Selenium, a browser based test automation tool.
+
Integrated testing involves recording a user's interaction with the system into a script that can be replayed. The testing framework then compares the system's response with the expected response and passes or fails the test. The PHPUnit testing framework that we use has the ability to work with [http://seleniumhq.org/ Selenium], a browser based test automation tool. Writing a functional test using Selenium is documented [http://docs.joomla.org/Functional_Testing#Writing_Functional_Tests here].
  
 
==== Test Objects ====
 
==== Test Objects ====
Line 37: Line 44:
  
 
=== Unit Testing in Joomla! ===
 
=== Unit Testing in Joomla! ===
Unit testing capabilities in Joomla are still at an early stage. The intention is to define more standards for developing tests, and then to expand the scope of available tests.
+
Unit testing capabilities in Joomla are steadily developing. [Add note re amount of coverage for platform by unit tests.]
  
For an overview of unit testing status in Joomla visit the [[Unit_Testing_Status]] page.
+
The [https://github.com/joomla/joomla-platform Joomla Platform GitHub repository] contains PHPUnit tests within the /tests subdirectory. See [http://docs.joomla.org/Running_Automated_Tests_for_Version_1.6 Running Automated Tests for Version 1.6] for instructions on setting up unit testing in your IDE.
 
+
The SVN repository contains code under the /testing path. /testing/trunk contains code based on the SimpleTest framework. In early December 2007, the development team elected to move to the [http://www.phpunit.de PHPUnit] framework.
+
 
+
Work on using PHPUnit has been done in '''/testing/branches/2007-12-17'''. Some new tests have been added, many old tests from the SimpleTest days are completely broken.
+
 
+
At this point, PHPUnit based tests only run in a command line environment.
+
  
 
==== The Unit Test Team ====
 
==== The Unit Test Team ====
Line 53: Line 54:
  
 
If you want to get started on unit testing, get in touch with Alan Langford (instance) or Ray Tsai (mihu). Either of us will be happy to help out.
 
If you want to get started on unit testing, get in touch with Alan Langford (instance) or Ray Tsai (mihu). Either of us will be happy to help out.
 +
 
==== Current Work ====
 
==== Current Work ====
 
* There is no longer any need to patch the main code to enable unit tests.
 
* There is no longer any need to patch the main code to enable unit tests.
 
* Basic techniques for mock objects are defined.
 
* Basic techniques for mock objects are defined.
 
* Strategies for dealing with local configuration is not yet complete, but there is a plan.
 
* Strategies for dealing with local configuration is not yet complete, but there is a plan.
* Files of the form class-sequence-type-test.php, for example JObject-0000-class-test.php use PHPUnit.
+
* Files of the form class-sequence-type-Test.php, for example JObject-0000-class-Test.php use PHPUnit.
 
* The JDate tests present a good example of a data-driven test, but they won't run on the current 1.5 code base (there are some proposed API changes as a result of unit test development).
 
* The JDate tests present a good example of a data-driven test, but they won't run on the current 1.5 code base (there are some proposed API changes as a result of unit test development).
 
* Previously functional tests, such as JFTP, haven't been moved to the PHPUnit environment yet.
 
* Previously functional tests, such as JFTP, haven't been moved to the PHPUnit environment yet.
 +
* The custom test runner is no longer needed. The current tests will run with the latest SVN version of PHPUnit 3.4. This code will eventually become PHPUnit 3.4.1.  Thanks to Sebastian Bergmann for his excellent work on an excellent project!
  
 
==== Writing Unit Tests ====
 
==== Writing Unit Tests ====
Line 73: Line 76:
 
/unittest/sample-simpletest.php.txt
 
/unittest/sample-simpletest.php.txt
  
Here are some example tests: [[Unit_Testing_--_a_Simple_Example|Simple Example]], [[Unit_Testing_--_Data_Driven_Example|Data Driven Example]], [[Unit_Testing_--_Plugin_Example|Plugin Example]].
+
Here are some example tests: [[Unit_Testing_--_a_Simple_Example|Simple Example]], [[Unit_Testing_--_Data_Driven_Example|Data Driven Example]], [[Unit_Testing_--_Plugin_Example|Plugin Example]], [[Unit_Testing_--_UI_Example|UI Example]].
  
 
==== Running Unit Tests ====
 
==== Running Unit Tests ====
Test files follow the form class-sequence-type-test.php, for example JObject-0000-class-test.php. For tests that are not class based, the first element refers to the object being tested. An example of this is the e-mail cloaking plugin test, which is called emailcloak-0000-mode1-test.php.
+
Test files follow the form classnameTest.php, for example JObjectTest.php. For tests that are not class based, use the name of the file being tested.
 
+
Joomla unit tests use a customized test "runner". Every test directory should have a "runtests.php" file. Runtests has a few options, mostly to allow the selection of specific tests. The command line options are:
+
 
+
'''--class-exclude ''regex'''''
+
 
+
e.g. --class-exclude /JDate.*/ Skips tests that have a class part that match the regular expression.
+
 
+
'''--class-filter ''regex'''''
+
 
+
e.g. --class-filter /JDate/ Selects only tests that have a class part that match the regular expression.
+
 
+
'''--debug'''
+
 
+
Turns on additional debugging output.
+
 
+
'''--help'''
+
 
+
Prints information on options and exits.
+
 
+
'''--sequence-exclude ''regex'''''
+
 
+
Skips tests that have a sequence part that match the regular expression.
+
 
+
'''--sequence-filter ''regex'''''
+
 
+
Selects only tests that have a sequence part that matches the regular expression.
+
 
+
'''--test-exclude ''regex'''''
+
 
+
Skips tests that have a test part that match the regular expression.
+
 
+
'''--test-filter ''regex'''''
+
 
+
Selects only tests that have a test part that match the regular expression.
+
  
'''TODO:''' Expand command line parsing to add other features of the PHPUnit framework, such as output formats, code metrics reports, etc.
+
Joomla unit tests use the standard PHPUnit test runner.  See http://www.phpunit.de for documentation.
  
 
== How to Get it Running ==
 
== How to Get it Running ==
Line 119: Line 88:
  
 
To get the unit tests to run on your Joomla! installation, perform the following steps:
 
To get the unit tests to run on your Joomla! installation, perform the following steps:
* Create an instance of your Joomla! installation
+
* Create an instance of your Joomla! installation. Since you will be using SVN to check out the testing project, you don't want to check out the Joomla! with SVN. Instead, simply unpack a normal Joomla! archive. If you are using Eclipse, you can create a folder in your Eclipse workspace for the Joomla! installation, but don't create an Eclipse project yet.
* In the root checkout (or export) the latest version of the unit test code from SVN ''"/testing/branches/2007-12-17"'' to your installation base. This will create a ''"/unittest"'' sub-folder under your joomla installation.
+
* In the root checkout (or export) the latest version of the unit test code from SVN ''"/testing/trunk/1.5/unittest"'' or ''"/testing/trunk/1.6/unittest"'' to your installation base. This will create a ''"/unittest"'' sub-folder under your joomla installation. If you are using Eclipse, Import the project from the SVN and use the same folder in your Eclipse workspace you used above.
* Create a ''"TestConfiguration.php"'' file. You can copy a the file from ''"TestConfiguration-dist.php"'' that is part of the package you just have checked out.
+
* From the command line, change to the unittest directory.
* If you want to run a subset of the commands, change to the directory that contains those tests.
+
* Run the unit test from the command prompt using the following command: <code>phpunit tests</code>
* Run the unit test from the command using the following command:
+
  
php runtests.php
+
The unit test will the run, and the results are rendered. You will see a series of dots for each passed test and other letters for failed tests.  
  
The unit test will the run, and the results are rendered.
+
See http://www.phpunit.de/manual/current/en/textui.html for help using the --filter switch to run only certain tests.  There are also many other command line switches you can use to get results in various formats.
  
 
== Troubleshooting ==
 
== Troubleshooting ==
  
The provided configurations should work out of the box. We have seen problems with it (currently the cause is unknown). If you get an error like blow, the solution is pretty easy.
+
The provided configurations should work out of the box. We have seen problems with it (currently the cause is unknown). If you get an error like below, the solution is pretty easy.
  
 
  file=/var/www/unittest/runtests.php
 
  file=/var/www/unittest/runtests.php
Line 151: Line 119:
  
 
==See also==
 
==See also==
[http://www.phpunit.de/pocket_guide/3.2/en/index.html PHPUnit Pocket Guide]
+
[http://www.phpunit.de/manual/current/en/ PHPUnit Manual]
  
 +
[[Category:Bug Squad]]
 
[[Category:Development]]
 
[[Category:Development]]
 +
[[Category:Testing]]
 +
[[Category:Automated Testing]]

Revision as of 19:11, 2 February 2012

Note that this article is badly out of date.

Contents

News and Updates

2011 10 26 Use version 3.5.15 of PHPUnit. Please see reference material on github.

2009 10 06: Tests now depend on the SVN version of PHPUnit 3.4.1.

2008 06 24: Update to reflect move of PHPUnit code from branch to trunk (former trunk now in /branches/old_simpletest).

2008 06 22: Add information on limiting tests by version.

2008 06 21: Added --class-exclude, --sequence-exclude, and --test-exclude options.

2008 06 21: PHPUnit has been updated to version 3.2.21 with SVN rev 10436. Please update.

Unit Testing

Unit testing is not only an essential part of a good Quality Control program, it is an aid to development as well. Writing new tests before writing code helps focus the developer on the problem at hand. The practice also encourages writing smaller, more loosely coupled, more reusable, and more maintainable code units and these benefits often outweigh the benefits gained by treating unit tests solely as a QC tool. When used in this manner correctness becomes more a by-product of the process than the goal. For a good general discussion of unit testing, visit the Wikipedia article.

Unit Testing in Open Source

Open source projects, with multiple developers working in parallel around the world, can greatly benefit from unit testing. The main benefits are:

  • Unit tests help highlight cases where seemingly minor changes cause unexpected breakage.
  • Unit tests help clearly specify how a class should behave.
  • Unit tests can expose design flaws very early in development.
  • Unit tests make great examples. They are a great place for developers to learn how to use the code.


The Testing Hierarchy: Unit, Subsystem, Integrated

Software testing systems usually run through a spectrum from "pure" unit tests through to fully integrated systems tests. We've described low level unit tests above. Integrated testing typically involves some sort of script that simulates user actions and then verifies that the result matches what's expected. This sort of "end to end" test verifies that all parts of the system are working correctly.

It's unfortunate that there is no clear nomenclature to describe all the intermediate stages of testing. The next stage beyond testing a single unit of code is subsystem testing. A subsystem test verifies that two or more units of code are interacting correctly to produce the desired result. In the simplest case, a subsystem test can be created simply by replacing mock objects with real objects and running unit tests on the top level module. In practise, this tends to not work as well as expected, because the original unit test data wasn't designed for a subsystem test, or because the nature of the test cases needs to be changed in order to fully test the subsystem. After all there is little point in simply repeating the unit test cases; the objective of a subsystem test should be to test boundary conditions and special cases that would be difficult to duplicate in unit tests.

Once a subsystem has been tested, it can be integrated into a larger system, which is still a subset of the whole product. Tests can be written for larger and larger subsystems, but at each stage the complexity of the tests increases. At some point, the effort required to hand craft tests exceeds the benefits of running them. This is where integrated testing comes in.

Integrated testing involves recording a user's interaction with the system into a script that can be replayed. The testing framework then compares the system's response with the expected response and passes or fails the test. The PHPUnit testing framework that we use has the ability to work with Selenium, a browser based test automation tool. Writing a functional test using Selenium is documented here.

Test Objects

The purpose of unit tests is to isolate a module of code. A test that tests only one thing provides better information than a test that involves several object interactions. But how do we isolate an object from its dependencies? By writing stub classes. xUnit Patterns defines a the hierarchy of dummy classes, ranging from simple to complex:

  • Dummy - Defines attributes and methods of a dummy class (not particularly useful in PHP).
  • Fake - Provides canned responses to method calls and fixed attribute values. Good for speed.
  • Stub - Allows the test to define responses to method calls (return values, exceptions) to simulate the dependent object.
  • Spy - A Fake or Stub that records method calls and parameters for later analysis.
  • Mock - A Fake or Stub with a set of expectations -- method calls and parameters -- that are automatically verified for correctness.

Unit Testing in Joomla!

Unit testing capabilities in Joomla are steadily developing. [Add note re amount of coverage for platform by unit tests.]

The Joomla Platform GitHub repository contains PHPUnit tests within the /tests subdirectory. See Running Automated Tests for Version 1.6 for instructions on setting up unit testing in your IDE.

The Unit Test Team

If you can commit to the Joomla code base, then you should consider yourself part of the unit test team!

Writing tests concurrently with code (or even before) is a good way to not only save development time, but a great tool for defending against regressions. Writing tests early in the development cycle also helps identify and resolve design issues sooner, which reduces refactoring.

If you want to get started on unit testing, get in touch with Alan Langford (instance) or Ray Tsai (mihu). Either of us will be happy to help out.

Current Work

  • There is no longer any need to patch the main code to enable unit tests.
  • Basic techniques for mock objects are defined.
  • Strategies for dealing with local configuration is not yet complete, but there is a plan.
  • Files of the form class-sequence-type-Test.php, for example JObject-0000-class-Test.php use PHPUnit.
  • The JDate tests present a good example of a data-driven test, but they won't run on the current 1.5 code base (there are some proposed API changes as a result of unit test development).
  • Previously functional tests, such as JFTP, haven't been moved to the PHPUnit environment yet.
  • The custom test runner is no longer needed. The current tests will run with the latest SVN version of PHPUnit 3.4. This code will eventually become PHPUnit 3.4.1. Thanks to Sebastian Bergmann for his excellent work on an excellent project!

Writing Unit Tests

At risk of stating the obvious, in the "purest" case the purpose of a unit test is to isolate a unit of code from its environment and to test the operation of that code.

This isolation is usually achieved by writing dummy classes that emulate the code unit's environment. These dummy objects can be passive, by simply simulating the environment, or they can be more active, keeping track of how they are being used by the test unit and reporting any variations from the expected behaviour. See Mock Objects in Joomla for a detailed example.

An interesting aspect of writing tests is that they become de facto detailed technical specifications of the interfaces between units of code. The fact that these specifications can be verified in an automated way makes them a superb resource when refactoring code.

The test code has a few templates designed to kick-start a test. They are:

/unittest/sample-datatest-php.txt /unittest/sample-simpletest.php.txt

Here are some example tests: Simple Example, Data Driven Example, Plugin Example, UI Example.

Running Unit Tests

Test files follow the form classnameTest.php, for example JObjectTest.php. For tests that are not class based, use the name of the file being tested.

Joomla unit tests use the standard PHPUnit test runner. See http://www.phpunit.de for documentation.

How to Get it Running

Before you start make sure you have installed PHPUnit and of course PHP (5!) properly...

To get the unit tests to run on your Joomla! installation, perform the following steps:

  • Create an instance of your Joomla! installation. Since you will be using SVN to check out the testing project, you don't want to check out the Joomla! with SVN. Instead, simply unpack a normal Joomla! archive. If you are using Eclipse, you can create a folder in your Eclipse workspace for the Joomla! installation, but don't create an Eclipse project yet.
  • In the root checkout (or export) the latest version of the unit test code from SVN "/testing/trunk/1.5/unittest" or "/testing/trunk/1.6/unittest" to your installation base. This will create a "/unittest" sub-folder under your joomla installation. If you are using Eclipse, Import the project from the SVN and use the same folder in your Eclipse workspace you used above.
  • From the command line, change to the unittest directory.
  • Run the unit test from the command prompt using the following command: phpunit tests

The unit test will the run, and the results are rendered. You will see a series of dots for each passed test and other letters for failed tests.

See http://www.phpunit.de/manual/current/en/textui.html for help using the --filter switch to run only certain tests. There are also many other command line switches you can use to get results in various formats.

Troubleshooting

The provided configurations should work out of the box. We have seen problems with it (currently the cause is unknown). If you get an error like below, the solution is pretty easy.

file=/var/www/unittest/runtests.php
posn=17
base=runtests.php
/var/www
 JPATH_BASE does not point to a valid Joomla! installation:
JPATH_BASE = /var/www
 Please modify your copy of "TestConfiguration.php"

Modify the "TestConfiguration.php" file and change the definition of the JPATH_BASE so it points to the path of you Joomla! installation, in the example below the Joomla! installation is installed at "/var/www/update".

define('JPATH_BASE', '/var/www/update');

Frequently Asked Questions

Why can't I use "phpunit testname.php" to run my tests?

The test facility has to do some work to be able to load the "Joomla!" framework and to be able to inject mock classes. It's difficult to do this from the PHPUnit test runner, so we built our own. Also, the Joomla test runner has specific options designed to make it easy to select specific tests. Over time we will add more functionality to the test runner so it has many of the capabilities of the phpunit command.

See also

PHPUnit Manual