System testing is an essential part of a good Quality Control program. For a good general discussion of system testing, visit the Wikipedia article.
Web applications have been notoriously difficult to do system testing with. It has traditionally required a great deal of time and effort. This can be a special challenge for open source projects because testing is often not very popular and designing tests that can be executed in a systematic way even less so. For a complicated application like Joomla!, this means that although we might do testing, we very seldom are able to perform extensive testing on each release that covers even a fraction of the elements in the CMS.
Relatively recently, a project called Selenium has arrived on the scene. This project makes performing automated system testing of web applications possible. This means that it is possible to devise a test and have it performed in an automated way without the requirement of somebody sitting in front of a browser and being a click monkey.
The benefits of system testing are:
In order to fully test an application, both system and unit testing are essential. Each approach has their benefits and weaknesses. Unit testing is very good for isolating small chunks of code and ensuring that they are behaving as expected. System testing, on the other hand, is geared towards looking at the system as a whole and ensuring that the units are behaving together in a way that achieves the desired effect.
The advantages of system testing is that system tests are easier to understand and conceptualize by people who are unfamiliar with the underlying code. With Selenium, tests can be designed by people who are not overly familiar with PHP - they only need to understand the application and what results should be expected.
The downside of system testing is that it will only tell you that something broke. It isn't as obvious with system testing exactly what went wrong or where the problem is.
If the purpose of unit tests is to isolate a module of code, then the purpose of system tests is to evaluate the entire system. This is much easier for those unfamiliar with what happens under the hood of Joomla! because one is not required to understand what happens during a Joomla! page request. A system test might evaluate, for example, if unpublishing an article works properly. Therefore, to design the test, it is only necessary to figure out how to change an article from Published to Unpublished, and how to verify that the article was actually unpublished.
I have recently been building a system testing architecture that will make it possible to write tests once, and to execute those tests on a variety of platforms and clients. I currently have infrastructure in place that makes it possible to run one command and execute system tests in one browser on one host system. I have designed the host system in such a way that it will run on 6 different PHP versions (stock Ubuntu PHP 5 and self-compiled PHP 4.3, 4.4, 5.0, 5.1 and 5.2) and 2 different web servers (Apache and Lighttpd). These are all currently serving from the same database.
The future steps will be to make it possible to run on multiple host systems (the requirement for this is to generalize the code that resets the database state after the tests are run) and to make it possible to use different clients (i.e. different systems with different web browsers). These steps should be fairly trivial, but I have not yet had time to focus on them.
This will eventually make it possible to run the designed tests on every major browser and on multiple server platforms (various PHP versions, various configurations, various operating systems and various web servers).
Selenium IDE makes it fairly easy to write system tests. It is a Firefox extension that operates as an Integrated Development Environment for creating system tests. In essence, it records your clicks as you navigate in your browser and encodes them into a series of commands that can control the browser. Tests recorded with Selenium IDE can be converted to tests within the Joomla repository and integrated into the test suite.
At of May 27, 2010, the latest version of Selenium IDE is 1.0.7. You can get it at http://seleniumhq.org/download/. Click on the download link and install like you would any Firefox plugin.
To get started, create a clean install of Joomla! with the standard sample data or update your local copy to the latest build and perform a re-installation.
You should see two tabs in your Selenium IDE window - one labeled Table, and one labeled Source. You don't really need to look at the Source window.
So, now our first test is done.
To run your test, you use the icons on the bar between the Base URL address bar and the Table and Source tabs. The two important buttons are the buttons with the Green Triangle with Three Solid rectangles and the one with the Green Triangle and One Solid Rectangle. The first one will execute the entire test suite and the second one will run the currently selected test case. Since we currently have only one test case, both of these currently do the same thing.
To run your test, press one of these buttons.
As you watch the Selenium IDE, you will see a yellow bar move down your list of steps. Once a step is completed, a successfully executed action (i.e. a button was successfully located and clicked) should show up in light green and a successful assertion should show up in darker green. Failed actions show up in light red and failed assertions show up in darker red.
Click on the Window Expander on the left to unhide the Test Case browser.
If your test completed successfully, you should see Runs: 1, Failures: 0 at the bottom of the Test Case browser.
The testing repository for the Selenium test suite for version 1.5 can be found here:
As of February 2010, the unit and system tests for version1.6 have been incorporated into the SVN trunk under a folder called "tests". More information about running the version 1.6 tests is available in the article Running Automated Tests for Version 1.6.
Note: If a login mask comes up, use
Instructions for running the system tests for version 1.6 can be found here: Running Automated Tests for Version 1.6.
For writing a more advanced Selenium testcase see here.