Running Automated Tests for the Joomla CMS
From Joomla! Documentation
This article has been updated during the Testing Sprint in October 2014 to reflect the latest information for running unit and system tests for Joomla version 3.x
When you checkout the staging branch of Joomla from Github.com (https://github.com/joomla/joomla-cms) you will see a folder called "tests". This folder contains unit and two types of system tests for the CMS. The unit tests use PHPUnit and the system tests use PHPUnit and Selenium. The unit tests perform tests on the framework and cms library files (libraries/cms and libraries/joomla) and on parts of com_finder. The Webdriver system tests run Joomla! from a browser and test it as a user would.
Before you can run the tests, you need to install some programs on your local workstation, as documented below. This document explains how to run those tests from your local workstation.
You may need to modify your PHP configuration. This is done by editing your
php.ini file and then restarting your Apache service.
CMS Unit Tests
Some of the CMS unit tests rely on the PHP extension Sqlite3. To run these tests, make sure that the following line in your php.ini file is uncommented:
- For Windows:
- For Linux:
On top of that, it might be necessary to increase your PHP memory limit, as some tests are consuming a lot of resources:
Webdriver System Tests
The Webdriver system tests require that the Curl extension is installed. Make sure the following line in your php.ini file is uncommented:
- For Windows:
- For Linux:
Both the unit and system tests rely on PHPUnit. PHPUnit is a testing framework that requires PHP to be installed on the local workstation. Please visit http://phpunit.de/ and follow the instructions to install the PHPUnit PHAR Archive. Make sure that you install the right version! Currently PHPUnit 5.4 does NOT work with Joomla's unit tests, you'll need PHPUnit 4.8 or 5.3.4.
Linux / OSX Users
To globally install the PHAR: $ wget https://phar.phpunit.de/phpunit-old.phar $ mv phpunit-old.phar phpunit.phar $ chmod +x phpunit.phar $ sudo mv phpunit.phar /usr/local/bin/phpunit
To see version number: $ phpunit --version
The instructions at http://phpunit.de/ may not work on Windows. Most likely, Windows users will not have the `wget` command used in the instructions. However, if msysgit is installed, curl should be available. If it is, then the simplest way to install PHPUnit on Windows is to just use this command:
curl https://phar.phpunit.de/phpunit.phar > _some_directory_in_your_path_/phpunit
This will download the phar file and store it without the phar extension so that you can run the command simply by running:
Running Unit Tests
Run Suite from XML File
Once PHPUnit is installed, it's easy to run the unit tests. The command that runs the tests is called
phpunit. This file is found in your PHP folder. For example, in Windows with Xampp, it would be
c:/xampp/php/phpunit. In Linux, with Xampp, it might be
/opt/lampp/bin/phpunit. If you add this to your operating system path, you can execute it from any folder without using the full path name.
When you run
phpunit it looks for a PHPUnit configuration XML file (either
phpunit.xml.dist). The CMS unit tests are run from the Joomla root folder, where you will find the
phpunit.xml.dist file. This file is included in the Git repository and can be used as is or customized (by making a copy called
To run all of the unit tests from the command line:
- Change Joomla root folder
- Execute the command
The following example runs the entire unit test suite on OS X.
MyMacbook:joomla-cms sniper$ phpunit PHPUnit 4.3.2 by Sebastian Bergmann. Configuration read from /Users/myself/joomla-cms/phpunit.xml .....S....................................................... 61 / 5373 ( 1%) ............................................................. 2745 / 5373 ( 51%) SSSSSSSSSSSSSSSSSSSSSSSS..................................... 5368 / 5373 ( 99%) ..... Time: 57.29 seconds, Memory: 186.75Mb OK, but incomplete, skipped, or risky tests! Tests: 5373, Assertions: 9914, Skipped: 157. Generating code coverage report in Clover XML format ... done Generating code coverage report in HTML format ... done
The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show. If you are generating code coverage or other logs, you can see the folders where these are generated in the XML file. In the default file, all of the logs are generated in
build/logs, with coverage information in a folder called
You can create your own phpunit.xml file by copying the phpunit.xml.dist file and making the desired changes. For example, if you don't want to create the coverage information, remove the
log type="coverage-html" element.
Run Selected Unit Tests
Running A Selected Suite
Joomla's unit tests are grouped into a number of different suites:
You can run unit tests grouped in a specific folder by appending the --testsuite parameter. The following example runs all of the tests in the database suite:
MyMacbook:joomla-cms sniper$ phpunit --testsuite database PHPUnit 4.3.2 by Sebastian Bergmann. Configuration read from /Users/myself/joomla-cms/phpunit.xml ............................................................... 63 / 354 ( 17%) ............................................................... 126 / 354 ( 35%) ..............................SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS 189 / 354 ( 53%) SSSSSSSSSSSSSSSSSSSSSSSSSSSSSSS................................ 252 / 354 ( 71%) ..................SSSSSS....................................... 315 / 354 ( 88%) ...........SSSSSSSSSSSSSSSSSSSSSSSSS... Time: 2.51 seconds, Memory: 20.00Mb OK, but incomplete, skipped, or risky tests! Tests: 354, Assertions: 718, Skipped: 95.
Running Tests In A Specific Folder
You can run unit tests one at a time or by folder. The following example runs all of the tests in the
MyMacbook:joomla-cms sniper$ phpunit tests/unit/suites/libraries/joomla/feed PHPUnit 4.3.2 by Sebastian Bergmann. Configuration read from /Users/myself/joomla-cms/phpunit.xml ................S.S............................................ 63 / 101 ( 62%) ...................................... Time: 335 ms, Memory: 8.50Mb OK, but incomplete, skipped, or risky tests! Tests: 101, Assertions: 201, Skipped: 2.
You can also specify a single test simply by specifying it's full path name in the command.
Running Tests Filtered By Name
You can run all unit tests which names are matching a specific naming pattern by appending the --filter parameter. The following example runs all tests starting with JModel like:
MyMacbook:joomla-cms sniper$ phpunit --filter JModel PHPUnit 4.3.2 by Sebastian Bergmann. Configuration read from /Users/myself/joomla-cms/phpunit.xml ................................................................. 65 / 95 ( 68%) .............................. Time: 4.58 seconds, Memory: 95.00Mb OK (95 tests, 140 assertions)
Running a single test
Running the Database Testsuites
By default, only the tests for Sqlite are run in the database testsuite, because all other tests for database systems like MySQL, PostgreSQL and MSSQL require a working database, valid credentials and some preparations. This example documents the necessary steps to get the MySQL tests running:
- create a new, empty database
- create a user that is allowed to access the created database
- import the default dataset, dumped in tests/unit/schema/mysql.sql, into the database
- copy the file
- uncomment and update the database dsn constants in the <php> block of your new phpunit.xml:
<?xml version="1.0" encoding="UTF-8"?> <phpunit bootstrap="tests/unit/bootstrap.php" colors="false"> <php> <const name="JTEST_DATABASE_MYSQL_DSN" value="host=YOURHOST;dbname=YOURDBNAME;user=YOURUSER;pass=YOURPASSWORD" /> <const name="JTEST_DATABASE_MYSQLI_DSN" value="host=YOURHOST;dbname=YOURDBNAME;user=YOURUSER;pass=YOURPASSWORD" /> </php> ...
Selenium RC System Tests
The older Selenium RC system tests are in the folder tests/system/suite. Before you can run the system tests, you need to install and download the Selenium Server program and configure your test environment, as shown below.
Download Selenium Server
Selenium Server (formerly Selenium RC) is the package that allows us to run Selenium tests from PHP (instead of Java). To download it, just go to the Selenium site http://seleniumhq.org/download/ and download the Selenium Server file (for example, selenium-server-standalone-2.25.0.jar) to a folder.
For example, in Windows you can create a folder called
C:\selenium and copy this file there. In that folder, create a Windows bat file (such as selenium.bat though the name does not matter) or Linux shell script that runs the following command:
java -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar
(Obviously you will need to use the exact name of the .jar file for the version you downloaded.) The -Xms and -Xmx arguments run the server program allocating more memory to Java than the default. That seems to help prevent intermittent errors when running a long suite of system tests.
If the Java executable is not on your path, then you will need to indicate the full path to it, like the following:
"c:\program files (x86)\Java\jre6\bin\java.exe" -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar
This program needs to be running in the background before you can run any Selenium functional tests. So just run this program from a Bat file and it will continue to run in the background in a console window until you close it (for example, with Ctrl+C).
Create a Selenium Configuration File
To run the Selenium tests, we have to tell Selenium how to navigate and login to our local Joomla! installation. This is done by creating a PHP file called
tests/system/servers/configdef.php. The Joomla! download includes a file called
tests/system/servers/config-def.php-dist. This is a sample file that you can use to create the real file. Copy this file to a file called
tests/system/servers/configdef.php and edit it to reflect your test systems configuration. There are comments in the file that tell you how to do this.
Run the System Test Suite
At this point, we are ready to actually run the tests. There are two steps to running a test. First, you need to make sure the Selenium Server process is running in the background. To do this, just execute the command (bat file or shell script) you created when you installed Selenium. This will continue to run (in a console window) until you cancel it. You do not need to stop and start this for each test. Just run it once and let it run in the background.
Once Selenium is running, you need to execute the system tests. These are executed the same way we did the unit tests, except for our starting directory.
To run all tests from the command line in Windows, change to the
tests/system folder and run the following command
By default, this will use the
tests/system/phpunit.xml.dist file that is included with the Joomla checkout. This will run all of the tests listed in the phpunit.xml.dist file. You should see two browser windows open. One will display Selenium commands that are being executed. The other will show the Joomla! website and the various screens that are being opened and closed as the tests are run. As each test is completed, you will see comments in the console window describing each step in the test.
Note that the entire suite of tests takes about one hour to run. When they are done, you will get a summary display showing how many tests were run and whether there were errors or failures. If there are errors or failures, the line of code from the test program that generated the error or failure will also show. An error occurs when the test encounters an actual PHP error. A failure occurs when the test gets a result that is different than expected (when one of the assert statements in the test is not as expected).
If you use the default phpunit.xml.dist file, a file
tests/system/suite/logs/junit.xml will be created at the conclusion of the test. This file can be interpreted by Eclipse and any other program that works with JUnit unit tests. It provides a graphical way to see the test results.
Run Selected Tests
As with the unit tests, you can run selected system tests by folder or file. For example, the following command will run all of the tests in the acl folder:
To run a the article0001Test.php tests:
Webdriver is the newer system test program from Selenium. In general, all new Joomla system tests should be written with Webdriver.
The Webdriver tests are in the folder
tests/system/webdriver. This folder has the following subfolders:
- Pages: Contains the page class files for the CMS.
- SeleniumClient: Contains the Nearsoft library files. These are files that allow us to write the Webdriver tests in PHP (instead of Java). They are maintained on Gihub at https://github.com/Nearsoft/PHP-SeleniumClient.
- tests: Contains the folders with the test programs. These use the Pages files.
There is also a file called
bootstrap.php in the webdriver folder. This is used to auto-load the required classes. Inside the tests folder we have a file called
JoomlaWebdriverTestCase.php. This file is the parent class for all Webdriver tests and has a number of useful methods.
Run All Webdriver Tests
Running the Webdriver tests is exactly the same as running the old system tests except for the starting directory. The steps are:
- Make sure the Selenium Server is running in the background.
- Change directory to the
- Execute the command:
This will by default use the configuration file
tests/system/webdriver/tests/phpunit.xml.dist, which is included with the CMS checkout. You can customize this by copying it to phpunit.xml and editing as desired. You do not need to delete the
phpunit.xml.dist file because the
phpunit.xml file takes priority if both are found.
Webdriver tests only open one browser window. Like the Selenium RC system tests, Webdriver tests are relatively slow -- similar to a very fast data entry person using the application.
Run Selected Webdriver Tests
You can run selected Webdriver tests by folder or by file just as you do for the other system tests or for the unit tests. The only difference is the starting folder.
- Make sure you run the tests on a clean database that has been installed with sample data. If you have made database changes, you should re-install Joomla! or otherwise put the database back to it's original post-installed state. When you run the entire system test suite, the first test does a clean re-install for you automatically. This tests the installation and also makes sure the database is in an expected state.
- If you find errors or failures when you run the tests, you can run the tests against the Joomla! master branch to see if the problems are in master or just in your branch.
In Eclipse, you can set up Debug and External Tools launch configurations that make it easy to debug or run unit and system tests. See Running Automated Tests from Eclipse.