Actions

Difference between revisions of "Running Automated Tests for the Joomla CMS"

From Joomla! Documentation

m (Install PHPUnit)
(Install PHPUnit)
(23 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
{{RightTOC}}
 
{{RightTOC}}
 
== Introduction ==
 
== Introduction ==
As of February 2010, a tests folder has been added to the Joomla! SVN download. This folder contains a growing library of unit and system (or functional) tests. The unit tests use PHPUnit and the system tests use PHPUnit and Selenium. The unit tests perform tests primarily on the Joomla! framework. The Selenium system tests actually run Joomla! from a browser and test it as a user would.
+
'''This article has been updated as of November 2012 to reflect the latest information for running unit and system tests for Joomla version 3.0.'''
  
Before you can run the tests, you need to install some programs on your local workstation, as documented below. This document explains how to run the these tests from your local workstation.
+
When you checkout the master branch of Joomla from Github.com (https://github.com/joomla/joomla-cms) you will see a folder called "tests". This folder contains unit and two types of system tests for the CMS. The unit tests use PHPUnit and the system tests use PHPUnit and Selenium. The unit tests perform tests on the CMS library files (libraries/cms). The Selenium and Webdriver system tests run Joomla! from a browser and test it as a user would.
  
== Install PHPUnit ==
+
As of 3.0, we are transitioning our system tests from the older Selenium RC (found in the <code>tests/system/suite</code> folder) to the newer Webdriver methodolgy (found in <code>tests/system/webdriver</code>). We will continue to use and maintain the older tests, but all new tests will be written using Webdriver.
Both the unit and functional tests rely on PHPUnit. PHPUnit is a testing framework that requires PHP to be installed on the local workstation. XAMP / WAMP users can use the PHP installation that comes with them. Otherwise, you may need to install PHP, and the PEAR package, on your local workstation. More detailed instructions can be found at [http://www.php.net/manual/en/install.php http://www.php.net/manual/en/install.php].
+
  
On most PHP installations, PNPUnit can be installed using the standard PEAR installation process.
+
Before you can run the tests, you need to install some programs on your local workstation, as documented below. This document explains how to run the these tests from your local workstation.
  
:''Important Note as of 13 October 2010: PUPUnit version 3.5 was recently released. This version appears to require some changes to the unit tests. For now, we are continuing to use version 3.4.15. To install that version, just add -3.4.15 to the end of the install commands below. For example:'' <code>pear install phpunit/PHPUnit-3.4.15</code>.
+
== PHP Requirements ==
 +
You may need to modify your PHP configuration. This is done by editing your <code>php.ini</code> file and then restarting your Apache service.
  
For example, on XAMPP version 1.7.3 on Windows, you would do the following commands:
+
=== CMS Unit Tests ===
 +
Some of the CMS unit tests rely on the PHP extension Sqlite3. To run these tests, make sure that the following line in your php.ini file is uncommented:
 +
* For Windows: <code>extension=php_sqlite3.dll</code>
 +
* For Linux: <code>extension=php_sqlite3.so</code>
  
* <code>cd c:\xampp\php</code>
+
=== Webdriver System Tests ===
# sudo pear channel-discover pear.phpunit.de
+
The Webdriver system tests require PHP version 5.3.8 or later. They also require that the Curl extension is installed. Make sure the following line in your php.ini file is uncommented:
# sudo pear channel-discover components.ez.no
+
* For Windows: <code>extension=php_curl.dll</code>
# sudo pear channel-discover pear.symfony-project.com
+
* For Linux: <code>extension=php_curl.so</code>
# sudo pear install -a phpunit/PHPUnit
+
  
If you get error about PEAR version too low, run:
+
== Install PHPUnit ==
 +
Both the unit and system tests rely on PHPUnit. PHPUnit is a testing framework that requires PHP to be installed on the local workstation. In the past, PEAR was needed to install PHPUnit. That has changed. Please visit http://phpunit.de/ and follow the instructions to install PHPUnit.
  
* <code>pear upgrade pear</code>
+
=== Windows Users ===
 +
The instructions at http://phpunit.de/ may not work on Windows. Most likely, Windows users will not have the `wget` command used in the instructions. However, if msysgit is installed, curl should be available. If it is, then the simplest way to install PHPUnit on Windows is to just use this command:
  
and repeat <code>pear install phpunit/PHPUnit</code>
+
    curl https://phar.phpunit.de/phpunit.phar > _some_directory_in_your_path_/phpunit
.
+
Windows users not using WAMP or XAMPP can use the same procedure, with substituting the location of your PHP installation for <code>cd c:\xampp\php</code>.
+
  
In Ubuntu Linux, use the following commands to install PHPUnit:
+
This will download the phar file and store it without the phar extension so that you can run the command simply by running:
 
+
   
* <code>sudo apt-get install php-pear</code>
+
    phpunit
* <code>sudo pear channel-update pear.php.net</code>
+
* <code>sudo pear upgrade-all</code>
+
* <code>sudo pear channel-discover pear.phpunit.de</code>
+
* <code>sudo pear channel-discover components.ez.no</code>
+
* <code>sudo pear channel-discover pear.symfony-project.com</code>
+
* <code>sudo pear install -a phpunit/PHPUnit</code>
+
 
+
In Mac OSX with XAMPP, use the following commands to install PHPUnit:
+
* <code>$sudo /Applications/XAMPP/xamppfiles/bin/pear channel-discover  pear.phpunit.de</code>
+
* <code>$sudo /Applications/XAMPP/xamppfiles/bin/pear channel-discover  pear.symfony-project.com</code>
+
* <code>$sudo /Applications/XAMPP/xamppfiles/bin/pear channel-discover  components.ez.no</code>
+
* <code>$sudo /Applications/XAMPP/xamppfiles/bin/pear install -a phpunit/PHPUnit</code>
+
 
+
If you are using MAMP, substitute the path to your PHP pear folder for <code>/Applications/XAMPP/xamppfiles/bin/</code>.
+
 
+
More detailed instructions can be found here: [http://www.phpunit.de/manual/3.0/en/installation.html http://www.phpunit.de/manual/3.0/en/installation.html].
+
 
+
== Set Up Configuration File ==
+
Many unit tests can run independent of an actual Joomla! instance or database. Other tests require a connection to a Joomla! database. For these tests to run correctly, you need to create a test config.php  file.
+
 
+
If you don't have a correct configuration file, you will see a message similar to the one below when you try to run certain unit tests:
+
 
+
  PHP Fatal error:  Class 'JElement' not found in
+
/local/www/joomla_trunk/libraries/joomla/html/parameter/element/list.php  on line 22
+
 
+
To create a configuration file:
+
 
+
# Copy the file <code>tests/unit/config.php-dist</code> to the name <code>tests/unit/config.php</code>.
+
# Edit the <code>config.php<code> file to match your test configuration. The configuration must point to a valid Joomla! 1.6 database. You can use any installed Joomla! 1.6 database.
+
# The values you oneed to edit are as follows:
+
::'''$user''': database user id (for example, 'root' or whatever login needed to connect to the db)
+
::'''$password''': database user password
+
::'''$db''': database name
+
::'''$tmp_path''': <Joomla! root>/tmp
+
::'''$log_path''': <Joomla! root>/log
+
: You can use your <code>configuration.php</code> file as a guide for these entries, if needed.
+
 
+
When you run unit tests that use the database, the database will become unusable for a normal Joomla! instance. There are two ways to fix this.
+
# Re-install Joomla! after the unit tests have run. This recreates the database and fixes any problems.
+
# Use a different database for the unit tests. For example, you can run the installation of Joomla! and give it a different database name and use this database name in the <code>tests/unit/config.php</code> file.
+
  
 
== Running Unit Tests ==
 
== Running Unit Tests ==
Once PHPUnit is installed, you can run the unit tests. To run a unit test from the command line:
+
===Run Suite from XML File===
# Change to the folder <code><joomla root>/tests/unit</code>
+
Once PHPUnit is installed, it's easy to run the unit tests. The command that runs the tests is called <code>phpunit</code>. This file is found in your PHP folder. For example, in Windows with Xampp, it would be <code>c:/xampp/php/phpunit</code>. In Linux, with Xampp, it might be <code>/opt/lampp/bin/phpunit</code>. If you add this to your operating system path, you can execute it from any folder without using the full path name.
# Execute the command <code>phpunit <test name or folder></code>
+
For example, to execute the test called "suite\libraries\joomla\utilities\JStringTest.php", you would type the command
+
* <code>phpunit suite\libraries\joomla\utilities\JStringTest.php</code>
+
  
When you run this, you will see output similar to the following:
+
When you run <code>phpunit</code> it looks for a PHPUnit configuration XML file (either <code>phpunit.xml</code> or <code>phpunit.xml.dist</code>). The CMS unit tests are run from the Joomla root folder, where you will find the <code>phpunit.xml.dist</code> file. This file is included in the Git repository and can be used as is or customized (by making a copy called <code>phpunit.xml</code>).  
<pre>
+
C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
+
joomla\utilities\jstringtest.php
+
PHPUnit 3.4.11 by Sebastian Bergmann.
+
  
............................................................ 60 / 89
+
To run all of the unit tests from the command line:
.............................
+
# Change Joomla root folder
 +
# Execute the command <code>phpunit</code>.  
  
Time: 26 seconds, Memory: 7.00Mb
+
The following example runs the entire unit test suite using the <code>phpunit.xml.dist</code> file in Windows.
  
OK (89 tests, 89 assertions)
+
C:\xampp\htdocs\joomla_development\cms-trunk>phpunit
</pre>
+
PHPUnit 3.6.11 by Sebastian Bergmann.
The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show.
+
 +
Configuration read from C:\xampp\htdocs\joomla_development\cms-trunk\phpunit.xml.dist
 +
 +
IIIIIIII....................SS.......S.........................  63 / 234 ( 26%)
 +
............................................ISSISSSSSSSSSSI.... 126 / 234 ( 53%)
 +
...IIIIIIIIII.I.I.I.IIIIIIIIIIIIIIIII..IIIIIIIIS.IIIIII..II..SS 189 / 234 ( 80%)
 +
SI..I........I...I...I...I.III.I.............
 +
 +
Time: 13 seconds, Memory: 21.25Mb
 +
 +
OK, but incomplete or skipped tests!
 +
Tests: 234, Assertions: 281, Incomplete: 67, Skipped: 19.
 +
 +
Generating code coverage report in Clover XML format ... done
 +
 +
Generating code coverage report in HTML format ... done
  
You can execute all of the tests in a folder by specifying a folder instead of a file. For example, to run all of the tests in the folder "suite\libraries\joomla\utilities", you would enter
+
The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show. If you are generating code coverage or other logs, you can see the folders where these are generated in the XML file. In the default file, all of the logs are generated in <code>build/logs</code>, with coverage information in a folder called <code>coverage</code>.
* <code>phpunit suite\libraries\joomla\utilities</code>
+
  
This command will produce output similar to that shown below:
+
You can create your own phpunit.xml file by copying the phpunit.xml.dist file and making the desired changes. For example, if you don't want to create the coverage information, remove the <code>log type="coverage-html"</code> element.
<pre>
+
C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
+
joomla\utilities
+
PHPUnit 3.4.11 by Sebastian Bergmann.
+
  
............................................................  60 / 317
+
===Run Selected Unit Tests===
............................................................ 120 / 317
+
You can run unit tests one at a time or by folder. The following example runs all of the tests in the <code>libraries/feed</code> folder:
............................................................ 180 / 317
+
............................................................ 240 / 317
+
............................................................ 300 / 317
+
...FFFF..........
+
  
Time: 27 seconds, Memory: 11.00Mb
+
C:\xampp\htdocs\joomla_development\cms-trunk>phpunit tests/unit/suites/libraries
 +
/feed
 +
PHPUnit 3.6.11 by Sebastian Bergmann.
 +
 +
Configuration read from C:\xampp\htdocs\joomla_development\cms-trunk\phpunit.xml
  
There were 4 failures:
+
....................SS.......S................................... 65 / 99 ( 65%)
 +
..................................
  
1) JUtilityTest::testGetHash
+
Time: 1 second, Memory: 8.25Mb
Failed asserting that two strings are equal.
+
--- Expected
+
+++ Actual
+
@@ @@
+
-ce114e4501d2f4e2dcea3e17b546f339
+
+0cbc6611f5540bd0809a388dc95a615b
+
  
 +
OK, but incomplete or skipped tests!
  
2) JUtilityTest::testGetToken with data set "default" (NULL, false)
+
You can also specify a single test simply by specifying it's full path name in the command.
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
+
<boolean:false>.
+
  
 +
== Selenium RC System Tests ==
 +
The older Selenium RC system tests are in the folder tests/system/suite. Before you can run the system tests, you need to install and download the Selenium Server program and configure your test environment, as shown below.
  
3) JUtilityTest::testGetToken with data set "false" (false, false)
+
=== Download Selenium Server ===
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
+
Selenium Server (formerly Selenium RC) is the package that allows us to run Selenium tests from PHP (instead of Java). To download it, just go to the Selenium site [http://seleniumhq.org/download/ http://seleniumhq.org/download/] and download the Selenium Server file (for example, selenium-server-standalone-2.25.0.jar) to a folder.
<boolean:false>.
+
  
 +
For example, in Windows if you create a folder called <code>C:\selenium</code> and copy this file there. In that folder, create a Windows bat file (such as selenium.bat though the name does not matter) or Linux shell script that runs the following command:
  
4) JUtilityTest::testGetToken with data set "true" (true, true)
+
<code>java -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar</code>
Expectation failed for method name is equal to <string:getFormToken> when invoke
+
d 1 time(s).
+
Method was expected to be called 1 times, actually called 0 times.
+
  
 
+
(Obviously you will need to use the exact name of the .jar file for the version you downloaded.) The -Xms and -Xmx arguments run the server program allocating more memory to Java than the default. That seems to help prevent intermittent errors when running a long suite of system tests.
FAILURES!
+
Tests: 317, Assertions: 361, Failures: 4.</pre>
+
In this case, we have 4 test failures with details about each.
+
 
+
== System Tests ==
+
Before you can run the system tests, you need to install and set up the Selenium RC package and configure your test environment, as shown below.
+
 
+
=== Install Selenium RC ===
+
Selenium RC is the package that allows you to run Selenium tests from PHP or other programming languages. To install it, just go to the Selenium site [http://seleniumhq.org/projects/remote-control http://seleniumhq.org/projects/remote-control] and download the package. Then unzip the file into a folder.  
+
 
+
For example, in Windows if you create a folder called <code>C:\selenium</code> and unzip this file there, it will create a folder called selenium-server-1.0.1. In that folder, create a Windows bat file (such as selenium.bat though the name does not matter) or Linux shell script that runs the following command:
+
 
+
<code>java -jar selenium-server.jar -browserSessionReuse</code>
+
  
 
If the Java executable is not on your path, then you will need to indicate the full path to it, like the following:
 
If the Java executable is not on your path, then you will need to indicate the full path to it, like the following:
  
<code>"c:\program files (x86)\Java\jre6\bin\java.exe" -jar selenium-server.jar -browserSessionReuse</code>  
+
<code>"c:\program files (x86)\Java\jre6\bin\java.exe" -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar</code>  
  
Note the argument <code>browserSessionReuse</code> is used to allow you to run multiple tests without closing and re-opening the browser each time. Save this file so you can easily find and execute it when needed.
+
This program needs to be running in the background before you can run any Selenium functional tests. So just run this program from a Bat file and it will continue to run in the background in a console window until you close it (for example, with Ctrl+C).
 
+
This program needs to be running in the background before you can run any Selenium functional tests.
+
 
+
If You are using Firefox 3.6.x, be sure to install Selenium RC version 1.0.3 (released February 2010) or later. Earlier versions of Selenium RC have problems running in Firefox 3.6.x.
+
  
 
=== Create a Selenium Configuration File ===
 
=== Create a Selenium Configuration File ===
To run the Selenium tests, we have to tell Selenium how to navigate and login to your local Joomla! installation. This is done by creating a PHP file called <code>configdef.php</code>. The Joomla! download includes a file called <code>tests/system/servers/config-def.php-dist</code>. This is a sample file that you can use to create the real file. Copy this file to a file called <code>tests/system/servers/configdef.php</code> and edit it to reflect your test systems configuration. There are comments in the file that tell you how to do this.
+
To run the Selenium tests, we have to tell Selenium how to navigate and login to our local Joomla! installation. This is done by creating a PHP file called <code>tests/system/servers/configdef.php</code>. The Joomla! download includes a file called <code>tests/system/servers/config-def.php-dist</code>. This is a sample file that you can use to create the real file. Copy this file to a file called <code>tests/system/servers/configdef.php</code> and edit it to reflect your test systems configuration. There are comments in the file that tell you how to do this.
  
 
=== Run the System Test Suite ===
 
=== Run the System Test Suite ===
At this point, we are ready to actually run the tests. There are two steps to running a test. First, you need to make sure the Selenium RC process is running in the background. To do this, just execute the command (bat file or shell script) you created when you installed Selenium RC. This will continue to run until you cancel it.
+
At this point, we are ready to actually run the tests. There are two steps to running a test. First, you need to make sure the Selenium Server process is running in the background. To do this, just execute the command (bat file or shell script) you created when you installed Selenium. This will continue to run (in a console window) until you cancel it. You do not need to stop and start this for each test. Just run it once and let it run in the background.
  
Once Selenium RC is running, you need to execute the functional tests. There are several ways you can do this.  
+
Once Selenium is running, you need to execute the system tests. These are executed the same way we did the unit tests, except for our starting directory.
  
To run all tests from the command line in Windows, change to the <code>tests/system</code> folder and run the following command (the phpunit.bat file comes from the PHPUnit installation):
+
To run all tests from the command line in Windows, change to the <code>tests/system</code> folder and run the following command  
  
<code>phpunit.bat --bootstrap servers\configdef.php suite\TestSuite.php </code>
+
<code>phpunit</code>
  
In Linux, the command is:
+
By default, this will use the <code>tests/system/phpunit.xml.dist</code> file that is included with the Joomla checkout. This will run all of the tests listed in the phpunit.xml.dist file. You should see two browser windows open. One will display Selenium commands that are being executed. The other will show the Joomla! website and the various screens that are being opened and closed as the tests are run. As each test is completed, you will see comments in the console window describing each step in the test.
  
<code>phpunit --bootstrap servers/configdef.php suite/TestSuite.php </code>
+
Note that the entire suite of tests takes about one hour to run. When they are done, you will get a summary display showing how many tests were run and whether there were errors or failures. If there are errors or failures, the line of code from the test program that generated the error or failure will also show. An error occurs when the test encounters an actual PHP error. A failure occurs when the test gets a result that is different than expected (when one of the assert statements in the test is not as expected).
  
This will run all of the tests in the TestSuite.php file. You should see a series of messages display in the console telling you what tests are being run and what the tests are doing. You should also see two browser windows open. One will display Selenium commands that are being executed. The other will show the Joomla! website and the various screens that are being opened and closed as the tests are run.  
+
If you use the default phpunit.xml.dist file, a file <code>tests/system/suite/logs/junit.xml</code> will be created at the conclusion of the test. This file can be interpreted by Eclipse and any other program that works with JUnit unit tests. It provides a graphical way to see the test results.
  
The tests will run for a few minutes, depending on the speed of your system. When they are done, you will get a summary display showing how many tests were run and whether there were errors or failures. If there are errors or failures, the line of code from the test program that generated the error or failure will also show.
+
===Run Selected Tests===
 +
As with the unit tests, you can run selected system tests by folder or file. For example, the following command will run all of the tests in the acl folder:
  
An error occurs when the test encounters an actual PHP error. A failure occurs when the test gets a result that is different than expected.
+
<code>phpunit suite/acl</code>
  
== Tips ==
+
To run a the article0001Test.php tests:
* Make sure you run the tests on a clean database that has been installed with sample data. If you have made database changes, you should re-install Joomla! or otherwise put the database back to it's original post-installed state.
+
* If you find errors or failures when you run the tests, you can run the tests against the Joomla! trunk to see if the problems are in trunk or just in your branch.
+
* If you maintain more than one local Joomla! project (for example, one for trunk, another for a branch), you will need to make sure to have a configdef.php file for each that has the right path and login information. You can watch the browser window while the tests are running to make sure you are testing the URL that you expect.
+
* If TestSuite fails with a red Error message when importing sample data (a Joomla install is the first step in TestSuite), you may need to increase the time allowed for the sample data import in the following line 57 of tests/system/suite/doInstall.php.
+
if ($second >= 15) $this->fail("timeout");
+
  
== Eclipse Users ==
+
<code>phpunit suite/articles/article0001Test.php</code>
In Eclipse, you can set up Debug and External Tools launch configurations that make it easy to debug or run unit and system tests.
+
  
=== Run a Unit Test ===
+
==Webdriver Tests==
You can create an External Tools launch configuration that lets you run any unit test simply by pointing to the test file and pressing the Run button. Here are the steps for Windows:
+
Webdriver is the newer system test program from Selenium. In general, all new Joomla system tests should be written with Webdriver.
* Open the Run&rarr;External Tools&rarr;External Tools Configuration menu and press the button labeled "New Launch Configuration".
+
* Enter a descriptive name, such as "Run Selected Unit Test".
+
* Enter the location that points to your phpunit.bat file (for example, <code>c:\xampp\php\phpunit.bat</code>).
+
* Set the Working Directory to <code>${project_loc}\tests\unit</code>
+
* Set the arguments to <code>c:\xampp\php\phpunit --verbose ${resource_loc}</code>. You can leave out the "--verbose" if you prefer the more compact display.
+
This is shown in the screenshot below.
+
  
[[Image:unit_test_tutorial_screenshot_20100405-01.png]]
+
The Webdriver tests are in the folder <code>tests/system/webdriver</code>. This folder has the following subfolders:
 +
* '''Pages:''' Contains the page class files for the CMS.
 +
* '''SeleniumClient:''' Contains the Nearsoft library files. These are files that allow us to write the Webdriver tests in PHP (instead of Java). They are maintained on Gihub at [https://github.com/Nearsoft/PHP-SeleniumClient https://github.com/Nearsoft/PHP-SeleniumClient].
 +
* '''tests:''' Contains the folders with the test programs. These use the Pages files.
  
Here are the steps for Mac OSX using XAMPP:
+
There is also a file called <code>bootstrap.php</code> in the webdriver folder. This is used to auto-load the required classes. Inside the tests folder we have a file called <code>JoomlaWebdriverTestCase.php</code>. This file is the parent class for all Webdriver tests and has a number of useful methods.
* Enter a descriptive name, such as "Run Selected Unit Test".
+
* Enter the  location that points to your phpunit executable file. For XAMPP, this will be <code>/Applications/XAMPP/xamppfiles/bin/phpunit</code>).
+
* Set the Working Directory to <code>${project_loc}/tests/unit</code>
+
* Set the  arguments to <code>  --verbose  "${resource_loc}"</code>. You can leave out the "--verbose" if you prefer the more compact  display.
+
This is shown in the screenshot below.
+
  
[[Image:Unit_test_tutorial_screenshot_201000708-01.png]]
+
===Run All Webdriver Tests===
 +
Running the Webdriver tests is exactly the same as running the old system tests except for the starting directory. The steps are:
 +
* Make sure the Selenium Server is running in the background.
 +
* Change directory to the <code>tests/system/webdriver/tests</code> folder.
 +
* Execute the command: <code>phpunit</code>
  
To run a unit test:
+
This will by default use the configuration file <code>tests/system/webdriver/tests/phpunit.xml.dist</code>, which is included with the CMS checkout. You can customize this by copying it to phpunit.xml and editing as desired. You do not need to delete the <code>phpunit.xml.dist</code> file because the <code>phpunit.xml</code> file takes priority if both are found.
# Select a unit test, either by selecting it in the PHP Explorer view or in the edit area of Eclipse.
+
# Run the External Launch configuration by selecting it from the drop-down list next to the External Launch Configuration button in the toolbar (shown below).
+
  
[[Image:Unit_test_tutorial_screenshot_20100406-01.png]]
+
Webdriver tests only open one browser window. Like the Selenium RC system tests, Webdriver tests are relatively slow -- similar to a very fast data entry person using the application.
  
Note that a test might normally take a 30-60 seconds to run. The output of the test will show in the Eclipse console.
+
===Run Selected Webdriver Tests===
 +
You can run selected Webdriver tests by folder or by file just as you do for the other system tests or for the unit tests. The only difference is the starting folder.
  
==== Run All Tests in a Folder ====
+
== Tips ==
You can run all of the unit tests in a folder using this same launch configuration. Just select a folder instead of a test file and all of the unit tests in that folder will run.
+
* Make sure you run the tests on a clean database that has been installed with sample data. If you have made database changes, you should re-install Joomla! or otherwise put the database back to it's original post-installed state. When you run the entire system test suite, the first test does a clean re-install for you automatically. This tests the installation and also makes sure the database is in an expected state.
 +
* If you find errors or failures when you run the tests, you can run the tests against the Joomla! master branch to see if the problems are in master or just in your branch.
  
=== Setting Up For Debugging ===
+
== Eclipse Users ==
 
+
In Eclipse, you can set up Debug and External Tools launch configurations that make it easy to debug or run unit and system tests. See [[Running Automated Tests from Eclipse|Running Automated Tests from Eclipse]].
==== Make Sure XDebug is Installed ====
+
As you would expect, XDebug must be installed and configured for use with Eclipse. See [[Setting_up_your_workstation_for_Joomla!_development#Edit_PHP.INI_File]] for detailed instructions.
+
 
+
==== Add PEAR Library to Workspace ====
+
Before you can debug a unit or system test, you have to add the PEAR library to your project or workspace. In this example, we will add PEAR to our workspace. That way, all projects will automatically have access to PHPUnit.
+
 
+
Here are the steps:
+
 
+
* Select Window&rarr;Preferences&rarr;PHP&rarr;PHP Libraries.
+
* Press the New button and add PEAR as shown below. Be sure to press the checkbox labeled "Add to environment".
+
+
[[Image:unit_test_tutorial_screenshot_20100406-01a.png]]
+
 
+
* Press OK, then press the Add External folder button and browse to your PEAR folder, as shown below. In this example, the folder is in <code>c:\xampp\php\PEAR</code>.
+
+
[[Image:unit_test_tutorial_screenshot_20100406-01b.png]]
+
 
+
* Press OK and the screen should show the external folder, as shown below, with the correct file path for your workstation.
+
+
[[Image:unit_test_tutorial_screenshot_20100406-01c.png]]
+
 
+
After this is completed go to project preferences - > Properties -> PHP Include path -> Libraries tab -> Add library. Select User library in popup, click Next and mark Pear[enviroment] as checked.
+
 
+
Also make sure your project folder is added in Source tab of your project's PHP Include path.
+
 
+
==== Configure the PHP Executable ====
+
Normally, you will want to debug unit and system tests as PHP scripts (as opposed to web pages). Before you can do this, you need to configure a PHP Executable in Eclipse. This is done as follows:
+
* Select Window&rarr;Preferences&rarr;PHP&rarr;PHP Executables. The screen below will show.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-05.png]]
+
 
+
* Press Add and complete the screen as shown below, substituting the correct file paths for your workstation.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-06.png]]
+
 
+
At this point, your workstation is set up for PHPUnit debugging. Now you just have to create the debug configurations as shown in the next section.
+
 
+
=== Debug a Single Unit Test ===
+
Once you have PEAR added to your libraries, it is easy to set up a debug configuration that allows you to debug any unit test. Here are the steps:
+
* Press the drop-down arrow next to the Debug button and select Debug Configurations from the menu, as shown below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-02.png]]
+
 
+
* In the Debug Configurations window, make sure that PHP Script is selected and press the New Launch configuration button as shown below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-17.png]]
+
 
+
* Fill out the PHP Script tab of the Debug Configurations as shown below.
+
** Name: a descriptive name like "Debug Selected Unit Test"
+
** PHP Debugger: XDebug
+
** PHP Executable: The one you set up earlier (there will only be one)
+
** PHP File: Use the Browse button to browse to the phpunit.php file in your <code>tests/system</code> folder.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-10.png]]
+
 
+
* Select the PHP Script Arguments tab and enter --verbose ${selected_resource_loc} as shown below.
+
** The "--verbose" option allows you to see more details when the tests are run.
+
** The "${selected_resource_loc}" passes the currently selected file at runtime, so you can use this configuration just by selecting the desired file and then running it.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-11.png]]
+
 
+
* Finally, select the Common tab and check the Debug checkbox under "Display in favorites menu", as shown below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-09a.png]]
+
 
+
At this point, you are ready to debug any unit test. All you need to do is:
+
# Select the desired test file either in the PHP Explorer view or in the editor.
+
# Select Debug Configurations from the Debug drop-down button and select the "Debug Selected Unit Test" option. Once you have run this once, it will show on your Debug favorites menu.
+
 
+
=== Run a Single System Test ===
+
You can create a configuration to make it easy to run any single test in your project. To do this:
+
* Open the Run&rarr;External Tools&rarr;External Tools Configuration menu and press the button labeled "New Launch Configuration"
+
* Enter the location that points to your phpunit.bat file (for example, <code>c:\xampp\php\phpunit.bat</code>). On Linux systems, this might be called "phpunit".
+
* Set the Working Directory to <code>${project_loc}/tests/system</code>
+
* Set the arguments to <code>--bootstrap servers/configdef.php ${selected_resource_loc}</code>.
+
This is shown in the screenshot below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-14.png]]
+
 
+
To use this configuration,
+
# Make sure the Selenium RC process is running (for example, by running the "selenium.bat" file you created when you installed Selenium RC earlier)
+
# Select the desired test file by clicking on it in the PHP Explorer view or in the edit area
+
# Start this launch configuration
+
The selected test file will be run and you will see the browser session open and run the test script. The test results will be reported in the Eclipse console.
+
 
+
=== Debug a Single System Test ===
+
The steps for creating a Debug Configuration for a Selenium system test are similar to those for the unit test.
+
* Open Debug Configurations from the drop-down menu next to the Debug Button on the toolbar, as shown below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-02.png]]
+
 
+
* In the Debug Configurations window, make sure that PHP Script is selected and press the New Launch configuration button as shown below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-17.png]]
+
 
+
* Fill out the Debug Configurations dialog box as shown below:
+
** Name: A descriptive name for the configuration, like "Debug Selected System Test".
+
** PHP Debugger: XDebug
+
** PHP Executable: The one you defined earlier (there probably is only one).
+
** PHP File: Browse to the phpunit.php file in your <code>tests/system</code> folder.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-12.png]]
+
 
+
* Select the PHP Script arguments and fill this out as shown below.
+
** --bootstrap servers/configdef.php points to the configuration file that passes the login information to Selenium.
+
** "${selected_resource_loc}" passes the location of the currently selected file in Eclipse to the command line.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-13.png]]
+
 
+
* Finally, select the Common tab and check the Debug option in the Display in favorites menu. This will add this configuration to your favorites.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-09a.png]]
+
 
+
At this point, you can debug any system test. To do so:
+
# Make sure the Selenium RC program is running in the background.
+
# Select the desired test by selecting it in the PHP Explorer or in the editor.
+
# Run this Debug Configuration by selecting the drop-down arrow next to the Debug button, then Debug Configurations, then the "Debug Selected System Test" configuration created above.
+
 
+
=== Run the System Test Suite ===
+
To create a launch configuration to run the Test Suite of the currently-selected Eclipse project:
+
* Enter the location that points to your phpunit.bat file. If you are running Linux, this file might be called phpunit.
+
* Set the Working Directory to <code>${project_loc}/tests\system</code>
+
* Set the arguments to <code>--bootstrap servers/configdef.php suite/TestSuite.php</code>.
+
This is shown in the screenshot below.
+
 
+
[[Image:unit_test_tutorial_screenshot_20100406-15.png]]
+
  
The console output will show in the Eclipse console. Note that you still need to have the Selenium RC program running in the background first, before running the suite. Note that you need to select a file in the current project before running this command so the <code>{project_loc}</code> variable is set correctly.
+
== Running tests on Ubuntu 12.04 ==
 +
phpUnit 3.8 will not install on Ubuntu 12.04 because it requires a later version of PHP than is supported on Ubuntu 12.04.  The solution that worked for me was to install an earlier version of phpUnit (3.7) using Composer.  I'm certain this is not the only way this can be done and it may not be the best, but it worked for me (eventually).
  
 +
# Check out Joomla from Github in the usual way (git clone).
 +
# Install Composer into the Joomla root directory (see instructions on Composer website).
 +
# Create a composer.json file containing <code>{"require-dev": {"phpunit/phpunit": "3.7.*"}}</code>
 +
# sudo php composer.phar install
 +
# php [path-to-Joomla]/vendor/phpunit/phpunit/composer/bin/phpunit [path-to-test] will run an individual test script.  Running all tests in a folder doesn't seem to work.
  
[[Category:Bug Squad]] [[Category:Development]] [[Category:Testing]] [[Category:Automated Testing]] [[Category:Joomla! 1.6]]
+
[[Category:Bug Squad]] [[Category:Development]] [[Category:Testing]] [[Category:Automated Testing]] [[Category:Joomla! 2.5]][[Category:Joomla! 3.x]]

Revision as of 20:38, 7 May 2014

Contents

Introduction

This article has been updated as of November 2012 to reflect the latest information for running unit and system tests for Joomla version 3.0.

When you checkout the master branch of Joomla from Github.com (https://github.com/joomla/joomla-cms) you will see a folder called "tests". This folder contains unit and two types of system tests for the CMS. The unit tests use PHPUnit and the system tests use PHPUnit and Selenium. The unit tests perform tests on the CMS library files (libraries/cms). The Selenium and Webdriver system tests run Joomla! from a browser and test it as a user would.

As of 3.0, we are transitioning our system tests from the older Selenium RC (found in the tests/system/suite folder) to the newer Webdriver methodolgy (found in tests/system/webdriver). We will continue to use and maintain the older tests, but all new tests will be written using Webdriver.

Before you can run the tests, you need to install some programs on your local workstation, as documented below. This document explains how to run the these tests from your local workstation.

PHP Requirements

You may need to modify your PHP configuration. This is done by editing your php.ini file and then restarting your Apache service.

CMS Unit Tests

Some of the CMS unit tests rely on the PHP extension Sqlite3. To run these tests, make sure that the following line in your php.ini file is uncommented:

  • For Windows: extension=php_sqlite3.dll
  • For Linux: extension=php_sqlite3.so

Webdriver System Tests

The Webdriver system tests require PHP version 5.3.8 or later. They also require that the Curl extension is installed. Make sure the following line in your php.ini file is uncommented:

  • For Windows: extension=php_curl.dll
  • For Linux: extension=php_curl.so

Install PHPUnit

Both the unit and system tests rely on PHPUnit. PHPUnit is a testing framework that requires PHP to be installed on the local workstation. In the past, PEAR was needed to install PHPUnit. That has changed. Please visit http://phpunit.de/ and follow the instructions to install PHPUnit.

Windows Users

The instructions at http://phpunit.de/ may not work on Windows. Most likely, Windows users will not have the `wget` command used in the instructions. However, if msysgit is installed, curl should be available. If it is, then the simplest way to install PHPUnit on Windows is to just use this command:

   curl https://phar.phpunit.de/phpunit.phar > _some_directory_in_your_path_/phpunit

This will download the phar file and store it without the phar extension so that you can run the command simply by running:

   phpunit

Running Unit Tests

Run Suite from XML File

Once PHPUnit is installed, it's easy to run the unit tests. The command that runs the tests is called phpunit. This file is found in your PHP folder. For example, in Windows with Xampp, it would be c:/xampp/php/phpunit. In Linux, with Xampp, it might be /opt/lampp/bin/phpunit. If you add this to your operating system path, you can execute it from any folder without using the full path name.

When you run phpunit it looks for a PHPUnit configuration XML file (either phpunit.xml or phpunit.xml.dist). The CMS unit tests are run from the Joomla root folder, where you will find the phpunit.xml.dist file. This file is included in the Git repository and can be used as is or customized (by making a copy called phpunit.xml).

To run all of the unit tests from the command line:

  1. Change Joomla root folder
  2. Execute the command phpunit.

The following example runs the entire unit test suite using the phpunit.xml.dist file in Windows.

C:\xampp\htdocs\joomla_development\cms-trunk>phpunit
PHPUnit 3.6.11 by Sebastian Bergmann.

Configuration read from C:\xampp\htdocs\joomla_development\cms-trunk\phpunit.xml.dist

IIIIIIII....................SS.......S.........................  63 / 234 ( 26%)
............................................ISSISSSSSSSSSSI.... 126 / 234 ( 53%)
...IIIIIIIIII.I.I.I.IIIIIIIIIIIIIIIII..IIIIIIIIS.IIIIII..II..SS 189 / 234 ( 80%)
SI..I........I...I...I...I.III.I.............

Time: 13 seconds, Memory: 21.25Mb

OK, but incomplete or skipped tests!
Tests: 234, Assertions: 281, Incomplete: 67, Skipped: 19. 

Generating code coverage report in Clover XML format ... done

Generating code coverage report in HTML format ... done

The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show. If you are generating code coverage or other logs, you can see the folders where these are generated in the XML file. In the default file, all of the logs are generated in build/logs, with coverage information in a folder called coverage.

You can create your own phpunit.xml file by copying the phpunit.xml.dist file and making the desired changes. For example, if you don't want to create the coverage information, remove the log type="coverage-html" element.

Run Selected Unit Tests

You can run unit tests one at a time or by folder. The following example runs all of the tests in the libraries/feed folder:

C:\xampp\htdocs\joomla_development\cms-trunk>phpunit tests/unit/suites/libraries
/feed
PHPUnit 3.6.11 by Sebastian Bergmann.

Configuration read from C:\xampp\htdocs\joomla_development\cms-trunk\phpunit.xml
....................SS.......S................................... 65 / 99 ( 65%)
..................................
Time: 1 second, Memory: 8.25Mb
OK, but incomplete or skipped tests!

You can also specify a single test simply by specifying it's full path name in the command.

Selenium RC System Tests

The older Selenium RC system tests are in the folder tests/system/suite. Before you can run the system tests, you need to install and download the Selenium Server program and configure your test environment, as shown below.

Download Selenium Server

Selenium Server (formerly Selenium RC) is the package that allows us to run Selenium tests from PHP (instead of Java). To download it, just go to the Selenium site http://seleniumhq.org/download/ and download the Selenium Server file (for example, selenium-server-standalone-2.25.0.jar) to a folder.

For example, in Windows if you create a folder called C:\selenium and copy this file there. In that folder, create a Windows bat file (such as selenium.bat though the name does not matter) or Linux shell script that runs the following command:

java -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar

(Obviously you will need to use the exact name of the .jar file for the version you downloaded.) The -Xms and -Xmx arguments run the server program allocating more memory to Java than the default. That seems to help prevent intermittent errors when running a long suite of system tests.

If the Java executable is not on your path, then you will need to indicate the full path to it, like the following:

"c:\program files (x86)\Java\jre6\bin\java.exe" -Xms40m -Xmx256m -jar selenium-server-2.25.0.jar

This program needs to be running in the background before you can run any Selenium functional tests. So just run this program from a Bat file and it will continue to run in the background in a console window until you close it (for example, with Ctrl+C).

Create a Selenium Configuration File

To run the Selenium tests, we have to tell Selenium how to navigate and login to our local Joomla! installation. This is done by creating a PHP file called tests/system/servers/configdef.php. The Joomla! download includes a file called tests/system/servers/config-def.php-dist. This is a sample file that you can use to create the real file. Copy this file to a file called tests/system/servers/configdef.php and edit it to reflect your test systems configuration. There are comments in the file that tell you how to do this.

Run the System Test Suite

At this point, we are ready to actually run the tests. There are two steps to running a test. First, you need to make sure the Selenium Server process is running in the background. To do this, just execute the command (bat file or shell script) you created when you installed Selenium. This will continue to run (in a console window) until you cancel it. You do not need to stop and start this for each test. Just run it once and let it run in the background.

Once Selenium is running, you need to execute the system tests. These are executed the same way we did the unit tests, except for our starting directory.

To run all tests from the command line in Windows, change to the tests/system folder and run the following command

phpunit

By default, this will use the tests/system/phpunit.xml.dist file that is included with the Joomla checkout. This will run all of the tests listed in the phpunit.xml.dist file. You should see two browser windows open. One will display Selenium commands that are being executed. The other will show the Joomla! website and the various screens that are being opened and closed as the tests are run. As each test is completed, you will see comments in the console window describing each step in the test.

Note that the entire suite of tests takes about one hour to run. When they are done, you will get a summary display showing how many tests were run and whether there were errors or failures. If there are errors or failures, the line of code from the test program that generated the error or failure will also show. An error occurs when the test encounters an actual PHP error. A failure occurs when the test gets a result that is different than expected (when one of the assert statements in the test is not as expected).

If you use the default phpunit.xml.dist file, a file tests/system/suite/logs/junit.xml will be created at the conclusion of the test. This file can be interpreted by Eclipse and any other program that works with JUnit unit tests. It provides a graphical way to see the test results.

Run Selected Tests

As with the unit tests, you can run selected system tests by folder or file. For example, the following command will run all of the tests in the acl folder:

phpunit suite/acl

To run a the article0001Test.php tests:

phpunit suite/articles/article0001Test.php

Webdriver Tests

Webdriver is the newer system test program from Selenium. In general, all new Joomla system tests should be written with Webdriver.

The Webdriver tests are in the folder tests/system/webdriver. This folder has the following subfolders:

  • Pages: Contains the page class files for the CMS.
  • SeleniumClient: Contains the Nearsoft library files. These are files that allow us to write the Webdriver tests in PHP (instead of Java). They are maintained on Gihub at https://github.com/Nearsoft/PHP-SeleniumClient.
  • tests: Contains the folders with the test programs. These use the Pages files.

There is also a file called bootstrap.php in the webdriver folder. This is used to auto-load the required classes. Inside the tests folder we have a file called JoomlaWebdriverTestCase.php. This file is the parent class for all Webdriver tests and has a number of useful methods.

Run All Webdriver Tests

Running the Webdriver tests is exactly the same as running the old system tests except for the starting directory. The steps are:

  • Make sure the Selenium Server is running in the background.
  • Change directory to the tests/system/webdriver/tests folder.
  • Execute the command: phpunit

This will by default use the configuration file tests/system/webdriver/tests/phpunit.xml.dist, which is included with the CMS checkout. You can customize this by copying it to phpunit.xml and editing as desired. You do not need to delete the phpunit.xml.dist file because the phpunit.xml file takes priority if both are found.

Webdriver tests only open one browser window. Like the Selenium RC system tests, Webdriver tests are relatively slow -- similar to a very fast data entry person using the application.

Run Selected Webdriver Tests

You can run selected Webdriver tests by folder or by file just as you do for the other system tests or for the unit tests. The only difference is the starting folder.

Tips

  • Make sure you run the tests on a clean database that has been installed with sample data. If you have made database changes, you should re-install Joomla! or otherwise put the database back to it's original post-installed state. When you run the entire system test suite, the first test does a clean re-install for you automatically. This tests the installation and also makes sure the database is in an expected state.
  • If you find errors or failures when you run the tests, you can run the tests against the Joomla! master branch to see if the problems are in master or just in your branch.

Eclipse Users

In Eclipse, you can set up Debug and External Tools launch configurations that make it easy to debug or run unit and system tests. See Running Automated Tests from Eclipse.

Running tests on Ubuntu 12.04

phpUnit 3.8 will not install on Ubuntu 12.04 because it requires a later version of PHP than is supported on Ubuntu 12.04. The solution that worked for me was to install an earlier version of phpUnit (3.7) using Composer. I'm certain this is not the only way this can be done and it may not be the best, but it worked for me (eventually).

  1. Check out Joomla from Github in the usual way (git clone).
  2. Install Composer into the Joomla root directory (see instructions on Composer website).
  3. Create a composer.json file containing {"require-dev": {"phpunit/phpunit": "3.7.*"}}
  4. sudo php composer.phar install
  5. php [path-to-Joomla]/vendor/phpunit/phpunit/composer/bin/phpunit [path-to-test] will run an individual test script. Running all tests in a folder doesn't seem to work.