Difference between revisions of "Running Automated Tests for the Joomla CMS"

From Joomla! Documentation

(fix up wording)
(Run Unit Tests section)
Line 31: Line 31:
  
 
More detailed instructions can be found here: [http://www.phpunit.de/manual/3.0/en/installation.html http://www.phpunit.de/manual/3.0/en/installation.html].
 
More detailed instructions can be found here: [http://www.phpunit.de/manual/3.0/en/installation.html http://www.phpunit.de/manual/3.0/en/installation.html].
 +
 +
== Running Unit Tests ==
 +
Once PHPUnit is installed, you can run the unit tests. To run a unit test from the command line:
 +
# Change to the folder <code><joomla root>/tests/unit</code>
 +
# Execute the command <code>phpunit <test name or folder></code>
 +
For example, to execute the test called "tests\unit\suite\libraries\joomla\utilities\JStringTest.php", you would type the command
 +
* <code>phpunit tests\unit\suite\libraries\joomla\utilities\JStringTest.php</code>
 +
 +
When you run this, you will see output similar to the following:
 +
<pre>
 +
C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
 +
joomla\utilities\jstringtest.php
 +
PHPUnit 3.4.11 by Sebastian Bergmann.
 +
 +
............................................................ 60 / 89
 +
.............................
 +
 +
Time: 26 seconds, Memory: 7.00Mb
 +
 +
OK (89 tests, 89 assertions)
 +
</pre>
 +
The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show.
 +
 +
You can execute all of the tests in a folder by specifying a folder instead of a file. For example, to run all of the tests in the folder "tests\unit\suite\libraries\joomla\utilities", you would enter
 +
* <code>phpunit tests\unit\suite\libraries\joomla\utilities</code>
 +
 +
This command will produce output similar to that shown below:
 +
<pre>
 +
C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
 +
joomla\utilities
 +
PHPUnit 3.4.11 by Sebastian Bergmann.
 +
 +
............................................................  60 / 317
 +
............................................................ 120 / 317
 +
............................................................ 180 / 317
 +
............................................................ 240 / 317
 +
............................................................ 300 / 317
 +
...FFFF..........
 +
 +
Time: 27 seconds, Memory: 11.00Mb
 +
 +
There were 4 failures:
 +
 +
1) JUtilityTest::testGetHash
 +
Failed asserting that two strings are equal.
 +
--- Expected
 +
+++ Actual
 +
@@ @@
 +
-ce114e4501d2f4e2dcea3e17b546f339
 +
+0cbc6611f5540bd0809a388dc95a615b
 +
 +
 +
2) JUtilityTest::testGetToken with data set "default" (NULL, false)
 +
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
 +
<boolean:false>.
 +
 +
 +
3) JUtilityTest::testGetToken with data set "false" (false, false)
 +
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
 +
<boolean:false>.
 +
 +
 +
4) JUtilityTest::testGetToken with data set "true" (true, true)
 +
Expectation failed for method name is equal to <string:getFormToken> when invoke
 +
d 1 time(s).
 +
Method was expected to be called 1 times, actually called 0 times.
 +
 +
 +
FAILURES!
 +
Tests: 317, Assertions: 361, Failures: 4.</pre>
 +
In this case, we have 4 test failures with details about each.
  
 
== Install Selenium RC ==
 
== Install Selenium RC ==

Revision as of 16:31, 5 April 2010

Introduction[edit]

As of February 2010, a tests folder has been added to the Joomla! SVN download. This folder contains a growing library of unit and system (or functional) tests. The unit tests use PHPUnit and the system tests use PHPUnit and Selenium. The unit tests perform tests primarily on the Joomla! framework. The Selenium system tests actually run Joomla! from a browser and test it as a user would.

Before you can run the tests, you need to install some programs on your local workstation, as documented below. This document explains how to run the these tests from your local workstation.

Install PHPUnit[edit]

Both the unit and functional tests rely on PHPUnit. PHPUnit is a testing framework that requires PHP to be installed on the local workstation. XAMP / WAMP users can use the PHP installation that comes with them. Otherwise, you may need to install PHP, and the PEAR package, on your local workstation. More detailed instructions can be found at http://www.php.net/manual/en/install.php.

On most PHP installations, PNPUnit can be installed using the standard PEAR installation process.

For example, on XAMPP version 1.7.3 on Windows, you would do the following commands:

  • cd c:\xampp\php
  • pear channel-discover pear.phpunit.de
  • pear channel-discover pear.symfony-project.com
  • pear install phpunit/PHPUnit

If you get error about PEAR version too low, run:

  • pear upgrade pear

and repeat pear install phpunit/PHPUnit . Windows users not using WAMP or XAMPP can use the same procedure, with substituting the location of your PHP installation for cd c:\xampp\php.

In Ubuntu Linux, use the following commands to install PHPUnit:

  • $sudo pear channel-discover pear.phpunit.de
  • $sudo pear channel-discover pear.symfony-project.com
  • $sudo pear install phpunit/PHPUnit

More detailed instructions can be found here: http://www.phpunit.de/manual/3.0/en/installation.html.

Running Unit Tests[edit]

Once PHPUnit is installed, you can run the unit tests. To run a unit test from the command line:

  1. Change to the folder <joomla root>/tests/unit
  2. Execute the command phpunit <test name or folder>

For example, to execute the test called "tests\unit\suite\libraries\joomla\utilities\JStringTest.php", you would type the command

  • phpunit tests\unit\suite\libraries\joomla\utilities\JStringTest.php

When you run this, you will see output similar to the following:

C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
joomla\utilities\jstringtest.php
PHPUnit 3.4.11 by Sebastian Bergmann.

............................................................ 60 / 89
.............................

Time: 26 seconds, Memory: 7.00Mb

OK (89 tests, 89 assertions)

The dots indicate a successful test. If you have errors or failures, they will show as "E" or "F" letters and a detailed message for each will show.

You can execute all of the tests in a folder by specifying a folder instead of a file. For example, to run all of the tests in the folder "tests\unit\suite\libraries\joomla\utilities", you would enter

  • phpunit tests\unit\suite\libraries\joomla\utilities

This command will produce output similar to that shown below:

C:\xampp\htdocs\joomla_development\j16_trunk\tests\unit>phpunit suite\libraries\
joomla\utilities
PHPUnit 3.4.11 by Sebastian Bergmann.

............................................................  60 / 317
............................................................ 120 / 317
............................................................ 180 / 317
............................................................ 240 / 317
............................................................ 300 / 317
...FFFF..........

Time: 27 seconds, Memory: 11.00Mb

There were 4 failures:

1) JUtilityTest::testGetHash
Failed asserting that two strings are equal.
--- Expected
+++ Actual
@@ @@
-ce114e4501d2f4e2dcea3e17b546f339
+0cbc6611f5540bd0809a388dc95a615b


2) JUtilityTest::testGetToken with data set "default" (NULL, false)
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
 <boolean:false>.


3) JUtilityTest::testGetToken with data set "false" (false, false)
Failed asserting that <string:adca734617ce829cc979492d6d037416> matches expected
 <boolean:false>.


4) JUtilityTest::testGetToken with data set "true" (true, true)
Expectation failed for method name is equal to <string:getFormToken> when invoke
d 1 time(s).
Method was expected to be called 1 times, actually called 0 times.


FAILURES!
Tests: 317, Assertions: 361, Failures: 4.

In this case, we have 4 test failures with details about each.

Install Selenium RC[edit]

Selenium RC is the package that allows you to run Selenium tests from PHP or other programming languages. To install it, just go to the Selenium site http://seleniumhq.org/projects/remote-control and download the package. Then unzip the file into a folder.

For example, in Windows if you create a folder called C:\selenium and unzip this file there, it will create a folder called selenium-server-1.0.1. In that folder, create a Windows bat file (such as selenium.bat though the name does not matter) or Linux shell script that runs the following command:

java -jar selenium-server.jar -browserSessionReuse

If the Java executable is not on your path, then you will need to indicate the full path to it, like the following:

c:\program files (x86)\Java\jre6\bin\java.exe" -jar selenium-server.jar -browserSessionReuse

Note the argument browserSessionReuse is used to allow you to run multiple tests without closing and re-opening the browser each time. Save this file so you can easily find and execute it when needed.

This program needs to be running in the background before you can run any Selenium functional tests.

If You are using Firefox 3.6.x you will run in problems with this configuration. Check out this link: http://www.qaautomation.net/?p=15 how to get it working on Firefox 3.6.x.

Create a Selenium Configuration File[edit]

To run the Selenium tests, we have to tell Selenium how to navigate and login to your local Joomla! installation. This is done by creating a PHP file called configdef.php. The Joomla! download includes a file called tests/system/servers/config-def.php-dist. This is a sample file that you can use to create the real file. Copy this file to a file called tests/system/servers/configdef.php and edit it to reflect your test systems configuration. There are comments in the file that tell you how to do this.

Run the Test Suite[edit]

At this point, we are ready to actually run the tests. There are two steps to running a test. First, you need to make sure the Selenium RC process is running in the background. To do this, just execute the command (bat file or shell script) you created when you installed Selenium RC. This will continue to run until you cancel it.

Once Selenium RC is running, you need to execute the functional tests. There are several ways you can do this.

To run all tests from the command line in Windows, change to the tests/system folder and run the following command (the phpunit.bat file comes from the PHPUnit installation):

phpunit.bat --bootstrap servers\configdef.php suite\TestSuite.php

In Linux, the command is:

phpunit --bootstrap servers/configdef.php suite/TestSuite.php

This will run all of the tests in the TestSuite.php file. You should see a series of messages display in the console telling you what tests are being run and what the tests are doing. You should also see two browser windows open. One will display Selenium commands that are being executed. The other will show the Joomla! website and the various screens that are being opened and closed as the tests are run.

The tests will run for a few minutes, depending on the speed of your system. When they are done, you will get a summary display showing how many tests were run and whether there were errors or failures. If there are errors or failures, the line of code from the test program that generated the error or failure will also show.

An error occurs when the test encounters an actual PHP error. A failure occurs when the test gets a result that is different than expected.

Tips[edit]

  • Make sure you run the tests on a clean database that has been installed with sample data. If you have made database changes, you should re-install Joomla! or otherwise put the database back to it's original post-installed state.
  • If you find errors or failures when you run the tests, you can run the tests against the Joomla! trunk to see if the problems are in trunk or just in your branch.
  • If you maintain more than one local Joomla! project (for example, one for trunk, another for a branch), you will need to make sure to have a configdef.php file for each that has the right path and login information. You can watch the browser window while the tests are running to make sure you are testing the URL that you expect.


Eclipse Users[edit]

You can use the External Tools Configurations option, found under the Run -> External Tools menu, to create launch configurations to run the tests.

  • For example, to create a launch configuration to run the Test Suite of the currently-selected Eclipse project, you would enter the location that points to your phpunit.bat file, set the Working Directory to "${project_loc}\tests\system", and set the arguments to --bootstrap servers\configdef.php suite\TestSuite.php. This is shown in the screenshot below.

File:Selenium tutorial screenshot 20100205.png

The argument suite\TestSuite.php identifies the test that you wish to run. To run a different test, change this argument to the test that you wish to run. For example, to run com_users user0001Test.php, you would use suite\com_users\user0001Test.php. Remember that we have already set the Working Directory as \tests\system.
Then you can run this from inside Eclipse and the console output will show in the Eclipse console. Note that you still need to have the Selenium RC program running in the background first, before running the suite. Note that you need to select a file in the current project before running this command so the {project_loc} variable is set correctly.
  • You can create a configuration to make it easy to run any single test in your project. To do this, create a new External Tools Configuration as shown below:

File:Selenium tutorial screenshot 20100211.png

This is the same as the configuration above except that the second argument is ${selected_resource_loc}. To use this configuration, select the desired test file by clicking on it in the PHP Explorer view or in the edit area. Then start this launch configuration. The selected test file will be run.