Actions

Difference between revisions of "System Testing"

From Joomla! Documentation

(One intermediate revision by one user not shown)
Line 1: Line 1:
{{Underconstruction}}
 
{{RightTOC}}
 
 
== News and Updates ==
 
== News and Updates ==
 
* 2009 07 27 Document created.
 
* 2009 07 27 Document created.
 
* 2010 05 24 Added Tips & Tricks section.
 
* 2010 05 24 Added Tips & Tricks section.
 
* 2010 05 28 Started content overhaul and integration of related docs for more logical flow.
 
* 2010 05 28 Started content overhaul and integration of related docs for more logical flow.
 +
* 2010 06 07 Removed the sections on Selenium IDE and merged those with [[Writing System Tests for Version 1.6]].
  
 
== Overview ==
 
== Overview ==
Line 10: Line 9:
 
For a good general discussion of system testing, visit the [http://en.wikipedia.org/wiki/System_Testing Wikipedia article].
 
For a good general discussion of system testing, visit the [http://en.wikipedia.org/wiki/System_Testing Wikipedia article].
  
=== System Testing in Open Source ===
+
== System Testing in Joomla! ==
Web applications have been notoriously difficult to do system testing with. It has traditionally required a great deal of time and effort. This can be a special challenge for open source projects because testing is often not very popular and designing tests that can be executed in a systematic way even less so. For a complicated application like Joomla!, this means that although we might do testing, we very seldom are able to perform extensive testing on each release that covers even a fraction of the elements in the CMS.
+
System testing in Joomla! uses an open-source program called [http://www.seleniumhq.org Selenium]. Selenium allows you to create a test script that can be run automatically. The test script actually runs Joomla! in a browser and does the same keystrokes and mouse clicks that a user would do. It can also test for conditions such as the value of text or the presence of HTML elements and report when a test fails or when there is an error in the program.
 
+
Relatively recently, a project called Selenium has arrived on the scene.  This project makes performing automated system testing of web applications possible. This means that it is possible to devise a test and have it performed in an automated way without the requirement of somebody sitting in front of a browser and being a click monkey.
+
 
+
 
The benefits of system testing are:
 
The benefits of system testing are:
* System tests help highlight cases where changes in one element of the system might cause breakage in other, unexpected areas.
+
* System tests help you test that the application is working correctly from the point of view of a user.
 
* System tests help clearly specify how the application should behave.
 
* System tests help clearly specify how the application should behave.
 +
* System tests help you test that changes made to one part of the application haven't created a bug somewhere else.
 +
* System tests can be run automatically (for example, each night) so that testing is done as the application is being developed.
  
=== System Testing versus Unit Testing ===
+
== System Testing versus Unit Testing ==
In order to fully test an application, both system and unit testing are essential. Each approach has their benefits and weaknesses. Unit testing is very good for isolating small chunks of code and ensuring that they are behaving as expected. System testing, on the other hand, is geared towards looking at the system as a whole and ensuring that the units are behaving together in a way that achieves the desired effect.
+
System Testing and Unit Testing are complementary test strategies. In Joomla!, Unit Testing is mainly used to test the framework classes (for example, the classes in libraries/joomla). Unit tests test that an individual method (also known as function) in a class does what it is supposed to do. For example, a unit test is used to test that the methods in the JString class work as expected. Unit testing provides confidence that the framework classes work as expected and allows you to refactor these classes (improve the code without changing the functionality) and still have confidence that they still work correctly.
 
+
The advantages of system testing is that system tests are easier to understand and conceptualize by people who are unfamiliar with the underlying code. With Selenium, tests can be designed by people who are not overly familiar with PHP - they only need to understand the application and what results should be expected.
+
 
+
The downside of system testing is that it will only tell you that something broke. It isn't as obvious with system testing exactly what went wrong or where the problem is.
+
 
+
==== Test Objects ====
+
If the purpose of unit tests is to isolate a module of code, then the purpose of system tests is to evaluate the entire system. This is much easier for those unfamiliar with what happens under the hood of Joomla! because one is not required to understand what happens during a Joomla! page request. A system test might evaluate, for example, if unpublishing an article works properly. Therefore, to design the test, it is only necessary to figure out how to change an article from Published to Unpublished, and how to verify that the article was actually unpublished.
+
 
+
=== System Testing in Joomla! ===
+
I have recently been building a system testing architecture that will make it possible to write tests once, and to execute those tests on a variety of platforms and clients. I currently have infrastructure in place that makes it possible to run one command and execute system tests in one browser on one host system. I have designed the host system in such a way that it will run on 6 different PHP versions (stock Ubuntu PHP 5 and self-compiled PHP 4.3, 4.4, 5.0, 5.1 and 5.2) and 2 different web servers (Apache and Lighttpd).  These are all currently serving from the same database.
+
 
+
The future steps will be to make it possible to run on multiple host systems (the requirement for this is to generalize the code that resets the database state after the tests are run) and to make it possible to use different clients (i.e. different systems with different web browsers). These steps should be fairly trivial, but I have not yet had time to focus on them.
+
 
+
This will eventually make it possible to run the designed tests on every major browser and on multiple server platforms (various PHP versions, various configurations, various operating systems and various web servers).
+
 
+
 
+
== Recording System Tests with Selenium IDE ==
+
Selenium IDE makes it fairly easy to write system tests. It is a Firefox extension that operates as an Integrated Development Environment for creating system tests. In essence, it records your clicks as you navigate in your browser and encodes them into a series of commands that can control the browser. Tests recorded with Selenium IDE can be converted to tests within the Joomla repository and integrated into the test suite.
+
 
+
=== Getting Selenium IDE ===
+
At of May 27, 2010, the latest version of Selenium IDE is 1.0.7.  You can get it at http://seleniumhq.org/download/. Click on the download link and install like you would any Firefox plugin.
+
 
+
==== Preparing your Environment ====
+
To get started, create a clean install of Joomla! with the standard sample data or update your local copy to the latest build and perform a re-installation.
+
 
+
=== Creating a Basic Test ===
+
# To start, open Firefox and browse to the home page of your Joomla! install.  For example, I setup my install on my localhost, and so the URL is http://127.0.0.1/selsampledata.
+
# Click on Tools -> Selenium IDE. You will notice that on the right of the window that appears near the top, that there is a red circle that is highlighted. This is the start/stop recording button. When you start Selenium IDE, recording starts right away.
+
#* For our first test, we are going to load the home page and check to make sure that all the items in the Main Menu are present.
+
#* The commands that we use to check that items are present are called assertions. Basically, you perform your actions and make assertions about the results.
+
#* In our case, we are going to use the command assertText. This command will read the text of a specified element and ensure it matches a particular value.
+
# Ensure that the Base URL displayed in the Selenium IDE window is the home page of your Joomla! install.
+
# Switch to the browser window that has your Joomla install open.
+
# Right click on each menu item and select 'Assert Text'.
+
# Switch to your Selenium IDE window and click the Red Button to stop recording.
+
 
+
You should see two tabs in your Selenium IDE window - one labeled Table, and one labeled Source. At this point you don't really need to look at the Source window.
+
 
+
[[Image:Selenium-screen.png]]
+
 
+
 
+
Our first test is done.
+
 
+
To run your test, you use the icons on the bar between the Base URL address bar and the Table and Source tabs. The two important buttons are the buttons with the Green Triangle with Three Solid rectangles and the one with the Green Triangle and One Solid Rectangle. The first one will execute the entire test suite and the second one will run the currently selected test case. Since we currently have only one test case, both of these currently do the same thing.
+
 
+
To run your test, press one of these buttons.
+
 
+
As you watch the Selenium IDE, you will see a yellow bar move down your list of steps. Once a step is completed, a successfully executed action (i.e. a button was successfully located and clicked) should show up in light green and a successful assertion should show up in darker green. Failed actions show up in light red and failed assertions show up in darker red.
+
 
+
Click on the Window Expander on the left to unhide the Test Case browser.
+
 
+
If your test completed successfully, you should see Runs: 1, Failures: 0 at the bottom of the Test Case browser.
+
 
+
Once recorded, you can export your test to a file for later use. In this example, we choose HTML so that we can easily share it with others. This is the recommended format to use for saving a Selenium Test.
+
 
+
[[Image:Selenium-export.png]]
+
 
+
== Converting Selenium IDE Tests ==
+
Tests recorded with Selenium IDE can be converted for inclusion in the /tests/system folder as well as integration in to the test suite.
+
 
+
We'll continue our example from above and convert the Selenium IDE test to a PHP file for inclusion in the /tests/system folder.
+
 
+
Since the test was exported in the HTML format, we can easily open it and re-play it in Selenium IDE.
+
 
+
Assuming that the test behaves as expected, we will convert it to PHP by selecting Options -> Format -> PHP - Selenium RC.
+
 
+
[[Image:Selenium-change-format.png]]
+
 
+
The test can now be be exported as PHP by selecting File -> Export Test Case As -> PHP - Selenium RC.
+
 
+
[[Image:Selenium-export-php.png]]
+
 
+
The converted Selenium IDE test now looks like:
+
<pre><?php
+
 
+
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
+
 
+
class Example extends PHPUnit_Extensions_SeleniumTestCase
+
{
+
  function setUp()
+
  {
+
    $this->setBrowser("*chrome");
+
    $this->setBrowserUrl("http://127.0.0.1/selsampledata/");
+
  }
+
 
+
  function testMyTestCase()
+
  {
+
    $this->open("/workspace/joomla-1-6-source/");
+
    $this->click("link=Using Joomla!");
+
    $this->waitForPageToLoad("30000");
+
    $this->assertEquals("Getting started", $this->getText("link=Getting started"));
+
    $this->assertTrue($this->isElementPresent("link=Extensions"));
+
    $this->click("link=The Joomla! Community");
+
    $this->waitForPageToLoad("30000");
+
    $this->assertEquals("The Joomla! Project", $this->getText("link=The Joomla! Project"));
+
    $this->click("//div[@id='breadcrumbs']/p/span/a");
+
    $this->waitForPageToLoad("30000");
+
  }
+
}
+
?></pre>
+
 
+
It is nearly ready to be added to <code>/tests/system</code>, but needs a few modifications before doing so.
+
 
+
* We need to add a comment block to the top
+
<pre>
+
<?php
+
/**
+
* @version    $Id: exmaple0001Test.php
+
* @package Joomla.SystemTest
+
* @copyright Copyright (C) 2005 - 2010 Open Source Matters, Inc. All rights reserved.
+
* @license GNU General Public License version 2 or later; see LICENSE.txt
+
* Tests creating a test
+
*/
+
</pre>
+
* We need to change <code>require_once 'PHPUnit/Extensions/SeleniumTestCase.php';</code> to <code>require_once 'SeleniumJoomlaTestCase.php';</code>
+
<pre>
+
require_once 'SeleniumJoomlaTestCase.php';
+
</pre>
+
* Change the class Selenium IDE uses to a new class, that extends SeleniumJoomlaTestCase, using the naming convention, written in CamelCase, of <code>subject + test number in the series (4 digits) + "Test"</code>.
+
** For example, our test that we are calling example would be:<code> Example + 0001 + Test</code> resulting in <code>Example0001Test</code>
+
<pre>class Example0001Test extends SeleniumJoomlaTestCase</pre>
+
* Remove the function setUp()
+
<pre>function setUp()
+
  {
+
    $this->setBrowser("*chrome");
+
    $this->setBrowserUrl("http://127.0.0.1/selsampledata/");
+
  }
+
</pre>
+
created by Selenium IDE and add <pre>$this->setUp();</pre> to our test function.
+
 
+
 
+
 
+
The converted test now looks like:
+
<pre><?php
+
/**
+
* @version    $Id: exmaple0001Test.php
+
* @package Joomla.SystemTest
+
* @copyright Copyright (C) 2005 - 2010 Open Source Matters, Inc. All rights reserved.
+
* @license GNU General Public License version 2 or later; see LICENSE.txt
+
* Tests creating a test
+
*/
+
 
+
require_once 'SeleniumJoomlaTestCase.php';
+
 
+
class Example0001Test extends SeleniumJoomlaTestCase
+
{
+
function testCreatDeleteGroup()
+
{
+
  $this->setUp();
+
        $this->open("/workspace/joomla-1-6-source/");
+
        $this->click("link=Using Joomla!");
+
        $this->waitForPageToLoad("30000");
+
        $this->assertEquals("Getting started", $this->getText("link=Getting started"));
+
        $this->assertTrue($this->isElementPresent("link=Extensions"));
+
        $this->click("link=The Joomla! Community");
+
        $this->waitForPageToLoad("30000");
+
        $this->assertEquals("The Joomla! Project", $this->getText("link=The Joomla! Project"));
+
        $this->click("//div[@id='breadcrumbs']/p/span/a");
+
        $this->waitForPageToLoad("30000");
+
        }
+
}
+
?></pre>
+
 
+
Your test can now be saved to the <code>/tests/system</code> folder within Joomla and run as a member of the test suite. See [[Running_Automated_Tests_for_Version_1.6]]
+
 
+
== Creating a Basic System Test ==
+
By leveraging the [[Selenium_Test_Case_Methods]], we can streamline automated system test with just a few lines of code.
+
 
+
For example, to test logging into and then out of the back end, we could use the following code:
+
<pre><?php
+
require_once 'SeleniumJoomlaTestCase.php';
+
 
+
class ControlPanelExample extends SeleniumJoomlaTestCase
+
{
+
function testExample()
+
{
+
$this->setUp();
+
$this->doAdminLogin();
+
$this->doAdminLogout();
+
}
+
}
+
?></pre>
+
 
+
* <code>require_once 'SeleniumJoomlaTestCase.php';?</code> allows use to utilize the methods that belong to the Selenium Joomla Test Case class
+
* <code>class ControlPanelExample extends SeleniumJoomlaTestCase</code> extends a class into our SeleniumJoomlaTestCase to implement the client/server protocol to talk to Selenium RC as well as specialized assertion methods for web testing.
+
* Each test is written as a function.
+
* <code>$this->setUp();</code> envokes our configuration from <code>tests/system/servers/configdef.php</code> see [[Running_Automated_Tests_for_Version_1.6#Create_a_Selenium_Configuration_File]]
+
* <code>$this->doAdminLogin();</code> is a Selenium Joomla Test Case method that logs into the back end.
+
* <code>$this->doAdminLogoutn();</code> is a Selenium Joomla Test Case method that logs out of the back end.
+
 
+
== Creating an Intermediate System Test ==
+
In this test, we are going to execute the test described at [[Intermediate_Selenium_Example]] and integrate it into the system tests suite in 1.6.
+
 
+
1. Open Joomla! 1.6 in FireFox and navigate to the home page.
+
 
+
2. Open Selenium IDE by going to the Tools menu and select Selenium IDE. Selenium will start recording automatically and the red record button will appear highlighted (see image below).
+
 
+
3. Verify that the Base URL displayed in Selenium, next to the record button, is that of your test site.
+
 
+
4. Perform the test by following the instructions described above (Note: there are some slight variations between the test described above and the one we performed. This is due to variation of 1.6 during the development process).
+
 
+
5. When you have completed your test, click the the red record button (see image below) to stop recording.
+
 
+
 
+
[[Image:Selenium-screen.png]]
+
 
+
 
+
6. Once recording has stopped, create a new PHP file in Eclipse (or your favorite code editor) and copy / paste the code from Selenium IDE to there. The contents of the example test is:
+
<pre><?php
+
 
+
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
+
 
+
class Example extends PHPUnit_Extensions_SeleniumTestCase
+
{
+
  function setUp()
+
  {
+
    $this->setBrowser("*chrome");
+
    $this->setBrowserUrl("http://change-this-to-the-site-you-are-testing/");
+
  }
+
 
+
  function testMyTestCase()
+
  {
+
    $this->open("/workspace/joomla-1-6-source/administrator/index.php?option=com_login");
+
    $this->type("mod-login-username", "admin");
+
    $this->click("link=Log in");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("link=User Manager");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("//li[@id='toolbar-new']/a/span");
+
    $this->waitForPageToLoad("30000");
+
    $this->type("jform_name", "My Test User");
+
    $this->type("jform_username", "TestUser");
+
    $this->type("jform_password", "password");
+
    $this->type("jform_password2", "password");
+
    $this->type("jform_email", "test@example.com");
+
    $this->click("1group_2");
+
    $this->click("link=Save & Close");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("Item successfully saved."));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->type("search", "TestUser");
+
    $this->click("//button[@type='submit']");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("TestUser"));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Log out");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("link=Go to site home page.");
+
    $this->waitForPageToLoad("30000");
+
    $this->type("modlgn_username", "TestUser");
+
    $this->type("modlgn_passwd", "password");
+
    $this->click("Submit");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("My Test User"));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Logout");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("//button[@type='submit']");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("link=Site Administrator");
+
    $this->waitForPageToLoad("30000");
+
    $this->type("mod-login-username", "admin");
+
    $this->click("link=Log in");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("//img[@alt='User Manager']");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("cb0");
+
    $this->click("link=Delete");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("1 item(s) successfully deleted."));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Log out");
+
    $this->waitForPageToLoad("30000");
+
  }
+
}
+
?></pre>
+
 
+
As noted in [[Intermediate_Selenium_Example]] Selenium does not record passwords that are entered. Additionally, we want to utilize [[Selenium_Test_Case_Methods]] to prevent coding path or login information in the test.
+
 
+
 
+
1. The first step in integrating this test into <code>tests/suite</code> is to call <code>SeleniumJoomlaTestCase.php</code> so that we can utilize the Selenium Test Case Methods. To do this, we will replace the first portion of our code:
+
<pre><?php
+
require_once 'PHPUnit/Extensions/SeleniumTestCase.php';
+
</pre>
+
 
+
with the following code
+
 
+
<pre><?php
+
require_once 'SeleniumJoomlaTestCase.php';</pre>
+
 
+
* We can now start utilizing the Selenium Joomla Test Case Methods.
+
 
+
 
+
2. We will now extend a class into our SeleniumJoomlaTestCase to implement the client/server protocol to talk to Selenium RC as well as specialized assertion methods for web testing.
+
 
+
This is done by replacing:
+
<pre>class Example extends PHPUnit_Extensions_SeleniumTestCase
+
{</pre>
+
 
+
with
+
 
+
<pre>class User0002Test extends SeleniumJoomlaTestCase
+
{</pre>
+
 
+
* The top portion of our test now looks like:
+
<pre><?php
+
require_once 'SeleniumJoomlaTestCase.php';
+
 
+
class User0002Test extends SeleniumJoomlaTestCase
+
{
+
 
+
  function testMyTestCase()
+
  {</pre>
+
 
+
* At this point, running the test would successfully launch a two browsers instances, one for Selenium Remote Control and one for test itself, but it would fail due to the lack of the password being recorded. Additionally, we want to be sure that none of the path information is hard coded in the test.
+
 
+
3. To improve our code, and achieve what we noted above, we would delete the function below:
+
 
+
<pre>function setUp()
+
  {
+
    $this->setBrowser("*chrome");
+
    $this->setBrowserUrl("http://change-this-to-the-site-you-are-testing/");
+
  }</pre>
+
and add the following to our testMyTestCase function
+
<pre> $this->setUp();</pre>
+
Additionaly, we can replace the following line of code:
+
<pre>$this->open("/workspace/joomla-1-6-source/administrator/index.php?option=com_login");</pre>
+
with
+
<pre>$this->gotoAdmin();</pre>
+
As you can see, this will prevent us from hard coding the path into the test.
+
 
+
We can now replace
+
<pre>$this->type("mod-login-username", "admin");
+
    $this->click("link=Log in");
+
    $this->waitForPageToLoad("30000");</pre>
+
with
+
<pre>$this->doAdminLogin();</pre>
+
 
+
At this point, running our test would succeed up until the point where we need to log into the administrative interface the second time. We'll fix that by replacing:
+
<pre>    $this->type("mod-login-username", "admin");
+
    $this->click("link=Log in");
+
    $this->waitForPageToLoad("30000");</pre>
+
 
+
with
+
 
+
<pre>$this->doAdminLogin();</pre>
+
 
+
Running our test should now successfully create, verify the creation of, and delete our new test user.
+
 
+
* To rely to the user what is happening, we want to add echo statements  to our code.
+
 
+
Our final test looks like:
+
<pre><?php
+
 
+
require_once 'SeleniumJoomlaTestCase.php';
+
 
+
class User0002Test extends SeleniumJoomlaTestCase
+
{
+
  function testMyTestCase()
+
  {
+
    echo("Starting testMyTestCase\n");
+
    $this->setUp();
+
    $this->gotoAdmin();
+
    $this->doAdminLogin();
+
    $this->click("link=User Manager");
+
    $this->waitForPageToLoad("30000");
+
    echo("Add new user.\n");
+
    $this->click("//li[@id='toolbar-new']/a/span");
+
    $this->waitForPageToLoad("30000");
+
    $this->type("jform_name", "My Test User");
+
    $this->type("jform_username", "TestUser");
+
    $this->type("jform_password", "password");
+
    $this->type("jform_password2", "password");
+
    $this->type("jform_email", "test@example.com");
+
    $this->click("1group_2");
+
    $this->click("link=Save & Close");
+
    $this->waitForPageToLoad("30000");
+
    echo("Verify existence of new user.\n");
+
    try {
+
        $this->assertTrue($this->isTextPresent("Item successfully saved."));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->type("search", "TestUser");
+
    $this->click("//button[@type='submit']");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("TestUser"));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Log out");
+
    $this->waitForPageToLoad("30000");
+
    echo("Go to home page.\n");   
+
    $this->click("link=Go to site home page.");
+
    $this->waitForPageToLoad("30000");
+
    echo("Log in as TestUser.\n");   
+
    $this->type("modlgn_username", "TestUser");
+
    $this->type("modlgn_passwd", "password");
+
    $this->click("Submit");
+
    $this->waitForPageToLoad("30000");
+
    echo("Verify existence of new user.\n");   
+
    try {
+
        $this->assertTrue($this->isTextPresent("My Test User"));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Logout");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("//button[@type='submit']");
+
    $this->waitForPageToLoad("30000");   
+
    $this->click("link=Site Administrator");
+
    $this->waitForPageToLoad("30000");
+
$this->doAdminLogin();
+
    $this->click("//img[@alt='User Manager']");
+
    $this->waitForPageToLoad("30000");
+
    $this->click("cb0");
+
    echo("Delete new user.\n");   
+
    $this->click("link=Delete");
+
    $this->waitForPageToLoad("30000");
+
    try {
+
        $this->assertTrue($this->isTextPresent("1 item(s) successfully deleted."));
+
    } catch (PHPUnit_Framework_AssertionFailedError $e) {
+
        array_push($this->verificationErrors, $e->toString());
+
    }
+
    $this->click("link=Log out");
+
    $this->waitForPageToLoad("30000");
+
  }
+
}
+
?>
+
</pre>
+
 
+
== Methods for Identifying Elements ==
+
 
+
== See Also ==
+
* [[System_Testing]]
+
* [[Running_Automated_Tests_for_Version_1.6]]
+
* [[System_Tests_Available_in_Version_1.6]]
+
 
+
==== Tips & Tricks====
+
* Undo all changes your test may make: All tests should make all efforts to leave the sample data and test site content as it was before the test started. In some cases, a test may break due to a change in the test site's content. For example, if you add a test user, delete the test user at the end of your test or series of tests that require that user.
+
 
+
* Maximize flexibility through test methods: In many tests, similar or identical actions are repeated. In these cases, these actions can be reproduced as a granular test method added to /tests/system/SeleniumJoomlaTestCases.php. Good examples include:
+
**doAdminLogin
+
**gotoAdmin
+
**gotoSite
+
(see [[Selenium_Test_Case_Methods|Selenium Test Case Methods]])
+
 
+
==== Running System Tests ====
+
 
+
=== Joomla Core Selenium testcases ===
+
The testing repository for the Selenium test suite for version 1.5 can be found here:
+
 
+
* [http://joomlacode.org/svn/joomla/development/releases/1.5/tests/system/ http://joomlacode.org/svn/joomla/development/releases/1.5/tests/system/]
+
 
+
As of February 2010, the unit and system tests for version1.6 have been incorporated into the SVN trunk under a folder called "tests". More information about running the version 1.6 tests is available in the article [[Running Automated Tests for Version 1.6]].
+
 
+
 
+
Note: If a login mask comes up, use </br>
+
:User: anonymous </br>
+
:Password: <empty>
+
 
+
== How to Get it Running ==
+
Instructions for running the system tests for version 1.6 can be found here: [[ Running_Automated_Tests_for_Version_1.6#System_Tests | Running Automated Tests for Version 1.6]].
+
 
+
== Troubleshooting ==
+
 
+
  
== Frequently Asked Questions ==
+
System tests test that the application works correctly from the user point of view. For example, a system test can test something simple, for example, that you can create a new menu item for a single article and show the menu item on the site. Or a system test can test something more detailed, for example that the parameters for a module all work as expected.
  
==See also==
+
Designing and creating system tests requires that you know how to use the application. It does not require that you understand how the program is written. Application knowledge is more important than programming knowledge, so system tests can be designed and written by people with less technical knowledge of PHP or the Joomla! framework.
For writing a more advanced Selenium testcase see [http://docs.joomla.org/Intermediate_Selenium_Example here].
+
  
[http://seleniumhq.org/docs/ Selenium documentation].
+
== Writing System Tests with Selenium ==
 +
For detailed documentation on writing system tests, please see the article [[Writing System Tests for Version 1.6]].
  
 
[[Category:Development]]
 
[[Category:Development]]
Line 512: Line 32:
 
[[Category:Bug Squad]]
 
[[Category:Bug Squad]]
 
[[Category:Automated Testing]]
 
[[Category:Automated Testing]]
<math>Insert formula here</math>
+
[[Category:Working Groups]]
--[[User:Betweenbrain|Betweenbrain]] 00:05, 31 May 2010 (UTC)
+

Revision as of 15:47, 29 September 2010

Contents

News and Updates

  • 2009 07 27 Document created.
  • 2010 05 24 Added Tips & Tricks section.
  • 2010 05 28 Started content overhaul and integration of related docs for more logical flow.
  • 2010 06 07 Removed the sections on Selenium IDE and merged those with Writing System Tests for Version 1.6.

Overview

System testing is an essential part of a good Quality Control program. For a good general discussion of system testing, visit the Wikipedia article.

System Testing in Joomla!

System testing in Joomla! uses an open-source program called Selenium. Selenium allows you to create a test script that can be run automatically. The test script actually runs Joomla! in a browser and does the same keystrokes and mouse clicks that a user would do. It can also test for conditions such as the value of text or the presence of HTML elements and report when a test fails or when there is an error in the program.

The benefits of system testing are:

  • System tests help you test that the application is working correctly from the point of view of a user.
  • System tests help clearly specify how the application should behave.
  • System tests help you test that changes made to one part of the application haven't created a bug somewhere else.
  • System tests can be run automatically (for example, each night) so that testing is done as the application is being developed.

System Testing versus Unit Testing

System Testing and Unit Testing are complementary test strategies. In Joomla!, Unit Testing is mainly used to test the framework classes (for example, the classes in libraries/joomla). Unit tests test that an individual method (also known as function) in a class does what it is supposed to do. For example, a unit test is used to test that the methods in the JString class work as expected. Unit testing provides confidence that the framework classes work as expected and allows you to refactor these classes (improve the code without changing the functionality) and still have confidence that they still work correctly.

System tests test that the application works correctly from the user point of view. For example, a system test can test something simple, for example, that you can create a new menu item for a single article and show the menu item on the site. Or a system test can test something more detailed, for example that the parameters for a module all work as expected.

Designing and creating system tests requires that you know how to use the application. It does not require that you understand how the program is written. Application knowledge is more important than programming knowledge, so system tests can be designed and written by people with less technical knowledge of PHP or the Joomla! framework.

Writing System Tests with Selenium

For detailed documentation on writing system tests, please see the article Writing System Tests for Version 1.6.