Testing Joomla Extensions with Codeception

Note: this page is under construction

What is Codeception?

Codeception PHP Testing Framework is designed to work just out of the box. This means its installation requires minimal steps and no external dependencies preinstalled (except PHP, of course). Only one configuration step should be taken and you are ready to test your web application from an eye of actual user.

See the following video of codeception in action testing Joomla:

Codeception provides possibilities to do Unit Testing or to Test APIs, however this page is focused in System Testing.

Why Use Codeception?

Why use Codeception when we already have Selenium IDE?

Codeception works great with Selenium. But with Codeception you can write your tests in PHP. The main reason is: Selenium IDE tests are tightly bound to XPath locators. If you ever change anything in layout tests will fall. Codeception locators are more stable. You can use names, labels, button names and CSS to match elements on page.

Features

• Selenium WebDriver integration
• Elements matched by name, CSS, XPath
• PageObjects and StepObjects included
• Powered by PHPUnit
• Data Cleanup
• Parallel Execution of tests
• BDD-style readable tests

Architecture of a test

The testing framework is built on the three layered testing architecture

Scenarios (Cept Class)

Cept or Test, it is the main class which contains steps of execution for the tests, Cept Class will call Step Class functions which will indirectly make use of page objects to perform operations

Example:

$I = new AcceptanceTester($scenario);
$I->wantTo('Install Joomla CMS');
$I->amOnPage('http://localhost/installation/index.php');
// I Wait for the text Main Configuration, meaning that the page is loaded
$I->waitForText('Main Configuration', 10, 'h3');
$I->click(JoomlaInstallationConfigurationPage::$elements['Language Selector']);
$I->click(JoomlaInstallationConfigurationPage::$elements['English (United Kingdom)']);
$I->fillField('Site Name', 'weblinks');
$I->fillField('Description', 'Site for testing Weblinks component');
$I->fillField('Admin Email', $cfg['Admin email']);
$I->fillField('Admin Username', $cfg['username']);
$I->fillField('Admin Password', $cfg['password']);
$I->fillField('Confirm Admin Password', $cfg['password']);
$I->click('Next');
...

Page Objects

Each page on your application has a class which we call page objects, each page class will have information about the URL, fields, Text present on the page. We use the best possible selector for any field on the page.

Example

class JoomlaAdministratorLoginManagerPage
{
	/**
	 * @var string Url of the page
	 */
	public static $URL = '/administrator/index.php';

	/**
	 * Array of Page elements indexed by descriptive name or label
	 *
	 * @var array
	 */
	public static $elements = array(
		'username' => "#mod-login-username",
		'password' => "#mod-login-password"
	);
}

Naming convention for Page Objects

When this document is created Codeception 2.0 was not having a PSR-4 compatible autoloader. Therefore we are creating a convention so with the future Codeception 2.1 with PSR-4 compatible autoloader (https://github.com/Codeception/Codeception/pull/1228) we are creating a naming convention for Pages so it will cost us a small effort to move to autoloaded classes in the future.

The naming convention works as follows:

APPLICATION (for example Joomla, Patchtester, Weblinks or your Extension name)
+ CLIENT (in Joomla we use it for defining if is Installation, Administrator or Frontend
+ CLIENT PART (In Joomla we use it following the menu are were it belongs: Components, Menus, System,...)
+ PAGE NAME (The Page name) 
+ Page

Examples:

JoomlaAdministratorLoginPage
JoomlaAdministratorSystemGlobalconfigurationPage
JoomlaInstallationConfigurationPage

And they are located with a folder estructure similar to:

Folder estructure for Pages in System Tests

Remind that in Codeception 2.0 you need to specify each folder in the acceptance/_bootstrap.php file:

<?php
// Here you can initialize variables that will be available to your tests
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/installation');
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/administrator');

Documentation

More info: http://codeception.com/docs/07-AdvancedUsage#PageObjects

Step Objects

Step Objects are classes which contains steps which are to be executed on the application to perform certain operations, step objects make use of page objects to perform these actions.

Example:

class JoomlaAdministratorLoginSteps extends \AcceptanceTester
{
	/**
	 * Function to execute an Admin Login for Joomla 3
	 *
	 * @return void
	 */
	public function doAdministratorLogin($user, $password)
	{
		$I = $this;
		$I->amOnPage(\JoomlaAdministratorLoginPage::$URL);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['username'], $user);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['password'], $password);
		$I->click('Log in');
		$I->see('Category Manager');
	}
}

More info: http://codeception.com/docs/07-AdvancedUsage#StepObjects

The Joomla Browser Module

The Joomla Browser is a Codeception Module that incorporates general functions to your tests like: - do administrator login - install joomla - install extension - ...

Find details about it at: https://github.com/joomla-projects/joomla-browser#table-of-contents

Getting started from scratch: what you need to download and install

To get the test environment running you need to have several components installed.

Very important: The downloaded GIT directory must be inside your htdocs folder of your local web-server. Otherwise the test will not work! The files from the 'weblinks' folder must be loadable via your local web-server with your browser.

local webserver

If you do not already have a local test server running you may want to check out: https://www.mamp.info/ https://www.apachefriends.org/ http://www.ampps.com/

If you are not familiar with a local web-server, learn that first before you advance futher.

Wget on Mac

For Mac Users Homebrew (package manager for OS X) is a very easy way to get wget running. Download Homebrew at: http://brew.sh/ After installation run the following command in your terminal

brew install wget

Java Runtime Environment (JRE)

The JRE has to be installed, so the tests will be executable. You find the recent versions at the Oracle Website. In end of May 2015 this is: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html later on that link could be obsolete and should be changed if there is a newer version out.

Newer Mac OS Versions are not shipped any more with the JRE, so they have to download that too.

Firefox

Selenium is testing in the browser Firefox. The selenium version is not that quick with the updates, as the Firefox browser, if the test is not executable, then it can be that you manually have to install an older version of Firefox. Sadly that can be a try and error thing. To get older versions of Firefox: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/ here you will find many folders. Just try the first one that came before the Firefox version that you have been using (and that did not work). In the version folders you will find folders for all the different operating systems.

‎

configure test system variables

Screenshot of file structure

In your the folder that you installed, downloaded, pulled (git) the project, you have to rename the "acceptance.suite.dist.yml" file to "acceptance.suite.yml" and configure the parameters to your system variables.

   config:
        JoomlaBrowser:
            url: 'http://localhost/tests/joomla-cms3'
            # the url that points to the joomla installation at /tests/system/joomla-cms
            browser: 'firefox'
            window_size: 1024x768
    [...]

test:acceptance

Once your machine is able to run tests (see The Joomla Browser Module), you are able to start working.

Check the contents of the /tests/ folder in weblinks and try to understand the structure of the tests. For each function you have a directory and sometimes a configuration file, both will have the corresponding name.

• FunctionName.yml - these files contain configurations for the function folders
• FunctionName - contains the FunctionName test
• joomla-cms3 - contains the latest Joomla! installation

After that check the tasks are at: https://github.com/joomla-extensions/weblinks/issues/47

The principles for writing a test: Readability and Performance

In Joomla we give priority to Readability over Performance, even when performance is a must.

To explain that let me show an example: Let's say that we want to fill a field Username in a login form.

<form action="administrator/index.php" method="post" id="form-login">
  <label for="username_field">Username</label>
  <input name="username_field" tabindex="1" id="username_field" type="text" size="15" />
  ...

the most performant way to fill that form field would be:

$I->fillField('#username_field', 'Mark');

And usually we would move that to a page object element ending in:

$I->fillField(\LoginPage::$username, 'Mark');

However Codeception is a smart framework and it adds some magic to it's methods (see: https://github.com/Codeception/Codeception/blob/2.0/src/Codeception/Module/WebDriver.php#L511-L513). It allows me to write more readable tests (keeping performance hight).

If we do this:

$I->fillField('Username', 'Mark');

Codeception will be smart and will search first in the page if there is any "LABEL" called Username. That is a fast operation, so performance will be affected just a little. And instead we have written less code than in previous examples.

Note: sometimes this tricks are not always possible, labels are not always there or there are some elements hidden that shows and hides depending on a Javascript. All these special cases would require a Page element.

Still there is a definitive method of using locators that made them readable and performant called: strict Locators see Testing_Joomla_Extensions_with_Codeception#Writting_reliable_locators.

With them we do:

$I->fillField(['id' => 'username_field'], 'Mark');

In conclusion we have more readable tests and less code to maintain while keeping performance. That is a win!

Setting Up Codeception

setting up the repository

clone the directory

1. Fork com_localise
3. Git clone git@github.com:*your-github-profile-name*/com_localise.git

Testing with Codeception

There are two ways to get and run codeception, via PHAR or via Composer. Using Composer is recommended (because we are using in some tests the Codeception 2.1 development version not available in the composer.phar).

• Using Composer

   You need to have Composer in your system, if not download it from here: https://getcomposer.org/

   Step 1: composer update

   Step 2: run Codeception by doing: php vendor/bin/codecept build

• Codeception.phar

   Linux Machine  : wget http://codeception.com/codecept.phar

   Windows Machine: Download from codecept.phar

   Step1 : php ./codecept.phar build

Running the Test

Run Selenium server

# Download
curl -O http://selenium-release.storage.googleapis.com/2.45/selenium-server-standalone-2.45.0.jar

# And start the Selenium Server
java -Xms40m -Xmx256m -jar /Applications/XAMPP/xamppfiles/htdocs/selenium/selenium-server-standalone-2.45.0.jar

Rename tests/acceptance.suite.dist.yml to tests/acceptance.suite.yml

Step 1: Modify the configuration at tests/acceptance.suite.yml to fit your server details.

Step 2: php codecept.phar run

with --steps to see a step-by-step report on the performed actions.

php vendor/bin/codecept run --steps

with --html. This command will run all tests for all suites, displaying the steps, and building HTML and XML reports. Reports will be store in tests/_output/ directory.

php vendor/bin/codecept run --html

To activate the debug mode in Codeception just add the parameter --debug when you run the tests:

php vendor/bin/codecept run acceptance --debug

Best Practices at Testing

Writting reliable locators

Writing good locators can be tricky. The Mozilla team has written an excellent guide titled Writing reliable locators for Selenium and WebDriver tests.

If the locator is an array, it should have a single element, with the key signifying the locator type (`id`, `name`, `css`, `xpath`, `link`, or `class`) and the value being the locator itself. This is called a "strict" locator. Examples:

* ['id' => 'foo'] matches `<div id="foo">`
* ['name' => 'foo'] matches `<div name="foo">`
* ['css' => 'input[type=input][value=foo]'] matches `<input type="input" value="foo">`
* ['xpath' => "//input[@type='submit'][contains(@value, 'foo')]"] matches `<input type="submit" value="foobar">`
* ['link' => 'Click here'] matches `<a href="google.com">Click here</a>`
* ['class' => 'foo'] matches `<div class="foo">`

How to use URLs

Codeception documentation suggests to use URL's as properties of a Page Object:

$I->amOnPage(JoomlaInstallationConfigurationPage::$URL);

Instead of hardcoding the url in the test:

$I->amOnPage('/administrator/index.php?option=com_banners&view...');

The pros of this approach is: - first way is more readable for non developers. - it helps to make tests more maintenable in the case of a url change in the system

However in joomla a change in an url is rare, and if it happens changing it will just be a Find&Replace task. I Joomla test we want to have the benefit of both, the url and a representative name that describes the page where we are going. That will make our tests more readable. Thus, we will write the urls the following way:

$I->amGoingTo('go to Create New Banner page at Joomla Administrator');
$I->amOnPage('/administrator/index.php?option=com_banners&view...');

Working with Hidden Elements

Selenium can't interact with elements that are not visible:

<a href="http://www.joomla.org" style="display: none;" id="joomla_site">go to joomla</a>

In this case you can't do:

$I->click('go to joomla');

Usually this happens when you need to wait for a Javascript animation that end up showing that element, so in this case you will need to do this:

$I->waitForElementVisible('#joomla_site');
$I->click('go to joomla');

When do not use contains

We do not recommend to use the function contains on XPath because is not specific and can create false positives at tests. See: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS/pull/99#issuecomment-69004191

Always add Strict Locators to your tests

Performance in tests is very important. You can reduce dramatically the time spent on a test with just adding a few extra locators. For example, the following code pertains to a test:

// see($text, $selector = null)
$I->see('Your article has been published');

In the previous code we are not facilitating the "Selector" so Codeception will have to search for the element in all the HTML. Doing this instead:

$I->see('Your article has been published', ['id' => 'messages']);

the tests went from: Time: 1.17 minutes to: Time: 49.18 seconds

You can see this video of a real example: http://www.youtube.com/watch?v=0JCv0BR2yZY

Losing 30 seconds in every Assertion can delay your tests dramatically. Therefore, remind always to add xPath locators to all your Assertions.

Using locators in $I->see

Do not use see without locator:

// Wrong:
$I->see('Your article has been published');

// Right:
$I->see('Your article has been published', ['id' => 'messages']);

Using locators in $I->waitForText

Do not use waitForText without locator:

// Wrong:
$I->waitForText('Finalisation');

// Right:
$I->waitForText('Finalisation', 10, ['css' => 'h3']);

Incase of an element has multiple css classes, When Using CSS Locators make sure you use more specific class as locator:

// Wrong:
$I->waitForText('Item Succesfully Created', 10, ['class' => 'alert']);

// Right:
$I->waitForText('Item Succesfully Created', 10, ['class' => 'alert-success']);

A page could have mutiple elements which can have alert css class associated, hence being more specific helps.

How much time should I wait for an Element or Text?

You may have noticed that the second parameter in the I waitForText is an integer that represents how much "time to wait" in seconds:

$I->waitForText('a text to wait for', 10, ['...' => '...']);

But, how much time is the right time to wait for a text?

PHP maximum execution time before returning a Fatal Error for timeout is set by default to 30 seconds. We suggest to set it for 10 seconds. A waitForText over 10 seconds will definitely mean that something is not performing well.

Quote and Double Quotes in Compex XPaths

Sometimes you will have to write XPaths that looks like:

//button[@onclick='Joomla.submitbutton("config.save.application.apply")']

When moved to a Page Object you will probably will face this situation:

Typical issue when you write a long XPath in a Page object

To avoid it remind to escape the quotes:

// Wrong
public static $elements = array(
   'Save' => "//button[@onclick="Joomla.submitbutton('config.save.application.apply')"]"
);
// Right
public static $elements = array(
   'Save' => "//button[@onclick=\"Joomla.submitbutton('config.save.application.apply')\"]"
);

Generating txt/html scenarios

Codeception allows you to generate user-friendly text scenarios from scenario-driven from Cest and Cept php test (see http://codeception.com/docs/reference/Commands#GenerateScenarios).

Don't use double quotes, use single quotes instead

Do not use double quotes (") for surrounding strings:

$I->waitForText("Congratulations! Joomla! is now installed.", 30);

If you do, when you generate the scenarios you will get this bad result:

I wait for text ""Congratulations! Joomla! is now installed.""," 30"

If you use single quotes instead:

$I->waitForText('Congratulations! Joomla! is now installed.', 30);

You will get the right expected result:

I wait for text 'Congratulations! Joomla! is now installed.'," 30"

The reason of not being able to use double quotes is due to a bug in Codeception humanize funtions, see issue: https://github.com/Codeception/Codeception/issues/1847

Do not leave spaces between parameters in Cepts

See the two lines:

// Wrong
$I->waitForText('Main Configuration', 10, 'h3');
// Good
$I->waitForText('Main Configuration',10,'h3');

If you do, when you generate the scenarios you will get this bad result:

I wait for text 'Main Configuration'," 10"," 'h3'

The reason that you are asked to not add spaces between parameters is due to a bug in Codeception humanize funtions, see issue: https://github.com/Codeception/Codeception/issues/1847