Joomla! Documentation™

Note: this page is under construction

What is Codeception?

Codeception PHP Testing Framework is designed to work just out of the box. This means its installation requires minimal steps and no external dependencies preinstalled (except PHP, of course). Only one configuration step should be taken and you are ready to test your web application from an eye of actual user.

See the following video of codeception in action testing Joomla:

Types of tests

Codeception provides possibilities to do Unit Testing or to Test APIs, however this page is focused in System Testing.

- System testing uses a production-like environment to verify the system as a whole, including user interfaces, databases, web servers, logging, etc.

- A unit test verifies a single behavior of a single unit – this can be a function, a method, or a class. A logical unit is something that can be reasoned about in isolation. The unit must be completely under your control. The test should limit the number of other units it touches on, and should not touch any part outside of your own system. As such, the test should only fail for one reason.

- Integration testing means we’re verifying that that different units are working together. It focuses on the interfaces between components. Some of these components may be from third parties. When they break, they signal that, although individual units may behave correctly, something changed in the API that caused failure in another place. Often, integration tests are slower than unit tests.

As mentioned at the beginning of this text, the current page is focused on System Tests and more specifically using Webdriver. The reason is that we came to the conclusion that this type of tests combined with the already existing Unit tests in Joomla are the most successful way to test our current code.

Why Use Codeception?

Why use Codeception when we already have Selenium IDE?

Codeception works great with Selenium. But with Codeception you can write your tests in PHP. The main reason is: Selenium IDE tests are tightly bound to XPath locators. If you ever change anything in layout tests will fall. Codeception locators are more stable. You can use names, labels, button names and CSS to match elements on page.

Features

• Selenium WebDriver integration
• Elements matched by name, CSS, XPath
• PageObjects and StepObjects included
• Powered by PHPUnit
• Data Cleanup
• Parallel Execution of tests
• BDD-style readable tests

Architecture of a test

The testing framework is built on the three layered testing architecture

Scenarios (Cept Class)

Cept or Test, it is the main class which contains steps of execution for the tests, Cept Class will call Step Class functions which will indirectly make use of page objects to perform operations

Example:

$I = new AcceptanceTester($scenario);
$I->wantTo('Install Joomla CMS');
$I->amOnPage('http://localhost/installation/index.php');
// I Wait for the text Main Configuration, meaning that the page is loaded
$I->waitForText('Main Configuration', 10, 'h3');
$I->click(JoomlaInstallationConfigurationPage::$elements['Language Selector']);
$I->click(JoomlaInstallationConfigurationPage::$elements['English (United Kingdom)']);
$I->fillField('Site Name', 'weblinks');
$I->fillField('Description', 'Site for testing Weblinks component');
$I->fillField('Admin Email', $cfg['Admin email']);
$I->fillField('Admin Username', $cfg['username']);
$I->fillField('Admin Password', $cfg['password']);
$I->fillField('Confirm Admin Password', $cfg['password']);
$I->click('Next');
...

Page Objects

Each page on your application has a class which we call page objects, each page class will have information about the URL, fields, Text present on the page. We use the best possible selector for any field on the page.

Example

class JoomlaAdministratorLoginManagerPage
{
	/**
	 * @var string Url of the page
	 */
	public static $URL = '/administrator/index.php';

	/**
	 * Array of Page elements indexed by descriptive name or label
	 *
	 * @var array
	 */
	public static $elements = array(
		'username' => "#mod-login-username",
		'password' => "#mod-login-password"
	);
}

Naming convention for Page Objects

When this document is created Codeception 2.0 was not having a PSR-4 compatible autoloader. Therefore we are creating a convention so with the future Codeception 2.1 with PSR-4 compatible autoloader (https://github.com/Codeception/Codeception/pull/1228) we are creating a naming convention for Pages so it will cost us a small effort to move to autoloaded classes in the future.

The naming convention works as follows:

APPLICATION (for example Joomla, Patchtester, Weblinks or your Extension name)
+ CLIENT (in Joomla we use it for defining if is Installation, Administrator or Frontend
+ CLIENT PART (In Joomla we use it following the menu are were it belongs: Components, Menus, System,...)
+ PAGE NAME (The Page name) 
+ Page

Examples:

JoomlaAdministratorLoginPage
JoomlaAdministratorSystemGlobalconfigurationPage
JoomlaInstallationConfigurationPage

And they are located with a folder estructure similar to:

Folder estructure for Pages in System Tests

Remind that in Codeception 2.0 you need to specify each folder in the acceptance/_bootstrap.php file:

<?php
// Here you can initialize variables that will be available to your tests
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/installation');
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/administrator');

Documentation

More info: http://codeception.com/docs/07-AdvancedUsage#PageObjects

Step Objects

Step Objects are classes which contains steps which are to be executed on the application to perform certain operations, step objects make use of page objects to perform these actions.

Example:

class JoomlaAdministratorLoginSteps extends \AcceptanceTester
{
	/**
	 * Function to execute an Admin Login for Joomla 3
	 *
	 * @return void
	 */
	public function doAdministratorLogin($user, $password)
	{
		$I = $this;
		$I->amOnPage(\JoomlaAdministratorLoginPage::$URL);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['username'], $user);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['password'], $password);
		$I->click('Log in');
		$I->see('Category Manager');
	}
}

More info: http://codeception.com/docs/07-AdvancedUsage#StepObjects

The Joomla Browser Module

The Joomla Browser is a Codeception Module that incorporates general functions to your tests like: - do administrator login - install joomla - install extension - ...

Find details about it at: https://github.com/joomla-projects/joomla-browser#table-of-contents

Getting started from scratch: what you need to download and install

To get the test environment running you need to have several components installed.

Important note: no matter if you are executing Joomla CMS Tests, Weblinks, or tests in other Joomla extensions, in any case you are asked to fork the repository into your local machine under your htdocs folder of your local web-server. The reason is that, the automated scripts that launches the tests use the repository files for the tests.

local webserver

If you do not already have a local test server running you may want to check out: https://www.mamp.info/ https://www.apachefriends.org/ http://www.ampps.com/

If you are not familiar with setting up a local web-server, learn that first before you advance futher.

Java Runtime Environment (JRE)

The JRE has to be installed, so the tests will be executable. You find the recent versions at the Oracle Website. In end of May 2015 this is: http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html later on that link could be obsolete and should be changed if there is a newer version out.

Newer Mac OS Versions are not shipped any more with the JRE, so they have to download that too.

Firefox

Selenium is testing in the browser Firefox. The selenium version is not that quick with the updates, as the Firefox browser, if the test is not executable, then it can be that you manually have to install an older version of Firefox. Sadly that can be a try and error thing. To get older versions of Firefox: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/ here you will find many folders. Just try the first one that came before the Firefox version that you have been using (and that did not work). In the version folders you will find folders for all the different operating systems.

‎

configure test system variables

Screenshot of file structure

In your the folder that you installed, downloaded, pulled (git) the project, you have to rename the "acceptance.suite.dist.yml" file to "acceptance.suite.yml" and configure the parameters to your system variables.

   config:
        JoomlaBrowser:
            url: 'http://localhost/tests/joomla-cms3'
            # the url that points to the joomla installation at /tests/system/joomla-cms
            browser: 'firefox'
            window_size: 1024x768
    [...]

test:acceptance

Once your machine is able to run tests (see The Joomla Browser Module), you are able to start working.

Check the contents of the /tests/ folder in weblinks and try to understand the structure of the tests. For each function you have a directory and sometimes a configuration file, both will have the corresponding name.

• FunctionName.yml - these files contain configurations for the function folders
• FunctionName - contains the FunctionName test
• joomla-cms3 - contains the latest Joomla! installation

After that check the tasks are at: https://github.com/joomla-extensions/weblinks/issues/47

The principles for writing a test: Readability and Performance

In Joomla we give priority to Readability over Performance, even when performance is a must.

To explain that let me show an example: Let's say that we want to fill a field Username in a login form.

<form action="administrator/index.php" method="post" id="form-login">
  <label for="username_field">Username</label>
  <input name="username_field" tabindex="1" id="username_field" type="text" size="15" />
  ...

the most performant way to fill that form field would be:

$I->fillField('#username_field', 'Mark');

And usually we would move that to a page object element ending in:

$I->fillField(\LoginPage::$username, 'Mark');

However Codeception is a smart framework and it adds some magic to it's methods (see: https://github.com/Codeception/Codeception/blob/2.0/src/Codeception/Module/WebDriver.php#L511-L513). It allows me to write more readable tests (keeping performance hight).

If we do this:

$I->fillField('Username', 'Mark');

Codeception will be smart and will search first in the page if there is any "LABEL" called Username. That is a fast operation, so performance will be affected just a little. And instead we have written less code than in previous examples.

Note: sometimes this tricks are not always possible, labels are not always there or there are some elements hidden that shows and hides depending on a Javascript. All these special cases would require a Page element.

Still there is a definitive method of using locators that made them readable and performant called: strict Locators see Testing_Joomla_Extensions_with_Codeception#Writting_reliable_locators.

With them we do:

$I->fillField(['id' => 'username_field'], 'Mark');

In conclusion we have more readable tests and less code to maintain while keeping performance. That is a win!

Setting Up Codeception

setting up the repository

clone the directory

1. Fork com_localise
3. Git clone git@github.com:*your-github-profile-name*/com_localise.git

Testing with Codeception

There are two ways to get and run codeception, via PHAR or via Composer. Using Composer is recommended (because we are using in some tests the Codeception 2.1 development version not available in the composer.phar).

• Using Composer

   You need to have Composer in your system, if not download it from here: https://getcomposer.org/

   Step 1: composer update

   Step 2: run Codeception by doing: php vendor/bin/codecept build

• Codeception.phar

   Linux Machine  : wget http://codeception.com/codecept.phar

   Windows Machine: Download from codecept.phar

   Step1 : php ./codecept.phar build

Running the Test

Run Selenium server

# Download
curl -O http://selenium-release.storage.googleapis.com/2.45/selenium-server-standalone-2.45.0.jar

# And start the Selenium Server
java -Xms40m -Xmx256m -jar /Applications/XAMPP/xamppfiles/htdocs/selenium/selenium-server-standalone-2.45.0.jar

Rename tests/acceptance.suite.dist.yml to tests/acceptance.suite.yml

Step 1: Modify the configuration at tests/acceptance.suite.yml to fit your server details.

Step 2: php codecept.phar run

with --steps to see a step-by-step report on the performed actions.

php vendor/bin/codecept run --steps

with --html. This command will run all tests for all suites, displaying the steps, and building HTML and XML reports. Reports will be store in tests/_output/ directory.

php vendor/bin/codecept run --html

To activate the debug mode in Codeception just add the parameter --debug when you run the tests:

php vendor/bin/codecept run acceptance --debug

Best Practices at Testing

Writting reliable locators

Writing good locators can be tricky. The Mozilla team has written an excellent guide titled Writing reliable locators for Selenium and WebDriver tests.

If the locator is an array, it should have a single element, with the key signifying the locator type (`id`, `name`, `css`, `xpath`, `link`, or `class`) and the value being the locator itself. This is called a "strict" locator. Examples:

* ['id' => 'foo'] matches `<div id="foo">`
* ['name' => 'foo'] matches `<div name="foo">`
* ['css' => 'input[type=input][value=foo]'] matches `<input type="input" value="foo">`
* ['xpath' => "//input[@type='submit'][contains(@value, 'foo')]"] matches `<input type="submit" value="foobar">`
* ['link' => 'Click here'] matches `<a href="google.com">Click here</a>`
* ['class' => 'foo'] matches `<div class="foo">`

How to use URLs

Codeception documentation suggests to use URL's as properties of a Page Object:

$I->amOnPage(JoomlaInstallationConfigurationPage::$URL);

Instead of hardcoding the url in the test:

$I->amOnPage('/administrator/index.php?option=com_banners&view...');

The pros of this approach is: - first way is more readable for non developers. - it helps to make tests more maintenable in the case of a url change in the system

However in joomla a change in an url is rare, and if it happens changing it will just be a Find&Replace task. I Joomla test we want to have the benefit of both, the url and a representative name that describes the page where we are going. That will make our tests more readable. Thus, we will write the urls the following way:

$I->amGoingTo('go to Create New Banner page at Joomla Administrator');
$I->amOnPage('/administrator/index.php?option=com_banners&view...');

Working with Hidden Elements

Selenium can't interact with elements that are not visible:

<a href="http://www.joomla.org" style="display: none;" id="joomla_site">go to joomla</a>

In this case you can't do:

$I->click('go to joomla');

Usually this happens when you need to wait for a Javascript animation that end up showing that element, so in this case you will need to do this:

$I->waitForElementVisible('#joomla_site');
$I->click('go to joomla');

When do not use contains

We do not recommend to use the function contains on XPath because is not specific and can create false positives at tests. See: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS/pull/99#issuecomment-69004191

Always add Strict Locators to your tests

Performance in tests is very important. You can reduce dramatically the time spent on a test with just adding a few extra locators. For example, the following code pertains to a test:

// see($text, $selector = null)
$I->see('Your article has been published');

In the previous code we are not facilitating the "Selector" so Codeception will have to search for the element in all the HTML. Doing this instead:

$I->see('Your article has been published', ['id' => 'messages']);

the tests went from: Time: 1.17 minutes to: Time: 49.18 seconds

You can see this video of a real example: http://www.youtube.com/watch?v=0JCv0BR2yZY

Losing 30 seconds in every Assertion can delay your tests dramatically. Therefore, remind always to add xPath locators to all your Assertions.

Allowed locator mechanisms are:

- ['id' => 'xxx']
- ['name' => 'xxx']
- ['css' => 'xxx']
- ['xpath' => 'xxx']
- ['link' => 'xxx']
- ['class' => 'xxx']

Using locators in $I->see

Do not use see without locator:

// Wrong:
$I->see('Your article has been published');

// Right:
$I->see('Your article has been published', ['id' => 'messages']);

Using locators in $I->waitForText

Do not use waitForText without locator:

// Wrong:
$I->waitForText('Finalisation');

// Right:
$I->waitForText('Finalisation', 10, ['css' => 'h3']);

Incase of an element has multiple css classes, When Using CSS Locators make sure you use more specific class as locator:

// Wrong:
$I->waitForText('Item Succesfully Created', 10, ['class' => 'alert']);

// Right:
$I->waitForText('Item Succesfully Created', 10, ['class' => 'alert-success']);

A page could have mutiple elements which can have alert css class associated, hence being more specific helps.

How much time should I wait for an Element or Text?

You may have noticed that the second parameter in the I waitForText is an integer that represents how much "time to wait" in seconds:

$I->waitForText('a text to wait for','60',['...' => '...']);

But, how much time is the right time to wait for a text?

PHP maximum execution time before returning a Fatal Error for timeout is set by default to 30 seconds. We suggest to set it for 60 seconds, because running the tests in slow computers can easily get to that time in some tasks.

If a waitForText command takes over 10 seconds, it will much probably mean that something is not performing well. Maybe a non Strict Locator is being used and Selenium takes to much time to find the element. However we have even identified cases were some steps take longer than 30 seconds in very slow computers, specifically installing joomla, see https://github.com/joomla-projects/joomla-browser/commit/b70f195140e599649cf98f051d7a31388c4f1632. For this cases we can't do much, other than recommend to add more memory to the local web server.

Quote and Double Quotes in Complex XPaths

Sometimes you will have to write XPaths that looks like:

//button[@onclick='Joomla.submitbutton("config.save.application.apply")']

When moved to a Page Object you will probably will face this situation:

Typical issue when you write a long XPath in a Page object

To avoid it remind to escape the quotes:

// Wrong
public static $elements = array(
   'Save' => "//button[@onclick="Joomla.submitbutton('config.save.application.apply')"]"
);
// Right
public static $elements = array(
   'Save' => "//button[@onclick=\"Joomla.submitbutton('config.save.application.apply')\"]"
);

Generating txt/html scenarios

Codeception allows you to generate user-friendly text scenarios from scenario-driven from Cest and Cept php test (see http://codeception.com/docs/reference/Commands#GenerateScenarios).

Don't use double quotes, use single quotes instead

Do not use double quotes (") for surrounding strings:

$I->waitForText("Congratulations! Joomla! is now installed.", 30);

If you do, when you generate the scenarios you will get this bad result:

I wait for text ""Congratulations! Joomla! is now installed.""," 30"

If you use single quotes instead:

$I->waitForText('Congratulations! Joomla! is now installed.', 30);

You will get the right expected result:

I wait for text 'Congratulations! Joomla! is now installed.'," 30"

The reason of not being able to use double quotes is due to a bug in Codeception humanize funtions, see issue: https://github.com/Codeception/Codeception/issues/1847

Do not leave spaces between parameters in Cepts

See the two lines:

// Wrong
$I->waitForText('Main Configuration', 10, 'h3');
// Good
$I->waitForText('Main Configuration',10,'h3');

If you do, when you generate the scenarios you will get this bad result:

I wait for text 'Main Configuration'," 10"," 'h3'

The reason that you are asked to not add spaces between parameters is due to a bug in Codeception humanize funtions, see issue: https://github.com/Codeception/Codeception/issues/1847

Write complex locator VS. modifying the output of the tested website

Sometimes we need to write a very complex xpath to locate a element in the page, for example:

$I->click(['xpath' =>  "//*[@id=\"menuList\"]/tbody/tr/td[2]/a[contains(text(), 'weblink')"])

wouldn't be much better to write something like this instead?

$I->click(['class' =>  "lc-menu_item_type_weblinks"])

When you find yourself writting a very complex locator in a test, is maybe because the markup of the Website is wrong. For this cases, is better to stop writing the test and reconsider the possibility of modifying the website that we are testing. There is nothing that prevents us to improve Joomla markup. Actually is the other way arround, with these fixes we are probably making Joomla more accessible. We have had experiences in the past, see for example this issue: https://github.com/joomla/joomla-cms/issues/7207.

In conclusion is much more convenient to add just a simple class to the element to easily locate it instead of becoming mad finding the right xPath. Remind that is important that our tests are readable, and that has to do with using readable locators.

In order to simplify locators used in Acceptance tests we add special locator classes into HTML of web pages. So not to confuse them with classes needed by CSS or JavaScript we use lc- prefix. Thus, if you find locator to be to complex, it would be much easier to edit HTML and add a new class into it.

$I->click('class' =>  "lc-comment-delete');

Where lc stands for locator.

Is important that the rest of Joomla coders know that this lc- prefix is used by the tests to prevent anyone from removing it.

This recommendation was originally suggested by Davert (the creator of Codeception) at http://phptest.club/t/codeception-tips-tricks/14/8.

Thoughts on Selenium IDE and the codeception Formatter

One can use the Selenium IDE with the codeception Formatter. Using the Selenium IDE to write the tests seems way easier than writing them in PHP (since you have immediate feedback on what's going on the page), but it also poses some issues. For the most part the Joomla-Browser has functions that we can use to do standard stuff like login, logout etc. If we do our tests with the Selenium IDE we cannot use those and our tests get filled up with stuff that we have already abstracted.

There is still use of the Selenium IDE though, we could use it to create the tests for specific functionality. Like just writing an article, setting publish_up and publish_down dates and then saving. When we export the result with the codeception formatter we would have the necessary PHP and we can add the rest of the stuff for the test to be fully automated (login, navigate to page etc, then execute the exported test). This could speed the creation of tests and we could use non-developers for this kind of work. However this is easier said than done. Due to the Javascript that we use in Joomla simple tests are not that simple. For example clicking in the texteditor and writing text in it is not registered by the IDE since the editor is created by TinyMCE... A way to add text to the editor is to use

Command | runScript
Target | tinyMCE.get('text_area_id_not_iframe_id').setContent('Your text') 

That is again a technical solution to a test that should have been easy in the first place. We get into the same trouble with selecting a date out of a Calendar field. So at the end it seems pretty difficult for non-developers to create tests that we could use, because most of the steps recorded by the Selenium IDE won't work a second time.

If we find a way to make the work with Javascript elements on the page easier, then Selenium IDE could be a great tool that could help us write far more tests, than we currently have in a less time.

As far the codeception Formatter is concerned - it hasn't been updated in 2 years, but if we find out how to use the Selenium IDE extending the formatter and building on top of it shouldn't be a show stopper.