Difference between revisions of "Testing Joomla Extensions with Codeception"

From Joomla! Documentation

 
(82 intermediate revisions by 14 users not shown)
Line 1: Line 1:
<nowiki>Note: this page is under construction</nowiki>
+
{{underconstruction}}
  
 
== What is Codeception? ==
 
== What is Codeception? ==
 
Codeception PHP Testing Framework is designed to work just out of the box. This means its installation requires minimal steps and no external dependencies preinstalled (except PHP, of course). Only one configuration step should be taken and you are ready to test your web application from an eye of actual user.
 
Codeception PHP Testing Framework is designed to work just out of the box. This means its installation requires minimal steps and no external dependencies preinstalled (except PHP, of course). Only one configuration step should be taken and you are ready to test your web application from an eye of actual user.
 +
 +
See the following video of Codeception in action testing Joomla:
 +
 +
{{#widget:YouTube|id=Vc1BJNnhsFM}}
 +
 +
=== Types of Tests ===
 +
Codeception provides possibilities to do Unit Testing or to Test APIs, however '''this page is focused in System Testing'''.
 +
 +
- '''System testing''' uses a production-like environment to verify the system as a whole, including user interfaces, databases, web servers, logging, etc.
 +
 +
- A '''unit test''' verifies a single behavior of a single unit – this can be a function, a method, or a class. A logical unit is something that can be reasoned about in isolation. The unit must be completely under your control. The test should limit the number of other units it touches on, and should not touch any part outside of your own system. As such, the test should only fail for one reason.
 +
 +
- '''Integration testing''' means we’re verifying that that different units are working together. It focuses on the interfaces between components. Some of these components may be from third parties. When they break, they signal that, although individual units may behave correctly, something changed in the API that caused failure in another place. Often, integration tests are slower than unit tests.
 +
 +
As mentioned at the beginning of this text, the current page is focused on '''System Tests''' and more specifically using WebDriver. The reason is that we came to the conclusion that this type of tests combined with the already existing Unit tests in Joomla are the most successful way to test our current code.
  
 
== Why Use Codeception? ==
 
== Why Use Codeception? ==
Line 18: Line 33:
 
* BDD-style readable tests
 
* BDD-style readable tests
  
== Structure of a test ==
+
== Architecture of a Test ==
 
The testing framework is built on the three layered testing architecture
 
The testing framework is built on the three layered testing architecture
  
Line 46: Line 61:
 
Each page on your application has a class which we call page objects, each page class will have information about the URL, fields, Text present on the page. We use the best possible selector for any field on the page.
 
Each page on your application has a class which we call page objects, each page class will have information about the URL, fields, Text present on the page. We use the best possible selector for any field on the page.
  
Example:
+
==== Example ====
 
<source lang="php">
 
<source lang="php">
 
class JoomlaAdministratorLoginManagerPage
 
class JoomlaAdministratorLoginManagerPage
 
{
 
{
 
/**
 
/**
* @var string Url of the page
+
* @var string URL of the page
 
*/
 
*/
 
public static $URL = '/administrator/index.php';
 
public static $URL = '/administrator/index.php';
Line 67: Line 82:
 
</source>
 
</source>
  
More info: http://codeception.com/docs/07-AdvancedUsage#PageObjects
+
==== Naming Convention for Page Objects ====
 +
When this document is created Codeception 2.0 was not having a PSR-4 compatible autoloader. Therefore we are creating a convention so with the future Codeception 2.1 with PSR-4 compatible autoloader (https://github.com/Codeception/Codeception/pull/1228) we are creating a naming convention for Pages so it will cost us a small effort to move to autoloaded classes in the future.
 +
 
 +
The naming convention works as follows:
 +
 
 +
APPLICATION (for example Joomla, Patchtester, Weblinks or your Extension name)
 +
+ CLIENT (in Joomla we use it for defining if is Installation, Administrator or Frontend
 +
+ CLIENT PART (In Joomla we use it following the menu are were it belongs: Components, Menus, System,...)
 +
+ PAGE NAME (The Page name)
 +
+ Page
 +
 
 +
Examples:
 +
 
 +
JoomlaAdministratorLoginPage
 +
JoomlaAdministratorSystemGlobalconfigurationPage
 +
JoomlaInstallationConfigurationPage
 +
 
 +
And they are located with a folder structure similar to:
 +
 
 +
[[File:System_tests_pages_folder_structure.png|framed|center|Folder structure for Pages in System Tests]]
 +
 
 +
Remind that in Codeception 2.0 you need to specify each folder in the acceptance/_bootstrap.php file:
 +
<source lang="php">
 +
<?php
 +
// Here you can initialize variables that will be available to your tests
 +
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/installation');
 +
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/administrator');
 +
</source>
 +
 
 +
==== Documentation ====
 +
More info: https://codeception.com/docs/07-AdvancedUsage#PageObjects
  
 
=== Step Objects ===
 
=== Step Objects ===
Line 93: Line 138:
 
</source>
 
</source>
  
More info: http://codeception.com/docs/07-AdvancedUsage#StepObjects
+
More info: https://codeception.com/docs/07-AdvancedUsage#StepObjects
 +
 
 +
=== The Joomla Browser Module ===
 +
The Joomla Browser is a Codeception Module that incorporates general functions to your tests like:
 +
- do administrator login
 +
- install Joomla
 +
- install extension
 +
- ...
 +
 
 +
Find details about it at: https://github.com/joomla-projects/joomla-browser#table-of-contents
 +
 
 +
== Getting Started from Scratch: What You Need to Download and Install ==
 +
To get the test environment running you need to have several components installed.
 +
 
 +
'''Important note:''' no matter if you are executing [https://github.com/joomla/joomla-cms/tree/staging/tests/codeception#testing-joomla-cms Joomla CMS Tests], [https://github.com/joomla-extensions/weblinks#tests Weblinks], or tests in other Joomla extensions, in any case you are asked to fork the repository into your local machine under your htdocs folder of your local web-server. The reason is that, the automated scripts that launches the tests use the repository files for the tests.
 +
 
 +
=== Local Webserver ===
 +
If you do not already have a local test server running you may want to check out:
 +
https://www.mamp.info/
 +
https://www.apachefriends.org/
 +
http://www.ampps.com/
 +
 
 +
If you are not familiar with setting up a local web-server, learn that first before you advance further.
 +
 
 +
=== Java Runtime Environment (JRE) ===
 +
The JRE has to be installed, so the tests will be executable. You find the recent versions at the [https://www.oracle.com/java/technologies/downloads/ Oracle Website]. When you do not want to create an Oracle account you can download JRE for free at https://www.java.com/en/download/.
 +
 
 +
Newer Mac OS Versions are not shipped any more with the JRE, so they have to download that too.
 +
 
 +
=== Firefox ===
 +
Selenium is testing in the browser Firefox. The selenium version is not that quick with the updates, as the Firefox browser, if the test is not executable, then it can be that you manually have to install an older version of Firefox.
 +
Sadly that can be a try and error thing. To get older versions of Firefox: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/ here you will find many folders. Just try the first one that came before the Firefox version that you have been using (and that did not work). In the version folders you will find folders for all the different operating systems.
 +
 
 +
=== Configure Test System Variables ===
 +
[[File:Acceptance.suite.dist.yml rename.png|thumbnail|left|Screenshot of file structure]]
 +
 
 +
In your the folder that you installed, downloaded, pulled (git) the project, you have to rename the "''acceptance.suite.dist.yml''" file to "''acceptance.suite.yml''" and configure the parameters to your system variables.
 +
<source lang="php">
 +
  config:
 +
        JoomlaBrowser:
 +
            url: 'http://localhost/tests/joomla-cms3'
 +
            # the url that points to the joomla installation at /tests/system/joomla-cms
 +
            browser: 'firefox'
 +
            window_size: 1024x768
 +
    [...]
 +
</source>
 +
[[File:Run Command vendor bin robo test-acceptance.png|thumbnail|test:acceptance]]
 +
Once your machine is able to run tests (see ''The Joomla Browser Module''), you are able to start working.
 +
 
 +
Check the contents of the /tests/ folder in weblinks and try to understand the structure of the tests.
 +
For each function you have a directory and sometimes a configuration file, both will have the corresponding name.
 +
* FunctionName.yml - these files contain configurations for the function folders
 +
* FunctionName - contains the ''FunctionName'' test
 +
* joomla-cms3 - contains the latest Joomla! installation
 +
 
 +
After that check the tasks are at: https://github.com/joomla-extensions/weblinks/issues/47
 +
 
 +
== The Principles for Writing a Test: Readability and Performance ==
 +
In Joomla we give priority to Readability over Performance, even when performance is a must.
 +
 
 +
To explain that let me show an example: Let's say that we want to fill a field Username in a login form.
 +
 
 +
<source lang="html4strict">
 +
<form action="administrator/index.php" method="post" id="form-login">
 +
  <label for="username_field">Username</label>
 +
  <input name="username_field" tabindex="1" id="username_field" type="text" size="15" />
 +
  ...
 +
</source>
 +
 
 +
the most performant way to fill that form field would be:
 +
 
 +
<source lang="php">
 +
$I->fillField('#username_field', 'Mark');
 +
</source>
 +
 
 +
And usually we would move that to a page object element ending in:
 +
<source lang="php">
 +
$I->fillField(\LoginPage::$username, 'Mark');
 +
</source>
 +
 
 +
However Codeception is a smart framework and it adds some magic to it's methods (see: https://github.com/Codeception/Codeception/blob/2.0/src/Codeception/Module/WebDriver.php#L511-L513). It allows me to write more readable tests (keeping performance hight).
 +
 
 +
If we do this:
 +
<source lang="php">
 +
$I->fillField('Username', 'Mark');
 +
</source>
 +
 
 +
Codeception will be smart and will search first in the page if there is any "LABEL" called Username. That is a fast operation, so performance will be affected just a little. And instead we have written less code than in previous examples.
 +
 
 +
Note: sometimes this tricks are not always possible, labels are not always there or there are some elements hidden that shows and hides depending on a JavaScript. All these special cases would require a Page element.
 +
 
 +
Still there is a definitive method of using locators that made them readable and performant called: strict Locators see [[Testing_Joomla_Extensions_with_Codeception#Writing_reliable_locators]].
 +
 
 +
With them we do:
 +
 
 +
<source lang="php">
 +
$I->fillField(['id' => 'username_field'], 'Mark');
 +
</source>
 +
 
 +
In conclusion we have more readable tests and less code to maintain while keeping performance. That is a win!
  
 
== Setting Up Codeception ==
 
== Setting Up Codeception ==
=== setting up the repository ===
+
=== Setting Up the Repository ===
 
'''clone the directory'''
 
'''clone the directory'''
  
# Fork [https://github.com/joomla-projects/com_localise com_localise]
+
# Fork [https://github.com/joomla-extensions/weblinks/ com_weblinks] (read about how to fork a repository at https://help.github.com/articles/fork-a-repo/)
#
+
# Git clone git@github.com:*your-github-profile-name*/com_weblinks.git
# Git clone git@github.com:*your-github-profile-name*/com_localise.git
 
  
 
=== Testing with Codeception ===
 
=== Testing with Codeception ===
  
There are two ways to get and run codeception, via PHAR or via Composer. Using '''Composer is recommended''' (because we are using in some tests the Codeception 2.1 development version not available in the composer.phar).
+
There are two ways to get and run Codeception, via PHAR or via Composer. Using '''Composer is recommended''' (because we are using in some tests the Codeception 2.1 development version not available in the codeception.phar).
  
 
*'''Using Composer'''
 
*'''Using Composer'''
Line 117: Line 260:
 
*'''Codeception.phar'''
 
*'''Codeception.phar'''
  
     Linux Machine : wget http://codeception.com/codecept.phar
+
     Linux Machine : wget https://codeception.com/codecept.phar
  
     Windows Machine: Download from [http://codeception.com/install codecept.phar]
+
     Windows Machine: Download from [https://codeception.com/install codecept.phar]
  
 
     Step1 : php ./codecept.phar build
 
     Step1 : php ./codecept.phar build
 +
 +
LINUX USERS:
 +
After using either Composer or Codeception, you have to change your LAMP configurations.
 +
 +
1.  If you see phpmyadmin error of wrong permissions on configuration file. (This can be ignored but this would be useful for some people who face the same issue as me.)
 +
$ chmod 0755 /etc/apache2/envvars
 +
change permissions of Apache2
 +
$ nano /etc/apache2/envvars
 +
            change:export APACHE_RUN_GROUP=username
 +
                  export APACHE_RUN_USER=username
 +
change ownership of /var/www/html
 +
$ chown -R username:username /var/www/html
 +
This will recursively change the permissions of all folders and sub-folders in html
 +
 +
>And then restart Apache2
 +
$ sudo service apache2 restart
 +
 +
If it still doesn't work
 +
$ sudo chmod -R 777 /var/www/html
 +
 +
>Restart Apache2
 +
$ sudo service apache2 restart
 +
 +
>If it still doesn't work and when you run this command
 +
$ tests/codeception/vendor/bin/robo run:tests
 +
and you see index.html getting downloaded on your browser, just check whether some other server like nginx (in my case) or some other isn't occupying Apache2's port. You can find that by using the command
 +
$ netstat -ntlp    (This will show you the active ports on your system)
 +
If the Apache2's port is occupied, you either remove the other server or stop the other server.
  
 
=== Running the Test ===
 
=== Running the Test ===
 +
Find the complete instructions for running Weblinks tests at: https://github.com/joomla-extensions/weblinks#tests
 +
 +
== Best Practices at Testing ==
 +
=== Writing reliable locators ===
 +
Writing good locators can be tricky. The Mozilla team has written an excellent guide titled [https://blog.mozilla.org/webqa/2013/09/26/writing-reliable-locators-for-selenium-and-webdriver-tests/ Writing reliable locators for Selenium and WebDriver tests].
 +
 +
If the locator is an array, it should have a single element, with the key signifying the locator type (`id`, `name`, `css`, `xpath`, `link`, or `class`) and the value being the locator itself. This is called a "strict" locator. Examples:
  
Rename tests/acceptance.suite.dist.yml to tests/acceptance.suite.yml
+
<pre>
<source lang="bash">
+
* ['id' => 'foo'] matches `<div id="foo">`
Step 1: Modify the configuration at tests/acceptance.suite.yml to fit your server details.
+
* ['name' => 'foo'] matches `<div name="foo">`
 +
* ['css' => 'input[type=input][value=foo]'] matches `<input type="input" value="foo">`
 +
* ['xpath' => "//input[@type='submit'][contains(@value, 'foo')]"] matches `<input type="submit" value="foobar">`
 +
* ['link' => 'Click here'] matches `<a href="google.com">Click here</a>`
 +
* ['class' => 'foo'] matches `<div class="foo">`
 +
</pre>
 +
 +
=== How to Use URLs ===
 +
Codeception documentation suggests to use URLs as properties of a Page Object:
 +
 
 +
<source lang="php">
 +
$I->amOnPage(JoomlaInstallationConfigurationPage::$URL);
 
</source>
 
</source>
  
<source lang="bash">
+
Instead of hardcoding the url in the test:
Step 2: php codecept.phar run
+
<source lang="php">
</source>
+
$I->amOnPage('/administrator/index.php?option=com_banners&view...');
with --steps to see a step-by-step report on the performed actions.
+
</source>  
 +
 
 +
The pros of this approach is:
 +
- first way is more readable for non developers.
 +
- it helps to make tests more maintainable in the case of a URL change in the system
 +
 
 +
However in Joomla a change in an URL is rare, and if it happens changing it will just be a Find&Replace task. I Joomla test we want to have the benefit of both, the URL and a representative name that describes the page where we are going. That will make our tests more readable. Thus, we will write the URLs the following way:
  
<source lang="bash">
+
<source lang="php">
php vendor/bin/codecept run --steps
+
$I->amGoingTo('go to Create New Banner page at Joomla Administrator');
 +
$I->amOnPage('/administrator/index.php?option=com_banners&view...');
 
</source>
 
</source>
  
with --html. This command will run all tests for all suites, displaying the steps, and building HTML and XML reports. Reports will be store in tests/_output/ directory.
+
=== Working with Hidden Elements ===
 +
Selenium can't interact with elements that are not visible:
  
<source lang="bash">
+
<source lang="html4strict">
php vendor/bin/codecept run --html
+
<a href="https://www.joomla.org" style="display: none;" id="joomla_site">go to joomla</a>
 
</source>
 
</source>
  
To activate the debug mode in Codeception just add the parameter --debug when you run the tests:
+
In this case you can't do:
 +
<source lang="php">
 +
$I->click('go to joomla');
 +
</source>
  
<source lang="bash">
+
Usually this happens when you need to wait for a JavaScript animation that end up showing that element, so in this case you will need to do this:
php vendor/bin/codecept run acceptance --debug
+
<source lang="php">
 +
$I->waitForElementVisible('#joomla_site');
 +
$I->click('go to joomla');
 
</source>
 
</source>
  
== Best Practices at Testing ==
+
=== When Not to Use Contains ===
=== When do not use contains ===
 
 
We do not recommend to use the function contains on XPath because is not specific and can create false positives at tests.
 
We do not recommend to use the function contains on XPath because is not specific and can create false positives at tests.
 
See: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS/pull/99#issuecomment-69004191
 
See: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS/pull/99#issuecomment-69004191
=== Always add xPath locators to your tests ===
+
=== Always add Strict Locators to your tests ===
 
Performance in tests is very important. You can reduce dramatically the time spent on a test with just adding a few extra locators. For example, the following code pertains to a test:
 
Performance in tests is very important. You can reduce dramatically the time spent on a test with just adding a few extra locators. For example, the following code pertains to a test:
  
Line 166: Line 367:
  
 
<source lang="php">
 
<source lang="php">
$I->see('Your article has been published', '#messages');
+
$I->see('Your article has been published', ['id' => 'messages']);
 
</source>
 
</source>
  
Line 172: Line 373:
 
to: Time: 49.18 seconds
 
to: Time: 49.18 seconds
  
You can see this video of a real example: http://www.youtube.com/watch?v=0JCv0BR2yZY
+
You can see this video of a real example: https://www.youtube.com/watch?v=0JCv0BR2yZY
 +
 
 +
Losing 30 seconds in every Assertion can delay your tests dramatically. Therefore, remind always to add XPath locators to all your Assertions.
 +
 
 +
Allowed locator mechanisms are:
 +
 
 +
<source lang="php">
 +
- ['id' => 'xxx']
 +
- ['name' => 'xxx']
 +
- ['css' => 'xxx']
 +
- ['xpath' => 'xxx']
 +
- ['link' => 'xxx']
 +
- ['class' => 'xxx']
 +
</source>
 +
 
 +
More info: https://codeception.com/docs/modules/WebDriver#Locating-Elements
 +
 
 +
==== Using Locators in $I->see ====
 +
Do not use ''see'' without locator:
 +
<source lang="php">
 +
// Wrong:
 +
$I->see('Your article has been published');
 +
 
 +
// Right:
 +
$I->see('Your article has been published', ['id' => 'messages']);
 +
</source>
 +
 
 +
==== Using Locators in $I->waitForText ====
 +
Do not use ''waitForText'' without locator:
 +
<source lang="php">
 +
// Wrong:
 +
$I->waitForText('Finalisation');
 +
 
 +
// Right:
 +
$I->waitForText('Finalisation', 10, ['css' => 'h3']);
 +
</source>
 +
 
 +
In the case of an element that has multiple CSS classes, make sure you use more specific class as the locator:
 +
<source lang="php">
 +
// Wrong:
 +
$I->waitForText('Item Successfully Created', 10, ['class' => 'alert']);
 +
 
 +
// Right:
 +
$I->waitForText('Item Successfully Created', 10, ['class' => 'alert-success']);
 +
</source>
 +
A page could have multiple elements which can have ''alert'' CSS class associated, hence being more specific helps.
 +
 
 +
==== How Much Time Should I Wait for an Element or Text? ====
 +
You may have noticed that the second parameter in the I waitForText is an integer that represents how much "time to wait" in seconds:
 +
 
 +
<source lang="php">
 +
$I->waitForText('a text to wait for','60',['...' => '...']);
 +
</source>
 +
 
 +
But, how much time is the right time to wait for a text?
 +
 
 +
PHP maximum execution time before returning a Fatal Error for timeout is set by default to 30 seconds. '''We suggest to set it for 60 seconds''', because running the tests in slow computers can easily get to that time in some tasks.
 +
 
 +
If a waitForText command takes over 10 seconds, it will much probably mean that something is not performing well. Maybe a non [[Testing Joomla Extensions with Codeception#Always_add_Strict_Locators_to_your_tests|Strict Locator]] is being used and Selenium takes to much time to find the element. However we have even identified cases were some steps take longer than 30 seconds in very slow computers, specifically installing Joomla, see https://github.com/joomla-projects/joomla-browser/commit/b70f195140e599649cf98f051d7a31388c4f1632. For this cases we can't do much, other than recommend to add more memory to the local web server.
 +
 
 +
=== Quote and Double Quotes in Complex XPaths ===
 +
Sometimes you will have to write XPaths that looks like:
 +
 
 +
//button[@onclick='Joomla.submitbutton("config.save.application.apply")']
 +
 
 +
When moved to a Page Object you will probably will face this situation:
 +
 
 +
[[File:System_tests_error_escaping_quotes_on_xpaths.png|framed|center|Typical issue when you write a long XPath in a Page object]]
 +
 
 +
To avoid it remind to escape the quotes:
 +
 
 +
<source lang="php">
 +
// Wrong
 +
public static $elements = array(
 +
  'Save' => "//button[@onclick="Joomla.submitbutton('config.save.application.apply')"]"
 +
);
 +
// Right
 +
public static $elements = array(
 +
  'Save' => "//button[@onclick=\"Joomla.submitbutton('config.save.application.apply')\"]"
 +
);
 +
</source>
 +
 
 +
===Generating txt/html Scenarios===
 +
Codeception allows you to generate user-friendly text scenarios from scenario-driven from Cest and Cept PHP test (see https://codeception.com/docs/reference/Commands#GenerateScenarios).
 +
 
 +
===Don't Use Double Quotes; Use Single Quotes Instead===
 +
Do not use double quotes (") for surrounding strings:
 +
<source lang="php">
 +
$I->waitForText("Congratulations! Joomla! is now installed.", 30);
 +
</source>
 +
 
 +
If you do, when you [[Testing_Joomla_Extensions_with_Codeception#Generating_txt.2Fhtml_scenarios|generate the scenarios]] you will get this bad result:
 +
 
 +
I wait for text ""Congratulations! Joomla! is now installed.""," 30"
 +
 
 +
If you use single quotes instead:
 +
 
 +
<source lang="php">
 +
$I->waitForText('Congratulations! Joomla! is now installed.', 30);
 +
</source>
 +
 
 +
You will get the right expected result:
 +
 
 +
I wait for text 'Congratulations! Joomla! is now installed.'," 30"
 +
 
 +
The reason of not being able to use double quotes is due to a bug in Codeception humanize functions, see issue: https://github.com/Codeception/Codeception/issues/1847
 +
 
 +
===Do Not Leave Spaces Between Parameters in Cepts===
 +
See the two lines:
 +
<source lang="php">
 +
// Wrong
 +
$I->waitForText('Main Configuration', 10, 'h3');
 +
// Good
 +
$I->waitForText('Main Configuration',10,'h3');
 +
</source>
 +
 
 +
If you do, when you [[Testing_Joomla_Extensions_with_Codeception#Generating_txt.2Fhtml_scenarios|generate the scenarios]] you will get this bad result:
 +
 
 +
I wait for text 'Main Configuration'," 10"," 'h3'
 +
 
 +
The reason that you are asked to not add spaces between parameters is due to a bug in Codeception humanize functions, see issue: https://github.com/Codeception/Codeception/issues/1847
 +
 
 +
===Write Complex Locator Vs. Modifying the Output of the Tested Website===
 +
Sometimes we need to write a very complex XPath to locate a element in the page, for example:
 +
 
 +
$I->click(['xpath' =>  "//*[@id=\"menuList\"]/tbody/tr/td[2]/a[contains(text(), 'weblink')"])
 +
 
 +
wouldn't be much better to write something like this instead?
 +
 
 +
$I->click(['class' => "test-weblinks-menu_item_type_weblinks"])
 +
 
 +
When you find yourself writing a very complex locator in a test, is maybe because the markup of the Website is wrong. For this cases, is better to stop writing the test and reconsider the possibility of modifying the website that we are testing. There is nothing that prevents us to improve Joomla markup. Actually is the other way around, with these fixes we are probably making Joomla more accessible.
 +
We have had experiences in the past, see for example this issue: https://github.com/joomla/joomla-cms/issues/7207.
 +
 
 +
In conclusion is much more convenient to add just a simple class to the element to easily locate it instead of becoming mad finding the right XPath. Remind that is important that our tests are readable, and that has to do with using readable locators.
 +
 
 +
In order to simplify locators used in Acceptance tests we add special locator classes into HTML of web pages. So not to confuse them with classes needed by CSS or JavaScript we use '''test-*projectname*-''' prefix. Thus, if you find locator to be to complex, it would be much easier to edit HTML and add a new class into it.
 +
 
 +
$I->click('class' =>  "test-projectname-comment-delete');
 +
 
 +
Where ''projectname'' stands for the name of the project in which we are adding this ''locator''.
 +
 
 +
It is important that the rest of Joomla coders know that this '''test-projectname-''' prefix is used by the tests to prevent anyone from removing it.
 +
 
 +
This recommendation was originally suggested by Davert (the creator of Codeception) at http://phptest.club/t/codeception-tips-tricks/14/8.
 +
 
 +
===Thoughts on Selenium IDE and the Codeception Formatter===
 +
Selenium IDE is a Firefox extension that allows to generate tests in a interactive way:
 +
 
 +
[[File:Seleniumide.png|thumbnail]]
 +
 
 +
One can use the [http://www.seleniumhq.org/projects/ide/ Selenium IDE] with the [https://addons.mozilla.org/en-US/firefox/addon/codeception-formatters/ codeception Formatter] to export the tests to Codeception Framework Format.
 +
 
 +
[[File:exportcodeception.png|thumbnail]]
 +
 +
Using the Selenium IDE to write the tests seems way easier than writing them in PHP (since you have immediate feedback on what's going on the page), but it also poses some issues.
 +
 
 +
For the most part the Joomla-Browser has functions that we can use to do standard stuff like login, logout etc. If we do our tests with the Selenium IDE we cannot use those and our tests get filled up with stuff that we have already abstracted.
 +
 
 +
There is still use of the Selenium IDE though, we could use it to create the tests for specific functionality. Like just writing an article, setting publish_up and publish_down dates and then saving. When we export the result with the Codeception Formatter we would have the necessary PHP and we can add the rest of the stuff for the test to be fully automated (login, navigate to page etc, then execute the exported test). This could speed the creation of tests and we could use non-developers for this kind of work. However this is easier said than done. Due to the JavaScript that we use in Joomla simple tests are not that simple. For example clicking in the text editor and writing text in it is not registered by the IDE since the editor is created by TinyMCE... A way to add text to the editor is to use
 +
 
 +
Command | runScript
 +
Target | tinyMCE.get('text_area_id_not_iframe_id').setContent('Your text')
 +
 
 +
That is again a technical solution to a test that should have been easy in the first place.
 +
We get into the same trouble with selecting a date out of a Calendar field. So at the end it seems pretty difficult for non-developers to create tests that we could use, because most of the steps recorded by the Selenium IDE won't work a second time.
 +
 
 +
If we find a way to make the work with JavaScript elements on the page easier, then Selenium IDE could be a great tool that could help us write far more tests, than we currently have in a less time.
 +
 
 +
As far the Codeception Formatter is concerned - it hasn't been updated in 2 years, but if we find out how to use the Selenium IDE extending the Formatter and building on top of it shouldn't be a show stopper.
  
Losing 30 seconds in every Assertion can delay your tests dramatically. Therefore, remind always to add xPath locators to all your Assertions.
 
 
[[Category:Automated Testing]]
 
[[Category:Automated Testing]]
 +
<noinclude>[[Category:Needs to be marked for translation]]</noinclude>
 +
 +
== BDD Testing Using Gherkin and Codeception ==
 +
 +
BDD (Behavior-driven development)  in its grandest sense is about communication and viewing your software as a system with behaviour. Using BDD approach to write test cases make test cases easy and understandable. BDD test cases can be understood by non developers and testers.
 +
 +
=== What is the Gherkin Language ===
 +
Gherkin is the language that codeception uses to define test cases. Test cases were designed to be non-technical and human readable, and collectively describes use cases relating to a software system.
 +
 +
=== Gherkin Syntax ===
 +
Gherkin is language in which we need to write statements in simple English but we need to follow below syntax guidelines:
 +
 +
* Gherkin is a line-oriented language most lines start with a special keyword.
 +
* Lines can be comment by using hash sign(#).
 +
* Gherkin test cases are saved in “.feature” file format.
 +
* You can write gherkin in any editor (Like PhpStorm, Sublime,Gherkin editor, and any IDE).
 +
* Use can use Tags (“@”) to organize features and scenarios.
 +
 +
=== Feature ===
 +
Descriptive text of what is desired from all the test cases inside that “.feature” file. Basically it is a description of feature which is going to cover by below test cases.
 +
 +
'''The general format of a feature is:'''
 +
 +
<source lang="ini">
 +
Feature: <short description>
 +
In order to…
 +
As a…
 +
I need to...
 +
</source>
 +
 +
Within the definition of a feature you can provide a plain text description of the story. You can use any format for this, but having some sort of template makes it easier to see the important bits of information with a quick glance. This format focuses us on three important questions:
 +
 +
* Who's using the system?
 +
* What are they doing?
 +
* Why do they care?
 +
 +
=== A Sample Story ===
 +
<source lang="ini">
 +
Feature: users
 +
In order to manage users in the web
 +
As an owner
 +
I need to create edit block unblock and delete user
 +
</source>
 +
 +
=== Background ===
 +
Background contains set of steps that all scenario need to perform at start.
 +
 +
'''Here’s an Example of Background'''
 +
<source lang="ini">
 +
Background:
 +
Given Joomla CMS is installed
 +
When Login into Joomla administrator with username "admin" and password "admin"
 +
Then I see administrator dashboard
 +
</source>
 +
 +
=== Scenario ===
 +
Description of test case what test case is going to test, In short it is test case name with some description. A scenario is a sequence of steps through the feature that exercises one path.
 +
 +
'''A scenario is made up of 3 sections related to the 3 types of steps'''
 +
 +
# '''Given''': This sets up preconditions, or context, for the scenario.
 +
#
 +
# '''When''': This is what the feature is talking about, the action, the behaviour that we're focused on.
 +
#
 +
# '''Then''': This checks postconditions. It verifies that the right thing happen in the When stage.
 +
 +
'''Here’s an Example of scenario'''
 +
 +
<source lang="ini">
 +
Scenario: Create User from Backend
 +
Given There is a add user link
 +
When I create new user with fields Name "register", Login Name "register", Password "register" and  Email 
 +
            "register@gmail.com"
 +
And  I Save the  user
 +
Then I see the "User successfully saved." message
 +
</source>
 +
 +
'''And''': If we have consecutive ''“given, when or then”'' then instead of using these we use ''“And”'' statement to make it more readable.
 +
 +
'''But''': Same as ''“And”'' but used in negative prospect.
 +
 +
<source lang="ini">
 +
Scenario: Create user from frontend
 +
Given I click on the link "Create an account"
 +
And I create a user with fields Name "patel", Username "patel", Password "patel" and Email "patel@gmail.com"
 +
When I press the "Register"
 +
Then I see "Could not instantiate mail function." message
 +
And  user is created
 +
</source>
 +
 +
=== Given/When/Then/But/And Best Practice ===
 +
The scenarios within our `.feature files` consist of '''steps''', also known as ''givens'', ''whens'', ''buts'', ''thens'' and ''ands''.
 +
 +
'''Technically''' these kind of steps are the same. When we executing the feature, each step is matched to a PHP function.
 +
 +
''' Example:'''<br />
 +
`Given There is an user link,`<br />
 +
`When I see the user edit view tabs`<br />
 +
and<br />
 +
`Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"`<br /><br />
 +
are the steps in the Scenario: <br /><br />
 +
`Verify available tabs in com_users `<br />
 +
of the feature-file `/tests/codeception/acceptance/users.feature`. <br />
 +
<br />
 +
 +
<source lang="ini">
 +
  Scenario: Verify available tabs in com_users
 +
    Given There is an user link
 +
    When I see the user edit view tabs
 +
    Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"
 +
</source>
 +
 +
The step <br />
 +
'''`Given There is an user link`'''<br />
 +
is matched to the PHP function '''`thereIsAAddUserLink()`''',<br />
 +
the step<br />
 +
'''`When I see the user edit view tabs`'''<br />
 +
is matched to the PHP function '''`iSeeTheUsereditViewTabs()`'''<br />
 +
and the step <br />
 +
'''`Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"`'''<br />
 +
is matched to the PHP function '''`iCheckAvailableTabs($tab1, $tab2, $tab3)`'''<br />
 +
of the file '''`/tests/codeception/_support/Step/Acceptance/Administrator/User.php`''' <br />
 +
<br />
 +
 +
<source lang="php">
 +
 +
public function thereIsAAddUserLink()
 +
{
 +
$I = $this;
 +
 +
$I->amOnPage(UserManagerPage::$url);
 +
$I->adminPage->clickToolbarButton('New');
 +
}
 +
 +
 +
public function iSeeTheUsereditViewTabs()
 +
{
 +
$I = $this;
 +
 +
$I->adminPage->clickToolbarButton('New');
 +
$I->adminPage->waitForPageTitle('Users');
 +
}
 +
public function iCheckAvailableTabs($tab1, $tab2, $tab3)
 +
{
 +
$I = $this;
 +
 +
$I->adminPage->verifyAvailableTabs([$tab1, $tab2, $tab3]);
 +
}
 +
</source>
 +
<br />
 +
'''So, why are there these different step names?'''<br /><br />
 +
Behaviour-driven development (BDD) means that you can turn an idea for a requirement into code simply, as long as the requirement is specific enough that everyone knows what’s going on. <br />
 +
We need a way to describe the requirement so that everyone – user and developer – have a common understanding of the scope of  the software.<br />
 +
<br />
 +
'''Our tests can only be successful if we have the same semantic understanding of the steps Given, When and Then!'''<br />
 +
<br />
 +
==== Given====
 +
The purpose of Given is to put the system in a known state. What is the context of the system? What pre-condition must be true before the action is tested? What relevant data is in the system?<br />
 +
 +
Important in this context: In most of our .feature files we use the the section Background. The background is run before each of the scenarios in our .feature file and also has impact on the state.<br />
 +
 +
''' Examples:'''<br />
 +
Given There is a add content link<br />
 +
Given I search and select the user with user name "register"<br />
 +
Given I am on the User Manager page<br />
 +
Given User "User Three" did login at least once<br />
 +
 +
==== When====
 +
The purpose of When is to describe the key action that will be performed. What will be tested? What relevant input data will be tested when the action occurs?
 +
 +
''' Examples:'''<br />
 +
This could be only one action like in the Scenario<br />
 +
Delete ACL
 +
 +
<source lang="ini">
 +
  Scenario: Delete ACL
 +
    Given I search and select the Access Level with name "Acl Two"
 +
    When I Delete the Access level "Acl Two"
 +
    Then I should see "No Matching Results" for deleted user "Acl Two"
 +
</source>
 +
 +
This could multiple actions in succession like in the Scenario<br />
 +
Check if block and activation are working:
 +
 +
<source lang="ini">
 +
  Scenario: Check if block and activation are working
 +
    Given I am on the User Manager page
 +
    When I unblock the user "User Two"
 +
    And I activate the user "User Two"
 +
    And I login with user "user2" with password "pass2" in frontend
 +
    Then I should see the message "Hi User Two,"
 +
</source>
 +
 +
==== Then ====
 +
The purpose of Then is to observe outcomes. The observations should be related to the business value/benefit in our feature description. In response to the action, what is the observable outcome? What are the expected results? What is the post-condition or output data observable by the user?
 +
 +
''' Examples:'''<br />
 +
Then I should see the access level "Acl Two" is created<br />
 +
Then Login in backend with username "User One" and password "pass1"<br />
 +
 +
'''Please note:''' We should only verify outcome that is observable for the user. It is not enough to verify that all entries in the database are correct!
 +
 +
==== And, But ====
 +
If we have several `givens`, `whens` or `thens` we could write
 +
 +
<source lang="ini">
 +
  Scenario: Create User without username fails
 +
    Given There is a add user link
 +
    When I don't fill Login Name but fulfill remaining mandatory fields: Name "User Two", Password "pass2" and Email "user2@example.com"
 +
    When I Save the user
 +
    Then I see the title "Users: New"
 +
    Then I see the alert error "Invalid field:  Login Name"
 +
</source>
 +
 +
But we make it read more fluently by writing
 +
 +
<source lang="ini">
 +
Scenario: Create User without username fails
 +
    Given There is a add user link
 +
    When I don't fill Login Name but fulfill remaining mandatory fields: Name "User Two", Password "pass2" and Email "user2@example.com"
 +
    And I Save the user
 +
    Then I see the title "Users: New"
 +
    But I see the alert error "Invalid field:  Login Name"
 +
</source>
 +
 +
Steps beginning with And or But are exactly the same kind of steps as all the others.
 +
 +
===Implementing Steps===
 +
'''Run this command'''
 +
 +
''tests/codeception/vendor/bin/codecept gherkin:snippets acceptance''
 +
 +
That will output all the "step" methods in .feature files.
 +
Copy all this snippets and put into step file.
 +
 +
[[File:Gherkin-steps-en.jpg]]
 +
 +
'''How to Create a step file?'''
 +
 +
''tests/codeception/vendor/bin/codecept generate:stepobject acceptance  Administrator/users''
 +
 +
'''Execute feature file'''
 +
 +
''tests/codeception/vendor/bin/codecept run tests/acceptance/users.feature --steps''
 +
 +
'''Execute feature file using robo'''
 +
 +
''tests/codeception/vendor/bin/robo run:tests''

Latest revision as of 18:50, 1 April 2022

Documentation all together tranparent small.png
Under Construction

This article or section is in the process of an expansion or major restructuring. You are welcome to assist in its construction by editing it as well. If this article or section has not been edited in several days, please remove this template.
This article was last edited by Togepi-together (talk| contribs) 23 months ago. (Purge)


What is Codeception?[edit]

Codeception PHP Testing Framework is designed to work just out of the box. This means its installation requires minimal steps and no external dependencies preinstalled (except PHP, of course). Only one configuration step should be taken and you are ready to test your web application from an eye of actual user.

See the following video of Codeception in action testing Joomla:

Types of Tests[edit]

Codeception provides possibilities to do Unit Testing or to Test APIs, however this page is focused in System Testing.

- System testing uses a production-like environment to verify the system as a whole, including user interfaces, databases, web servers, logging, etc.

- A unit test verifies a single behavior of a single unit – this can be a function, a method, or a class. A logical unit is something that can be reasoned about in isolation. The unit must be completely under your control. The test should limit the number of other units it touches on, and should not touch any part outside of your own system. As such, the test should only fail for one reason.

- Integration testing means we’re verifying that that different units are working together. It focuses on the interfaces between components. Some of these components may be from third parties. When they break, they signal that, although individual units may behave correctly, something changed in the API that caused failure in another place. Often, integration tests are slower than unit tests.

As mentioned at the beginning of this text, the current page is focused on System Tests and more specifically using WebDriver. The reason is that we came to the conclusion that this type of tests combined with the already existing Unit tests in Joomla are the most successful way to test our current code.

Why Use Codeception?[edit]

Why use Codeception when we already have Selenium IDE?

Codeception works great with Selenium. But with Codeception you can write your tests in PHP. The main reason is: Selenium IDE tests are tightly bound to XPath locators. If you ever change anything in layout tests will fall. Codeception locators are more stable. You can use names, labels, button names and CSS to match elements on page.

Features

  • Selenium WebDriver integration
  • Elements matched by name, CSS, XPath
  • PageObjects and StepObjects included
  • Powered by PHPUnit
  • Data Cleanup
  • Parallel Execution of tests
  • BDD-style readable tests

Architecture of a Test[edit]

The testing framework is built on the three layered testing architecture

Scenarios (Cept Class)[edit]

Cept or Test, it is the main class which contains steps of execution for the tests, Cept Class will call Step Class functions which will indirectly make use of page objects to perform operations

Example:

$I = new AcceptanceTester($scenario);
$I->wantTo('Install Joomla CMS');
$I->amOnPage('http://localhost/installation/index.php');
// I Wait for the text Main Configuration, meaning that the page is loaded
$I->waitForText('Main Configuration', 10, 'h3');
$I->click(JoomlaInstallationConfigurationPage::$elements['Language Selector']);
$I->click(JoomlaInstallationConfigurationPage::$elements['English (United Kingdom)']);
$I->fillField('Site Name', 'weblinks');
$I->fillField('Description', 'Site for testing Weblinks component');
$I->fillField('Admin Email', $cfg['Admin email']);
$I->fillField('Admin Username', $cfg['username']);
$I->fillField('Admin Password', $cfg['password']);
$I->fillField('Confirm Admin Password', $cfg['password']);
$I->click('Next');
...

Page Objects[edit]

Each page on your application has a class which we call page objects, each page class will have information about the URL, fields, Text present on the page. We use the best possible selector for any field on the page.

Example[edit]

class JoomlaAdministratorLoginManagerPage
{
	/**
	 * @var string URL of the page
	 */
	public static $URL = '/administrator/index.php';

	/**
	 * Array of Page elements indexed by descriptive name or label
	 *
	 * @var array
	 */
	public static $elements = array(
		'username' => "#mod-login-username",
		'password' => "#mod-login-password"
	);
}

Naming Convention for Page Objects[edit]

When this document is created Codeception 2.0 was not having a PSR-4 compatible autoloader. Therefore we are creating a convention so with the future Codeception 2.1 with PSR-4 compatible autoloader (https://github.com/Codeception/Codeception/pull/1228) we are creating a naming convention for Pages so it will cost us a small effort to move to autoloaded classes in the future.

The naming convention works as follows:

APPLICATION (for example Joomla, Patchtester, Weblinks or your Extension name)
+ CLIENT (in Joomla we use it for defining if is Installation, Administrator or Frontend
+ CLIENT PART (In Joomla we use it following the menu are were it belongs: Components, Menus, System,...)
+ PAGE NAME (The Page name) 
+ Page

Examples:

JoomlaAdministratorLoginPage
JoomlaAdministratorSystemGlobalconfigurationPage
JoomlaInstallationConfigurationPage

And they are located with a folder structure similar to:

Folder structure for Pages in System Tests

Remind that in Codeception 2.0 you need to specify each folder in the acceptance/_bootstrap.php file:

<?php
// Here you can initialize variables that will be available to your tests
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/installation');
\Codeception\Util\Autoload::registerSuffix('Page', __DIR__.DIRECTORY_SEPARATOR.'_pages/joomla/administrator');

Documentation[edit]

More info: https://codeception.com/docs/07-AdvancedUsage#PageObjects

Step Objects[edit]

Step Objects are classes which contains steps which are to be executed on the application to perform certain operations, step objects make use of page objects to perform these actions.

Example:

class JoomlaAdministratorLoginSteps extends \AcceptanceTester
{
	/**
	 * Function to execute an Admin Login for Joomla 3
	 *
	 * @return void
	 */
	public function doAdministratorLogin($user, $password)
	{
		$I = $this;
		$I->amOnPage(\JoomlaAdministratorLoginPage::$URL);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['username'], $user);
		$I->fillField(\JoomlaAdministratorLoginPage::$elements['password'], $password);
		$I->click('Log in');
		$I->see('Category Manager');
	}
}

More info: https://codeception.com/docs/07-AdvancedUsage#StepObjects

The Joomla Browser Module[edit]

The Joomla Browser is a Codeception Module that incorporates general functions to your tests like: - do administrator login - install Joomla - install extension - ...

Find details about it at: https://github.com/joomla-projects/joomla-browser#table-of-contents

Getting Started from Scratch: What You Need to Download and Install[edit]

To get the test environment running you need to have several components installed.

Important note: no matter if you are executing Joomla CMS Tests, Weblinks, or tests in other Joomla extensions, in any case you are asked to fork the repository into your local machine under your htdocs folder of your local web-server. The reason is that, the automated scripts that launches the tests use the repository files for the tests.

Local Webserver[edit]

If you do not already have a local test server running you may want to check out: https://www.mamp.info/ https://www.apachefriends.org/ http://www.ampps.com/

If you are not familiar with setting up a local web-server, learn that first before you advance further.

Java Runtime Environment (JRE)[edit]

The JRE has to be installed, so the tests will be executable. You find the recent versions at the Oracle Website. When you do not want to create an Oracle account you can download JRE for free at https://www.java.com/en/download/.

Newer Mac OS Versions are not shipped any more with the JRE, so they have to download that too.

Firefox[edit]

Selenium is testing in the browser Firefox. The selenium version is not that quick with the updates, as the Firefox browser, if the test is not executable, then it can be that you manually have to install an older version of Firefox. Sadly that can be a try and error thing. To get older versions of Firefox: https://ftp.mozilla.org/pub/mozilla.org/firefox/releases/ here you will find many folders. Just try the first one that came before the Firefox version that you have been using (and that did not work). In the version folders you will find folders for all the different operating systems.

Configure Test System Variables[edit]

Screenshot of file structure

In your the folder that you installed, downloaded, pulled (git) the project, you have to rename the "acceptance.suite.dist.yml" file to "acceptance.suite.yml" and configure the parameters to your system variables.

   config:
        JoomlaBrowser:
            url: 'http://localhost/tests/joomla-cms3'
            # the url that points to the joomla installation at /tests/system/joomla-cms
            browser: 'firefox'
            window_size: 1024x768
    [...]
test:acceptance

Once your machine is able to run tests (see The Joomla Browser Module), you are able to start working.

Check the contents of the /tests/ folder in weblinks and try to understand the structure of the tests. For each function you have a directory and sometimes a configuration file, both will have the corresponding name.

  • FunctionName.yml - these files contain configurations for the function folders
  • FunctionName - contains the FunctionName test
  • joomla-cms3 - contains the latest Joomla! installation

After that check the tasks are at: https://github.com/joomla-extensions/weblinks/issues/47

The Principles for Writing a Test: Readability and Performance[edit]

In Joomla we give priority to Readability over Performance, even when performance is a must.

To explain that let me show an example: Let's say that we want to fill a field Username in a login form.

<form action="administrator/index.php" method="post" id="form-login">
  <label for="username_field">Username</label>
  <input name="username_field" tabindex="1" id="username_field" type="text" size="15" />
  ...

the most performant way to fill that form field would be:

$I->fillField('#username_field', 'Mark');

And usually we would move that to a page object element ending in:

$I->fillField(\LoginPage::$username, 'Mark');

However Codeception is a smart framework and it adds some magic to it's methods (see: https://github.com/Codeception/Codeception/blob/2.0/src/Codeception/Module/WebDriver.php#L511-L513). It allows me to write more readable tests (keeping performance hight).

If we do this:

$I->fillField('Username', 'Mark');

Codeception will be smart and will search first in the page if there is any "LABEL" called Username. That is a fast operation, so performance will be affected just a little. And instead we have written less code than in previous examples.

Note: sometimes this tricks are not always possible, labels are not always there or there are some elements hidden that shows and hides depending on a JavaScript. All these special cases would require a Page element.

Still there is a definitive method of using locators that made them readable and performant called: strict Locators see Testing_Joomla_Extensions_with_Codeception#Writing_reliable_locators.

With them we do:

$I->fillField(['id' => 'username_field'], 'Mark');

In conclusion we have more readable tests and less code to maintain while keeping performance. That is a win!

Setting Up Codeception[edit]

Setting Up the Repository[edit]

clone the directory

  1. Fork com_weblinks (read about how to fork a repository at https://help.github.com/articles/fork-a-repo/)
  2. Git clone git@github.com:*your-github-profile-name*/com_weblinks.git

Testing with Codeception[edit]

There are two ways to get and run Codeception, via PHAR or via Composer. Using Composer is recommended (because we are using in some tests the Codeception 2.1 development version not available in the codeception.phar).

  • Using Composer
   You need to have Composer in your system, if not download it from here: https://getcomposer.org/
   Step 1: composer update
   Step 2: run Codeception by doing: php vendor/bin/codecept build
  • Codeception.phar
   Linux Machine : wget https://codeception.com/codecept.phar
   Windows Machine: Download from codecept.phar
   Step1 : php ./codecept.phar build

LINUX USERS: After using either Composer or Codeception, you have to change your LAMP configurations.

1. If you see phpmyadmin error of wrong permissions on configuration file. (This can be ignored but this would be useful for some people who face the same issue as me.)

$ chmod 0755 /etc/apache2/envvars 

change permissions of Apache2

$ nano /etc/apache2/envvars 
           change:export APACHE_RUN_GROUP=username 
                  export APACHE_RUN_USER=username 

change ownership of /var/www/html

$ chown -R username:username /var/www/html 

This will recursively change the permissions of all folders and sub-folders in html

>And then restart Apache2

$ sudo service apache2 restart

If it still doesn't work

$ sudo chmod -R 777 /var/www/html

>Restart Apache2

$ sudo service apache2 restart 

>If it still doesn't work and when you run this command

$ tests/codeception/vendor/bin/robo run:tests 

and you see index.html getting downloaded on your browser, just check whether some other server like nginx (in my case) or some other isn't occupying Apache2's port. You can find that by using the command

$ netstat -ntlp    (This will show you the active ports on your system)

If the Apache2's port is occupied, you either remove the other server or stop the other server.

Running the Test[edit]

Find the complete instructions for running Weblinks tests at: https://github.com/joomla-extensions/weblinks#tests

Best Practices at Testing[edit]

Writing reliable locators[edit]

Writing good locators can be tricky. The Mozilla team has written an excellent guide titled Writing reliable locators for Selenium and WebDriver tests.

If the locator is an array, it should have a single element, with the key signifying the locator type (`id`, `name`, `css`, `xpath`, `link`, or `class`) and the value being the locator itself. This is called a "strict" locator. Examples:

* ['id' => 'foo'] matches `<div id="foo">`
* ['name' => 'foo'] matches `<div name="foo">`
* ['css' => 'input[type=input][value=foo]'] matches `<input type="input" value="foo">`
* ['xpath' => "//input[@type='submit'][contains(@value, 'foo')]"] matches `<input type="submit" value="foobar">`
* ['link' => 'Click here'] matches `<a href="google.com">Click here</a>`
* ['class' => 'foo'] matches `<div class="foo">`

How to Use URLs[edit]

Codeception documentation suggests to use URLs as properties of a Page Object:

$I->amOnPage(JoomlaInstallationConfigurationPage::$URL);

Instead of hardcoding the url in the test:

$I->amOnPage('/administrator/index.php?option=com_banners&view...');

The pros of this approach is: - first way is more readable for non developers. - it helps to make tests more maintainable in the case of a URL change in the system

However in Joomla a change in an URL is rare, and if it happens changing it will just be a Find&Replace task. I Joomla test we want to have the benefit of both, the URL and a representative name that describes the page where we are going. That will make our tests more readable. Thus, we will write the URLs the following way:

$I->amGoingTo('go to Create New Banner page at Joomla Administrator');
$I->amOnPage('/administrator/index.php?option=com_banners&view...');

Working with Hidden Elements[edit]

Selenium can't interact with elements that are not visible:

<a href="https://www.joomla.org" style="display: none;" id="joomla_site">go to joomla</a>

In this case you can't do:

$I->click('go to joomla');

Usually this happens when you need to wait for a JavaScript animation that end up showing that element, so in this case you will need to do this:

$I->waitForElementVisible('#joomla_site');
$I->click('go to joomla');

When Not to Use Contains[edit]

We do not recommend to use the function contains on XPath because is not specific and can create false positives at tests. See: https://github.com/joomla-projects/GSOC-Webdriver_system_tests_for_CMS/pull/99#issuecomment-69004191

Always add Strict Locators to your tests[edit]

Performance in tests is very important. You can reduce dramatically the time spent on a test with just adding a few extra locators. For example, the following code pertains to a test:

// see($text, $selector = null)
$I->see('Your article has been published');

In the previous code we are not facilitating the "Selector" so Codeception will have to search for the element in all the HTML. Doing this instead:

$I->see('Your article has been published', ['id' => 'messages']);

the tests went from: Time: 1.17 minutes to: Time: 49.18 seconds

You can see this video of a real example: https://www.youtube.com/watch?v=0JCv0BR2yZY

Losing 30 seconds in every Assertion can delay your tests dramatically. Therefore, remind always to add XPath locators to all your Assertions.

Allowed locator mechanisms are:

- ['id' => 'xxx']
- ['name' => 'xxx']
- ['css' => 'xxx']
- ['xpath' => 'xxx']
- ['link' => 'xxx']
- ['class' => 'xxx']

More info: https://codeception.com/docs/modules/WebDriver#Locating-Elements

Using Locators in $I->see[edit]

Do not use see without locator:

// Wrong:
$I->see('Your article has been published');

// Right:
$I->see('Your article has been published', ['id' => 'messages']);

Using Locators in $I->waitForText[edit]

Do not use waitForText without locator:

// Wrong:
$I->waitForText('Finalisation');

// Right:
$I->waitForText('Finalisation', 10, ['css' => 'h3']);

In the case of an element that has multiple CSS classes, make sure you use more specific class as the locator:

// Wrong:
$I->waitForText('Item Successfully Created', 10, ['class' => 'alert']);

// Right:
$I->waitForText('Item Successfully Created', 10, ['class' => 'alert-success']);

A page could have multiple elements which can have alert CSS class associated, hence being more specific helps.

How Much Time Should I Wait for an Element or Text?[edit]

You may have noticed that the second parameter in the I waitForText is an integer that represents how much "time to wait" in seconds:

$I->waitForText('a text to wait for','60',['...' => '...']);

But, how much time is the right time to wait for a text?

PHP maximum execution time before returning a Fatal Error for timeout is set by default to 30 seconds. We suggest to set it for 60 seconds, because running the tests in slow computers can easily get to that time in some tasks.

If a waitForText command takes over 10 seconds, it will much probably mean that something is not performing well. Maybe a non Strict Locator is being used and Selenium takes to much time to find the element. However we have even identified cases were some steps take longer than 30 seconds in very slow computers, specifically installing Joomla, see https://github.com/joomla-projects/joomla-browser/commit/b70f195140e599649cf98f051d7a31388c4f1632. For this cases we can't do much, other than recommend to add more memory to the local web server.

Quote and Double Quotes in Complex XPaths[edit]

Sometimes you will have to write XPaths that looks like:

//button[@onclick='Joomla.submitbutton("config.save.application.apply")']

When moved to a Page Object you will probably will face this situation:

Typical issue when you write a long XPath in a Page object

To avoid it remind to escape the quotes:

// Wrong
public static $elements = array(
   'Save' => "//button[@onclick="Joomla.submitbutton('config.save.application.apply')"]"
);
// Right
public static $elements = array(
   'Save' => "//button[@onclick=\"Joomla.submitbutton('config.save.application.apply')\"]"
);

Generating txt/html Scenarios[edit]

Codeception allows you to generate user-friendly text scenarios from scenario-driven from Cest and Cept PHP test (see https://codeception.com/docs/reference/Commands#GenerateScenarios).

Don't Use Double Quotes; Use Single Quotes Instead[edit]

Do not use double quotes (") for surrounding strings:

$I->waitForText("Congratulations! Joomla! is now installed.", 30);

If you do, when you generate the scenarios you will get this bad result:

I wait for text ""Congratulations! Joomla! is now installed.""," 30"

If you use single quotes instead:

$I->waitForText('Congratulations! Joomla! is now installed.', 30);

You will get the right expected result:

I wait for text 'Congratulations! Joomla! is now installed.'," 30"

The reason of not being able to use double quotes is due to a bug in Codeception humanize functions, see issue: https://github.com/Codeception/Codeception/issues/1847

Do Not Leave Spaces Between Parameters in Cepts[edit]

See the two lines:

// Wrong
$I->waitForText('Main Configuration', 10, 'h3');
// Good
$I->waitForText('Main Configuration',10,'h3');

If you do, when you generate the scenarios you will get this bad result:

I wait for text 'Main Configuration'," 10"," 'h3'

The reason that you are asked to not add spaces between parameters is due to a bug in Codeception humanize functions, see issue: https://github.com/Codeception/Codeception/issues/1847

Write Complex Locator Vs. Modifying the Output of the Tested Website[edit]

Sometimes we need to write a very complex XPath to locate a element in the page, for example:

$I->click(['xpath' =>  "//*[@id=\"menuList\"]/tbody/tr/td[2]/a[contains(text(), 'weblink')"])

wouldn't be much better to write something like this instead?

$I->click(['class' => "test-weblinks-menu_item_type_weblinks"])

When you find yourself writing a very complex locator in a test, is maybe because the markup of the Website is wrong. For this cases, is better to stop writing the test and reconsider the possibility of modifying the website that we are testing. There is nothing that prevents us to improve Joomla markup. Actually is the other way around, with these fixes we are probably making Joomla more accessible. We have had experiences in the past, see for example this issue: https://github.com/joomla/joomla-cms/issues/7207.

In conclusion is much more convenient to add just a simple class to the element to easily locate it instead of becoming mad finding the right XPath. Remind that is important that our tests are readable, and that has to do with using readable locators.

In order to simplify locators used in Acceptance tests we add special locator classes into HTML of web pages. So not to confuse them with classes needed by CSS or JavaScript we use test-*projectname*- prefix. Thus, if you find locator to be to complex, it would be much easier to edit HTML and add a new class into it.

$I->click('class' =>  "test-projectname-comment-delete');

Where projectname stands for the name of the project in which we are adding this locator.

It is important that the rest of Joomla coders know that this test-projectname- prefix is used by the tests to prevent anyone from removing it.

This recommendation was originally suggested by Davert (the creator of Codeception) at http://phptest.club/t/codeception-tips-tricks/14/8.

Thoughts on Selenium IDE and the Codeception Formatter[edit]

Selenium IDE is a Firefox extension that allows to generate tests in a interactive way:

Seleniumide.png

One can use the Selenium IDE with the codeception Formatter to export the tests to Codeception Framework Format.

Exportcodeception.png

Using the Selenium IDE to write the tests seems way easier than writing them in PHP (since you have immediate feedback on what's going on the page), but it also poses some issues.

For the most part the Joomla-Browser has functions that we can use to do standard stuff like login, logout etc. If we do our tests with the Selenium IDE we cannot use those and our tests get filled up with stuff that we have already abstracted.

There is still use of the Selenium IDE though, we could use it to create the tests for specific functionality. Like just writing an article, setting publish_up and publish_down dates and then saving. When we export the result with the Codeception Formatter we would have the necessary PHP and we can add the rest of the stuff for the test to be fully automated (login, navigate to page etc, then execute the exported test). This could speed the creation of tests and we could use non-developers for this kind of work. However this is easier said than done. Due to the JavaScript that we use in Joomla simple tests are not that simple. For example clicking in the text editor and writing text in it is not registered by the IDE since the editor is created by TinyMCE... A way to add text to the editor is to use

Command | runScript
Target | tinyMCE.get('text_area_id_not_iframe_id').setContent('Your text') 

That is again a technical solution to a test that should have been easy in the first place. We get into the same trouble with selecting a date out of a Calendar field. So at the end it seems pretty difficult for non-developers to create tests that we could use, because most of the steps recorded by the Selenium IDE won't work a second time.

If we find a way to make the work with JavaScript elements on the page easier, then Selenium IDE could be a great tool that could help us write far more tests, than we currently have in a less time.

As far the Codeception Formatter is concerned - it hasn't been updated in 2 years, but if we find out how to use the Selenium IDE extending the Formatter and building on top of it shouldn't be a show stopper.

BDD Testing Using Gherkin and Codeception[edit]

BDD (Behavior-driven development) in its grandest sense is about communication and viewing your software as a system with behaviour. Using BDD approach to write test cases make test cases easy and understandable. BDD test cases can be understood by non developers and testers.

What is the Gherkin Language[edit]

Gherkin is the language that codeception uses to define test cases. Test cases were designed to be non-technical and human readable, and collectively describes use cases relating to a software system.

Gherkin Syntax[edit]

Gherkin is language in which we need to write statements in simple English but we need to follow below syntax guidelines:

  • Gherkin is a line-oriented language most lines start with a special keyword.
  • Lines can be comment by using hash sign(#).
  • Gherkin test cases are saved in “.feature” file format.
  • You can write gherkin in any editor (Like PhpStorm, Sublime,Gherkin editor, and any IDE).
  • Use can use Tags (“@”) to organize features and scenarios.

Feature[edit]

Descriptive text of what is desired from all the test cases inside that “.feature” file. Basically it is a description of feature which is going to cover by below test cases.

The general format of a feature is:

Feature: <short description>
 In order to…
 As a…
 I need to...

Within the definition of a feature you can provide a plain text description of the story. You can use any format for this, but having some sort of template makes it easier to see the important bits of information with a quick glance. This format focuses us on three important questions:

  • Who's using the system?
  • What are they doing?
  • Why do they care?

A Sample Story[edit]

Feature: users
 In order to manage users in the web
 As an owner
 I need to create edit block unblock and delete user

Background[edit]

Background contains set of steps that all scenario need to perform at start.

Here’s an Example of Background

Background:
 Given Joomla CMS is installed
 When Login into Joomla administrator with username "admin" and password "admin"
 Then I see administrator dashboard

Scenario[edit]

Description of test case what test case is going to test, In short it is test case name with some description. A scenario is a sequence of steps through the feature that exercises one path.

A scenario is made up of 3 sections related to the 3 types of steps

  1. Given: This sets up preconditions, or context, for the scenario.
  2. When: This is what the feature is talking about, the action, the behaviour that we're focused on.
  3. Then: This checks postconditions. It verifies that the right thing happen in the When stage.

Here’s an Example of scenario

Scenario: Create User from Backend
 Given There is a add user link
 When I create new user with fields Name "register", Login Name "register", Password "register" and   Email   
             "register@gmail.com"
 And  I Save the  user
 Then I see the "User successfully saved." message

And: If we have consecutive “given, when or then” then instead of using these we use “And” statement to make it more readable.

But: Same as “And” but used in negative prospect.

Scenario: Create user from frontend 
 Given I click on the link "Create an account"
 And I create a user with fields Name "patel", Username "patel", Password "patel" and Email "patel@gmail.com"
 When I press the "Register"
 Then I see "Could not instantiate mail function." message
 And  user is created

Given/When/Then/But/And Best Practice[edit]

The scenarios within our `.feature files` consist of steps, also known as givens, whens, buts, thens and ands.

Technically these kind of steps are the same. When we executing the feature, each step is matched to a PHP function.

Example:
`Given There is an user link,`
`When I see the user edit view tabs`
and
`Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"`

are the steps in the Scenario:

`Verify available tabs in com_users `
of the feature-file `/tests/codeception/acceptance/users.feature`.

  Scenario: Verify available tabs in com_users
    Given There is an user link
    When I see the user edit view tabs
    Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"

The step
`Given There is an user link`
is matched to the PHP function `thereIsAAddUserLink()`,
the step
`When I see the user edit view tabs`
is matched to the PHP function `iSeeTheUsereditViewTabs()`
and the step
`Then I check available tabs "Account Details", "Assigned User Groups" and "Basic Settings"`
is matched to the PHP function `iCheckAvailableTabs($tab1, $tab2, $tab3)`
of the file `/tests/codeception/_support/Step/Acceptance/Administrator/User.php`

	public function thereIsAAddUserLink()
	{
		$I = $this;

		$I->amOnPage(UserManagerPage::$url);
		$I->adminPage->clickToolbarButton('New');
	}


	public function iSeeTheUsereditViewTabs()
	{
		$I = $this;

		$I->adminPage->clickToolbarButton('New');
		$I->adminPage->waitForPageTitle('Users');
	}
	public function iCheckAvailableTabs($tab1, $tab2, $tab3)
	{
		$I = $this;

		$I->adminPage->verifyAvailableTabs([$tab1, $tab2, $tab3]);
	}


So, why are there these different step names?

Behaviour-driven development (BDD) means that you can turn an idea for a requirement into code simply, as long as the requirement is specific enough that everyone knows what’s going on.
We need a way to describe the requirement so that everyone – user and developer – have a common understanding of the scope of the software.

Our tests can only be successful if we have the same semantic understanding of the steps Given, When and Then!

Given[edit]

The purpose of Given is to put the system in a known state. What is the context of the system? What pre-condition must be true before the action is tested? What relevant data is in the system?

Important in this context: In most of our .feature files we use the the section Background. The background is run before each of the scenarios in our .feature file and also has impact on the state.

Examples:
Given There is a add content link
Given I search and select the user with user name "register"
Given I am on the User Manager page
Given User "User Three" did login at least once

When[edit]

The purpose of When is to describe the key action that will be performed. What will be tested? What relevant input data will be tested when the action occurs?

Examples:
This could be only one action like in the Scenario
Delete ACL

  Scenario: Delete ACL
    Given I search and select the Access Level with name "Acl Two"
    When I Delete the Access level "Acl Two"
    Then I should see "No Matching Results" for deleted user "Acl Two"

This could multiple actions in succession like in the Scenario
Check if block and activation are working:

  Scenario: Check if block and activation are working
    Given I am on the User Manager page
    When I unblock the user "User Two"
    And I activate the user "User Two"
    And I login with user "user2" with password "pass2" in frontend
    Then I should see the message "Hi User Two,"

Then[edit]

The purpose of Then is to observe outcomes. The observations should be related to the business value/benefit in our feature description. In response to the action, what is the observable outcome? What are the expected results? What is the post-condition or output data observable by the user?

Examples:
Then I should see the access level "Acl Two" is created
Then Login in backend with username "User One" and password "pass1"

Please note: We should only verify outcome that is observable for the user. It is not enough to verify that all entries in the database are correct!

And, But[edit]

If we have several `givens`, `whens` or `thens` we could write

  Scenario: Create User without username fails
    Given There is a add user link
    When I don't fill Login Name but fulfill remaining mandatory fields: Name "User Two", Password "pass2" and Email "user2@example.com"
    When I Save the user
    Then I see the title "Users: New"
    Then I see the alert error "Invalid field:  Login Name"

But we make it read more fluently by writing

 Scenario: Create User without username fails
    Given There is a add user link
    When I don't fill Login Name but fulfill remaining mandatory fields: Name "User Two", Password "pass2" and Email "user2@example.com"
    And I Save the user
    Then I see the title "Users: New"
    But I see the alert error "Invalid field:  Login Name"

Steps beginning with And or But are exactly the same kind of steps as all the others.

Implementing Steps[edit]

Run this command

tests/codeception/vendor/bin/codecept gherkin:snippets acceptance

That will output all the "step" methods in .feature files. Copy all this snippets and put into step file.

Gherkin-steps-en.jpg

How to Create a step file?

tests/codeception/vendor/bin/codecept generate:stepobject acceptance Administrator/users

Execute feature file

tests/codeception/vendor/bin/codecept run tests/acceptance/users.feature --steps

Execute feature file using robo

tests/codeception/vendor/bin/robo run:tests