Parallel Testing

GSoC 2017
Parallel Testing Project Documentation

Joomla!

3.x

Introduction[edit]

The general purpose of the project is getting early feedback for every newly submitted code for Joomla extensions. By creating a fully working environment to run tests for Joomla, so that they can run in a parallel (container-based) environment, increasing speed and coverage of PHP and Joomla versions. The key requirement here is speed, it is crucial to get early feedback on new PRs. In order to achieve this, tests not dependent on each other will be run on different containers simultaneously.

The scope of the project is to integrate the Joomla! Weblinks Extension existing tests into the new testing environment. Therefore, pre-installed Joomla containers need to be used in order to be able to run the tests in a short time.

The expected results are to automate the testing environment from its creation (by running containers with Joomla/PHP combinations and testing selenium containers), to parallel test execution (by coordination test runs in parallel and taking into account their dependencies) and reporting (storing logs and screenshots for each failed test).

Prerequisites[edit]

In order to run the parallel testing module, make sure you have docker installed and running. On top of that, be sure that the user that will run the project does not need sudo to run a docker command.

All the dependencies are loaded through composer and all other prerequisites are loaded in containers by the virtualisation project: memcached, mysql, Joomla PHP servers, selenium servers.

Last, but not least, in order to load the tests and their dependencies, they need to be written in a special way for each extension. Currently for Weblinks this is the template example:

install/InstallWeblinksCest.php:installWeblinks:
  administrator/AdministratorCategoriesCest.php:administratorCreateCategoryWithoutTitleFails:
  administrator/AdministratorCategoriesCest.php:administratorVerifyAvailableTabs:
  administrator/AdministratorSmartSearchCest.php:administratorEnableSmartsearchWeblinksPlugin:
  administrator/AdministratorSmartSearchCest.php:administratorPurgeIndex:
  administrator/AdministratorSmartSearchCest.php:administratorCreateWeblink:
  administrator/AdministratorSmartSearchCest.php:administratorRunTheIndexer:
  administrator/AdministratorSmartSearchCest.php:administratorDisableContentPlugin:
  administrator/AdministratorSmartSearchCest.php:administratorDisableSmartsearchWeblinksPlugin:
  administrator/AdministratorSmartSearchCest.php:cleanUp:
  administrator/AdministratorWeblinksCest.php:administratorVerifyAvailableTabs:
  administrator/AdministratorWeblinksCest.php:administratorCreateWeblink:
    administrator/AdministratorWeblinksCest.php:administratorTrashWeblink:
    administrator/AdministratorWeblinksCest.php:administratorDeleteWeblink:
    administrator/AdministratorWeblinksCest.php:administratorCreateWeblinkWithoutTitleFails:
  frontend/FrontendWeblinksCest.php:createWeblinkAndConfirmFrontend:
  frontend/FrontendWeblinksCest.php:hitsAreNotIncrementedIfCountClicksIsOff:
  frontend/FrontendWeblinksCest.php:hitsAreIncrementedIfCountClicksIsOn:

Install main project dependencies[edit]

The first step in testing the parallel testing project, is cloning its main repository, joomla-testing, on the container-test branch.
The only other step is to navigate in its main folder and run composer install.

Start the testing environment and run the acceptance tests for Weblinks[edit]

First of all, the purpose of the project is to run the acceptance tests for an extension and in the future to be integrated in a CI tool, such as Travis. Therefore, Windows was not considered for the first iteration of the project, as it can only be used for development and local testing, not production.

In order to make the use of the project easy, a single Robo command is needed to run everything.

In the root folder, run the following command:

vendor/bin/robo run:coordinator $repoOwner $repoName $repoBranch > coordinator.log 2>&1

Where

$repoOwner, $repoName, $repoBranch

shall be replaced with the corresponding desired extension information.

In order to use the project for the Weblinks extension, the following tested command can be used:

vendor/bin/robo run:coordinator isacandrei weblinks container-test > coordinator.log 2>&1

The tests results are stored in the coordinator.log file:

start
apachev7p1v3p7_1 execute install/InstallWeblinksCest.php:installWeblinks with client seleniumv0_1
apachev7p1v3p7_1 assign admin/adminCategoriesCest.php:adminCreateCategoryWithoutTitleFails
apachev7p1v3p7_1 assign admin/adminCategoriesCest.php:adminVerifyAvailableTabs
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminEnableSmartsearchWeblinksPlugin
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminEnableSmartsearchWeblinksPlugin with client seleniumv2_1
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminPurgeIndex
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminCreateWeblink
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminRunTheIndexer
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminPurgeIndex with client seleniumv2_1
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminDisableContentPlugin
apachev7p1v3p7_1 execute admin/adminCategoriesCest.php:adminCreateCategoryWithoutTitleFails with client seleniumv0_1
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:adminDisableSmartsearchWeblinksPlugin
apachev7p1v3p7_1 execute admin/adminCategoriesCest.php:adminVerifyAvailableTabs with client seleniumv1_1
apachev7p1v3p7_1 assign admin/adminSmartSearchCest.php:cleanUp
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminCreateWeblink with client seleniumv2_1
apachev7p1v3p7_1 assign admin/adminWeblinksCest.php:adminVerifyAvailableTabs
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminRunTheIndexer with client seleniumv0_1
apachev7p1v3p7_1 assign admin/adminWeblinksCest.php:adminCreateWeblink
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminDisableContentPlugin with client seleniumv1_1
apachev7p1v3p7_1 assign frontend/FrontendWeblinksCest.php:createWeblinkAndConfirmFrontend
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:adminDisableSmartsearchWeblinksPlugin with client seleniumv2_1
apachev7p1v3p7_1 assign frontend/FrontendWeblinksCest.php:hitsAreNotIncrementedIfCountClicksIsOff
apachev7p1v3p7_1 execute admin/adminSmartSearchCest.php:cleanUp with client seleniumv0_1
apachev7p1v3p7_1 assign frontend/FrontendWeblinksCest.php:hitsAreIncrementedIfCountClicksIsOn
apachev7p1v3p7_1 execute admin/adminWeblinksCest.php:adminVerifyAvailableTabs with client seleniumv1_1
apachev7p1v3p7_1 execute frontend/FrontendWeblinksCest.php:createWeblinkAndConfirmFrontend with client seleniumv0_1
apachev7p1v3p7_1 execute frontend/FrontendWeblinksCest.php:hitsAreIncrementedIfCountClicksIsOn with client seleniumv0_1
apachev7p1v3p7_1 execute frontend/FrontendWeblinksCest.php:hitsAreNotIncrementedIfCountClicksIsOff with client seleniumv1_1
apachev7p1v3p7_1 execute admin/adminWeblinksCest.php:adminCreateWeblink with client seleniumv2_1
apachev7p1v3p7_1 assign admin/adminWeblinksCest.php:adminTrashWeblink
apachev7p1v3p7_1 assign admin/adminWeblinksCest.php:adminDeleteWeblink
apachev7p1v3p7_1 assign admin/adminWeblinksCest.php:adminCreateWeblinkWithoutTitleFails
apachev7p1v3p7_1 execute admin/adminWeblinksCest.php:adminDeleteWeblink with client seleniumv1_1
apachev7p1v3p7_1 execute admin/adminWeblinksCest.php:adminTrashWeblink with client seleniumv0_1
apachev7p1v3p7_1 execute admin/adminWeblinksCest.php:adminCreateWeblinkWithoutTitleFails with client seleniumv2_1
Results on server apachev7p1v3p7_1 are:
Executed18
Failed 0
Total 18

And the detailed codeception logs, screenshots and HTML code for failed tasks are stored in separate folders for each server in the _output folder of the tested extension:

The magic inside[edit]

Virtualisation[edit]

The virtualisation repository has been updated and now supports the Memcached container. The way virtualisation works, is by having 4 default xml configuration files:

database.xml
default.xml
selenium.xml
network.xml

Each of them defining configuration for the database, the Joomla servers, memcached, selenium containers and last but not least, the network they all work on.

In order to be easily used in other projects (such as this one), an API has been defined for the virtualisation project. The API expects and environment configuration which is used to extend and override the configuration in the above defined xmls.

This would be a typical environment configuration:

        $env = array(
            'php' => ['5.4', '5.5', '5.6', '7.0', '7.1'],
            'joomla' => ['3.6'],
            'selenium.no' => 3,
            'extension.path' => $tmpDir . '/extension',
            'host.dockyard' => '.tmp/dockyard',
        );

In this way, all the containers needed for parallel testing are created.

Selection List[edit]

The selection list main purpose is to load and maintain the order in the tests needed to be run for an extension. It makes sure that it serves for running only test ready for execution, ensuring their dependencies already succeeded, or marking the task as failed otherwise.

In order to maintain the test's’ status, three(four) flags are used:

final class Flag
{
    const NO_FLAG = 0;
    const ASSIGNED = 1;
    const EXECUTED = 2;
    const FAILED = 3;
}

Now, before the selection list is able to load the tests, the extension should have defined a tests.yml file for their acceptance tests. The file should contain all the tests needed to be run, and the indentation represents the dependencies between tests. If a test has two disjunct dependencies, then it needs to be written twice, indented accordingly to the required dependencies. For Weblinks, there is no such case, but the solution is presented nevertheless. Below is an example of a shortened tests.yml file for weblinks:

install/InstallWeblinksCest.php:installWeblinks:
  administrator/AdminCategoriesCest.php:adminCreateCategoryWithoutTitleFails:
  administrator/AdminCategoriesCest.php:adminCreateCategory:
    administrator/AdminCategoriesCest.php:adminPublishCategory:
      administrator/AdminCategoriesCest.php:adminUnpublishCategory:
  administrator/AdminCategoriesCest.php:adminVerifyAvailableTabs:
  administrator/AdminWeblinksCest.php:adminCreateWeblink:
    administrator/AdminWeblinksCest.php:adminTrashWeblink:

A recursive read function has been defined to read the yml file and store the tests data in memory. The data is not normalised, for the ease of future operation, tasks being stored both in a simple list (for the ease of checking their dependencies) and in a map having their flag as key (for the ease of managing the selection process). An exponential improvement in complexity has been made with just a scalar increase in memory (*2).

After the read has been done, the next important responsibility of the selection list is to “pop()” tests ready for execution. If there is one task that has no flag and all its dependencies is executed then it is returned, otherwise, “false” is returned.

Because all the behaviour of this project (tests run) is async, an isFinished method has been defined. This gives us the information when to write in the logs the overall results of the tests for each selection list, which ultimately represents a server.

Main Coordinator[edit]

Tests need to run asynchronous, therefore, in order to keep the information required for their execution, such as the selection lists, a cache (Memcached) is used a persistent storage. Therefore, MainCoordinator queries the cache every time it requires to perform actions on the selection lists and writes the information back afterwards. In order to avoid the concurrency issues, a simple locking system has been implemented.

The first responsibility of MainCoordinator is to prepare the data required for tests execution. First, it loads the selection lists, then it creates the runQueue and manageQueue. As discussed in the initial proposition, runQueue stores the tasks that are ready for execution, but in the same time is limited to the number of clients. manageQueue is used to keep the tests coverage balanced across the servers, therefore ensuring maximum efficiency. Furthermore, codeception configuration files are created for each server in order to reuse the same weblinks clone and store failed screenshots and logs separately.

With the use of the virtualisation API, the testing environment is created and started. Before tests are running, MainCoordinator waits for the database initialisation to finish.

Lastly, MainCoordinator is in charge of filling the execution queue and running the available tasks. The filling of the execution queue is done until the maximum capacity is reached, and the tests from the server who last executed one have priority. The running of tasks is performed as long as clients (selenium containers) are available and as long as tasks in the execution queue are available.

Robo Run Test Task[edit]

In order to make the tests async, a robo task had to be created. The execution of a task is basically a “docker exec” command which runs a codeception test inside a client container. The result is then verified and in case of success, the test is marked with the “executed” flag and the whole fill and run flow is reloaded, all async, without knowing the status of the other clients or tests at the moment. In case of failure, an additional log is stored with all the codeception output.

Robo Run Coordinator Task[edit]

The design of the project implies it to be run by Travis or other CI tool. Therefore a robo task would be best suited to the job. It expects as arguments the repository owner, name and branch and shall be used as follows:

vendor/bin/robo run:coordinator isacandrei weblinks container-test

In this version, the environment configuration needs to be defined inside the implementation of the robo task, but later it shall be defined externally in order to allow the different needs of testing.

Overall diagram[edit]

PR Commits[edit]

Future Improvements[edit]

Integrate with Travis
Fix all tests for newer Joomla! Versions
Update Selenium container
Fix tests for php 5.5, 5.6
Fix the rest of acceptance tests for Weblinks to run on Joomla 3.7+ (2 out of 25 still fail)