PR Testing Platform

The printable version is no longer supported and may have rendering errors. Please update your browser bookmarks and please use the default browser print function instead.

GSoC 2017
Pull Requests Testing Platform
Project Documentation

Joomla!

3.x

Introduction[edit]

The main problem/limitation this project aims to solve is the constraints there are still regarding explorative tests like hardware problems, time consumption, repetitive and somewhat complex setup and deployment efforts (complex for most of the testers with less technical knowledge).

This limitation leads to a certain loss in human resources for testing Joomla, and that is certainly not ideal, even more for such an inclusive, multicultural and community oriented organization like Joomla! so this project aims to bring us closer to those ideals.

The main goal here is to automate the creation/setup of a testing environment for Joomla! CMS and with this shift the tester’s focus entirely to what really matters and test Joomla! instances almost instantaneously.

In the end the tester won’t need to configure his/her environment manually and would no longer need a more technical knowledge regarding the setup and deployment processes, would also benefit more technically advanced users as well because it also saves them from having a local Joomla! Installation and saves them time as well.

We have a tests requests website based on the issue tracker with GitHub authentication where the user can request a test for a PR in particular, after choosing the PR to test the user can choose the PHP version and then waits for the Joomla! instance to be prepared having a view for the instance with all the info regarding that instance plus a no-patch instance for the target branch which the user can use for comparison with the PR instance.

You can check the working prototype here: PR Testing Platform

Before going into the setup process in more detail let's go over each part of the platform.

The platform is composed of 3 parts, the Jenkins setup, the multi-container setup based on Devilbox and the website for the tests requests.

Jenkins[edit]

Jenkins's role here is to be our connection to the Github API and the repository, it's intended to build any incoming Pull Requests / Commits. Jenkins is a cross platform CI tool for automation with a large base of plugins which support building, deployment and automation of any project.

For this project the Github Pull Request builder plugin is used in order to build pull requests.

We are running builds in Jenkins server based on a set of events like creating a PR, pushing commits, commenting PRs, etc. A Jenkins project will be created to run these builds for Github pull request events like create, push, comment, close and reopen.

Besides the main job to build the PRs there will also be another Jenkins job to build the PR's target branch for comparison later on the tests requests website.

Multi-container Setup[edit]

For the multi-container setup Docker Compose was used and the platform stack was based on Devilbox which is a highly customisable LAMP/LEMP and MEAN stack replacement based purely on docker and docker-compose that supports an unlimited number of projects for which vhosts and DNS records are created automatically. You can learn more about Devilbox on its github repository documentation

I heavily modified the Devilbox stack in order to allow having 3 different PHP version services each with its webserver behind a proxy which distributes the requests according to the php version embedded in the instances URL. The next diagram exemplifies how the services stack is set up according to the docker-compose:

So in the docker compose for this project there are 14 containers running, each one with a function:

Bind DNS server container: The devilbox uses its own DNS server internally (your host computer can also use it for Auto-DNS) in order to resolve custom project domains defined by TLD_SUFFIX in the .env file. To also be able to reach the internet from inside the container there must be some kind of upstream DNS server to ask for queries.

PHP 5.6, 7.0 and 7.1 containers: The php containers are the center of all the containers, everything happens in there and all the instances will be located inside these 3 containers. All the php modules and tools will be made available inside. The remote ports and remote sockets are made available to the php container by using socat. It uses socat to forward the remote mysql, pgsql, redis and memcached ports (3306, 5432, 6379 and 11211 respectively) on each of these 4 containers to its own at 127.0.0.1 (127.0.0.1:3306, 127.0.0.1:5432, 127.0.0.1:6379, 127.0.0.1:11211).

Webserver container for each PHP version: These are the Apache/Nginx webserver (currently only working with Nginx, changed only config files from Nginx for this project) containers for each of the PHP version containers, each one is linked to the respective php-fpm container through each php-fpm socket, each on the same port (9000).

Jenkins container: Jenkins was added to the service stack for this docker compose. This is the container where everything Jenkins related will be located, the Jenkins jobs and the workspace where the code will be pulled into for each job.

JWilder Nginx Proxy: This is the container for the automated nginx proxy for Docker containers using docker-gen which sets up a container running nginx and docker-gen. docker-gen generates reverse proxy configs for nginx and reloads nginx when containers are started and stopped. This reverse proxy was needed in order to forward the requests correctly according to the php version of the URL that's being accessed. There was a problem with knowing which of the webservers to send the requests because there was no way to know to what version an URL corresponded. I solved that issue by placing the php version on the url and using a regular expression in the VIRTUAL_HOST to proxy the requests for each php version to the right webserver container (''~^56.*\.dbox-tests\.ml$$'', ''~^70.*\.dbox-tests\.ml$$'' and ''~^71.*\.dbox-tests\.ml$$'').

Letsencrypt Nginx Proxy Companion: Container for SSL Support using letsencrypt. The letsencrypt-nginx-proxy-companion is a lightweight companion container for the nginx-proxy. It allows the creation/renewal of Let's Encrypt certificates automatically side by side with JWilder's nginx proxy. Currently it only generates a certificate for the main tests requests website and for Jenkins, the Joomla! instances URLs have no SSl certificates yet because the way the proxy e configured with the regular expressions the only way to have SSL certificates for those instances without constantly having to bring down the docker compose would be to issue wildcard SSL certificates and letsencrypt will only allow this in 2018

Memcached, PostgreSQL, MySQL and Redis containers: These are the containers for the PostgreSQL and MySQL databases, Memcached memory object caching system and Redis which is an open source, in-memory data structure store, used as a database, cache and message broker.

Besides all of these services in the docker compose, there is also a bridge docker network in order to allow the containers that are connected to it to communicate with each other via IP address. Each container has an assigned internal IP address and is connected to that network in the docker-compose file.

###############################
# NETWORK
###############################
networks:
  app_net:
    driver: bridge
    driver_opts:
      com.docker.network.enable_ipv6: "false"
    ipam:
      driver: default
      config:
        - subnet: 172.16.238.0/24
          gateway: 172.16.238.1

To bring up the docker compose just execute the docker-compose up -d command, to bring down the docker compose execute the docker-compose down command and to look at the logs from the containers execute docker-compose logs -f or docker-compose logs command. Also to list the containers running from the docker compose run docker-compose ps. Should you want to update the images from each container execute the update-docker.sh script in the docker-compose base folder.

Tests Requests Website[edit]

The part from the last phase is the Tests Requests Website which was based on the Issue Tracker, basically added an MVC app to the tracker for the Joomla! instances, changed a bit the script to fetch the issues from the issue tracker to only fetch PRs and added a field to the Issues table to know which PRs are mergeable or not fetching only the mergeable ones (the ones with no conflicts).

In each instance view there is some info about the instance like PHP version, target branch, link to the PR and login credentials. You also have the buttons to go to the instance and another to an instance with the target branch for that PR in order to compare the Patched Joomla! instance with the No Patch instance.

The instances are added to and removed from its respective PHP version container by using docker exec to enter those containers from inside the container where the website is located. This is achievable by sharing the docker socket descriptor file and the docker executable as read only to that container in order to be able to use docker commands within a container. JWilder nginx proxy does something similar in order to automatically generate the configuration file based on the existent containers in the same network.

The website is located inside one of the PHP version containers (PHP 7.1).

Initial Setup[edit]

After downloading the code from the project's repository you have to setup Docker and Docker Compose on your machine or server and setup the permissions on the folders and files. It's important from this point forward that all steps are thoroughly followed. It's also important to create a Github Repository where you can make PRs with Joomla code and a Github user for tests.

Docker & Docker Compose[edit]

Docker CE[edit]

First thing to install is Docker, consider for these instructions an Ubuntu server. The documentation explains very well the steps to install Docker CE, you can check here for Ubuntu and for other OS's check here.

For installing on Ubuntu follow the next steps which are the same you can find in the docker documentation:

Update the apt package index:
```
$ sudo apt update
```

Install packages to allow apt to use a repository over HTTPS:

$ sudo apt install \
    apt-transport-https \
    ca-certificates \
    curl \
    software-properties-common

Add Docker’s official GPG key:

$ curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

Verify that the key fingerprint is 9DC8 5822 9FC7 DD38 854A E2D8 8D81 803C 0EBF CD88 .

$ sudo apt-key fingerprint 0EBFCD88
pub   4096R/0EBFCD88 2017-02-22
      Key fingerprint = 9DC8 5822 9FC7 DD38 854A  E2D8 8D81 803C 0EBF CD88
uid                  Docker Release (CE deb) <docker@docker.com>
sub   4096R/F273FCD8 2017-02-22

Use the following command to set up the stable repository. You always need the stable repository, even if you want to install builds from the edge or test repositories as well. To add the edge or test repository, add the word edge or test (or both) after the word stable in the commands below.
Note: The lsb_release -cs sub-command below returns the name of your Ubuntu distribution, such as xenial. Sometimes, in a distribution like Linux Mint, you might have to change $(lsb_release -cs) to your parent Ubuntu distribution. For example, if you are using Linux Mint Rafaela, you could use trusty.
1. amd64:
```
$ sudo add-apt-repository \
   "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
```
2. armhf:
```
$ sudo add-apt-repository \
   "deb [arch=armhf] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
```
3. s390x:
```
$ sudo add-apt-repository \
   "deb [arch=s390x] https://download.docker.com/linux/ubuntu \
   $(lsb_release -cs) \
   stable"
```

Now to install Docker CE:

Update the apt package index:
```
$ sudo apt update
```
Install the latest version of Docker CE.
```
$ sudo apt install docker-ce
```

Docker Compose[edit]

Now to install Docker Compose you can check the tutorial here or follow the next few steps that are explained in the tutorial:

First check the latest stable release here.

Copy the release name (1.15.0 for example) and execute the next command by placing the right release:

$ sudo curl -o /usr/local/bin/docker-compose -L "https://github.com/docker/compose/releases/download/1.15.0/docker-compose-$(uname -s)-$(uname -m)"

Next set the permissions:

$ sudo chmod +x /usr/local/bin/docker-compose

Then verify if the installation was successful by checking the version:
```
$ docker-compose -v
```
This should print out the version that was installed like this:
```
docker-compose version 1.15.0, build e12f3b9
```

Folders and Files Permissions[edit]

Now that Docker is installed the next step, and a very important one, is to correctly set up the permissions on the folders, sub-folders and files on the docker-compose.yml base folder.

cd over to where the docker-compose.yml file is located.
Give 755 permissions to every folder and subfolder in the folder you are in.
```
$ find ./ -type d -exec chmod 755 {} \;
```
Give 644 permissions to every file in every folder and subfolder in the folder you are in.
```
$ find ./ -type f -exec chmod 644 {} \;
```
Add your user to www-data group and make www-data user and group owner of the project folder:
```
$ usermod -a -G www-data youruser && chown -R www-data:www-data ./
```
Add your user to the docker group and add the user as owner of the docker socket file so that the user inside the container can execute docker commands:
```
$ usermod -aG docker youruser && chown 1000:docker /var/run/docker.sock
```
Still in the docker-compose.yml base folder give 777 permissions to all bash scripts there and to all files inside the files/ folder.
```
$ chmod 777 *.sh && chmod -R 777 files/
```

Docker Compose Environment Setup[edit]

Now that the initial setup is done time to configure our docker-compose.yml and .env files before bringing the docker compose online.

Edit the .env file and look for the TLD_SUFFIX environment variable, place your domain suffix (for a domain.com you would place TLD_SUFFIX=com) and then save the file.
Edit the .env file, look for the MYSQL_ROOT_PASSWORD and the PGSQL_ROOT_PASSWORD environment variables and choose a password for the mysql and postgres users (it is recommended that a strong password is used with alfa-numeric digits, caps and no-caps characters).
Edit the docker-compose.yml and look for the LETSENCRYPT_EMAIL environment variable occurrences throughout the file and replace email@example.com with your email for the ssl certificates generation/renewal, look for any occurrence of the VIRTUAL_HOST and LETSENCRYPT_HOST environment variables in the services and replace dbox-tests by your domain and ml by your domain suffix (domain.suffix) and save the file.
Next execute $ docker-compose up -d command to bring up the docker compose. It will then start pulling the docker images in order to create the containers.
After it is done bringing the docker compose up, give ownership of the jenkins/ folder in the data/ folder to your user in order for the container to allow the Jenkins container to freely write/rewrite changes on files: $ chown -R 1000:1000 data/jenkins/
Execute the root_bash71.sh bash file to enter the php 7.1 container as root user, give 755 permissions to the dbox-tests folder and everything inside (dbox-tests is you domain name, you have to change this to your own domain), make the devilbox user inside the container its owner and exit the container:
```
$ ./root_bash71.sh

root@php-7.1.8 in /shared/httpd $ chmod -R 755 dbox-tests/ && chown -R devilbox:devilbox dbox-tests/
root@php-7.1.8 in /shared/httpd $ exit
```

Jenkins Setup[edit]

Jenkins GUI Setup[edit]

Head to jenkins.yourdomain.com to start the GUI setup.

To unlock Jenkins follow the instructions given on the GUI after accessing on the browser. Get the initial password from the Jenkins folder while on the docker-compose.yml base folder and copy/paste it onto the field where it asks for the password:
```
$ cat data/jenkins/secrets/initialAdminPassword
```
Select "Install Suggested Plugins" to install the default plugins, then either create a new user or continue as admin logging in with username admin and password ab56a379e3154a38acfc46b91bc0c25a and change it by clicking the "admin" tab on the right hand side of screen and "Configure" option in the menu.
Afterwards head to Manage Jenkins -> Configure Global Security and select the "Safe HTML" option under "Markup Formatter" section. Without this, the "Build History" section of the Jenkins jobs won't render HTML links.

Install Plugins[edit]

Go to jenkins.yourdomain.com, then to Manage Jenkins -> Manage Plugins and check if the "Git plugin", "GitHub plugin", "GitHub Pull Request Builder" and "Rebuilder" plugins are or are not installed, if not find and tick those plugins for installation, click the "Install without restart" button and restart Jenkins.

Configure Jenkins Globally[edit]

Head over to jenkins.yourdomain.com, then to Manage Jenkins -> Configure System and here add the Jenkins URL (https://jenkins.yourdomain.com/) to the "Jenkins URL" field in "Jenkins Location" and to the "Jenkins URL override" field in the "GitHub Pull Request Builder" section.
Next click the "Add" button located next to the select box associated with "Credentials" label, select "Username with password" as "Kind" field, type in the github credentials for the bot account on the "Username" and "Password" fields and add the username into the "Admin list" textbox in order to be an allowed admin user.

Configure Github Web Hooks[edit]

At this point it will be necessary to configure the Github webhooks which will listen for the events we need for building the repository.

First thing will be to obtain the web hook URL. Go to https://jenkins.yourdomain.com/, Manage Jenkins -> Configure System and under the "GitHub" section, click "Advanced" button, tick "Specify another hook url for GitHub configuration" tickbox and obtain the URL https://jenkins.yourdomain.com/github-webhook/, then untick it again and exit settings without saving.
After obtaining the URL next step is to enable the web hook URL in Github. Head to the Github repository for this test, click "Settings" tab, "Webhooks" menu option, "Add webhook" button and type https://jenkins.yourdomain.com/ghprbhook/ into the "Payload URL" field, select x-www-form-urlencoded in "Content Type" field, tick "Let me select individual events." option and tick just Issue comment, Pull request and Push options and click "Add webhook" button. If you see a green tick icon next to the webhook URL after refreshing the page it means that the link is working.
Final step missing is enabling the Jenkins (Github) plugin service. Head to the Github repository, click the "Settings" tab, click "Integrations & services" menu option, click "Add service" button and find Jenkins (GitHub plugin) service and type the web hook URL that was obtained in the beginning (https://jenkins.yourdomain.com/github-webhook/) into the "Payload URL" field and click the "Add service" button.

Create Jenkins Projects[edit]

The next steps are to create the two necessary Jenkins projects/jobs for this project, 1 for building the PRs upon changes and pull each PR code into its own subfolder and another one which also builds the PRs upon changes but pulls the code for that PR's target branch. This way we can have the patch/no patch instances when testing PRs for comparison.

Main Github PR Builder Project[edit]

This is the main project for pulling the PR code.

Create Project[edit]

Go to https://jenkins.yourdomain.com/, click on "New item", type a name for the project in the "Enter an item name" field (it's best to name it after the name of the repository), select "Freestyle project" option and click "OK" to finish.

Configure Project[edit]

Go to https://jenkins.yourdomain.com/, click on the link for the job that was created in the list, click "Configure" and in the "General" tab tick "GitHub project" option, type the Github repository URL in "Project URL" field and tick "This project is parameterized" option, click "Add Parameter", select "String Parameter" and type "sha1" in the "Name" field. This will change default "Build Now" option to "Build with Parameters" in the project's menu. By default it will run the last build or if a commit hash value from the "Build History" list or GitHub pull request is passed, it will run a specific build.
In the "Source Code Management" tab tick "Git" option, type the Github repository URL in the "Repository URL" field, in the "Credentials" select box, select the option that was previously created under Configure Jenkins Globally section, click the "Advanced" button then type origin in "Name" and +refs/pull/*:refs/remotes/origin/pr/* in "Refspec" field and type ${sha1} in "Branch Specifier" field. Also on "Additional Behaviours" click "Add", choose "Check out to a sub-directory" option and in "Local subdirectory for repo" place ${sha1}. This last option is meant for having the PR code be pulled into separate folders with a folder per PR.
In "Build Triggers" tab tick "GitHub Pull Request Builder" option. You'll see that your previous configs will appear there. Then tick "Use github hooks for build triggering" option. After clicking the "Advanced" button you can add users to the allowed Admins in "Admin list" and place a trigger phrase so that a build can be triggered after a specific comment from one of the admins.
In the "Build Environment" tab you can tick the "Set GitHub commit status with custom context and message (Must configure upstream job using GHPRB trigger)" option and then set the commit status URL, build triggered and started messages to anything you want as well as the build result messages.

Target Branch Github PR Builder Project[edit]

This is the project for pulling the PR's target branch code.

Create Project[edit]

Go to https://jenkins.yourdomain.com/, click on "New item", type a name for the project in the "Enter an item name" field (place the same name as your main project and add "-branches" like "project-name-branches" or any other name you want as long as you place the names of the projects in the script files to add/remove instances and target branch in the files/ folder), select "Freestyle project" option and click "OK" to finish.

Configure Project[edit]

Go to https://jenkins.yourdomain.com/, click on the link for the job that was created in the list, click "Configure" and in the "General" tab tick "GitHub project" option, type the Github repository URL in "Project URL" field and tick "This project is parameterized" option, click "Add Parameter", select "String Parameter" and type "ghprbTargetBranch" in the "Name" field. This will change default "Build Now" option to "Build with Parameters" in the project's menu. By default it will run the last build or if a commit hash value from the "Build History" list or GitHub pull request is passed, it will run a specific build.
In the "Source Code Management" tab tick "Git" option, type the Github repository URL in the "Repository URL" field, in the "Credentials" select box, select the option that was previously created under the Configure Jenkins Globally section, click the "Advanced" button then type branches in "Name" and +refs/heads/*:refs/remotes/branches/* in "Refspec" field and type ${ghprbTargetBranch} in "Branch Specifier" field. Also on "Additional Behaviours" click "Add", choose "Check out to a sub-directory" option and in "Local subdirectory for repo" place ${ghprbTargetBranch}. This last option is meant for having the PR's target branch code be pulled into separate folders with a folder for the target branch.
In "Build Triggers" tab tick "GitHub Pull Request Builder" option. You'll see that your previous configs will appear there. Then tick "Use github hooks for build triggering" option. After clicking the "Advanced" button you can add users to the allowed Admins in "Admin list" and place a trigger phrase so that a build can be triggered after a specific comment from one of the admins.
In the "Build Environment" tab you can tick the "Set GitHub commit status with custom context and message (Must configure upstream job using GHPRB trigger)" option and then set the commit status URL, build triggered and started messages to anything you want as well as the build result messages.

The projects are now fully configured with all the right settings and the webhooks successfully configured so now if you create a Pull Request on the repository or a new commit to an existent PR Jenkins will automatically build it. Also if the admins comment the trigger phrase on a PR a build will also be triggered for that PR.

Tests Requests Website Setup[edit]

Head over to the docker-compose.yml file base folder, execute the bash71.sh script to enter the container as devilbox user, while inside cd to the website folder (dbox-tests for this project) and install the project dependencies (composer install).
```
$ ./bash71.sh

devilbox@php-7.1.8 in /shared/httpd $ cd dbox-tests/ && composer install
```
After installing the dependencies give 777 permissions to the logs folder and to the htdocs/images/avatars folder in order to allow reads and writes to the github avatars and log files and exit the container.
```
devilbox@php-7.1.8 in /shared/httpd/dbox-tests $ chmod -R 777 logs/ && chmod -R 777 htdocs/images/avatars
```
Head over to the etc/ folder in the website folder (data/www/php71/dbox-tests/etc/), edit the config.json file insert your database info like db driver, host(127.0.0.1), database user, database password, database name, tables prefix for that database and save the file.
After inserting the database info on the config.json file you have to insert the Github info in the same file, but first you should create a "bot" user for this project and have an org and repository created within that org in Github meant to hold the PRs with Joomla code as referred before.
After creating the repo you'll have to create an oath Github app with your bot account (go to profile settings, OAuth Apps, register a new application, place an application name and your URL with your domain - https://dbox-tests.ml/ for this project - in the Homepage URL and in the authorization callback URL and register the application).
Then you must copy your app's client ID and client secret from the oauth app created in the previous step and introduce them in the config.json file. Afterwards in accounts place your github "bot" account username and password (you can place as many accounts as you want, they are meant for the authenticated requests made to the Github API from the website) and save the file.
Edit the mysql.sql file in the etc/ folder as well, head over to the #__tracker_projects table insert and introduce your repository info. Save the file afterwards.
Give 777 permissions to the tracker.php file in the cli/ folder from the website base folder.
```
$ chmod 777 cli/tracker.php
```
In the website base folder execute $ cli/tracker.php install in order to install the website app and create the tables by importing the mysql.sql file to the specified database in the config.json file.
In the website base folder execute $ cli/tracker.php get project and select option 1 whenever prompted in order to obtain the latest data from the repository.
You are now set to go and can now head over to the website and test it by creating/deleting instances from PRs.
To bring up the docker compose just execute the docker-compose up -d command, to bring down the docker compose execute the docker-compose down command and to look at the logs from the containers execute docker-compose logs -f or docker-compose logs command. Also to list the containers running from the docker compose run docker-compose ps. Should you want to update the images from each container execute the update-docker.sh script in the docker-compose base folder.

Issues[edit]

Note: If there are any issues while attempting to set up the platform or any suggestions to the platform itself or to the documentation don't hesitate to drop an issue at the project's repository.

Future improvements[edit]

Get the platform working with the Joomla! CMS repository instead of the tests repository created for this project.
Make the cron jobs to delete instances that are expired after 1 hour.
Update the website DB either by executing the tracker.php script for fetching latest data with cron jobs as well, by executing the script when a Jenkins build is done, or a combination of both.
Fix the issues currently with the Joomla! CMS PostgreSQL dump file in order to allow choosing the DBMS (MySQL or PostgreSQL) besides the PHP version.
Solve the lack of SSL certificates for the Joomla! instances URLs when letsencrypt releases the functionality to generate wildcard certificates in 2018.

PR Testing Platform

From Joomla! Documentation

Contents

Introduction[edit]

Jenkins[edit]

Multi-container Setup[edit]

Tests Requests Website[edit]

Initial Setup[edit]

Docker & Docker Compose[edit]

Docker CE[edit]

Docker Compose[edit]

Folders and Files Permissions[edit]

Docker Compose Environment Setup[edit]

Jenkins Setup[edit]

Jenkins GUI Setup[edit]

Install Plugins[edit]

Configure Jenkins Globally[edit]

Configure Github Web Hooks[edit]

Create Jenkins Projects[edit]

Main Github PR Builder Project[edit]

Create Project[edit]

Configure Project[edit]

Target Branch Github PR Builder Project[edit]

Create Project[edit]

Configure Project[edit]

Tests Requests Website Setup[edit]

Issues[edit]

Future improvements[edit]