Joomla:Release procedure and checklist

Content is Incomplete

This is a general checklist for managing releases of Joomla! and the Joomla! Framework. This list is best suited for a Minor Release or Maintenance Release, however the steps will generally apply to a Major Release as well but will require some judgement calls from the release manager and/or leadership team.

This document is a work in progress. The information on this page is currently undergoing a major revision.

CMS Release Cycle

Issue Triage, Patch Review, Merging Patches

The release cycle for a new software version begins immediately after the previous one is completed. In the immediate hours and days after a release, the issue tracker and support forums, in close coordination with the Bug Squad and CMS Release Team, should be monitored for reports of potential regressions or critical bugs that would warrant a quick follow-on release to address the issue. It is generally suggested to wait 48-72 hours before resuming merging of reviewed and ready for commit pull requests to keep the release branch "clean" and ready for a release (note this is only a suggestion, if need be a release manager could create a branch from the release tag however this involves a bit more effort therefore this suggestion is for simplicity's sake).

Throughout the release cycle, the release manager should be involved with reviewing patches and merging reviewed patches to the appropriate release branch. As of this writing (where 3.9.1 is the present release), the following branches are generally active:

• staging: This is the main release branch and is where most activity occurs, generally only bug fixes and "general" maintenance work should merge to this branch although minor features may be merged at the discretion of the release manager and/or leadership team
• 3.10-dev: This is the release branch for the next minor release (the branch's name will always be the version it represents), new features should be merged to this branch
• 4.0-dev: This is the release branch for the next major release (the branch's name will always be the version it represents)

Branch Merging

All branches should remain in-sync and all work should be regularly merged forward. Given the above example, the following is the recommended merging workflow (in general, the suggested workflow is that staging is merged to the next minor branch and the next minor branch is merged to the next major branch):

• staging should be merged into 3.10-dev
• 3.10-dev should be merged to 4.0-dev

It is suggested that at least once per week branch merging occurs.

Preparing To Release

When scheduling and preparing a release, the timeline should be communicated with the Production and Marketing & Communication Department leadership so that all teams are prepared to support the release. Note that when scheduling releases, consideration should be given to holidays around the world and the impact of issuing a release near those periods considered. Helpful resources include http://interfaith-calendar.org/ and https://www.timeanddate.com/holidays/

Review Release Milestone and Blockers

On GitHub, check the milestone for the release you are preparing to create and for any open items with the release-blocker label. If there are remaining open items, review them to determine if they are items that should be included in the release or if they can be rescheduled to a later release.

Release Schedule

Minor Release

Given the nature of a minor release, the following is the suggested release timeline:

• Six weeks prior to planned stable release: Tag the first Beta release
  • At this point the release should be considered feature complete and new features should be deferred to the next minor release
• Weekly Beta releases should follow this until the Release Candidate
• Two weeks prior to planned stable release: Tag the Release Candidate
  • At this point the release branch (generally staging) should remain closed for committing except for high priority bug fixes and language updates
  • With the Release Candidate, language freeze is also in effect and no language changes are permitted unless there is an urgent need (which should be communicated with the translations coordinator); note that if necessary the language freeze may be declared earlier but the Release Candidate should be considered the "no later than" time
• One week prior to planned stable release: If necessary, create a second Release Candidate
• Planned release date: Tag the stable release

Maintenance Release

Maintenance releases operate on a 4-6 week cycle and the following is the suggested release timeline:

• 4-5 days before planned release date: Tag a Release Candidate
  • With the Release Candidate, language freeze is also in effect and no language changes are permitted unless there is an urgent need (which should be communicated with the translations coordinator); note that if necessary the language freeze may be declared earlier but the Release Candidate should be considered the "no later than" time
• Planned release date: Tag the stable release

Reviewing Release Changes

Before tagging a release, the list of changes in the release should be reviewed by the release manager and/or leadership team. The list of changes can be found using GitHub's compare function with the URL https://github.com/joomla/joomla-cms/compare/<version>...staging which will compare the changes from <version> to the current staging branch (for example, https://github.com/joomla/joomla-cms/compare/3.9.0...3.9.1-rc is the comparison from the 3.9.0 release tag to the 3.9.1-rc tag). Key purposes of this review include:

• Ensuring all patches are compliant with Semantic Versioning (SemVer) (i.e. no backward compatibility breaks in minor or maintenance releases and no features in maintenance releases); SemVer violations should be cleared with the Production Department leadership
• Ensuring all code revisions are compatible with Joomla's minimum supported versions (i.e. no PHP 5.4 syntax in 3.x releases where PHP 5.3 is the minimum supported version, exceptions may be made if introducing a code path and files that will only be executed on newer PHP versions)
  • Joomla's testing integrations should not be relied on in full for this as there are large portions of the Joomla code base which do not have automated test coverage, and as of this writing, there is not a code linting process in place checking for potential issues
• Ensuring features and "significant" changes are documented

Security Fixes

From time to time, security fixes must be included in a release. Prior to every release, the release manager should check with the Security Strike Team to determine if security fixes are required, and if so ensure this is communicated to all members involved with release work (i.e. the marketing team liaison who assists with the publication of release related material). When a security release is necessary, an internal Release Candidate should be built for testing by the Security Strike Team, CMS Release Team, and CMS Maintenance Team with all security patches included. These teams should also receive information about the vulnerabilities being addressed to ensure there are no regressions in the impacted areas.

Note, when a security fix involves a Joomla Framework package, the Framework package should be patched first and a release created of the affected package then the CMS updated with composer update.

Building a Release

Building a release starts with checking that any files and folders that have been deleted are included in the arrays in the installer script at administrator/components/com_admin/script.php. There is a helper script in the repository to assist with this located at build/deleted_file_check.php. Running php build/deleted_file_check.php with its required --from option will write a file to build/deleted_files.txt with the list of files deleted since the from reference. Please see the PHP file for all documentation about this script.

The following steps require the individual to create commits and tags. It is suggested that these commits and tags be GPG signed as an extra security step but this is not a mandatory requirement as of this writing.

If creating a security release, at this point you should apply all patches provided by the security team.

Once it is confirmed that the deleted files listing is correct, the build/bump.php script should be executed to set the new version number. This script will also perform maintenance steps such as correcting the copyright year ranges for all files and replacing the __DEPLOY_VERSION__ placeholder. Please see the file for all documentation about this script.

After running this script, review all changes with git diff as a final check before creating your release commit. Once you are satisfied that all changes are correct, commit your changes locally. Once committed, create the release tag locally; all release tags should be the version number you are creating and should match the output of Joomla\CMS\Version::getShortVersion() (so if you have created 3.9.0-beta2 then the tag should be named 3.9.0-beta2).

Once the tag has been created, you will run the build/build.php script to generate the release packages. To ensure a clean build is produced, the script makes an archive of the tag you are building using the git archive command and extracts this to the build/tmp/<timestamp> directory. Depending on the patch version number, the following packages will be created in the build/tmp/packages:

• 0: Full installation and upgrade packages
• 1: Full installation, upgrade packages, and patch packages for x.y.0 to x.y.1
• 2+: Full installation, upgrade packages, patch packages for x.y.0 to x.y.<new> and x.y.<new-1> to x.y.<new> (i.e. when creating 3.8.8 this will create a patch package suitable for all 3.8.x releases and an optimized patch package for 3.8.7 to 3.8.8)

Additionally, the script will generate a build/tmp/checksums.txt file with the checksums for all generated packages.

After the packages are created, you should validate that you are able to install and update a site using the ZIP packages (as these are the most frequently used type and served by our update system).

Publishing a Release

Release Packages

Depending on the type of release being created, you will publish the release packages in different locations.

Alpha, Beta, Release Candidate

The packages for these releases are uploaded to GitHub. When ready to publish, you should push the release tag you created to GitHub and use their interface to create a release from the tag and upload the packages. There is a text template used for the releases, generally you can copy a previous release's text and adjust it to the version you are publishing.

Stable

The packages for stable releases are uploaded to multiple locations, this is in part driven by the fact that the Amazon Web Services (AWS) S3 region used by Joomla is blocked in some locations therefore download mirrors are available when necessary.

Joomla! Downloads Portal

The canonical source for all stable releases is the Joomla! Downloads Portal. The packages served from the site use Joomla's AWS account. You will need to log into the AWS console to upload the packages to S3 under the joomla-official-downloads bucket. The exact path inside the bucket is joomladownloads/joomla<version> where <version> is the major version number (i.e. joomla3), note for compatibility with Akeeba Release System folders for each individual release are not used.

Joomla! Update Server

The Joomla! Update Server is used as a mirror for the download packages. After connecting to the server, you should create a new folder at www/releases/<version> (i.e. www/releases/3.9.1 for the 3.9.1 release) and upload the packages here.

GitHub

The packages are also uploaded to GitHub. This is similar to the steps for alpha, beta, and release candidate releases above. Note that you only need to push the release tag at this point (i.e. git push --tags), you can wait to push the updated release branch until after the release is available.

Announcements

joomla.org

For all stable releases, the main release announcement is published to the Joomla! Official News > Project Release News category on the main joomla.org website.

For a minor release, an updated version of the landing page at joomla.org/3 is published. Generally, this site is backed up from the sandbox location and transferred to the live domain shortly before release, this transfer should be coordinated with the Webmasters Team.

developer.joomla.org

For security releases, all accompanying articles are published to the News > Security Centre category on the Joomla! Developer Network. As these articles are referenced from the main release announcement, these should be published at the same time.

Update Server

The update server must be updated to alert websites that a core update is available, this subdomain is managed on GitHub at https://github.com/joomla/update.joomla.org. Prior to each release, a pull request should be prepared with necessary changes for the release (generally this updates the version number, announcement URL, package URLs, and package checksums for the stable release and updating the version number for the nightly builds), see https://github.com/joomla/update.joomla.org/pull/117 as an example of this update for the 3.9.1 release. Pushes to the master branch of this repo are automatically deployed thanks to a Jenkins job, notifications for many of the Jenkins jobs are sent to the .org Build Notifications channel on RingCentral Glip so you should ensure a success notification was sent here otherwise check if there was an issue with the job and/or manually deploy the updated files.

Note that the update.joomla.org subdomain is behind a CDN and it generally takes 5-10 minutes for all nodes to update, if need be you can access update1.joomla.org to verify the right files are deployed without using the CDN.

Post Release Actions

Once the release is published, there are several additional tasks that should be completed shortly after.

Post-Release Version Bump

The version should be bumped to the next version scheduled for release using the build/bump.php script (i.e. after the 3.9.1 release this would be bumped to 3.9.2-dev).

Branch Merging

After incrementing the version number, you should merge all the release changes forward to other active branches (especially in the case of a security release). Ensure you pay careful attention while resolving merge conflicts, when making a merge that includes a version bump generally all files which contain the version string and/or release date will be in conflict (as of this writing this is 8 files).

Push Updated Branches

When ready, you should directly push your updates to the Git repo. If you elect to use pull requests, the pull requests should NEVER be squashed or rebased otherwise this will wipe out Git's awareness of the merged commits and will make your next branch merges even more complex. Likewise, as an example, if the commit preparing the 3.9.1 release and bumping the version string to 3.9.2-dev were squash merged as a pull request, then the commit that the 3.9.1 release was tagged from would not be in the repository.

Update Release Milestones

On GitHub, all merged items are assigned to a milestone to help organize items. With each release, the milestones should be updated to ensure that the milestone for the release you created has been closed and a milestone for the next release is open.

Website Updates

With all code updates pushed, there are several website updates that need to be triggered.

api.joomla.org

With each release, an updated build of the API documentation should be deployed to the api.joomla.org website. These updates are triggered with the Joomla! API > Deploy Docs - CMS job on Jenkins. As this job relies on the release tags from the CMS repo, this can only be done after the release tag has been pushed.

developer.joomla.org

After a release, the Joomla! Project Roadmap article in the Uncategorised category should be updated to reflect the newly released version and updated links. The version whitelist for the statistics server should also be updated to allow the next version to be submitted; if you have not already, you should have a clone of the https://github.com/joomla/statistics-server repository on your local system. Once cloned and set up, run the bin/stats tags:joomla command to regenerate the Joomla version whitelist. As this command relies on the release tags from the CMS repo, this can only be done after the release tag has been pushed. The updated versions/joomla.json should be committed and pushed, and a Jenkins job will deploy the updated site resources.

Additional Considerations

The release manager or a delegate is suggested to be in communication with the Webmasters Team to ensure the joomla.org websites are updated in a timely manner. When a security release is scheduled, it is encouraged to share the update release package with the team so that our sites may be updated before the fix is made public to mitigate the chance of the vulnerability being used to exploit one of our sites.

Framework Release Cycle

The packages of the Joomla Framework are on an independent lifecycle meaning each package can be updated and released independently of all others. Generally the only restriction is with major releases, which should coincide with the CMS release cycle (i.e. CMS 3.x uses Framework 1.x, CMS 4.x will use Framework 2.x therefore Framework 2.0 stable releases should not be made until the CMS is in its beta phase at the earliest).

Tagging a Release

Most packages can be tagged and released through the Joomla! Framework > Package Release job on Jenkins, however as of this writing this job only supports tagging releases from a repository's master branch. To tag a release, select the package you wish to update and specify the version you are releasing in the job's parameters and run the job; all releases should follow Semantic Versioning to set the correct number.

Once the job has completed, you should verify the new release is visible on Packagist. Within a day, the new update should be visible on the Joomla Framework Status page (the database is updated via cron job reading from Packagist's API), if the updated version isn't seen on the site in a relatively timely manner you should clone the website locally and debug the packagist:sync:releases command.

Security Releases

When a security release is created, a pull request should be sent to the FriendsOfPHP/security-advisories repository with relevant details. This repository is used by many tools in the PHP and Composer ecosystem to alert users of security issues. https://github.com/FriendsOfPHP/security-advisories/pull/134 is an example of this pull request.