Joomla

Release procedure and checklist

From Joomla! Documentation

(Redirected from Release procedure and checklist)

This is a general procedure explanation for managing releases of Joomla! and the Joomla! Framework. This list is best suited for a Minor Release or Maintenance Release, however the steps will generally apply to a Major Release as well but will require some judgement calls from the release manager and/or leadership team.

There is a more formal checklist available on GitHub for release leads to use that is intended to complement the steps described in this page.

Issue Triage, Patch Review, Merging Patches[edit]

The release cycle for a new software version begins immediately after the previous one is completed. In the immediate hours and days after a release, the issue tracker and support forums, in close coordination with the Bug Squad and CMS Release Team, should be monitored for reports of potential regressions or critical bugs that would warrant a quick follow-on release to address the issue. It is generally suggested to wait 48-72 hours before resuming merging of reviewed and ready for commit pull requests to keep the release branch "clean" and ready for a release (note this is only a suggestion, if need be a release manager could create a branch from the release tag however this involves a bit more effort therefore this suggestion is for simplicity's sake).

Throughout the release cycle, the release manager should be involved with reviewing patches and merging reviewed patches to the appropriate release branch. As of this writing (where 4.0.6 is the present release), the following branches are generally active:

  • 3.10-dev: This is the 3.x branch, generally only bug fixes and "general" maintenance work should merge to this branch.
  • 4.0-dev: This is the 4.0.x branch - there are no more intended releases to this branch unless required by security fixes.
  • 4.1-dev: This is the main release branch and is where most activity occurs, generally only bug fixes and "general" maintenance work should merge to this branch although minor features may be merged at the discretion of the release manager and/or leadership team. All code from 3.10-dev is merged into this branch.
  • 4.2-dev: This is the release branch for the next minor release (the branch's name will always be the version it represents), new features should be merged to this branch

Branch Merging[edit]

All branches should remain in-sync and all work should be regularly merged forward. Given the above example, the following is the recommended merging workflow (in general, the suggested workflow is that 3.10-dev is merged to the current 4.x-dev branch and then through to the next minor branch):

  • 3.10-dev should be merged into 4.0-dev
  • 4.0-dev should be merged to 4.1-dev
  • 4.1-dev should be merged to 4.2-dev

It is suggested that at least once per week branch merging occurs.

Preparing To Release[edit]

When scheduling and preparing a release, the timeline should be communicated with the Production and Marketing & Communication Department leadership so that all teams are prepared to support the release. Note that when scheduling releases, consideration should be given to holidays around the world and the impact of issuing a release near those periods considered. Helpful resources include http://interfaith-calendar.org/ and https://www.timeanddate.com/holidays/

Review Release Milestone and Blockers[edit]

On GitHub, check the milestone for the release you are preparing to create and for any open items with the release-blocker label. If there are remaining open items, review them to determine if they are items that should be included in the release or if they can be rescheduled to a later release.

Release Schedule[edit]

Minor Release[edit]

Given the nature of a minor release, the following is the suggested release timeline:

  • Six weeks prior to planned stable release: Tag the first Beta release
    • At this point the release should be considered feature complete and new features should be deferred to the next minor release
  • Weekly Beta releases should follow this until the Release Candidate
  • Two weeks prior to planned stable release: Tag the Release Candidate
    • At this point the release branch(es) should remain closed for committing except for high priority bug fixes and language updates
    • With the Release Candidate, language freeze is also in effect and no language changes are permitted unless there is an urgent need (which should be communicated with the translations coordinator); note that if necessary the language freeze may be declared earlier but the Release Candidate should be considered the "no later than" time
    • Inform the CMS Release Team and call out anything that you would like extra testing on
  • One week prior to planned stable release: If necessary, create a second Release Candidate
  • 2-3 days before release: Build final packages - share with CMS Release Team and do release preparation activities.
  • Planned release date: Tag the stable release

Maintenance Release[edit]

Maintenance releases operate on a 4-6 week cycle and the following is the suggested release timeline:

  • 9-10 days before planned release date: Tag a Release Candidate
    • With the Release Candidate, language freeze is also in effect and no language changes are permitted unless there is an urgent need (which should be communicated with the translations coordinator); note that if necessary the language freeze may be declared earlier but the Release Candidate should be considered the "no later than" time
    • Inform the CMS Release Team and call out anything that you would like extra testing on
  • 2-3 days before release: Build final packages - share with CMS Release Team and do release preparation activities.
  • Planned release date: Tag the stable release

Reviewing Release Changes[edit]

Before tagging a release, the list of changes in the release should be reviewed by the release manager and/or leadership team. The list of changes can be found using GitHub's compare function with the URL https://github.com/joomla/joomla-cms/compare/<version>...4.1-dev which will compare the changes from <version> to the current 4.1-dev branch (for example, https://github.com/joomla/joomla-cms/compare/3.9.0...3.9.1-rc is the comparison from the 3.9.0 release tag to the 3.9.1-rc tag). Key purposes of this review include:

  • Ensuring all patches are compliant with Semantic Versioning (SemVer) (i.e. no backward compatibility breaks in minor or maintenance releases and no features in maintenance releases); SemVer violations should be cleared with the Production Department leadership
  • Ensuring all code revisions are compatible with Joomla's minimum supported versions (i.e. no PHP 5.4 syntax in 3.x releases where PHP 5.3 is the minimum supported version, exceptions may be made if introducing a code path and files that will only be executed on newer PHP versions)
    • Joomla's testing integrations should not be relied on in full for this as there are large portions of the Joomla code base which do not have automated test coverage, and as of this writing, there is not a code linting process in place checking for potential issues
  • Ensuring features and "significant" changes are documented

Security Fixes[edit]

From time to time, security fixes must be included in a release. Prior to every release, the release manager should check with the Security Strike Team to determine if security fixes are required, and if so ensure this is communicated to all members involved with release work (i.e. the marketing team liaison who assists with the publication of release related material). When a security release is necessary, an internal Release Candidate should be built for testing by the Security Strike Team, CMS Release Team, and CMS Maintenance Team with all security patches included. These teams should also receive information about the vulnerabilities being addressed to ensure there are no regressions in the impacted areas.

Note, when a security fix involves a Joomla Framework package, the Framework package should be patched first and a release created of the affected package then the CMS updated with composer update.

Sometimes security fixes are required for composer dependencies that are out of support. In such cases we fork the 3rd party dependency into https://github.com/joomla-backports and then apply security fixes there in a joomla-backports branch (and override the repository in composer.json and as well as the branch reference here. Links are a sample for PHPMailer overrides in Joomla 3.10).

Release FAQ[edit]

Create Release FAQ Category on the Joomla Documentation site. (i.e. Clone the contents of an existing page such as FAQ and update the version number to your required version)

Building a Release[edit]

Building a release starts with checking that any files and folders that have been deleted are included in the arrays in the installer script at administrator/components/com_admin/script.php. There is a helper script in the repository to assist with this located at build/deleted_file_check.php. Running php build/deleted_file_check.php with its required --from option will write a file to build/deleted_files.txt with the list of files deleted since the from reference. Please see the PHP file for all documentation about this script.

The following steps require the individual to create commits and tags. It is suggested that these commits and tags be GPG signed as an extra security step but this is not a mandatory requirement as of this writing.

If creating a security release, at this point you should apply all patches provided by the security team.

Once it is confirmed that the deleted files listing is correct, the build/bump.php script should be executed to set the new version number. This script will also perform maintenance steps such as correcting the copyright year ranges for all files and replacing the __DEPLOY_VERSION__ placeholder. Please see the file for all documentation about this script.

After running this script, review all changes with git diff as a final check before creating your release commit. Once you are satisfied that all changes are correct, commit your changes locally. Once committed, create the release tag locally; all release tags should be the version number you are creating and should match the output of Joomla\CMS\Version::getShortVersion() (so if you have created 3.9.0-beta2 then the tag should be named 3.9.0-beta2).

Once the tag has been created, you will run the build/build.php script to generate the release packages. To ensure a clean build is produced, the script makes an archive of the tag you are building using the git archive command and extracts this to the build/tmp/<timestamp> directory. Depending on the patch version number, the following packages will be created in the build/tmp/packages:

  • 0: Full installation and upgrade packages
  • 1: Full installation, upgrade packages, and patch packages for x.y.0 to x.y.1
  • 2+: Full installation, upgrade packages, patch packages for x.y.0 to x.y.<new> and x.y.<new-1> to x.y.<new> (i.e. when creating 3.8.8 this will create a patch package suitable for all 3.8.x releases and an optimized patch package for 3.8.7 to 3.8.8)

Additionally, the script will generate a build/tmp/checksums.txt file with the checksums for all generated packages. build/tmp/github_release.txt is also produced which is a text template used for the GitHub releases.

After the packages are created, you should validate that you are able to install and update a site using the ZIP packages (as these are the most frequently used type and served by our update system).

Publishing a Release[edit]

Release Packages[edit]

Depending on the type of release being created, you will publish the release packages in different locations.

Alpha, Beta, Release Candidate[edit]

The packages for these releases are uploaded to GitHub. When ready to publish, you should push the release tag you created to GitHub and use their interface to create a release from the tag and upload the packages.

Stable[edit]

The packages for stable releases are uploaded to multiple locations, this is in part driven by the fact that the Amazon Web Services (AWS) S3 region used by Joomla is blocked in some locations therefore download mirrors are available when necessary.

Note that the Joomla! Downloads Portal release can be done in advance (as long as the release is kept unpublished until release day). The Update server and GitHub releases are public by default and therefore should only be done at the time of release.

Joomla! Downloads Portal[edit]

The canonical source for all stable releases is the Joomla! Downloads Portal. The packages served from the site use Joomla's AWS account. You will need to log into the AWS console to upload the packages to S3 under the joomla-official-downloads bucket. The exact path inside the bucket is joomladownloads/joomla<version> where <version> is the major version number (i.e. joomla3), note for compatibility with Akeeba Release System folders for each individual release are not used. You can use AWS CLI to upload these files - but MFA is still required - follow the procedure documented here or use a tool to automate this such as AWS-MFA

Once files have been uploaded to S3 then you need to generate the release in Akeeba Release System on the Downloads Portal. To generate all required items quickly it's recommended to use the cli/ars-create-cms-release.php script - the source for this can be found on Github.

Note that if you need to rebuild a package - for example because issues are found in the release candidate - you do not need to re-create the ARS resources but simply delete and reupload the S3 packages then use the cli/ars-get-hashes.php script on the downloads site (which simply update the checksums for each item in the release) - source code for this is also located at Github.

Joomla! Update Server[edit]

The Joomla! Update Server is used as a mirror for the download packages. After connecting to the server, you should create a new folder at www/releases/<version> (i.e. www/releases/3.9.1 for the 3.9.1 release) and upload the packages here.

GitHub[edit]

The packages are also uploaded to GitHub. This is similar to the steps for alpha, beta, and release candidate releases above. Note that you only need to push the release tag at this point (i.e. git push --tags), you can wait to push the updated release branch until after the release is available.

Announcements[edit]

joomla.org[edit]

For all stable releases, the main release announcement is published to the Joomla! Official News > Project Release News category on the main joomla.org website. This is primarily handled by the Marketing team - however release leads are expected to contribute to the Bug fixes and Improvements section as well as highlighting any other information that may need to be called out as part of release notes. After release the Social Media team need to be made aware to ensure the release is available on social media - use the Contact the Social Media Team Channel on Glip.

For a minor release, an updated version of the landing page at joomla.org/4 (previously joomla.org/3) is published. Generally, this site is backed up from the sandbox location and transferred to the live domain shortly before release, this transfer should be coordinated with the Webmasters Team.

developer.joomla.org[edit]

For security releases, all accompanying articles are published to the News > Security Centre category on the Joomla! Developer Network. As these articles are referenced from the main release announcement, these should be published at the same time.

Update Server[edit]

The update server must be updated to alert websites that a core update is available, this subdomain is managed on GitHub at https://github.com/joomla/update.joomla.org. Prior to each release, a pull request should be prepared with necessary changes for the release (generally this updates the version number, announcement URL, package URLs, and package checksums for the stable release and updating the version number for the nightly builds), see https://github.com/joomla/update.joomla.org/pull/117 as an example of this update for the 3.9.1 release. Pushes to the master branch of this repo are automatically deployed thanks to a Jenkins job, notifications for many of the Jenkins jobs are sent to the .org Build Notifications channel on RingCentral Glip so you should ensure a success notification was sent here otherwise check if there was an issue with the job and/or manually deploy the updated files.

Note that the update.joomla.org subdomain is behind a CDN and it generally takes 5-10 minutes for all nodes to update, if need be you can access update1.joomla.org to verify the right files are deployed without using the CDN. Note files cache times aren't consistent and often we've seen issues where the list update file is the new version and the extension.xml file is on the old version.

Post Release Actions[edit]

Once the release is published, there are several additional tasks that should be completed shortly after.

Post-Release Version Bump[edit]

The version should be bumped to the next version scheduled for release using the build/bump.php script (i.e. after the 3.9.1 release this would be bumped to 3.9.2-dev).

Branch Merging[edit]

After incrementing the version number, you should merge all the release changes forward to other active branches (especially in the case of a security release). Ensure you pay careful attention while resolving merge conflicts, when making a merge that includes a version bump generally all files which contain the version string and/or release date will be in conflict (as of this writing this is 8 files).

Push Updated Branches[edit]

When ready, you should directly push your updates to the Git repo. If you elect to use pull requests, the pull requests should NEVER be squashed or rebased otherwise this will wipe out Git's awareness of the merged commits and will make your next branch merges even more complex. Likewise, as an example, if the commit preparing the 3.9.1 release and bumping the version string to 3.9.2-dev were squash merged as a pull request, then the commit that the 3.9.1 release was tagged from would not be in the repository.

Update Release Milestones[edit]

On GitHub, all merged items are assigned to a milestone to help organize items. With each release, the milestones should be updated to ensure that the milestone for the release you created has been closed and a milestone for the next release is open.

Regenerate Nightly Builds[edit]

The nightly builds for any branch updated during the release process should be regenerated to include the release's changes. These can be manually triggered within the CMS Packaging folder on the Jenkins server.

Website Updates[edit]

With all code updates pushed, there are several website updates that need to be triggered.

api.joomla.org[edit]

With each release, an updated build of the API documentation should be deployed to the api.joomla.org website. These updates are triggered with the Joomla! API > Deploy Docs - CMS job on Jenkins. As this job relies on the release tags from the CMS repo, this can only be done after the release tag has been pushed.

developer.joomla.org[edit]

After a release, the Joomla! Project Roadmap article in the Uncategorised category should be updated to reflect the newly released version and updated links, as well as adding expected release dates for the next release.

The version whitelist for the statistics server should also be updated to allow the next version to be submitted; if you have not already, you should have a clone of the https://github.com/joomla/statistics-server repository on your local system. Once cloned and set up, run the bin/stats tags:joomla command to regenerate the Joomla version whitelist. As this command relies on the release tags from the CMS repo, this can only be done after the release tag has been pushed. The updated versions/joomla.json should be committed and pushed, and a Jenkins job will deploy the updated site resources.

Additional Considerations[edit]

The release manager or a delegate is suggested to be in communication with the Webmasters Team to ensure the joomla.org websites are updated in a timely manner. When a security release is scheduled, it is encouraged to share the update release package with the team so that our sites may be updated before the fix is made public to mitigate the chance of the vulnerability being used to exploit one of our sites. It also provides additional testing for security fixes on real world sites in addition to the tests performed by the CMS Release Team.

Framework Release Cycle[edit]

The packages of the Joomla Framework are on an independent lifecycle meaning each package can be updated and released independently of all others. Generally the only restriction is with major releases, which should coincide with the CMS release cycle (i.e. CMS 3.x uses Framework 1.x, CMS 4.x will use Framework 2.x therefore Framework 2.0 stable releases should not be made until the CMS is in its beta phase at the earliest).

Tagging a Release[edit]

Most packages can be tagged and released through the Joomla! Framework > Package Release job on Jenkins, however as of this writing this job only supports tagging releases from a repository's master branch. To tag a release, select the package you wish to update and specify the version you are releasing in the job's parameters and run the job; all releases should follow Semantic Versioning to set the correct number.

Once the job has completed, you should verify the new release is visible on Packagist. Within a day, the new update should be visible on the Joomla Framework Status page (the database is updated via cron job reading from Packagist's API), if the updated version isn't seen on the site in a relatively timely manner you should clone the website locally and debug the packagist:sync:releases command.

Security Releases[edit]

When a security release is created, a pull request should be sent to the FriendsOfPHP/security-advisories repository with relevant details. This repository is used by many tools in the PHP and Composer ecosystem to alert users of security issues. https://github.com/FriendsOfPHP/security-advisories/pull/134 is an example of this pull request.

Onboarding Release Leads[edit]

Release Leads will need access to a number of resources. The following are required:

  • Account on Github (added to the CMS Release Leads team for the Joomla Org (which also provides jenkins access) and the CMS Release Leads team for the Joomla Backport org.
  • The Forum (Ensure users are aware of release threads)
  • Glip (CMS Maintainers, CMS Release Leads, WebMasters Team, Production, CMS Release Team)
  • Introduction with Translation, JSST and Webmaster Liasons
  • SSH keys added to Update and Downloads servers (including VPN Access)
  • Downloads Website (Release Leads -> CMS ACL Group)
  • Developer Website (? ACL Group)
  • AWS Account in Release_Managers Group. User must reset password and setup MFA.
  • Drone Admins