Release Management

This page is the Xen Project’s process document for running a hypervisor release. It is written for the Release Manager, but maintainers, committers, and contributors are encouraged to read it so they know what to expect during a release cycle.

For the release schedule and development cycle in general — cadence, cut-off date, feature freeze, release candidates — see Release Planning. This page covers the roles involved, the Release Manager’s operational duties, and the email templates the Release Manager uses.

Why this page exists

The Release Manager changes from release to release. Without a written-down process, each new Release Manager rediscovers the same lessons and repeats the same mistakes. This page is the project’s record of that process.

The email templates near the end use placeholders — X.Y for the release, rcZ for the release candidate, $RM, $DATE, $REASON — which you fill in before sending.

The release cycle from the Release Manager’s side

A release divides into a development period of roughly six months and a freeze period of about two months. During development, contributors post and commit patches. Two dates matter:

  • Last posting date — the date by which a series must have been posted for the first time if it is to be considered for the release. Normally about two weeks before the cut-off date.

  • Cut-off date — the date by which feature patches must be committed. The freeze period begins after it.

The freeze can run shorter or longer than two months. If it runs long, it eats into the next development period. The precise schedule is set afresh each cycle by the Release Manager. For a March release, give particular thought to the Chinese New Year holiday, which then typically falls during the freeze — the freeze should probably be extended to compensate.

The reasons for the eight-month cadence were discussed on xen-devel in July 2018.

Roles

Release Manager

A trusted developer who owns the release process. The goal is a high-quality release that does not slip too far. Workload is light during the development window and grows through the freeze. The Release Manager tracks issues, arranges release candidates, negotiates with stakeholders, balances competing needs, and makes the difficult calls. During the freeze the Release Manager effectively owns the staging branch — committers act on their wishes.

Maintainers

Trusted developers responsible for specific components in xen.git. They are expected to respond to patches and questions about their components promptly, especially during the freeze.

Committers

The subset of maintainers who can push to xen.git. During the development window they push as they see fit. During the freeze they transfer branch ownership to the Release Manager and act on their wishes, which normally means a patch needs a Release Ack before it can go in.

Contributors

Expected to respond quickly to any problems with code they submitted during the cycle. Failing that, the Release Manager may revert the change, declare the feature unsupported, or take whatever action they deem appropriate.

Security Team

Operates independently, with limited visibility due to the sensitive nature of the work. The best the Release Manager can do is set aside time in the schedule for security issues to be fixed.

Release Technician

Tags the trees, prepares tarballs, and produces the release deliverables. Acts on the Release Manager’s instructions — keep the communication as clear as it can be.

Community Manager

Owns the xenproject.org infrastructure. Responsible for updating the web archives and wiki pages and for coordinating with the PR personnel.

PR personnel

Coordinate with external reporters to publish the release announcement. The Release Manager must be certain the release is going out on a given date before giving them the signal to proceed — there is a point of no return once they schedule a date with reporters.

Development period

  • Send a monthly development update email to xen-devel. It contains the release timeline, the major work items, and anything else the Release Manager sees fit to include. The usual way to produce it is to keep a structured plain-text list of work items — subject, owner, owner’s email, optional series version — grouped by area (hypervisor, x86, Arm, completed), and generate the email from that list, with the item owners on Bcc.

  • Send reminders one week before the last posting date and one week before the cut-off date.

  • Consider adding the key dates to your calendar.

  • Occasionally check the status of the staging branch and make sure it gets timely pushes to master.

Freeze period

At or before the very start of the freeze, agree a schedule of release-candidate test days with the Community Manager. Once the freeze starts, ownership of staging automatically transfers to the Release Manager, who can say “not releasing now” because of too many bugs, “until someone fixes these”, or “no more patches until X, Y and Z happen”.

Each release candidate follows the same loop:

  1. Check the status of the tree. If it is good, ask the Release Technician to make a release candidate.

  2. Send an email to xen-devel, xen-users, and xen-announce to announce the release candidate.

  3. Branch and/or reopen the tree for further feature submission if appropriate.

  4. Collect and track reported issues, determine their severity, and prod the relevant developers and maintainers to fix them.

  5. When fix patches are posted, determine whether each is good to include.

  6. Go back to step 1.

In the early release candidates it is normally fine to hand staging back to the committers so they can commit bug fixes at will. As the release approaches, the bar for accepting a patch gets higher and higher. Communicate clearly when committers can commit at will and when a formal Release Ack is needed.

At the same time, work with the Community Manager, PR personnel, and contributors to gather the feature list and discuss the support status of new features with stakeholders. Help prepare the press release and write a blog post for the release. Specifically:

  1. Collate a list of major changes, in collaboration between the Release Manager, PR personnel, and key contributors. This should not be done on a public mailing list, to minimise the risk of release-related media stories appearing before the release date.

  2. PR personnel identify feature highlights, a theme for the press release, companies providing supporting quotes, and media outlets to reach out to, and manage the creation of the press release in private.

  3. The Community Manager drafts the blog post with help from PR personnel and the Release Manager; it is published under the name of the Release Manager.

  4. The Community Manager creates release documentation — acknowledgements, feature list, man pages, release notes — accessible via a release category. This can be done in public.

  5. PR personnel get stakeholder and Advisory Board approval for the press release, one to two weeks before the release.

Make sure the key people for doing the release — especially the Community Manager, Release Manager, and Release Technician — will be available around the planned release date, or have named a capable substitute.

Final checklist

When you think all pending issues are fixed and Xen is ready to be released from the last release candidate:

  1. Send a commit moratorium email to committers.

  2. Check all the trees (mini-os, qemu-xen, seabios, ovmf, and so on) have the correct commits and all security patches applied. Tooling is provided for this.

  3. Negotiate release date options with PR personnel. Typically three to four days are needed to line up press briefings under embargo; PR personnel also weigh industry events. Press releases are typically done mid-week (Tuesday–Thursday).

  4. Select the release date.

  5. Specify the dates regarding support and security support in SUPPORT.md.

  6. Check with the relevant stakeholders (typically the Community Manager) whether the release documentation and PR material are in good shape.

  7. Obtain a formal go-ahead from the Community Manager and the Release Technician. Ask them to dry-run their checklist and confirm everything is OK. If not, arrange another release candidate and restart this checklist.

  8. Do not commit to a release date until the exact xen.git commit id to be released is known and has been satisfactorily tested.

  9. Give PR personnel the final go-ahead and instruct the Release Technician to make the release deliverables (tags and tarballs — usually in place the day before the release). At this point PR collateral goes to reporters, typically two to three working days before the release date, and publications cannot be undone without questions being asked and risk of negative PR. It is acceptable to make a xen-devel announcement before the PR release date (blog, xen-announce, press release).

  10. Make the announcement on the various mailing lists and publish the blog post.

Allow for contingencies. It is not uncommon for last-minute bugs, security or otherwise, to be discovered. Providing a fix takes time and testing the fix takes more — allow at least one week from getting a fix to getting a push. For security bugs, coordinate with the Security Team to adjust the dates according to the security policy.

Handing over

If there is a new Release Manager for the next release, make sure the following happen for them:

  1. Access to the community test infrastructure is granted. In the common case the public pages at logs.test-lab.xenproject.org will suffice.

  2. Access to the mailing-list moderation panel is granted.

  3. An account for blog.xenproject.org is created. The new Release Manager can create the account themselves; it might be necessary to adjust the access rights afterwards.

  4. An account for wiki.xenproject.org is created. As with the blog account, the new Release Manager can create it themselves, with access rights adjusted afterwards if needed.

Email templates

If you want specific actions from committers, make sure you CC committers@.

Release candidate announcement

Subject: Xen X.Y rcZ

Hi all,

Xen X.Y rcZ is tagged. You can check that out from xen.git:

https://xenbits.xen.org/git-http/xen.git X.Y.0-rcZ

For your convenience there is also a tarball at:
https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z

And the signature is at:
https://downloads.xenproject.org/release/xen/X.Y.0-rcZ/xen-X.Y.0-rcZ.tar.[glx]z.sig

Please send bug reports and test reports to xen-devel@lists.xenproject.org.
When sending bug reports, please CC relevant maintainers and me
(abc@xyz.com).

As a reminder, there will be another Xen Test Day.

See instructions on: URL_TO_TEST_INSTRUCTIONS

Forego control of the tree

Subject: No Release Ack needed before RcX

Committers,

The tree is in good state. No release ack is needed before RcX. Please commit
bug fixes at will.

$RM

Commit moratorium

Subject: Commit moratorium for $REASON

Committers,

Please don't push any new patch to staging because $REASON.

Another email will be sent once the moratorium is lifted.

$RM

Lift commit moratorium

Subject: Commit moratorium is lifted for $REASON

Committers,

The commit moratorium is lifted, please commit patches that are already
Release-acked.

$RM

Reminder of last posting date

Subject: Last posting date for Xen X.Y is $DATE

Hi all,

The last posting date for Xen X.Y is $DATE. If you want your features to be
included for the release, please make sure they are posted for the first
time before $DATE.

$RM

Reminder of cut-off date

Subject: Cut-off date for Xen X.Y is $DATE

Hi all,

The cut-off date for Xen X.Y is $DATE. If you want your features to be
included for the release, please make sure they are committed by $DATE.

$RM

Release announcement

Subject: [ANNOUNCEMENT] Xen X.Y is released

Dear community members,

I'm pleased to announce that Xen X.Y.0 is released.

Please find the tarball and its signature at:

https://xenproject.org/downloads/xen-archives/xen-project-xy-series/xen-project-xy0.html

You can also check out the tag in xen.git:

  https://xenbits.xen.org/git-http/xen.git RELEASE-X.Y.0

Git checkout and build instructions can be found at:

https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes#Build_Requirements

Release notes can be found at:

  https://wiki.xenproject.org/wiki/Xen_Project_X.Y_Release_Notes

A summary for X.Y release documents can be found at:

  https://wiki.xenproject.org/wiki/Category:Xen_X.Y

Technical blog post for X.Y can be found at:

 URL_TO_BLOG

Thanks everyone who contributed to this release. This release would
not have happened without all the awesome contributions from around
the globe.

Regards,

$RM (on behalf of the Xen Project Hypervisor team)