Development Cycle
[edit] Executive Summary
After many years of a traditional release model (where we define a number of new features and publish the release once they have all been implemented,) the MySQL server has now a new dynamic model for its development.
The main goal of this model is to have more frequent and timely GA releases of the MySQL server, which allow a faster integration of new features and complex changes. There is a presentation by Tomas Ulin about The New MySQL Release Model available from the MySQL University page.
Users who are only interested in maintaining a stable server will not be affected. Users who want to try out new versions will now have a broader choice and more frequent releases for testing.
Contributors and early adopters will be pleased to notice that this more dynamic model makes their life easier and their choices more interesting.
The basic principles of this model are:
- The trunk is always at least in beta quality.
- A milestone starts in beta quality (never in alpha) with a merge between the trunk and a stage tree;
- Milestone releases, with RC quality, released every three to six months.
- Integration windows between milestones allow the insertion of new features from staging trees
- GA releases happen every 12 to 18 months;
- There are not more than two releases in active support.
Presentation slides:
[edit] Background and The Concept of the trunk
MySQL has a long history of maintaining multiple development branches. Back in 2002, when 4.0 version became stable, we already had both 4.1 and 5.0 in alpha. Later, 5.1 was already under development before 5.0 got stabilized. Before adopting this new model, we had a similar situation with 6.0 and 6.1.
The need for maintaining multiple development branches stems from large, long-running projects, such as stored procedures, partitioning, online backup, foreign keys. These projects have a long history, and commitments are often made for them long before a delivery timeline becomes clear.
Maintenance of multiple actively developed versions leads to excessive merges, when each bugfix needs to be merged up from the GA tree to all development trunks, with unnecessary feature dependencies, when a complete feature has to wait for a remaining alpha code to stabilize. This creates consequent restrictions on the scale of changes, with temptations to add features to GA releases.
It could be argued that these problems stem from having a low quality code accepted into the main tree in the first place. However, the concept of version-tagged trees and early assignment of features to versions implies that a feature gets a "home version" as early as possible, and, consequently, favours version-driven planning.
This release model aims at releasing a stable server more often. With this model, it should be easy to get a good patch into the main tree. Merges should not be excessive, and should not restrict our ability to do re-engineering. Development cycles must become shorter.
The way we achieve these goals, is to move to a paradigm with a single main tree -- the trunk -- which has often, repetitive changes in its life cycle, and regularly reaches a stable state.
[edit] Life cycle of the trunk. Milestones
A period in the life cycle of the trunk when it is first open to accept a few new features (the Integration Window), then closed to features and only open to bugfixes, and finally matures to reach RC quality, forms a milestone. When the trunk with the new features, which were accepted at the beginning of the milestone, matures to stable state (RC quality), it's said that the milestone is reached. In order to shorten the period necessary to reach a milestone, only few complete features are accepted at a time. Any RC qualifying bugs that are discovered during the stabilization period is fixed before the end of the period, and if not all RC qualifying bugs are fixed in a set time frame, the unstable feature is taken out of the trunk.
In total, a milestone cycle, from IW to stabilization to RC quality, is somewhere between 3 to 6 months.
[edit] Preparing the Milestone, staging features for the Integration Window
In preparation for the Integration Window opening, features are brought together in a staging tree, integration testing is performed, and all beta qualifying bugs discovered are fixed by feature owners. If not all beta qualifying bugs can be fixed in time, the unstable features are removed from the staging tree.
[edit] Trunk acceptance criteria
The owner of the current milestone (release manager) is responsible for delivering it on time. In order to do that, she selects for inclusion features that are sufficiently mature.
In order to be accepted into the trunk a patch needs to:
- have a bug report or a worklog task for it;
- fulfill the requirements of the architecture process and pass code reviews;
- pass the QA sign-off;
- pass the regression test suite (normally within the staging tree);
- have no open beta qualifying bugs.
In other words, any larger changes, e.g. complex bug fixes or code refactoring efforts, need to mature in a feature tree and staging tree before it can get into the trunk.
Once approached by the milestone owner, the feature owners can accept the proposition and work to make sure that their feature meets the acceptance criteria by the time the milestone IW opens.
If these criteria are met in time, the patch can be added to the staging tree (or in some cases directly to the Trunk).
Only a few features are added in a single milestone. A sizable feature, like online backup, or performance schema, or foreign keys, deserves a milestone all for itself.
[edit] Taking features out
MySQL had a history of accepting alpha quality code into the main tree. It wouldn't become apparent that the code is not ready until it sees the light of the day and is reviewed by key engineers/testers. Integration testing also can reveal design problems formerly unknown of.
If a feature that was added in scope of a milestone doesn't pass the integration testing, i.e. by the time the IW open still has beta-level bugs open against it, it's taken out of the staging tree and matures in a local tree.
It's the responsibility of the owner of the feature, with help of the merge captain1 , to remove it from the staging tree.
1 The merge captain is an engineer who pulls features from the stage tree and checks if they pass the acceptance criteria.
The following ways to remove features from the staging tree or the Trunk are possible:
- It should be possible to either completely disable it (if it's a plug-in), or #ifdef it out. A feature that is not mature is #ifdef'ed or disabled.
- All the source files and feature-related changes are removed from the source tree;
- The entire milestone is abandoned. The trunk is rolled back to the previous stable state, a new milestone is started, and the process starts anew for all features except the offending one.
[edit] Releases during a milestone
At the end of a milestone cycle, before new features of the next milestone are accepted to the Trunk, a last release of the milestone is released. We call this the Milestone Release. Prior to this release we will be doing regular public releases of the milestone candidates to enable community testing. These releases of a milestone starts directly after the IW closes, and the "beta1" release of the milestone is produced. Followed by "beta2", "beta3", etc.
[edit] GA releases
A Milestone Release is a candidate for becoming a GA release. Well in advance of completing a milestone, a decision will be made if this Milestone should become GA.
A milestone is brought into GA by branching off the Trunk into a release Branch. From the release branch release candidates and finally GA releases are produced. This release branch is maintained throughout the lifecycle of the GA release, and it is also there that the bugfixing occurs.
[edit] At a glance - before and after
[edit] DOWN FROM THIS POINT, STILL DRAFT
[edit] ************** Developent Cycle details *************
This section is still a living document where further details on the development cycle is being as still being worked upon.
[edit] Support
We can have a limited number of versions in active support. Consequently, we won't be able to fully support all milestones. It is suggested that only the current stable milestone is supported. The level of support commitment is to be developed by the support team.
[edit] Fixing milestone bugs
To reduce merge work, we'll be able to fix bugs in the current milestone only. If a bug is discovered in a previous closed (achieved) milestone, the bugfix goes directly into the trunk and is released as part of the current milestone.
[edit] Backward compatibility and feature deprecation
There is no backward compatibility between milestones. Features may move in and out.
[edit] Where do all the alpha features live? Staging trees
Since the trunk sets a fairly high quality standard, and certain features may require significant changes, a notion of staging trees are introduced. Patches can flow in any direction between staging trees, and all decisions about what's in a tree makes the owner of the staging tree, which can be anyone. It's expected that most teams will have a staging tree for their development work, and some teams and individuals will share a staging tree if they work on overlapping functionality.
[edit] Roles and responsibilities
- Milestone owner: decides on the milestone definition (what feature gets in), gets management/community credit to set milestone duration
- Merge captain: verifies that all patches in the patch set that is accepted into the trunk satisfies the acceptance criteria. Works with technical team merge captains to accept features in. Is responsible for keeping the main trunk pushbuild green.
- Feature developer: an engineer or a team, including QA, working on a feature in a local tree. Ensures that the feature meets the trunk acceptance criteria. Feature developer owns the feature tree. Once a feature is in the trunk, before the trunk gets mature, the feature developer is assigned to work on bugs in the trunk.
- Staging tree owner: a host that owns a tree that has multiple alpha features. Pulls from feature trees and from the trunk and follows up to make sure everything works together. Never pushes anywhere.
[edit] Appendix A. Terminology
- A milestone - a set of features, a landmark that we aim to reach. A milestone is reached when all features accepted into the trunk for this milestone are stable. Until a milestone is reached, the trunk is not open for new features.
- A milestone release - the last release of a certain milestone before the trunk transitions back into alpha. E.g. when a merge window closes, we may decide to do a first release (build) of the milestone, e.g. 5.5.1. Then we stabilize the trunk further, and build and make available to community again. That will be 5.5.2. As soon as the trunk reached stability level we feel is good enough, we make the final build -- the release of the milestone, and the trunk transitions to alpha state of the next milestone.
- A GA release -- at some point, we decide that a certain milestone is a source of a long-time support version we would like to make available to our customers. We clone off the latest stable milestone and tag it as a GA.
[edit] Appendix B. Frequently asked questions
Q. How does the concept with a single trunk maps to the current tree situation?
A. 5.5 becomes the trunk. 6.0, 6.1-fk and 6.1-globalization become staging trees. Alpha features are taken out of the trunk into new staging trees. The result reaches maturity state and is released as the first milestone.
Q. How is a milestone different from 5.1 or 6.0 or any other release, done right?
A. Limited support. Limited life time. Lack of backward or forward compatibility. Fixes for some late bugs in a milestone are most likely to get published in the next milestone only, unless there is a volunteer to backport them.
Q. How is a milestone is different from a development tree snapshot?
A. Milestones are released when they are stable. We aim to shorten the release cycle by frequently (every few months) stabilizing the trunk. Each new milestone contains only few new features, all of which passed integration testing, whereas a snapshot of the trunk contains everything that passed a code review.
Q. Why is it important to have a single trunk? Why can't we have multiple development trees, as in the past?
A. Merging. Re-engineering code. Massive changes. Feature integration. Undesirable dependencies and inability to release due to alpha-quality features blocking stable features.
Q. With milestone-based planning, it can be difficult to produce marketing materials that promise particular features in a version: MySQL story with "Cobra", "Jaguar" or "Fortress" labels attached to future versions will become nonsensical.
A. Yes.
Q. Who makes the decision whether my feature is included into the trunk?
A. There is no single authority to make that decision. The three key participants in making of it are:
- Feature owner: needs to have a feature ready, and commit to fixing bugs in it after it gets included into a staging tree for the next milestone or into the trunk.
- QA responsible for the feature and for the milestone: their job is to test the feature, report bugs in it, and most importantly, verify that the feature passes QA acceptance criteria. If a feature doesn't pass acceptance criteria after it's been pushed into the trunk, it's removed from the trunk
- Milestone owner, for the milestone in which the feature will land. Together with the merge captain on the milestone, the owner of the milestone is making the milestone happen, coordinating with feature owners, QA and other engaged parties.
Other arrangement violate of the described process.
Q. How does a single development trunk works when attempting to work on multiple parallel releases?
A. It does not. We don't try to work on multiple parallel releases any more. Working on multiple parallel features is still okay, in feature trees.
Q. What happens with the trunk when it's time to release a GA version?
A. Please see the PDF with the merge diagram in "Merge flow" section.
[edit] Appendix C. References
- http://wiki.postgresql.org/wiki/CommitFest_Help
- http://www.youtube.com/watch?v=L2SED6sewRw
- http://en.wikipedia.org/wiki/Linux_kernel#Development_model
- http://ldn.linuxfoundation.org/book/2-how-development-process-works
- http://ldn.linuxfoundation.org/book/how-participate-linux-community
- http://ldn.linuxfoundation.org/book/23-how-patches-get-kernel
- http://ldn.linuxfoundation.org/book/24-staging-trees
- http://httpd.apache.org/dev/guidelines.html
[edit] Appendix D. Comments and criticism
Put your comments here in a separate section.
[edit] Top-down merge flow
I would like to strongly encourage moving to the top-down merge flow. I have written a long article about why top-down flow is better. In brief, it reduces risk and bureaucracy, encourages cleaner code, and saves time. --Timothy Smith 01:36, 3 April 2009 (CEST)
Totally agree with Timothy for three reasons:
- Today we merge bugfixes bottom-up and features top-down, which leads to collisions, and wasted time
- It is more important that the bugfix is in the latest revision and every revision after that, than in an early revision (of some merges are forgotten)
- It is most important to have the bugfix fixed correctly in the latest revision and every revision after that, than in an early revision (e.g. a bug with a common signature is fixed in 5.0 code, but the same signature also exist in added feature in 6.0, a bottom-up merge would fix only one of the places)
"Because this is how we have always done it on MySQL" is not an argument for the opposite, just an observation.
-- Jørgen Austvik 15:42, 27 April 2009 (CEST)