Release process for arXiv-NG

This section describes the major stages of new software components that we release as part of the core arXiv platform. These stages are intended as a guide, to be adapted as appropriate given the nature of the software and other prevailing factors.

The decision to release software to production rests exclusively with the arXiv IT Team Lead and/or Lead Architect.

Minor Releases

Minor releases fix major bugs, and may introduce some new functionality in a backwards-compatible way. Many of the considerations that apply to major releases also apply to minor releases: attention to quality and performance goals (including accessibility), communication strategy, etc. Minor releases need not proceed through the same alpha/beta/production workflow. Effort should be made (but is not required) to solicit dev team review prior to production releases of minor versions.

Major releases

Pre-Alpha

Prior to the first related sprint, the team working toward a release should address the following items.

  1. Milestone review

    • Toward what milestone does this release contribute?

    • Is the milestone scoped clearly and appropriately?

    • Is the release scoped clearly and appropriately? What are the feature goals for this sprint?

    • What are the dependencies of this release?

  2. Identify/review quality and performance goals:

    • Accessibility. Minimum: WCAG 2.0 level A or better.

    • Code quality goals (linting, documentation, static checking, test coverage).

    • Performance goals (error rates, latency, etc).

  3. Identify opportunities for incorporating user feedback, as applicable:

    • Quantitative input, related to performance goals.

    • Qualitative input, e.g. surveys, focus groups, etc.

  4. Set a minimum and maximum duration for alpha review; preferably in the ballpark of one to two weeks. This should be long enough to get broad feedback, but short enough to keep development moving.

  5. User input:

    • Do we have a clear understanding of user expectations and requirements?

    • What additional user input is necessary or would improve the quality of this release?

    • Plan and implement collection of user input, e.g. through surveys, focus groups, usability studies, etc.

Alpha: Does It Work?

Source code for an alpha release is tagged with M.m-alpha.

  1. The majority of and/or highest priority functional goals for the release have been met.

  2. Bugs are expected to exist, and known bugs may not yet have been fully addressed.

  3. The release is deployed for internal review. This may be on the on-premises “beta” server, or in a staging environment in the cloud.

  4. Dev and operations teams all participate in review, and report bugs, improvement requests, and other feedback to the feature team. A two-stage alpha review will first target the core operations team, and then broaden testing to volunteer moderators and close partners.

  5. Feature team begins to prioritize and fix bugs, performs load testing, and works to evaluate the release against quality and performance goals.

Pre-Beta

During and after alpha testing, the release team should address the follow items:

  1. Identify fixes, remaining features, required for beta release.

  2. Performance testing, such as load tests.

  3. Address remaining quality deficiencies, e.g. low test coverage, incomplete documentation, etc.

  4. Accessibility review: complete checklist for target WCAG 2.0 level.

  5. Developer review: provide an opportunity for other developers who are not on the release team to provide constructive feedback.

    • The primary objective of the developer review is to ensure that the code is of sufficient quality and documentation is sufficiently complete to be comprehensible to other developers.

    • Feedback should focus on increasing quality and comprehension, and identifying any critical flaws that may have been overlooked.

    • Developer review is usually facilitated by a PR from develop to the master branch.

  6. Architectural review: the Lead System Architect will review the software for consistency with existing or planned components, and for adherence to design and performance constraints. This review may surface refactoring or other revision that will need to take place before the software can be released.

  7. Decide on minimum/maximum duration for beta testing. This may be longer than alpha testing, but not indefinite.

  8. Review and implement mechanisms for collecting user input (if applicable).

  9. Review all # TODO items in code and address in some manner.

Beta: Did we get it right?

Source code for an alpha release is tagged with M.m-beta.

  1. The release is feature-complete (or very nearly so), and bugs/changes identified in alpha testing have been addressed.

  2. The release is deployed for stakeholder/public review. This will be on the production arxiv.org cluster, or in a publicly accessible cluster in AWS.

  3. Releases that replace classic functionality will usually be deployed in parallel to the routes/interfaces that they replace. For example, a release that replaces /search would be deployed at /search-beta.

  4. A news item is added to arxiv.org briefly describing the beta release, foreshadowing future related milestones, and providing a link to information about providing feedback.

    • This might be in conjunction with a blog post, tweets, emails, etc., depending on the volume of input desired.

    • This may be done in multiple cycles, e.g. with the first cycle announced only to a small group of users/stakeholders, and a subsequent cycle announced more broadly.

  5. The feature team works to address emergent bugs, and continues to evaluate quality and performance goals in light of live traffic in the production environment.

  6. The feature team specifies a rollback strategy if the transition out of beta to full production release is unsuccessful. This may lead to one or more sprints to address issues brought up in beta, generating release candidates for internal dev team review.

Live: It’s Not Over!

The release is feature-complete, and the dev and ops teams are satisfied that

quality and performance goals have been met.

If the release is replacing a classic interface/endpoint, a decision may be made to deploy the release at a different route. E.g. old route: /find, new route /search. In that case, all references to the old route/interface would be updated to the new route/interface. A deprecation notice with end date would be placed at the old route and other communications channels (news item, tweet, blog entry, etc).

We expect that the initial release may surface unexpected bugs, or additional improvements/features that can be implemented relatively quickly. The feature team should anticipate one or more minor releases shortly after the initial major release to address these items should they arise.