Ubuntu Explained: How to ensure security and stability in cloud instances—part 1

The LTS philosophy, releases, updates and repositories explained

Since we launched Ubuntu Pro’s Expanded Security Maintenance for additional packages, and we now integrate more closely with public cloud update management tools, more people have been asking us about the Ubuntu archive and how this is divided and security patched. In this blog we will therefore explain Ubuntu releases, how the Ubuntu archive is divided into different sections and how we deliver Ubuntu updates.

This blog is the first in a three-part series by John Chittum (Engineering Manager, CPC) and Aaron Whitehouse (Senior Public Cloud Enablement Director) on how to keep your Ubuntu cloud instances secure and stable. The first step to understanding how to keep your Ubuntu instances stable and up-to-date is to understand the philosophy of Ubuntu’s Long Term Support (LTS) releases, how the archive is structured and how packages are updated.

Ubuntu releases

Creating an Ubuntu release

Every six months, Canonical releases a new version of Ubuntu, named after the year and month it was released (YY.MM). Every two years is a “Long Term Support” (LTS) release, which is maintained, security patched and supported by Canonical for a much longer time. The Ubuntu Release Cycle page is a great resource to understand the cadence of Ubuntu releases:

The world of free and open source software is constantly moving and the software that makes up Ubuntu is continuously being updated with new features, bug fixes and security patches. When releasing Ubuntu, we need to balance providing the latest features with stability and security. In the lead up to each release, Ubuntu takes a copy of the software versions at that date to provide the starting ground for the release and any future updates.

Updating packages in a release

Once we have created a release, the general rule is that we only apply security and bug fixes to those versions, rather than new features. This keeps the release as stable as possible. To give a concrete example, in Ubuntu 20.04 Focal Fossa, we chose sudo 1.8.31 as the long-term supported sudo. Sudo in Ubuntu 20.04 will only adopt a new version in exceptional circumstances during the lifecycle of the release. If Ubuntu’s security team patch a security vulnerability, the update will normally fix the specific issue in the version of sudo that was in the release, rather than bringing in the latest version of sudo. That means that the sudo version will still show as 1.8.31 (as that was the version in the release), but we will increment the -1ubuntu1 suffix to show the additional patch that we applied.

Continuing the example, the upstream project did not fix CVE-2021-3156 until 1.9.5p2 but we included it as a security update for Focal in 1.8.31-1ubuntu1.2. That 1.8.31-1ubuntu1.2 package contains the security fix even though the version looks lower than 1.9.5p2. Some security teams can find this confusing and some security tools implement it incorrectly. For an authoritative source of what packages may have outstanding vulnerabilities, check the Ubuntu CVE Tracker.

Non-security updates

Non-security updates to Ubuntu releases need to follow the Stable Release Updates (SRU) policy. This policy has strong statements regarding what is allowed to go into a release. The goal is to ensure stability and security for platforms, and not introduce breaking changes. If a user requires a newer major version of an application than the one that came with the release, the options are:

• move on to a later release of Ubuntu;
• use a snapped version of the application or a version from backports if available; or
• build from source and maintain.

Some third-party projects/vendors offer PPAs containing Ubuntu debs. There is some risk in adding a third-party PPA as (unlike a snap) packages from a PPA could replace other important packages on your system.  You can find more information on this, and some practical advice on limiting risk, here.

The archive repository pockets

Ubuntu builds on the Debian architecture. Debian archives are split into “distributions”/”suites” (the release) and “components”. Distributions are large areas of the archive, and components are slices. Example (using both the one-line Debian source format and deb-822 format):

One-Line:

Deb-822:

Enabled: yes
Types: deb deb-src
URIs: http://archive.ubuntu.com/ubuntu
Suites: jammy
Components: main

Ubuntu publishes updated packages to two different components: $SUITE-updates and $SUITE-security. These two components have different rules and we update them at different times.

-security

The Ubuntu Security team triage and assess whether an update is a significant security issue. Security updates are placed in the security pocket first. The Security team also publish CVEs and Ubuntu Security Notices on their pages. The Security team patch the highest version of the package from either the -updates or the -security pocket. They release this patched version to the -security pocket. As such, -security can end up containing some changes from -updates. This is necessary to ensure all users (including those using -updates) can always upgrade to the security-updated version (the version number has to be strictly greater than anything prior to that from both -updates and -security).

-updates

New package versions published first to the -updates component are bug fixes. We will only introduce functionality in exceptional cases, as specified in the Stable Release Update (SRU) procedure discussed below. There are some exceptions for specific packages detailed in that policy. One exception is where upstream produce new “microreleases” that bundle patches together. If an upstream produces microreleases with tight requirements and testing, we may bring in these microreleases rather than individual patches. One such example is DPDK.

-proposed

Discussed further below, -proposed is where packages are placed for testing before entering -updates.

Expanded Security Maintenance pockets

Expanded Security Maintenance (ESM) updates are provided as part of Ubuntu Pro and appear as additional sources in /etc/apt/sources.list.d/. Both esm-infra and esm-apps have a -security and -updates pocket that broadly follows the approach above for ESM packages. You can find further detail on Ubuntu Pro and the difference between esm-infra and esm-apps in the Ubuntu Pro FAQ. You can launch Ubuntu Pro, or in-place upgrade your existing Ubuntu Server instances, on AWS, Azure or Google Cloud Platform. The cloud platform will meter and bill these for you natively, through your normal cloud account.

Publication process for Ubuntu updates

Now that we have covered the pockets of the archive, let’s cover how updated packages reach those pockets. The maintainers of packages in Ubuntu publish them atomically over time.

One important piece of information is that the security and updates components may not be in sync. We will publish all updates published to -security to -updates as well, lagging a few days. However, we will normally only publish -updates packages to -security if we have also patched a security vulnerability in that package.

Stable Release Updates

The Stable Release Updates (SRU) policy sets out the process to publish an updated package to -updates. The SRU policy aims to minimise regressions and further bugs when applying patches to stable versions of packages. All changes being made to a Debian package in a stable Ubuntu release must have an accompanying bug. The bug report must explain the issue being fixed, regression possibilities, and testing procedures. Once we have applied a patch, we will normally “age” the package in -proposed for at least seven days. This gives testers a chance to verify the bug is addressed and no regressions found. We generally see regressions as a very poor user experience and will often count them as release blockers. No policy or testing is perfect; however, following this procedure helps Ubuntu ensure stability.

Critical and embargoed CVEs

We publish critical or embargoed CVEs, and some coordinated release date (CRD) CVEs, directly to -security. There is not a -security-proposed pocket. There is a public PPA, but the PPA is only used for immediate testing. We do not publish embargoed CVEs publicly prior to the CRD date. This is to ensure we deliver the fix to end-users at the same time the exploit is publicly published.

Testing

Packages generally have a test suite included. The Ubuntu process for including a package in the main repository (MIR), for example, requires that the package has a non-trivial test suite. Tests are generally expected to be run at package build time, making the build fail if the tests fail. The package should also contain autopkgtests, with results published publicly in several locations (including the bug). These tests include unit, functional, and integration tests. We also run a further test called a reverse dependency, or rdep, test. This triggers tests of all packages that depend upon the package under test, with the proposed package installed. This is to ensure that a change to the package does not cause an issue with an already-published package.

Conclusion

The Ubuntu release and archive processes aim to balance stability and security. The starting point is that packages will not receive new features, but will only receive security and bug fixes. We hope this blog has helped you to understand the philosophy and the structure of the Ubuntu archive. In the next blogs in this series, we will build on this to cover managing updates of Ubuntu cloud instances. Keep an eye out for new posts.

Update: See Part 2 of this blog series on best practices for updating individual Ubuntu instances. Part 3 then extends this to cover best practices for updating multiple Ubuntu instances while maximising stability of workloads on top.