Manage FIPS-enabled Linux machines at scale with Landscape 23.03

You or your organisation are tasked with hardening your workstations and servers, where do you begin? Installing Ubuntu and applying all the security patches is a good place to start, but what else is needed? The National Institute of Standards and Technology (NIST), a cybersecurity agency established in 1988, has published a series of security requirements for cryptographic modules since 1993. Instead of approaching hardening from a blank slate, anyone can benefit from NIST’s ongoing work in under 5 minutes, and have the strongest cryptography and hardening posture, when using Ubuntu.

FIPS, Ubuntu Pro, and Landscape: a brief introduction

The data security and regulatory compliance guardrails published by NIST are known as Federal Information Processing Standards (FIPS). Configuring Ubuntu to comply with FIPS provides data protection and risk management assurances. If you’re interested in having an auditable and cryptographically secure posture, or if FIPS is a contractually specified requirement for you or your organisation, this article will explain how to enable and manage FIPS on a single machine, or on many machines at scale. By following the short steps outlined in this article, your entire Ubuntu estate, inclusive of your Linux systems management software, will comply with FIPS. It’s worth noting that Canonical’s tooling to automate FIPS makes one-way changes to the system. The goal is to apply automations to improve your security and compliance posture, and there is no corresponding automation to revert these changes.

Ubuntu is published regularly in one of two types of versions. Interim releases happen every 6 months, and long term support (LTS) releases are published every 2 years. Features are delivered rapidly through interim releases, and are matured and promoted to LTS releases. The software used to build Ubuntu is contained in the Main repository, and comprises approximately 2,300 packages. Since Ubuntu’s inception, Canonical has provided free security patches for these 2,300 packages for the standard support period, which is 6 months for interim releases and 5 years for LTS releases.

Since 2017, Canonical has provided expanded security maintenance (ESM) for the 2,300 packages in the Main Repository, and gave the Ubuntu 12.04 LTS users an extended runway of an additional 5 years to transition their workloads to the next LTS. This provided users with a stable and secure operating system, and continues today with a deep history of reliable and uninterrupted updates, and upgrades.

Beyond the Main repository, Ubuntu also provides optional access to a vast and growing assortment of 23,000 open source packages in the Universe repository, and a rich selection of proprietary and non-open source software in a Multiverse repository. Historically, Ubuntu users relied upon open source software maintainers to publish security patches to the Ubuntu universe repository, but these open source projects didn’t all have 10 year security patching windows, like Ubuntu. This year, Canonical includes security patching for all software in the Ubuntu Universe Repository, with the release of  Ubuntu Pro. Ubuntu users don’t have to maintain a tangle of software repositories to fetch their open source software from trusted channels. With Ubuntu Pro enabled, Canonical triages vulnerabilities, handles the fixes, and back ports the fixes as needed. There is no tangle of software repositories, the user just has to install the software from main or universe. Beyond access to a broader scope of security patches, Ubuntu Pro also includes a variety of other features, including access to any Landscape edition.

Landscape is Canonical’s systems management solution for individuals or organisations that use Ubuntu. It provides access to a wide range of administrative functions that encompass inventory, automation, hardening, compliance, reporting, and software distribution. In the context of FIPS, Landscape can inventory cryptography software that is used across your estate, and also be configured to monitor FIPS compliance over time. The automation to conform with FIPS on Ubuntu can be orchestrated at scale, targeting your entire Ubuntu estate. This capability is available in all three available Landscape editions:

• Self-hosted Landscape can be installed and maintained by anyone, anywhere.
• Managed Landscape is installed and maintained by Canonical, either on-premises or on a public cloud.
• Landscape SaaS is installed and maintained by Canonical on Canonical’s cloud, and available as a cloud service.

Landscape is a client-server systems management solution. Landscape Client runs on Ubuntu machines, allowing them to be centrally managed through Landscape Server. Landscape Server relies on PostgreSQL, RabbitMQ, Apache or HAProxy, Postfix, and other open source software to perform its functions. Before installing and configuring any software in a FIPS environment, it is important to perform the FIPS hardening step first. Landscape Server is an excellent example of an application which relies on cryptography software to communicate securely with machines over a network. The high-level considerations for installing Landscape Server on a FIPS compliant machine are relevant for any other application.

Perform FIPS hardening before installing applications

If the installed applications use cryptographic components and dependencies, the machine may not conform to FIPS if the hardening sequence is not correct. Ideally, FIPS hardening is performed as the first order of business when launching a machine, before installing and configuring any other software. On major public clouds this is streamlined by Ubuntu LTS Pro FIPS Server images, which are in a FIPS certified state as soon as you boot them up. You can verify what Ubuntu Pro entitlements are enabled by looking the Pro Client status output:

sudo pro status

Turn on FIPS-mode on Ubuntu

All Ubuntu machines include the Pro Client, which includes a 1-step FIPS hardening option for Ubuntu Pro subscribers. You can ensure you have this tool by updating your system and installing the ubuntu-advantage-tools package:

sudo apt update && sudo apt install -y ubuntu-advantage-tools

An Ubuntu Pro token is a string of characters, you can get your token from the Your Subscriptions tab of the Ubuntu Pro dashboard. You can attach your token to your machine like this:

sudo pro attach TOKEN # replace "TOKEN" with your Ubuntu Pro token

The difference between a FIPS certified and FIPS compliant configuration is the security patching posture. FIPS certification is a lengthy process, and software is certified at an exact version number. A FIPS compliant configuration on Ubuntu will take the certified software and apply security patches, without changing the major version of any package.

To enable a FIPS certified configuration, without security patches, run:

sudo pro enable fips

Due to the lack of security patches, this is not a recommended configuration. Unpatched vulnerabilities can be actively exploited, and in the context of cryptography software, this can put sensitive information at risk at a large scale. Instead, Canonical strongly recommends the FIPS compliant configuration, with security patches:

sudo pro enable fips-updates

Once these commands have been run, a system reboot is necessary to fully enable a FIPS configuration:

sudo reboot

Launch Ubuntu LTS Pro FIPS Server on public clouds

All of these steps can be omitted when using Ubuntu Pro FIPS images on a public cloud. These Ubuntu Pro FIPS images do not require a token, and FIPS does not have to be manually turned on, Ubuntu is in FIPS-mode out of the box. Ubuntu Pro FIPS server images are available on Amazon Web Services, Microsoft Azure, and Google Cloud Platform. These machines will be booted up in a FIPS certified mode, and the Pro Client can upgrade the machine to a FIPS compliant mode, with security patches:

sudo pro enable fips-updates

Enable FIPS at scale

There are one of two starting points when enabling FIPS at scale:

1. Machines have already been deployed, Ubuntu is already installed
2. Machines have yet to be deployed, nothing is installed yet

When machines have already been provisioned, and Ubuntu is installed, any push-based configuration management solution can be used to deploy the Landscape Client and run Pro Client commands to enable FIPS. Once machines are enrolled in Landscape, Landscape is a very capable Linux systems management solution. Landscape can configure and inventory users, groups, machines, software, and policies with point and click ease.

When machines have yet to be deployed, leveraging a configuration management solution like cloud-init is the easiest way to install Landscape Client and configure FIPS at first boot. Cloud-init YAML configuration files can be included within the installation medium, or side-loaded with provisioning solutions like MAAS.

Before enabling FIPS at scale, it makes sense to have the Landscape’s server component already deployed.

Install Landscape 23.03 on a FIPS hardened machine

Landscape Server can be installed on a single machine using the landscape-server-quickstart package, installable via the apt package manager. This package contains the Landscape Server application, and declares the following dependencies: PostgreSQL, RabbitMQ, and an Apache web server to serve pages on behalf of the Landscape application server.

Install prerequisites

The software-properties-common package is installed to get the add-apt-packages command line utility, this is necessary to add the PPA which contains the Landscape Server software to the machine. The add-apt-repository command is available in the software-properties-common package, which can be installed using this command:

sudo apt-get update && sudo apt-get install software-properties-common -y

Set the FQDN

Before installing Landscape Server, it is important to set the fully qualified domain name (FQDN) of the machine correctly. The FQDN is the address you will type into a web browser to access the Landscape dashboard, and is composed of a hostname and a domain name. Using the FQDN landscape-fips.rajanpatel.com as an example, run the following commands to set the HOST_NAME, DOMAIN, and FQDN variables:

HOST_NAME=landscape-fips
DOMAIN=rajanpatel.com
FQDN=$HOST_NAME.$DOMAIN

Replace the values of HOST_NAME and DOMAIN to reflect what you are using for your Landscape installation. It’s important to set the HOST_NAME, DOMAIN, and FQDN variables as shown above, and not skip those steps, because those variables will be consumed by other commands later on. When Landscape Server is installed, it will read the machine’s hostname and use it in the Apache configuration. Set the hostname with this command:

sudo hostnamectl set-hostname "$FQDN"

How to install landscape-server-quickstart with FIPS enabled

Next, add the repository for Landscape Server, and install it. The PPA for Landscape Beta is: ppa:landscape/self-hosted-beta. The PPA for the current stable version of Landscape is: ppa:landscape/self-hosted-23.03. These commands will install the stable version:

sudo add-apt-repository ppa:landscape/self-hosted-23.03 -y
sudo apt-get update
sudo DEBIAN_FRONTEND=noninteractive apt-get install landscape-server-quickstart -y

This could take approximately 300 seconds, depending upon the available network bandwidth, and the specifications of your machine.

How to install an SSL certificate with FIPS enabled

Landscape relies on secure communication to happen between Landscape Client and Landscape Server. Landscape’s web based dashboard and low-level HTTPS API use an SSL certificate for encrypted data transfer. Out of the box, Landscape will generate and use a self-signed SSL certificate. A self-signed SSL certificate has no external validation of identity, is vulnerable to man-in-the-middle (MITM) attacks, and requires extra configuration steps for limited protective value. Certbot is a command line utility which makes acquiring and renewing SSL certificates from LetsEncrypt an easy, free, and automated process. Certbot can be installed from either of two package managers native to Ubuntu: apt and snap.

Snap packages are designed to provide a high level of isolation and encapsulation for applications. To achieve this, snaps have the ability to include their own versions of dependencies, which are available only within the confines of the snap package. If a snap package ships dependencies that do not conform to FIPS, the machine has run afoul of the regulations. There is a certificate generation step in the Landscape Server installation, and leveraging the FIPS conformant OpenSSL libraries is appropriate. If the FIPS hardening step happened after Landscape Server was installed, the certificates would be caught in a FIPS audit.

When installing Certbot as a snap, the snap package includes the libcrypt dependency. Certificates generated by the Certbot snap may not use a FIPS compliant version of libcrypt. Using Certbot installed with the apt package manager on a FIPS hardened Ubuntu machine will use the system-level libcrypt, and conform with FIPS.

To conform with FIPS, install Certbot using the apt package manager:

sudo apt-get install certbot python3-certbot-apache -y

If your Landscape instance has a public IP, and your FQDN resolves to that public IP, acquiring a valid SSL certificate from LetsEncrypt is a 1 line activity. Change [email protected] to an email address where certificate renewal reminders can be sent.

sudo certbot --non-interactive --apache --no-redirect --agree-tos --email [email protected] --domains $FQDN

How to configure Postfix with FIPS enabled

At this point, visiting https://HOST_NAME.DOMAIN (in my example: https://landscape-fips.rajanpatel.com) prompts for creation of Landscape’s first Global Administrator account. To add administrators, click on Settings and set a valid outgoing email address. By default, the email address will be prefilled with noreply@HOST_NAME.DOMAIN (in my example: [email protected]). Once the email address is set to noreply@DOMAIN (or in my case: [email protected]), configure Postfix, so Landscape can send emails that will not get rejected or sent to spam.

Mailjet, SendGrid, Amazon SES, and Google are some of many email service providers that can be configured to work with Postfix. It is necessary to set the following additional variables that will be consumed alongside the DOMAIN and FQDN variables that were set earlier. Each of these email service providers have instructions for configuring Postfix, and your minimum configuration will typically entail setting the SMTP_HOST, SMTP_PORT, SMTP_USERNAME, and SMTP_PASSWORD variables, and some additional authentication and TLS configurations.

The following settings will work with SendGrid:

SMTP_HOST='smtp.sendgrid.net'
SMTP_PORT='587'
SMTP_USERNAME='apikey' # 'apikey' is the correct username for SendGrid
SMTP_PASSWORD='' # Use an API Key from: https://app.sendgrid.com/settings/api_keys

Postconf is a utility that configures the /etc/postfix/main.cf file:

sudo postconf -e myhostname="$FQDN"
sudo postconf -e mydomain="$DOMAIN"
sudo postconf -e myorigin="$DOMAIN"
sudo postconf -e masquerade_domains="$DOMAIN"
sudo postconf -e mydestination=localhost
sudo postconf -e default_transport=smtp
sudo postconf -e relay_transport=smtp
sudo postconf -e relayhost="[${SMTP_HOST}]:${SMTP_PORT}"
sudo postconf -e smtp_sasl_auth_enable=yes
sudo postconf -e smtp_sasl_password_maps=hash:/etc/postfix/sasl_passwd
sudo postconf -e smtp_sasl_security_options=noanonymous
sudo postconf -e header_size_limit=4096000

SendGrid requires TLS encryption when connecting on Port 587, so the following additional configurations are required:

sudo postconf -e smtp_use_tls=yes
sudo postconf -e smtp_tls_security_level=encrypt
sudo postconf -e smtp_sasl_tls_security_options=noanonymous

By default, postfix uses MD5 hashes with the TLS for backward compatibility. In FIPS mode, the MD5 hashing function is not available. SHA-256 is a secure cryptographic hash function that can be used with FIPS, so explicitly settings the SMTP TLS fingerprint digest is necessary:

sudo postconf -e smtp_tls_fingerprint_digest=sha256

Write /etc/postfix/sasl_passwd with the authentication credentials, and generate a hashed version of that file:

sudo sh -c "echo \"[$SMTP_HOST]:$SMTP_PORT $SMTP_USERNAME:$SMTP_PASSWORD\" > /etc/postfix/sasl_passwd"
sudo postmap /etc/postfix/sasl_passwd

Remove /etc/postfix/sasl_passwd for security, to avoid keeping passwords stored on disk in plaintext, and restart Postfix for these settings to take effect:

sudo rm /etc/postfix/sasl_passwd
sudo /etc/init.d/postfix restart

Success

Once machines have been registered with Landscape, it is possible to use the remote script execution capability of Landscape to interact with all the machines you are managing. For example, you could run Pro Client commands to enable FIPS, FIPS updates, Livepatch, and a number of other Ubuntu Pro entitlements. The Landscape Scripts Github repository has some excellent examples that show what can be achieved beyond monitoring and managing FIPS configurations, and Landscape’s default functionality. If you are inspired, feel free to submit a Pull Request with scripts you find helpful for managing your Ubuntu estate.

It is very common for FIPS deployments to be air-gapped, with limited or zero external network connectivity. Canonical’s Field Engineering team has experience with these deployments, and can help you achieve the perfect blend of resiliency, redundancy, and scalability for you and your organisation.

In summary, enabling FIPS-mode for application servers, such as Landscape, is a straightforward process. The Pro Client tool allows you to implement FIPS on any Ubuntu machine with ease. Cloud-init provides a simple YAML based configuration management solution for provisioning machines with Landscape and FIPS, both on-premise and in the cloud. In the “Deploy fully configured VMs in minutes on Google Cloud, using gcloud CLI and cloud-init” article, I dissect a cloud-init.yaml file that provisions a FIPS compliant machine and deploys Landscape with the manual steps outlined above. Stay tuned by subscribing to our newsletter.