Although my time was very limited due to project reasons over the last months and I didn’t change anything within my own IT, I recently noticed that RedHat released a new version of RHEL. What had been changed in the new release compared to the current one, can be reviewed in the RHEL 9 release notes https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/9.0_release_notes/index (may require registration). I haven’t yet dug deeper into the changes, but the most important change that popped to my attention is, that RHEL now officially supports in-place upgrades. This leads me to the first action with RHEL 9, test the upgrade!
Note: I’m using the official RHEL OS in my IT environment (Red Hat Developer Subscription for Individuals). The steps mentioned in this blog post are based on this entitlement. If you use RHEL clones like Rocky Linux, AlmaLinux and the like, the procedure may differ. In this case, please check the clone documentation first to see what procedure is applicable here.
So, how does this work? First of all, it’s required to upgrade the existing system to the latest patch level within RHEL 8 which is 8.6 as of writing. Older versions are currently not supported for an in-place upgrade. So, if you still run RHEL 7.x machines, you won’t be able to upgrade to 8.6 and as a consequence, you still need to reinstall the machine for the upgrade.
Note: Before you start an upgrade, make sure you followed all good practices around upgrades like backups, tests and the like. Every upgrade has the potential to break the box it’s executed on, be aware of this and protect yourself accordingly. I don’t take any responsibility or accountability that the described steps work. They just did it in my environment.
Let’s start! First of all, the box needs to be prepared for the upgrade. The easiest way to achieve this is to use the following Ansible playbook:
---
- hosts: all
become: true
become_user: root
tasks:
- name: Enable BaseOS repo for RHEL 8
community.general.rhsm_repository:
name: rhel-8-for-x86_64-baseos-rpms
state: enabled
when:
- ansible_facts['distribution'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Enable AppStream repo for RHEL 8
community.general.rhsm_repository:
name: rhel-8-for-x86_64-appstream-rpms
state: enabled
when:
- ansible_facts['distribution'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Lock subscription-manager version to 8.6
community.general.rhsm_release:
release: "8.6"
when:
- ansible_facts['os_family'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Remove file DNF release lock
ansible.builtin.file:
path: /etc/dnf/vars/releasever
state: absent
- name: Install DNF version lock
ansible.builtin.dnf:
name:
- python3-dnf-plugin-versionlock
state: present
when:
- ansible_facts['os_family'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Remove DNF version locks
ansible.builtin.command: /usr/bin/dnf versionlock clear
when:
- ansible_facts['os_family'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Upgrade RHEL and clones from major release 8
ansible.builtin.dnf:
update_cache: true
name: "*"
state: latest
when:
- ansible_facts['os_family'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
- name: Install required packages for leapp upgrade
ansible.builtin.dnf:
name:
- leapp-upgrade
- cockpit-leapp
state: present
when:
- ansible_facts['os_family'] == "RedHat"
- ansible_facts['distribution_major_version']|int == 8
The playbook installs the required packages, locks the version to 8.6 and installs the latest updates. Once the playbook has prepared the box, the documentation recommends a reboot.
Now we can start with the upgrade itself. The central command for the upgrade is leapp and we’ll use it with different parameters to perform the upgrade. The journey starts with the pre-upgrade check.
> leapp preupgrade --target 9.0
============================================================
UPGRADE INHIBITED
============================================================
Upgrade has been inhibited due to the following problems:
1. Inhibitor: Firewalld Configuration AllowZoneDrifting Is Unsupported
2. Inhibitor: Missing required answers in the answer file
Consult the pre-upgrade report for details and possible remediation.
============================================================
UPGRADE INHIBITED
============================================================
In my case, I got two issues I needed to solve first. The first one is around a configuration option in firewalld that is not supported anymore with the new RHEL version. The fix was simple, just deactivating the feature and the issue disappeared from the pre-upgrade check. The second issue is an issue where the upgrade tool needs feedback from the user. In this case, if VDO drives are used:
> cat /var/log/leapp/answerfile 2
[check_vdo]
# Title: None
# Reason: Confirmation
# ========================== check_vdo.no_vdo_devices =========================
# Label: Are there no VDO devices on the system?
# Description: Enter True if there are no VDO devices on the system and False continue the upgrade. If the system has no VDO devices, then it is safe to continue the upgrade. If there are VDO devices they must all be converted to LVM management before the upgrade can proceed.
# Reason: Based on installed packages it is possible that VDO devices exist on the system. All VDO devices must be converted to being managed by LVM before the upgrade occurs. Because the 'vdo' package is not installed, Leapp cannot determine whether any VDO devices exist that have not yet been converted. If the devices are not converted and the upgrade proceeds the data on unconverted VDO devices will be inaccessible. If you have any doubts you should choose to install the 'vdo' package and re-run the upgrade process to check for unconverted VDO devices. If you are certain that the system has no VDO devices or that all VDO devices have been converted to LVM management you may opt to allow the upgrade to proceed.
# Type: bool
# Default: None
# Available choices: True/False
# Unanswered question. Uncomment the following line with your answer
# no_vdo_devices =
I don’t use VDO drives, so the answer was true (the question was interestingly if no VDO drives are used):
leapp answer --section check_vdo.no_vdo_devices=True
A second pre-upgrade check didn’t show issues anymore and the box was ready for the upgrade. The upgrade itself takes some time which can cause some unwanted situations, especially when the upgrade is performed remotely. To ensure that the upgrade is not aborted because of unstable or timed out network connections, for example, tmux can be used. To start the upgrade use the following command:
leapp upgrade --target 9.0
When the upgrade has been finished, the system needs to be rebooted. This will take some time as the reboot is part of the upgrade itself. Instead of the manual reboot, you can add the parameter --reboot to the leapp upgrade command.
Once the reboot is finished, some post-upgrade activities need to be carried out. First, check if the upgrade was successful:
cat /etc/redhat-release
Red Hat Enterprise Linux release 9.0 (Plow)
uname -r
5.14.0-70.10.1.el9_0.x86_64
subscription-manager list --installed
+-----------------------------------------+
Installed Product Status
+-----------------------------------------+
Product Name: Red Hat Enterprise Linux for x86_64
Product ID: 479
Version: 9.0
Arch: x86_64
Status: Subscribed
subscription-manager release
Release: 9.0
When everything points to RHEL 9, start the cleanup activities. First, remove all packages from the exclude list:
dnf config-manager --save --setopt exclude=''
Identify the remaining RHEL 8 packages and remove them if appropriate:
rpm -qa | grep -e '\.el[78]' | grep -vE '^(gpg-pubkey|libmodulemd|katello-ca-consumer)' | sort
Finally remove the leapp packages itself:
dnf remove leapp-deps-el9 leapp-repository-deps-el9
RHEL 9 come with some enhanced security settings. For example, SELINUX should be enforced by default. Make sure that no open issues belong to SELINUX:
ausearch -m AVC,USER_AVC -ts boot
Edit /etc/selinux/config and set SELINUX=enforcing. Reboot the box and check with getenforce the SELINUX status:
getenforce
Enforcing
Additionally, check the current system-wide cryptographic policy:
update-crypto-policies --show
DEFAULT
The policy should be either DEFAULT or FUTURE. FUTURE is the more secure variant but may also break with existing applications, especially old 3rd party applications. The documentation highlight a few more security-relevant settings and tools that I haven’t covered. The full story could be found under https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/upgrading_from_rhel_8_to_rhel_9/applying-security-policies_upgrading-from-rhel-8-to-rhel-9. The next chapter is about troubleshooting in case required.