How we got here

The Centos distribution has always been a strange beast: how could it be that the main exclusively paid Linux distribution (Red Hat Enterprise Linux, or RHEL) had a totally free clone?
Therefore, and for reasons I will elaborate below, it did not surprise me that the conditions of use for CentOS have changed, although the method, the time and the way everything was communicated will be a case-study on how-not-to-do…
I was curious about the two alternatives that immediately presented themselves, Rocky Linux and AlmaLinux, so I decided to test the migration of a CentOS 8 system to both, since everything ended up with the End-Of-Life of CentOS8 on December 2021 instead of 2029 as many people had planned.

Fragmentation?

As always in the FOSS world, there is never a single answer to a problem - it’s a strength, and a weakness. In this case, I believe both projects have their target audiences, which should allow both them to keep going in strength:

We’ll see if both survive in the long term…

The test

For this test, I started from a set of VMs that simulate a typical Windows service network (AD-DC, SMB, printing, Group Policies) based on CentOS8 and Samba (compiled from source).

The task consists on doing the sidegrade of this server (TJGGS) to both CentOS 8 alternatives, Rocky Linux and AlmaLinux, and test basic functionality.

Setting up

The computers are in the DASILVAT domain and domain users can log on to computers:

LOGONSERVER=\\TJGGS
USERDNSDOMAIN=DASILVAT.LOCAL
USERDOMAIN=DASILVAT

I logged in to the server, called TJGGS, with PuTTY, from the RSAT client.
Out of the box, this server is just I left in June 2020, when I finished the assigment for CINEL where I used this set of VMs.

login as: formando
Authenticating with public key "rsa-key-20200615"
Passphrase for key "rsa-key-20200615":
Last login: Sat Jun 27 20:31:21 2020 from 10.12.140.99
[formando@tjggs ~]$ uname -a
Linux tjggs 4.18.0-147.8.1.el8_1.x86_64 #1 SMP Thu Apr 9 13:49:54 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux

The first thing we have to do is a general update. And since the repositories haven’t been updated for more than a year, we’d better use --refresh

1

sudo dnf upgrade -y --refresh

[...]
Transaction Summary
====================================================================================================================================
Install 22 Packages
Upgrade 518 Packages
[...]

After a restart, we went from Centos 8.1 to 8.4:

1

uname -a

Linux tjggs 4.18.0-305.19.1.el8_4.x86_64 #1 SMP Wed Sep 15 15:39:39 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

In the Windows clients there was a warning that credentials needed to be reintroduced; logoff, change of password, and when trying to logon again, I got an error that the computer wasn’t registered in the domain.

Very well, I logged on to Client with the local Administrator and tried first Remove-Computer -Credential DASILVAT\Administrator and then Add-Computer; always Access is denied.
Then I logged in to TJGGS and checked the logs; sure enough, /var/log/messages had an indication that sysvol had the wrong permissions:

chdir_current_service: vfs_ChDir(/usr/local/samba/var/locks/sysvol) failed: Permission denied. Current token: uid=3000025, gid=3000026, 7 groups: 3000025 3000026 3000035 3000036 3000037 3000006 3000011

Despite the danger, I broke out the grease scrubber:

1
2

samba-tool ntacl sysvolreset
systemctl restart samba-ad-dc

Now systemctl status samba-ad-dc showed the service as active and without errors. So when I tried again to logon to the Client computer, I got in, and the user’s GPOs were applied.
Solving the same issue on RSAT was a tad more involved; first I had to take the computer out of the domain using the GUI (Computer Preferences as local Administrator), and then I was able to open an Admin PowerShell and use the commands:

1
2

Add-Computer -Credential DASILVAT\Administrator -Domain DASILVAT
Restart-Computer

However, I began to see another problem in samba-ad-dc’s log; maybe because I refreshed the sysvol ACLs, now the error comes from the shares' permissions:

chdir_current_service: vfs_ChDir(/mnt/dumdum/recursos) failed: Permission denied. Current token: uid=3000041, gid=3000026, 7 groups: 3000041 3000026 3000035 3000036 3000037 3000006 3000011

This token, of course, matches the computer that I just reintroduced into the domain:

1

wbinfo --sid-to-name $(wbinfo --uid-to-sid 3000041)

DASILVAT\WIN7-RSAT$ 1

This was fixed by adding the Domain Computers group to the share’s permissions (Read was enough)
Now that we have the system more or less stabilized, we can test TJGGS' migration from CentOS8 to Rocky Linux and to AlmaLinux

Rocky Linux

Rocky’s documentation page for the migration process is at https://docs.rockylinux.org/guides/migrate2rocky/, and it’s advisable to be logged-on as root.
To avoid the curl <url> -o <script>.sh method, which can be insecure if there is a Man-In-The-Middle attack, start by installing git:

1

dnf install git -y

Last metadata expiration check: 2:05:12 ago on Sun 31 Oct 2021 09:14:58 PM WET.
Package git-2.27.0-1.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

Download the migration scripts from the repo and run the migration script:

1
2
3

git clone https://github.com/rocky-linux/rocky-tools.git
chmod u+x ./rocky-tools/migrate2rocky/migrate2rocky.sh
./rocky-tools/migrate2rocky/migrate2rocky.sh -r

[...]
Done, please reboot your system.
A log of this installation can be found at /var/log/migrate2rocky.log

After restarting, GRUB has an entry for “Rocky Linux”:

hostnamectl’s output is:

A pre-existing user can still log on to the domain:

Using RSAT, I created a new user with the same characteristics and the same set of GPOs:

The new user also logged in and the same settings were applied, except for the shared printer which is a very brittle configuration in Samba; however, the correct wallpaper (specified by GPO from a network share) was loaded, so that wasn’t affected by any issues with permissions in the domain’s shares.

AlmaLinux

To keep the test even, a second clone of TJGGS was used, still on CentOS but with the Samba issues solved (at least, those I noticed)
AlmaLinux’s migration process isn’t as well documented as Rocky’s; instead of an explanatory page, the Migrate button at https://almalinux.org links directly to the migration scripton the Github repository, where the documentation of the process left to the README. Not ideal.
On the other hand, it’s not proposed to locally clone the repository, and the only suggestion is downloading the script with curl. Even less ideal, and plainly insecure.
So, let’s repeat the safer process, as we did for Rocky.
Open a root shell e clone the repo:

1
2
3

git clone https://github.com/AlmaLinux/almalinux-deploy.git
chmod u+x ./almalinux-deploy/almalinux-deploy.sh
./almalinux-deploy/almalinux-deploy.sh

Check root privileges OK
Check centos-8.x86_64 is supported OK
Download RPM-GPG-KEY-AlmaLinux OK
Import RPM-GPG-KEY-AlmaLinux to RPM DB OK
Download almalinux-release package OK
Verify almalinux-release package OK
Your OS is supported OK
Remove OS specific rpm packages OK
[...]
All Secure Boot related packages which were released by not AlmaLinux are reinstalledOK
Migration to AlmaLinux is completed

And restart the TJGGS server

Final thoughts

As promised, both Rocky Linux and AlmaLinux were able to make the sidegrade of a Centos8 server with some services (samba-ad-dc, chrony, mdm RAIDs, SSHd) without major issues.
The initial problems with permissions were certainly caused by me during the initial configuration of the server, because I remember that I had problems with the permissions of the sysvol which prevented some GPOs from working properly; basically the problem went on for this long (but no more, hopefully).

Editorial

The Free and Open Source world has always experienced a problem caused by “Free as in Freedom” and “Free as in Beer”. And if Being Free has a relative value for a lot of people, Being For Free is a much larger attractive and it can be measured directly, so the FOSS world, at least in its early years, did little to stress that of the two possibilities, Freedom is FOSS’s main objective.
Obviously nothing is “for free” (free lunches and all that), everything has a cost and if we aren’t paying, someone else is paying for us, whether in human effort, CPU cycles, storage, connectivity, energy, hardware… So when FOSS projects reach some maturity and market position, they try to entice users to contribute something (anything!) that benefits the project directly, more than mere disclosure and promotion.
In this way, the companies behind many FOSS projects sell services and support, but are also closing certain conveniences behind these contracts, to instill users to pay money for the FOSS product. These extra conveniences (beyond simple support) usually are:

• Limited functionality: certain functions which add value in an enterprise setting are developed on a non-FOSS model and are only unlocked against payment of a license or purchase of packaged hardware
  • OPNsense, among others; notably, almost all providers of ZFS-based storage appliances (like TrueNAS) lock clustering and High-Availability behind the purchase of hardware bundled with the FOSS software.
• Access to updates: only the binaries of version x.0.0 are publicly available, and any updates in the form of pre-compiled binaries are only available for users with a Support Agreement; however, usually the FOSS license used mandates acess to all source code, which allows anyone that is up to it to try to compile the update.
  • Bareos, OPNsense work like this
• Access to stable versions: as a complete reversal of the previous option, only the nightly build of the actual state of the development process is made publicly available, without any type of functionality tests (except the assurance that the build didn’t break); in this case, to get a stable(r) version it’s safer to get the source for the nightly of the previous major version, which will be better tested and whose updates should be just bug fixes - but even then, without any testing for regressions.
  • VyOS

RedHat decided to use a variant of this last option: instead of only making the CentOS nightlies available, which they knew was too much (and they already had Fedora Rawhide for that), they decided that CentOS would be accessible only as the development branch of the next minor version of RHEL

• Instead of CentOS version 8.4 being released after RHEL 8.4, while RHEL is at 8.4 CentOS Stream will be testing the next version 8.5; and only when CentOS Stream 8.5 is found to be stable it will be converted into RHEL 8.5, while CentOS Stream proceeds with testing the corrections slated for 8.6.
• Even better, whoever wants to test the features of RHEL 9, can already try to install CentOS Stream 9…
  In this way, users of CentOS will be contributing with testing (and hopefully bug reports), therefore paying something for their use of code developed and packaged by RedHat.