Completed missionMigrating and upgrading GitLab Community Edition

Mission summary

For a startup with strong confidentiality requirements, Cloud Algebra stepped in to secure the migration and upgrade of a GitLab Community Edition instance hosted on their own private infrastructure. After migrating 300+ GB, several services and 20 successive upgrades (7 of them major), the client now enjoys a healthy, modern working environment.

Engagement details

Context

The client, a startup mindful of the confidentiality of its know-how, hosts its source code on its own infrastructure with GitLab CE (Community Edition). GitLab CE is an open-source platform for managing, versioning and reviewing source code, with built-in automation and CI/CD features. The client’s GitLab CE instance and its operating system had not been updated for several years and were past their end-of-support dates. Moreover, being stuck on an old version, the development teams unfortunately could not benefit from the latest features and improvements.

To avoid any cyber risk, achieve regulatory compliance and modernise its development environment, the client decided to tackle this technical debt with expert support to make the operation reliable.

Scope

The engagement consisted in migrating and upgrading a self-hosted GitLab Community Edition instance:

• on the one hand: migrate the GitLab instance (service and data) from an old shared server to a new dedicated virtual server;
• on the other hand: upgrade the legacy GitLab version (v13.7.3) and OS (Ubuntu 20.04 LTS) to GitLab v18.1.1 on Ubuntu 24.04 LTS.

Challenges encountered

The client had run into several difficulties attempting such a substantial GitLab CE migration and upgrade:

• A saturated source disk, the GitLab CE instance being too large (many repositories, attachments and metadata);
• Corruption risks due to:
  • data migrated without the proper precautions for GitLab CE’s sensitive internal services (Gitaly, Redis, Prometheus, PostgreSQL);
  • major Ubuntu upgrades (2 major releases);
  • dependencies between the intermediate upgrades. Indeed, 18 upgrades must be performed one by one to go from GitLab CE 13.7.3 to 18.1.1.
• Difficulty interpreting changelogs to know whether their GitLab CE usage was affected and how to adapt;
• A lack of expertise, methodology and above all time to tackle these topics in-house.

Cloud Algebra’s operating procedure

• Audit of the existing GitLab CE instance: services in use, integrations, configuration, consumption, …;
• Preparation of the target system: virtualisation, OS, updates, network configuration, user access;
• Upfront test of the GitLab CE backup and restore procedure;
• Upfront test of the GitLab CE upgrade procedure;
• Planning of the switchover to the new GitLab CE instance, then switchover;
• Quality control: post-migration and post-upgrade acceptance testing;
• Engagement report: documentation and general recommendations;
• Warranty and support: 10 working days.

Key points of the migration

First, some figures to grasp the problem:

• Number of machines: 1
• Installation type: Omnibus
• Replication / distribution: none
• Number of users: 20+
• Number of repositories: 150+
• Data volume (excluding repositories): 500+ MB
• Repository volume: 300+ GB
• Number of source Gitaly storages: 3
• Runners: 1

Then, a few technical difficulties that departed from the traditional “happy path”:

• The documentation for the initial GitLab version was partial and did not cover backups and restores;
• The initial GitLab version being too old, files uploaded to the Package Registry were not included in the assisted backup. This had to be detected, and then the files migrated manually;
• The source system’s disks being saturated, it was not possible to migrate GitLab CE the traditional way with the gitlab-backup create and gitlab-backup restore commands, for lack of space to create an archive. The Git repositories therefore had to be migrated manually, taking every precaution to avoid corruption related to Gitaly and to UID/GID ownership;
• The source system had more Gitaly storages (3 disks) than the target (2 disks). To proceed with the migration, the equivalent of 2 old disks was merged onto a single new disk.

Key points of the GitLab upgrade

For backward-compatibility reasons, GitLab cannot be upgraded in one go. It must be done iteratively, one upgrade at a time, following an exact sequence of successive upgrades known as the “Upgrade Path”. The following path was followed:

• 13.7.3 > 13.7.9 > 13.8.8 > 13.12.5 > 14.0.12 > 14.3.6 > 14.9.5 > 14.10.5 > 15.0.5 > 15.4.6 > 15.11.13 > 16.3.9 > 16.7.10 > 16.11.10 > 17.1.8 > 17.3.7 > 17.5.5 > 17.8.7 > 17.11.4 > 18.1.1

Each step requires checks and sometimes manual, case-by-case actions to accommodate breaking changes and non-backward-compatible updates. Here is a sample of the pitfalls encountered:

• Mandatory removal of old Elasticsearch-related entries;
• Rewriting the Gitaly storage configuration in the new format, the old one having been dropped;
• Moving from OpenSSL 1.1 to OpenSSL 3.0 and from TLS 1.1 to TLS 1.2, hence the need to update all clients.

Fortunately, the GitLab Runner could be migrated and upgraded much more easily, in a single pass.

Key points of the major Ubuntu upgrades

Once GitLab was up to date, we took the opportunity to upgrade Ubuntu to simplify future GitLab upgrades. Two major upgrades took place:

• Upgrade from 20.04 to 22.04
• Upgrade from 22.04 to 24.04

For each upgrade, manual work on the PostgreSQL database was required, as it risked corruption from a glibc update and non-backward-compatible collation changes. The database indexes and collations were therefore fully refreshed.

Other information

Beyond the key points above, it was also necessary to:

• update network redirections to the correct IP addresses,
• update SSH known hosts,
• renew users’ Personal Access Tokens,
• update all obsolete curl and git clients.

To confirm that the migrated, up-to-date instance was 100% functional, we ran a full test with the instance’s users, successfully.

Conclusion

Following our engagement, the client expressed their satisfaction and their interest in tailored ongoing support.

Regular maintenance is essential to:

• benefit from gradual modernisation of the working environment;
• avoid accumulating technical debt;
• guard against the security flaws and vulnerabilities discovered every day.

Infrastructure maintenance demands constant vigilance and a preparation effort that can be hard for a company to shoulder alone. A managed-infrastructure company such as Cloud Algebra removes both obstacles. Our technology watch and preparation are pooled across all our clients. This economy of scale lets us be more competitive and migrate your systems more reliably and quickly than in-house resources.

Get support, stay up to date and stay well protected against cyber risks!

All our projects