Of Mice and Migrations
Migrations are a complex process. They suck up your resources, can be costly, and disrupt your business. In the worst-case scenario, they may cause severe repercussions if something goes wrong, causing downtime and data loss.
Transformations are even more perilous undertakings as they face change resistance. So, who in their right mind would perform a migration and a transformation simultaneously?
Well, in fact, we at Eficode always recommend taking the bold step and using every migration as an opportunity to implement change—upgrade to better tools, improve your processes, bolster your security posture, and so on. Every migration that doesn’t contain at least an element of transformation is a lost opportunity to improve how you do your business.
Here are some things you can nearly always improve when you migrate:
- Automation: There is hardly ever too much automation in a large organization’s processes.
- Security: Given the ever-evolving cybersecurity landscape, it’s unlikely that you’ll ever reach a security posture that’s too good.
- Process optimization: Identifying bottlenecks or redundancies in workflows and streamlining them for faster delivery and improved collaboration.
- Cultural change: Encouraging new ways of working, fostering innovation, and aligning teams to adopt new practices and tools effectively.
Sounds tantalizing? Surely it’s more expensive, right? Well, yes and no. Planning for change always takes effort and requires resources, but it’s also an investment into future gains in productivity. However, combining migration and transformation doesn’t necessarily mean you increase the complexity of the operation. With a good assessment of your current situation, a clear understanding of the desired end state, proper resourcing, and good advance planning, a migration transformation can help you improve your organization's work and performance.
In one of our recent migration transformations, we helped a customer achieve a deeper maturity level in platform engineering, drive up their identity and access management systems, and implement self-help migration tooling embedded in their platform—all while keeping their critical systems at minimal downtime. We’d like to share this story with all of our customers.
Migration as a part of platform engineering
The customer in question had an expired Bitbucket Server that needed to be migrated to GitHub Enterprise Cloud with Enterprise Managed User ASAP.
For Bitbucket, there was no centralized access management and no governance policy in place. In addition, all the work done to administer the product and handle user management was done manually. In short, it was a typical mess: A tool that had been neglected for years reached a state where it was not only a bottleneck for development work but also posed compliance and security risks.
The customer’s desired end state was to move to GitHub while implementing fundamental changes to the way they structure their codebase and manage users:
- Migration from their decaying Bitbucket to GitHub Enterprise Cloud with Enterprise Managed Users.
- Adopt C4, a software architecture framework.
- Implement an IAM model based on Zero Trust concepts such as Role Based Access Control, Just Enough Access and full management of an identity lifecycle.
With over 4000 repositories and 58 developer teams, there is a lot of change to implement in a single go. But a year after starting the work, the customer has reached the desired end state—and not only reached it, but reached it with ease and no major dumpster fires.
Help yourself migration
Given the complexity of the migration and transformation process, we aimed to have as much standardization and automation as possible. To achieve this, we decided to leverage the existing customer tooling (namely Backstage, Platform API, Entra ID, and MyAccess). The existing migration tooling was integrated into the customer-managed tooling to build a "Help yourself migration” experience in the customer’s internal development platform, Backstage. The end users had to input the required information in a GUI provided in Backstage to trigger an End-to-End pipeline that handled the entire lifecycle of their repository migration. The automated migration pipeline then handled everything from start to finish:
- Migrated git repositories and associated metadata (issues, PRs, attachments, etc.) from Bitbucket to GitHub to conform to the C4 model.
- Set all the branches in the source repository into read-only mode during the migration.
- Conflict resolution on the fly.
- Large file storage content migration.
- Stripped user access from all users and groups in the Bitbucket environment (Effective lockdown of the source environment).
- Rename the repository to add a prefix to the repo name before moving it into the Archival project (e.g., PRJ.<repo_name>).
- Validated repository data integrity after the migration (verifying each latest commit hash value for each branch between source and target repositories).
- Created RBAC conforming GitHub teams and Entra ID groups that were then provisioned into the GitHub environment to seamlessly migrate users into the new environment.
- Logged the whole process into Grafana.
- Added the correct settings to the GitHub repository as part of the post-migration process.
Before the production migrations, the entire migration process was thoroughly tested and documented to provide confidence in running the migrations at a large scale. Support was provided during the migration process to troubleshoot any possible issues as a safety measure. During the process, we didn't face any significant errors; only a few issues were spotted (which were all fixed within the 24-hour timeframe).
As a result, not only was the customer data migrated from a legacy tool into a new GitHub Enterprise Cloud, but all of this was done with essentially close to zero downtime and disruption.
At the same time, a cultural change was implemented. Strict access controls were implemented to limit actions like repository creation to service accounts (SAs) and organization admins, establishing a centralized governance model based on Zero Trust principles.
Users now have to submit requests through MyAccess to perform any actions related to repositories. Deleting repositories was restricted and could only be done through a formal request and review process. Access to repositories also needs to be requested to ensure permissions are managed effectively. Overall, everything is automated, centralized, monitored, and logged.
After the migration
The project delivered what was promised: The customer data was moved from Bitbucket to GitHub on time and without disruption to the business. In addition, a large selection of equally important transformations were implemented.
Improved security and compliance posture
The identity and access management processes were overhauled completely. Currently, they are based on Zero Trust principles. Combined with the automation of IAM processes, they drive the organization towards an enhanced security posture and help maintain compliance with relevant frameworks such as ISO 270001 and NIS2.
Automation of IT processes
An integral part of the migration process was to build automation around user provisioning and setting up resources needed for development work.
The newly established automated processes promote consistency, efficiency, and security, driving the standards of DevSecOps up while freeing the DevSecOps team to concentrate on high-value tasks instead of repeating mundane tasks such as repository creation, user management requests, etc.
Organizational change
The migration was used to shift the paradigm of the software architecture framework, improving development practices and making the large estate of code more manageable and understandable.
Deeper embrace of platform engineering
The migration process was built on existing developer platform tooling (Backstage, Platform API) and leveraged to the full extent. As a consequence, the organization gained a new level of maturity in terms of platform engineering.
Published: Dec 20, 2024
