United Airlines clears CI/CD pipelines for takeoff
In an industry heavily affected by pandemic disruptions, United Airlines brought in a new orchestration tool to navigate fluctuating software delivery requirements.
As it weathered the disruption of the COVID-19 pandemic, United Airlines sought to tame sprawling continuous integration/continuous deployment pipelines so that development teams could keep up with the pace of change.
Like the rest of the airline industry, United Airlines weathered a sharp decline in travel at the height of the COVID-19 pandemic public health emergency, followed by an upsurge in travel as restrictions eased. This bred chaos among hundreds of development teams and CI/CD pipelines that struggled to push software updates and new tools to airport employees quickly. At the same time, the airline's IT infrastructure was also undergoing upheaval as it migrated apps to Kubernetes in the public cloud.
United soon had a need for a different kind of air traffic control -- one that would establish governance over and prioritize its software delivery without adding major delays to what developers were already doing.
"Doing DevOps is different from doing DevOps at scale," said Rajeswari Koppala, senior manager of DevOps airport operations technology at United Airlines, based in Chicago. "Certain tools, they work really well at the application level if you have a 15-member group to support. But to support an organization when you have hundreds of teams coming together, you really need tools that can scale."
From build delays to streamlined self-service
United began efforts to centralize and standardize its software delivery a year ago, as it migrated apps from servers in its own data centers to the Amazon Elastic Container Service and Elastic Kubernetes Service. It also needed to add self-service access for developers to new cloud-native tools such as Kubernetes Helm charts, Karpenter node autoscaling and Terraform infrastructure as code.
By then, United was facing performance lags with its deployments of tools such as JetBrains' TeamCity that ran on servers on the premises. Its CI stages were particularly scattered, which meant some issues surfaced relatively late in the software delivery process, where they became more complex and took longer to fix. The airline also needed to impose security policies, compliance guidelines and role-based access on its systems.
After evaluating vendors such as Octopus Deploy, GitLab and Armory's supported version of Spinnaker, United chose tools from Harness -- Harness CI and Harness CD -- to add fresh automation to its existing CI/CD pipelines in late 2022. United chose Harness in part because of the breadth of its product line, which it could tie in with existing workflows at different points depending on what tools developers preferred.
Rajeswari Koppala
"Harness has a bunch of offerings available for different stages of the software delivery lifecycle, like CI, CD, infrastructure as code, feature flags, Security Test Orchestration, Test Intelligence, et cetera," Koppala said. "Users can pick the offering that is appropriate for them, and it does not force any dependency. For example, a user can still choose GitHub Actions [to automate] infrastructure as code and have CI/CD done through Harness."
Harness is a SaaS product but has a component called a Harness Delegate that supports flexible scaling of the infrastructure that runs some CI pipelines within United's AWS virtual private clouds. United used this to replace certain TeamCity deployments with which it had experienced performance issues.
"Since Harness [Delegate] is deployed on Kubernetes, it can scale, and I can add resources on demand, which was not the case with on-premises [tools]," Koppala said. "You don't have to wait for a build agent to become available to trigger anything. It's always available."
A typical CI process at United consists of unit testing, static and dynamic code scans, a build image scan, and a push to JFrog Artifactory. That previously took up to 22 minutes, but it now takes about five minutes, Koppala said. Developers can also access CI/CD pipeline resources through an automated system, further cutting wait times.
Koppala's DevOps team created Harness templates for each programming language used by developers, such as .NET Core, Java, Node.js and Angular. It then created a standardized automation process for each one, which would be triggered by developers submitting a ticket through Atlassian's Jira issue tracking system.
"We give the CI pipeline on demand to them, so we're not investing our DevOps effort in writing a CI pipeline again and again," Koppala said.
Self-healing on the horizon, but change requires persuasion
Many vendors offer integrations and governance over multiple kinds of CI/CD pipelines and associated tools without replacing them as well, from CloudBees to Atlassian and GitLab. All the vendors United considered offered products it could have used, Koppala said.
Where Harness also stood out was in the fact that it offers AI-driven automated rollbacks for flawed deployments, which will soon become part of United's CI/CD pipelines as well.
Certain tools, they work really well at the application level if you have a 15-member group to support. But to support an organization when you have hundreds of teams coming together, you really need tools that can scale.
Rajeswari KoppalaSenior manager of DevOps Airport Operations Technology, United Airlines
"We want to get to a stage where our deployments can self-heal if there is any discrepancy with service performance after the deployment," Koppala said.
United recently finished proof-of-concept testing an integration between Harness and its Datadog observability tool that will underpin automated rollbacks and will soon begin making it available to application teams.
Automated rollbacks are also a feature of Spinnaker. At the time they were evaluating products, however, United IT teams felt it lacked easy setup for air-gapped environments.
In the meantime, adapting to new ways of working remains as much a human challenge as a technical one, Koppala said. Harness has an engineering maturity assessment feature in beta, a spokesperson for the vendor confirmed this month -- a tool for which Koppala said she sees a need.
"When you ask [teams] to move to a new standard, they need answers for, 'Why we do need to adapt?' And it's very difficult … to assess every application team and provide a report without having any baseline," she said. "Tools like Harness … should have another feature to assess the maturity of any application."
Beth Pariseau, senior news writer at TechTarget, is an award-winning veteran of IT journalism. She can be reached at [email protected] or on Twitter @PariseauTT.
