Flavijus Piliponis â stock.ado
XZ backdoor discovery reveals Linux supply chain attack
A maintainer for XZ, a popular open source compression library for Linux distributions, compromised the open source project over the course of two years.
A backdoor was discovered in a widely used compression library for Linux distributions that could give unauthorized users access to infected systems.
On March 29 Andres Freund, a PostgreSQL developer at Microsoft, revealed he had discovered a backdoor hidden in the open source liblzma package for XZ, a popular compression library used in many Linux distributions. In a disclosure on the Open Source Security Mailing List, Freund wrote that he observed odd behavior in liblzma running on Debian in recent weeks, such as slow logins via SSH and surprisingly high CPU usage for SSHD processes. After further investigation, he concluded that the upstream XZ repository was compromised in a supply chain attack.
Fruend traced the malicious code and discovered the backdoor was committed to the open source project through legitimate-looking updates by an XZ maintainer, later identified as Jia Tan. "Given the activity over several weeks, the committer is either directly involved or there was some quite severe compromise of their system," he wrote in the disclosure. "Unfortunately, the latter looks like the less likely explanation, given they communicated on various lists about the 'fixes' mentioned above."
Freund also found that the backdoor, which is present in XZ Utils versions 5.6.0 and 5.6.1, was contained in only the archived download packages, known as "tarballs," that are released upstream. The malicious code was not present in Git distributions.
Red Hat noted in a security advisory Friday that the backdoor, which is tracked as CVE-2024-3094 with a CVSS score of 10, is two stages. "The Git distribution lacks the M4 macro that triggers the build of the malicious code," the advisory said. "The second-stage artifacts are present in the Git repository for the injection during the build time, in case the malicious M4 macro is present. Without the merge into the build, the 2nd-stage file is innocuous."
In a blog post Friday, Tenable researchers said the backdoor affected several Linux distributions, including Fedora Rawhide and Fedora 41. It also had impacts on Debain testing, and unstable and experimental distributions versions 5.5.1alpha-0.1 to 5.6.1-1.
Other affected distributions include openSUSE Tumbleweed and openSUSE MicroOSm, which was backdoored between March 7 and March 28; Kali Linux, which was backdoored between March 26 and March 28; and Arch Linux installation medium 2024.03.01, as well as virtual machine images 20240301.218094 and 20240315.221711 and container images created between Feb. 24 and March 28.
It's unclear how many organizations have downloaded the distributions with the malicious XZ versions, or if any of those organizations were breached through the backdoor.
CISA issued an alert Friday highlighting Red Hat's advisory and urging Linux developers and users to downgrade to unaffected versions of XZ Utils, such as version 5.4.6 stable, and to search for any malicious activity in their environments.
Other popular Linux distributions, including Red Hat Enterprise, SUSE Linux Enterprise and Leap, Amazon Linux and Ubuntu, have not been compromised by the backdoor.
"Luckily xz 5.6.0 and 5.6.1 have not yet widely been integrated by Linux distributions, and where they have, mostly in pre-release versions," Freund wrote.
Attack timeline
The backdoor was created through an elaborate, multiyear social engineering attack, according to multiple cybersecurity vendors and threat analysts. At the center was GitHub user JiaT75, also known as Jia Tan. This developer contributed to the XZ library over two years, built trust within the development community, and ultimately gained privileges to maintain the XZ repository and merge commits.
According to a timeline created by programmer Evan Boehs, Tan made their GitHub account in November 2021 and created a pull request in libarchive that replaced one piece of code with a less safe variant under the cover of an error text change. In April 2022, Jia Tan submitted a patch to XZ via a mailing list and another user, "Jigar Kumar" -- perhaps an alias or a co-conspirator -- pressured maintainers for the patch to be merged.
Jigar Kumar then put pressure on a maintainer of the XZ project to add another maintainer to freely make commits. Tan began making regular contributions to the project at this point, and apparently gained enough trust to merge their own code in early January 2023. Tan then, over the course of months and into early 2024, took action to further their own influence within the project and build the infrastructure for the backdoor.
The backdoor was ultimately finalized in the early part of this year before being discovered in March.
CVE assigned
The XZ backdoor was assigned a CVE and is listed on the NIST National Vulnerability Database, which is not typical for malware.
"Through a series of complex obfuscations, the liblzma build process extracts a prebuilt object file from a disguised test file existing in the source code, which is then used to modify specific functions in the liblzma code," the NIST listing read. "This results in a modified liblzma library that can be used by any software linked against this library, intercepting and modifying the data interaction with this library."
Although it is a complex incident still being researched, Michael Skelton, Bugcrowd vice president of security operations and hacker success, referred to the assignment of a CVE as a proactive and precautionary approach.
"It's important to clarify that CVEs are not assigned to malware or backdoors themselves, but rather to the vulnerabilities that such malicious entities might exploit," Skelton told TechTarget Editorial. "By assigning a CVE to a suspected yet unproven vulnerability, the community acknowledges the potential for exploitation and prioritizes the security of the ecosystem, ensuring that precautions are taken even in the absence of immediate proof of exploitability."
Scott Caveza, staff research engineer at Tenable, agreed that the issuing of a CVE is less typical in this case, but said Red Hat's choice to issue one "makes identifying affected systems much easier."
"Each affected or non-affected Linux distribution can respond to their customers with a common identifier to designate this backdoor, and concerned parties are not trying to search a multitude of possible phrases and instead can refer to this situation with a CVE identifier," Caveza said. "While this is a bit of an uncommon scenario, when the CVE program was first introduced in 1999, it's unlikely that supply chain attacks were even a consideration at the time."
Open source security consequences
In a Bugcrowd blog post on the backdoor, Skelton highlighted the repercussions of the discovery and the potential fallout for other open source projects.
"One of the consequences of this finding is that Jia Tan contributed not only to this library, but also others, leading to a larger review effort being needed to see if other components may also be in a currently vulnerable state," he wrote. "Beyond that, the question becomes how to avoid these issues in the supply chain -- or whether they can be avoided at all. Open source software is the backbone of many services, and companies that we rely upon -- and not everything can be closely scrutinized to the level of detail needed to avoid these attacks."
Omkhar Arasaratnam, general manager of the Open Source Software Foundation, wrote in a blog post Saturday that the nature of open source software allowed the backdoor to be discovered, reported and addressed quickly, and that the number of systems affected by the backdoor is "relatively low."
"Situations like this remind us all that we need to remain vigilant within the open source software ecosystem," Arasaratnam wrote. "Open source is about well-intentioned humans donating their time and talents to help solve problems, and sadly this can be compromised."
Arasaratnam told TechTarget Editorial that for open source security, there are no perfect sets of tools, practices that would have prevented this type of supply chain attack. "We may one day get to a point where we do have open source security capabilities to help secure open source software," he said. "I think that will require an all-hands approach using a tapestry of technologies and processes to help maintainers and consumers stay secure when producing and using digital public goods like open source software."
John Bambenek, president at security consultancy Bambenek Consulting, similarly noted that "we don't have great tools to deal with attack vectors introduced in commonly used open source software," such as those like XZ.
"That said, we are getting much better at detecting malicious activity in the wild, and because this hit so many organizations at once, we were able to quickly contain the threat," he said. "There is only so much we can do for software maintained by volunteers, and it is clear we need to find ways to get better quickly, or this will keep happening."
Rob Wright is a longtime reporter and senior news director for TechTarget Editorial's security team. He drives breaking infosec news and trends coverage. Have a tip? Email him. Alexander Culafi is a senior information security news writer and podcast host for TechTarget Editorial.
