Bas' Take on Tech: xz-utils Thriller
Malicious Actors in Open Source: What happened to xz-utils and why you should care.
Howdy,
thanks for reading my tech newsletter!
It’s a long time since the last issue – life happened. The more I’m happy to be back on track with the latest tech news!
🧑💻 xz-utils
– CVE-2024-3094
It is no exaggeration to say that what happened to the supposedly unremarkable "lz
" library is one of the most serious security vulnerabilities in computer history. The special thing about it is that it was certainly intentional. The details point to a state actor and the goal was nothing less than the infiltration of the entire infrastructure.
What is it about?
"lz
" or, more precisely, the "liblzma
" library is software used to compress data. "OpenSSH" is software used for remote maintenance of Linux servers. Although OpenSSH does not utilize liblzma
directly, thousands, perhaps millions, of Linux servers were affected by a security vulnerability that would have allowed arbitrary code to be executed on other systems without authorization. This is because many large distributions use liblzma
to enable systemd
notifications in OpenSSH.
Via this detour, it has now almost been possible to introduce malware that would have ended in a security apocalypse: using a secret key, the authentication in OpenSSH would have been bypassed and an attacker would have been able to execute arbitrary code and take over foreign systems in this way.
What happened?
What happened reads like an agent thriller: Lasse Collin, the maintainer of "lz
", reported overwork and mental health issues on the "xz
" mailing list. He was put under pressure by some strangers via the mailing list to find a co-maintainer. Collin later accepted a new maintainer: Jia Tan, a previously unknown person who has been active on GitHub since 2021. Tan used the trust of the community, which he had gained through some trivial commits, to disable certain security checks for self-introduced problems in "oss-fuzz
", an automated security checker for open-source software maintained by Google. With malicious code hidden in precompiled binaries and quite sophisticated commits to disable security checks that were barely recognizable as such, it was possible to infiltrate versions 5.6.0 and 5.6.1 with the vulnerability.
The first commit from 2021 was already suspicious: Here, Jia Tan inserted an error message into the "libarchive
" project, which also replaced the "safe_fprint
" function with an insecure variant.
With further social engineering attacks, previously unknown community members attempted to persuade package maintainers of large distributions to update by praising the compromised version.
It was only by chance that Microsoft developer Andres Freud noticed in March 2023 that the login process for OpenSSH was "much slower". People who, like Freud, do not have a particular penchant for performance optimization would probably never have noticed these milliseconds.
The Aftermath
The good news is that the compromised version has not made it into the stable versions of the major distributions. However, it is included in some beta releases of Debian, Fedora and Alpine. If the gap had been discovered later, it would have been an earthquake. Accordingly, the security community is keen to clear up the case. Evan Boehs, for example, has compiled a timeline of the events in question, which is currently still being updated. JFrog has a list of the affected distributions and Rhea Karty here on Substack has an analysis of the commit history of Jia Tan's account. The admittedly easy-to-falsify timestamps suggest that the affected commits could originate from Eastern Europe. If this is the case, the overlap in time between Tan's first appearance and the start of the conflict between Russia and Ukraine would be remarkable, to say the least.
My take
The xz-utils
incident brings several interesting points to light:
First - this is the good news - the community was able to react quickly enough to avoid a disaster. Even if the gap that had been planned for years was a chance discovery by a developer outside the project, the control by attentive and capable developers worked just in time in this case. Chapeau and bravo to the open-source community.
Second, the dependencies of modern software, even when they affect critical infrastructure, are complex and difficult to understand. Individual - smaller - libraries such as the xz-utils
in this case are partly maintained by individuals. The bus factor and the possibility of human error are simply too high here.
Third, complex attacks by potent attackers, such as organized crime in the state services, are commonplace. There is an uneasy feeling that the incident in question is not an isolated case. Our software is under attack. Everywhere, at all times and for a wide variety of interests.
Fourth, every community has an Achilles' heel: Social interactions. In this case, it was obviously quite easy to gain trust and thus infiltrate a project with malicious intent. More quality control is needed here, especially with unknown "newcomers".
Fifth, the present case has shown that malicious code is not only hidden in the actual source code, but also in the depths of test suites, automatic build processes or distributors' patch methodologies. Testing, packaging and distributing software is sometimes more complex than the actual software itself. We need to move towards an open, less complex and, above all, reproducible standard for CI/CD processes.
Further Readings
notes, honeypot, and exploit demo for the
xz
backdoor (CVE-2024-3094
)Infographic on X
Critizism on
xz
Bash Script to detect affected versions
Another timeline of events
Underhanded C Code – A contest to hide malicious attempts in innocent-looking code
📰 Some interesting reads
Those old enough to remember Sourceforge, might be interested in this article outlining how GitHub managed to replace it as the centralised hub for Open Source code.
A reflection of the US Department of Justice antitrust case against Apple.
Sam Bankman-Fried, CEO and co-founder of FTX has been sentenced to 25 years.
The market value of freshly IPO’ed shares of Donald Trump’s “truth.social” dropped by 21.5%.
🚀 What else?
I have changed this newsletter from “occasional” to at least once every two weeks. The new format not only contains curated news and insights from the tech world but also sets the stage for fellow developers.
I will ask people about their journeys in the tech world and also in life. I’m pretty sure there is much to learn from listening to other people’s experiences, and I am excited to share that with you!
Of course, I need your help with this.
Please tell me if you want to share your story with my 600+ readers!
What is something you learned?
What are some books that recently influenced you?
What do you want to share with others about your life, your productivity, your career, and your health?
Reach out to me on this form.
Let me know how I can make the newsletter better.
I set up a short feedback form here.
It would mean the world to me if you spent a minute or two filling it out!
Best,