How Linkerd became resilient to CVE-2023-44487, a HTTP/2 DDOS vulnerability, six months prior to its disclosure
Yesterday, CVE-2023-44487, a DDOS vulnerability in many HTTP/2 implementations, was disclosed. This is a very interesting attack involving the specifics of how HTTP/2 multiplexes concurrent requests on the same TCP connection, and there are several great writeups on how it works—see e.g. Cloudflare’s HTTP/2 Rapid Reset: deconstructing the record-breaking attack and Google’s How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack for details of how this attack works and the consequences.
We’re happy to report that due to Linkerd’s internal security policies and the security-awareness and rapid response of the Rust community, all recent versions of Linkerd are resilient to this class of DDOS attack. In fact, Linkerd has been resilient to these attacks since April of this year!
Specifically, versions of Linkerd that are resilient to CVE-2023-44487 include:
- All versions of Linkerd 2.14.x
- Linkerd 2.13.1 and all later minor versions of Linkerd 2.13
- Linkerd 2.12.5 and all later minor versions of Linkerd 2.12
Linkerd solved this vulnerability 6 months ago thanks in part to our dependency handling procedures, but mostly thanks to the the security-mindedness of the Rust community.
Let’s see just how this feat happened.
Linkerd is a security-first project
It’s no understatement to say that Linkerd treats security as a critical requirement. Organizations around the world rely on Linkerd for everything from protecting sensitive customer medical and financial data, to scheduling COVID tests, to building 911 call centers. For some people, Linkerd is quite literally a life-or-death project.
Part of that approach is the choice of technologies like Rust, of course, which allow us to avoid an entire class of buffer overflow exploits and other vulnerabilities that are endemic to languages like C and C++.
But another, just as important part is simply how seriously the project takes potential security vulnerabilities. Tracing the path to resolution for CVE-2023-44487 is a great example of that. Here’s how it happened:
This issue was first tracked as a vulnerability in the Rust community as
RUSTSEC-2023-0034 on
April 14, 2023. At that point it had actually
already been fixed in h2,
the underlying library that Linkerd uses to handle HTTP/2 requests, as a change
that had gone out
on April 12th, two days earlier.
The fix was published in
h2 v0.3.17. Linkerd
automatically flagged that dependency
on April 13th
through GitHub’s Dependabot, the automated
dependency tool that Linkerd uses to ensure it stays up-to-date with critical
dependencies. The Linkerd team published the update as
proxy release v2.198.1.
On April 13th, this new proxy version
was pulled into the main Linkerd repo.
On April 14th, we pushed it to
Linkerd 2.13.1—two
days after the underlying fix in h2, and the same day it was recognized as an
vulnerability in the Rust ecosystem. The fix also went out on
edge-23.4.2 on
April 21st, and from there it was in all future and stable releases.
In short: two days after the fix was made in the underlying Rust HTTP/2 library, it was already in the hands of Linkerd users as a stable release, and all Linkerd releases since April have been protected against this vulnerability. While this vulnerability is making the news this week, Linkerd adopters have been protected for almost 6 months.
h2, hyper, and more
But what is h2 anyways? The h2 library and
its companion hyper are two of the foundational libraries
that Linkerd uses to handle HTTP/2 requests in the proxy. HTTP/2 is used
extensively by Linkerd: not only does Linkerd proxy application-initiated HTTP/2
and gRPC (which uses HTTP/2) requests, Linkerd also transparently upgrades all
HTTP/1 communication in between two meshed pods to HTTP/2! This is part of
Linkerd’s magic: by using h2, we can can dramatically reduce TCP connection
usage and improve performance and resiliency for inter-service HTTP traffic
without the application needing to do anything.
Like much of the Rust async networking stack, these libraries represent the
pinnacle of modern network programming. The Linkerd team has been heavily
involved in h2 for years. Buoyant, the primary sponsor
of Linkerd, also sponsored significant portions of h2 development, and Linkerd
maintainers occupy the third and fourth spots on the
h2 contributor leaderboard.
But ultimately it is to the credit of h2 maintainer Sean McArthur that
CVE-2023-44487 was addressed six months ago,
allowing Linkerd to maintain its commitment to security and simplicity for its
users everywhere around the globe. A heartfelt thanks to you, Sean.
Linkerd is for everyone
Linkerd is a graduated project of the Cloud Native Computing Foundation. Linkerd is committed to open governance. If you have feature requests, questions, or comments, we’d love to have you join our rapidly-growing community! Linkerd is hosted on GitHub, and we have a thriving community on Slack, Twitter, and the mailing lists. Come and join the fun!
