Feb 21, 2024: Announcing Linkerd 2.15 with support for VM workloads, native sidecars, and SPIFFE! Read more »

How Linkerd became resilient to CVE-2023-44487, a HTTP/2 DDOS vulnerability, six months prior to its disclosure

William Morgan

William Morgan
October 12, 2023 • 4 min read

A fast-moving clock

Yesterday, CVE-2023-44487, a DDOS vulnerability in many HTTP/2 implementations, was disclosed. This is a very interesting attack involving the specifics of how HTTP/2 multiplexes concurrent requests on the same TCP connection, and there are several great writeups on how it works—see e.g. Cloudflare’s HTTP/2 Rapid Reset: deconstructing the record-breaking attack and Google’s How it works: The novel HTTP/2 ‘Rapid Reset’ DDoS attack for details of how this attack works and the consequences.

We’re happy to report that due to Linkerd’s internal security policies and the security-awareness and rapid response of the Rust community, all recent versions of Linkerd are resilient to this class of DDOS attack. In fact, Linkerd has been resilient to these attacks since April of this year!

Specifically, versions of Linkerd that are resilient to CVE-2023-44487 include:

  • All versions of Linkerd 2.14.x
  • Linkerd 2.13.1 and all later minor versions of Linkerd 2.13
  • Linkerd 2.12.5 and all later minor versions of Linkerd 2.12

Linkerd solved this vulnerability 6 months ago thanks in part to our dependency handling procedures, but mostly thanks to the the security-mindedness of the Rust community.

Let’s see just how this feat happened.

Linkerd is a security-first project

It’s no understatement to say that Linkerd treats security as a critical requirement. Organizations around the world rely on Linkerd for everything from protecting sensitive customer medical and financial data, to scheduling COVID tests, to building 911 call centers. For some people, Linkerd is quite literally a life-or-death project.

Part of that approach is the choice of technologies like Rust, of course, which allow us to avoid an entire class of buffer overflow exploits and other vulnerabilities that are endemic to languages like C and C++.

But another, just as important part is simply how seriously the project takes potential security vulnerabilities. Tracing the path to resolution for CVE-2023-44487 is a great example of that. Here’s how it happened:

This issue was first tracked as a vulnerability in the Rust community as RUSTSEC-2023-0034 on April 14, 2023. At that point it had actually already been fixed in h2, the underlying library that Linkerd uses to handle HTTP/2 requests, as a change that had gone out on April 12th, two days earlier.

The fix was published in h2 v0.3.17. Linkerd automatically flagged that dependency on April 13th through GitHub’s Dependabot, the automated dependency tool that Linkerd uses to ensure it stays up-to-date with critical dependencies. The Linkerd team published the update as proxy release v2.198.1.

On April 13th, this new proxy version was pulled into the main Linkerd repo. On April 14th, we pushed it to Linkerd 2.13.1—two days after the underlying fix in h2, and the same day it was recognized as an vulnerability in the Rust ecosystem. The fix also went out on edge-23.4.2 on April 21st, and from there it was in all future and stable releases.

In short: two days after the fix was made in the underlying Rust HTTP/2 library, it was already in the hands of Linkerd users as a stable release, and all Linkerd releases since April have been protected against this vulnerability. While this vulnerability is making the news this week, Linkerd adopters have been protected for almost 6 months.

h2, hyper, and more

But what is h2 anyways? The h2 library and its companion hyper are two of the foundational libraries that Linkerd uses to handle HTTP/2 requests in the proxy. HTTP/2 is used extensively by Linkerd: not only does Linkerd proxy application-initiated HTTP/2 and gRPC (which uses HTTP/2) requests, Linkerd also transparently upgrades all HTTP/1 communication in between two meshed pods to HTTP/2! This is part of Linkerd’s magic: by using h2, we can can dramatically reduce TCP connection usage and improve performance and resiliency for inter-service HTTP traffic without the application needing to do anything.

Like much of the Rust async networking stack, these libraries represent the pinnacle of modern network programming. The Linkerd team has been heavily involved in h2 for years. Buoyant, the primary sponsor of Linkerd, also sponsored significant portions of h2 development, and Linkerd maintainers occupy the third and fourth spots on the h2 contributor leaderboard.

But ultimately it is to the credit of h2 maintainer Sean McArthur that CVE-2023-44487 was addressed six months ago, allowing Linkerd to maintain its commitment to security and simplicity for its users everywhere around the globe. A heartfelt thanks to you, Sean.

Linkerd is for everyone

Linkerd is a graduated project of the Cloud Native Computing Foundation. Linkerd is committed to open governance. If you have feature requests, questions, or comments, we’d love to have you join our rapidly-growing community! Linkerd is hosted on GitHub, and we have a thriving community on Slack, Twitter, and the mailing lists. Come and join the fun!

(Photo by Djim Loic on Unsplash.)

Suggested Blog Posts