OpenSSL HollowByte DoS Flaw: 11-Byte Payload Freezes Server Memory

OpenSSL silently patched a denial-of-service vulnerability dubbed HollowByte that lets unauthenticated attackers freeze server memory with an 11-byte payload. There is no CVE assigned, no advisory published, and no changelog entry pointing to the fix. Organizations running unpatched OpenSSL should treat this as an urgent patching priority.

What Happened: OpenSSL HollowByte DoS Flaw Freezes Server Memory

On July 17, 2026, Okta’s Red Team published a technical breakdown of HollowByte, a flaw in the OpenSSL TLS handshake parser that has been lurking in every supported branch for an unknown period. The vulnerability is simple but devastating. An attacker sends an 11-byte TLS handshake header claiming a much larger message body is coming. The vulnerable server allocates the claimed memory immediately and then blocks forever waiting for data that never arrives.

Consequently, each connection costs the server up to 131 KB of permanently lost memory. Furthermore, the attacker can repeat this across thousands of connections until the server runs out of RAM and is killed by the OOM killer. The attack requires no authentication, no valid TLS session, and no cryptographic keys. Any server listening on port 443 with a vulnerable OpenSSL version is exposed.

Technical Details of the HollowByte Vulnerability

The TLS handshake protocol begins each message with a 4-byte header. Three of those bytes declare the length of the body that should follow. In vulnerable versions of OpenSSL, the server reads that length field and immediately calls grow_init_buf() followed by OPENSSL_clear_realloc(), which allocates the full claimed size through malloc(). Only after that allocation does the worker thread attempt to read the actual payload.

However, the attacker never sends the promised body. The worker thread blocks indefinitely. When the connection eventually drops, OpenSSL frees the buffer. On its own, this would be a classic Slowloris-style connection exhaustion attack. HollowByte introduces a nastier compounding effect due to how the GNU C Library handles memory.

Moreover, glibc does not return small and medium heap chunks to the operating system. It holds them for reuse. By launching waves of connections with randomized claimed sizes, the attacker fragments the heap in a way that prevents the allocator from ever reusing the freed blocks. The server’s Resident Set Size climbs continuously and stays elevated long after the attacker disconnects.

Real-World Impact in Okta’s Testing

• A 1 GB RAM test server running NGINX was OOM-killed after HollowByte froze 547 MB in fragmented memory.
• A 16 GB server lost 25% of total memory while the attack stayed below standard connection-limiting thresholds.
• The attack bandwidth was small enough to evade typical DDoS detection rules.

Affected Software and Platforms

OpenSSL is embedded in virtually every TLS-enabled application. Therefore, the following categories are at risk:

• Web servers: Apache HTTP Server, NGINX
• Language runtimes: Node.js, Python, Ruby, PHP
• Databases: MySQL, PostgreSQL
• Linux distributions shipping OpenSSL as the system TLS library

Notably, the DTLS handshake path was left unfixed because OpenSSL maintainers determined a proper fix would be too invasive. As of the July 2026 releases, DTLS handshakes still trust peer-declared lengths.

Business and Operational Impact

The HollowByte vulnerability poses a direct threat to service availability and infrastructure cost. Organizations running unpatched OpenSSL face several real-world consequences:

• Service disruption: Web applications, APIs, and databases become unreachable when memory is exhausted.
• Cascading failures: In containerized environments, OOM-killed processes trigger restart loops and potential cluster-wide pressure.
• Cloud cost inflation: Auto-scaling groups spin up replacement instances that are immediately re-targeted, driving compute bills higher.
• Detection gaps: Because the attack stays under connection-rate thresholds, standard DDoS defenses may not fire.
• Patch visibility gaps: Without a CVE, scanners cannot flag this vulnerability automatically. Organizations must manually verify their OpenSSL version.

Mitigation and Recommendations

Organizations should move quickly to verify and patch affected systems. The fix is already available in upstream OpenSSL releases dated June 9, 2026.

Immediate Actions for Defenders

1. Upgrade OpenSSL to one of the following fixed versions: 4.0.1, 3.6.3, 3.5.7, 3.4.6, or 3.0.21.
2. Restart all services that load OpenSSL. A library upgrade alone does not reclaim memory already trapped in running processes.
3. Verify downstream patches. Linux distributions typically backport fixes while keeping the original version string. Check package changelogs for pull requests 30792, 30793, and 30794.
4. Audit DTLS services. The DTLS handshake parser remains vulnerable. If you run DTLS endpoints, consider additional rate-limiting or network-level protections.
5. Monitor memory trends. Track Resident Set Size anomalies on TLS-terminating hosts. Sudden, unexplained RSS growth may indicate an active HollowByte campaign.

Detection Guidance

There is no public exploit code as of July 17, 2026. However, defenders can watch for the following indicators:

• TLS connections that send exactly 11 bytes and then idle indefinitely.
• Rapid memory growth on reverse proxy or application servers without matching traffic volume.
• Heap fragmentation patterns reported by memory profiling tools.

Bottom line: OpenSSL treated HollowByte as a hardening fix rather than a security vulnerability, which means your scanners and patch pipelines almost certainly missed it. Manually verify your OpenSSL version and restart every service that depends on it. Do not assume your June patch cycle caught this one.

Incident Summary

CVE ID / Incident:	No CVE assigned (HollowByte disclosed by Okta Red Team, July 17, 2026)
Affected Systems:	OpenSSL versions prior to 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21; NGINX, Apache, Node.js, Python, Ruby, PHP, MySQL, PostgreSQL, and all Linux distributions shipping OpenSSL
Disclosure Date:	July 17, 2026
Patch Status:	Fixed in OpenSSL 4.0.1, 3.6.3, 3.5.7, 3.4.6, and 3.0.21 (released June 9, 2026)
Severity:	High (unauthenticated DoS with permanent memory exhaustion)
Attack Vector:	Network, 11-byte malicious TLS handshake header
Prerequisites:	Server must expose a TLS endpoint using a vulnerable OpenSSL version

References

1. Okta Red Team, “OpenSSL HollowByte: A DoS Hiding in 11 Bytes,” Okta Security Blog, June 2026 (published July 17, 2026), https://sec.okta.com/articles/2026/06/openssl-hollowbtye-a-dos-hiding-in-11-bytes/, accessed July 17, 2026.
2. Bill Toulas, “HollowByte DDoS flaw bloats OpenSSL server memory with 11-byte payload,” BleepingComputer, July 17, 2026, https://www.bleepingcomputer.com/news/security/hollowbyte-ddos-flaw-bloats-openssl-server-memory-with-11-byte-payload/, accessed July 17, 2026.
3. The Hacker News, “OpenSSL HollowByte Flaw Could Freeze Server Memory with 11-Byte TLS Requests,” July 17, 2026, https://thehackernews.com/2026/07/openssl-hollowbyte-flaw-could-freeze.html, accessed July 17, 2026.
4. OpenSSL Project, GitHub Pull Request #30792, https://github.com/openssl/openssl/pull/30792, accessed July 17, 2026.
5. OpenSSL Project, OpenSSL 4.0.1 Release Notes, June 9, 2026, https://github.com/openssl/openssl/releases/tag/openssl-4.0.1, accessed July 17, 2026.