Repository navigation
Conversation
The bindings hand raw pointers into Python-owned buffers to C, so Python code could invalidate them or feed C bad sizes: - Hold a memoryview export on the window/dictionary bytearray and the internal input buffer for the object's lifetime, so resizing them raises BufferError instead of leaving C with a dangling pointer. - Pass f.readinto() a memoryview (it can't resize the input buffer) and reject a byte count that is None, negative, or larger than the buffer. - readinto() re-derives its output pointer after each f.readinto() call, since that may run code that resizes the caller's buffer. - Raise RuntimeError when a Compressor/Decompressor is used from 2 threads at once (the C calls run with the GIL released), or before __init__ ran. - Validate window (8-15) and literal (5-8) before they're truncated into 4-bit TampConf bitfields. None of these are reachable from compressed data alone. Throughput is unchanged (2 MB enwik8 compress/decompress, and 64-byte readinto calls). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The Cython bindings hand C raw pointers into Python-owned buffers, so ordinary Python code could invalidate them or feed C bad sizes. None of this is reachable from compressed data alone; it needs a caller or file-like object doing something unusual. Most cases crash the interpreter (segfault / bus error) rather than raising.
<unsigned char*>bytearraytakes no buffer export, so e.g.dictionary.clear()or.extend()after construction freed/moved storage C kept using. Both classes now hold amemoryviewon the window and internal input buffer for the object's lifetime, so a resize raisesBufferError.f.readinto()trusted: an over-reported count made C read past the input buffer, andreadintoreceived the internalbytearrayitself, so the commonbuf[:] = chunkidiom could shrink it. The decompressor now passes amemoryviewand rejectsNone, negative, or too-large counts.readinto(): the output pointer was cached acrossf.readinto(), which can run arbitrary code. It's now re-derived and clamped each iteration.__init__from another thread) crashed. Concurrent use now raisesRuntimeError(zlib/bz2/lzma guard the same case).__init__:Compressor.__new__(Compressor).write(...)ran C on an uninitialized struct; this now raisesRuntimeError.window/literalrange: values were truncated into 4-bitTampConfbitfields (window=24became 8 after allocating a 16 MiB window), so they're now validated before use.Performance
Measured on 2 MB of enwik8: compression, decompression via
read(), and 64-bytereadinto()calls are all unchanged. A first version tookreadinto(buf)as a typed memoryview and cost ~15% on small calls, so the signature staysbytearray.Tests
TestCythonBufferLifetimesintests/test_bug_regressions.py(7 tests). Each fails or crashes onmainexcept the output-resize one, which only crashes under guard malloc.🤖 Generated with Claude Code