Fix qUncompress for BE LLP64 platforms

Partially reverts 50661a9558ca3fe93bf569cc572b0fb2def5589d.

On big-endian, when long is 32-bits, but size_t is 64-bits, the
reinterpret_cast<ulongf*>(&len) from a size_t len is addressing the
MSBs of len, which, by construction, are 0.

I'm not sure such platforms exist (Windows is LLP64, but always LE
these days, all Unixes are LP64), but the reinterpret_cast is a code
smell, so do the length calculations in uLongf, which is zlib's
interface type, and avoid the cast, so the compiler can complain if we
get the type wrong, which it promptly did:

The static_assert I added a few commits back now informed me that the
doubling of len started to overflow, so replace the unconditional len
*= 2 by saturation arithmetic: if doubling would overflow, use
MaxDecompressedSize instead; if we fail with Z_BUF_ERROR even with
that buffer size, we just can't decompress and need to give up.

As a drive-by, change a C-style cast to a reinterpret_cast.

Task-number: QTBUG-104972
Change-Id: I91c311b297ef410b4c001d3528f1e9fd469d5def
Reviewed-by: Dmitry Shachnev <mitya57@gmail.com>
Reviewed-by: Thiago Macieira <thiago.macieira@intel.com>
(cherry picked from commit 63b31ca72e59039a8623e9adb832e2189b0bbad0)
Reviewed-by: Mårten Nordheim <marten.nordheim@qt.io>

1 files changed, 7 insertions, 10 deletions

diff --git a/src/corelib/text/qbytearray.cpp b/src/corelib/text/qbytearray.cpp
index 10c8def113..16b4eef69c 100644
--- a/src/corelib/text/qbytearray.cpp
+++ b/src/corelib/text/qbytearray.cpp
@@ -629,9 +629,8 @@ QByteArray qUncompress(const uchar* data, qsizetype nbytes)
             qWarning("qUncompress: Input data is corrupted");
         return QByteArray();
     }
-    size_t expectedSize = size_t((data[0] << 24) | (data[1] << 16) |
-                                 (data[2] <<  8) | (data[3]      ));
-    size_t len = qMax(expectedSize, 1ul);
+    const auto expectedSize = qFromBigEndian<quint32>(data);
+    uLong len = qMax(expectedSize, 1u);
     constexpr size_t MaxZLibSize = (std::numeric_limits<uLong>::max)();
     constexpr size_t MaxDecompressedSize = (std::min)(size_t(MaxByteArraySize), MaxZLibSize);
     if (len > MaxDecompressedSize)
@@ -644,7 +643,7 @@ QByteArray qUncompress(const uchar* data, qsizetype nbytes)
 
     forever {
         const auto alloc = len;
-        int res = ::uncompress((uchar*)d.data(), reinterpret_cast<uLongf*>(&len),
+        int res = ::uncompress(reinterpret_cast<uchar *>(d.data()), &len,
                                data+4, nbytes-4);
 
         switch (res) {
@@ -661,13 +660,11 @@ QByteArray qUncompress(const uchar* data, qsizetype nbytes)
             return QByteArray();
 
         case Z_BUF_ERROR:
-            static_assert(MaxDecompressedSize <= (std::numeric_limits<decltype(len)>::max)() / 2,
-                          "oops, next line may overflow");
-            len *= 2;
-            if (len > MaxDecompressedSize)
+            if (len == MaxDecompressedSize) // can't grow further
                 return invalidCompressedData();
-
-            d->reallocate(d->allocatedCapacity() * 2, QArrayData::Grow);
+            if (qMulOverflow<2>(len, &len))
+                len = MaxDecompressedSize;
+            d->reallocate(qsizetype(len), QArrayData::Grow); // cannot overflow!
             if (d.data() == nullptr) // reallocation failed
                 return invalidCompressedData();