huge memory allocation mostly

Created attachment 737353 [details]
oom-t2p_readwrite_pdf_image_tile

QA REPRODUCER:

tiff2pdf oom-t2p_readwrite_pdf_image_tile

will run out of memory even though the file is small. It should not run out of memory.

Created attachment 737354 [details]
oom-TIFFFetchStripThing

QA REPRODUCER:

tiff2pdf oom-TIFFFetchStripThing

should not run out of memory for this small file.

I do not get OOM neither with 4.0.10, 4.0.9 nor with 3.8.2.

Could you please double-check and eventually tell me which code streams do you see affected?

I tried also with 3.8.2 with security related patches (patch 8 to patch 90) commented out.

I tried also 4.0.8 32-bit and 3.8.2 32-bit. No OOM reproduced.

However, I am able to get the ASAN report from the upstream bug for 4.0.8 on i586. I do not get it for x86_64.

For non-ASAN build, the criteria may be as observed with 4.0.8 and 4.0.9 (i586, x86_64):

4.0.8:

$ /usr/bin/time -v tiff2pdf oom-TIFFFetchStripThing 2>&1 | grep 'Maximum'
        Maximum resident set size (kbytes): 2787664
$


4.0.9:

$ /usr/bin/time -v tiff2pdf oom-TIFFFetchStripThing 2>&1 | grep 'Maximum resident set size'
	Maximum resident set size (kbytes): 2940
$

Therefore the update to 4.0.9 fixed it. 

3.8.2:

$ /usr/bin/time -v tiff2pdf oom-TIFFFetchStripThing 2>&1 | grep 'Maximum resident set size'
	Maximum resident set size (kbytes): 7584
$

The testcase does not expose the issue for 3.8.2.


Reportedly fixed with:

https://gitlab.com/libtiff/libtiff/commit/5b7f711586f1fc7541abba85dfe2c6e90602f8ae


The code in 3.8.2 seem to be different to some extent. While I am not able to reproduce the issue there, I would consider that unaffected.

@Marcus, what do you think?

The results for second testcase is different:

1. I do not get ASAN report for 4.0.8 (i586).

For non ASAN builds:

4.0.10:

$  /usr/bin/time -v tiff2pdf oom-t2p_readwrite_pdf_image_tile 2>&1 | grep 'Maximum'
        Maximum resident set size (kbytes): 1969004
$

4.0.9:

$ /usr/bin/time -v tiff2pdf oom-t2p_readwrite_pdf_image_tile 2>&1 | grep 'Maximum'
        Maximum resident set size (kbytes): 1968840
$

4.0.8:

$ /usr/bin/time -v tiff2pdf oom-t2p_readwrite_pdf_image_tile 2>&1 | grep 'Maximum'
        Maximum resident set size (kbytes): 1968928
$

3.8.2:

$ /usr/bin/time -v tiff2pdf oom-t2p_readwrite_pdf_image_tile 2>&1 | grep 'Maximum'
        Maximum resident set size (kbytes): 7296
$

Also fifth comment in suggests it is not fixed, still:
http://bugzilla.maptools.org/show_bug.cgi?id=2725#c5

The second testcase exhibits an issue in tiff2pdf only, however, CVE-2017-12944 talks about an issue in the library.

it would probably need a SPLIT of the CVE. not sure if Mitre will assign a fresh one for the commandline tool, as the commandline tools are not considered that relevant.

I would also consider a commandline tool OOM DOS not a big problem.

Unlike I thought, the upstream bug is still opened (but without any activity for a year or so).

The issue can be seen also with tiffsplit:

$ /usr/bin/time -v tiffsplit oom-t2p_readwrite_pdf_image_tile 2>&1 | grep Maxim
        Maximum resident set size (kbytes): 1968672
$

For tiff2pdf, the allocation really happens int tiff2pdf tool (4.0.10/tools/tiff2pdf.c:2278):

                               buffer = (unsigned char*)
                                        _TIFFmalloc(t2p->tiff_datasize);

There are more such allocations depending on user input.

In tiffsplit, the situation is similar:

             TIFFGetField(in, TIFFTAG_TILEBYTECOUNTS, &bytecounts);
             [..]
             buf = (unsigned char *)_TIFFrealloc(buf, (tmsize_t)bytecounts[t]);

Perhaps tiff_datasize (bytecounts) input could be sanitized against input file size. I had proposed it in the upstream bug.

Summary:
--------
testcase1: TW,15,12: fixed by 4.0.9, rpm changelogs has to be amended, 11,10sp3: unaffected
testcase2: TW,15,12: affected, but seems to be a minor issue

SUSE-SU-2018:4008-1: An update that fixes 6 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1017693,1054594,1115717,990460
CVE References: CVE-2016-10092,CVE-2016-10093,CVE-2016-10094,CVE-2016-6223,CVE-2017-12944,CVE-2018-19210
Sources used:
SUSE Linux Enterprise Module for Packagehub Subpackages 15 (src):    tiff-4.0.9-5.20.1
SUSE Linux Enterprise Module for Open Buildservice Development Tools 15 (src):    tiff-4.0.9-5.20.1
SUSE Linux Enterprise Module for Desktop Applications 15 (src):    tiff-4.0.9-5.20.1
SUSE Linux Enterprise Module for Basesystem 15 (src):    tiff-4.0.9-5.20.1

openSUSE-SU-2018:4053-1: An update that fixes 6 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1017693,1054594,1115717,990460
CVE References: CVE-2016-10092,CVE-2016-10093,CVE-2016-10094,CVE-2016-6223,CVE-2017-12944,CVE-2018-19210
Sources used:
openSUSE Leap 15.0 (src):    tiff-4.0.9-lp150.4.12.1

SUSE-SU-2018:4191-1: An update that fixes 6 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1017693,1054594,1115717,990460
CVE References: CVE-2016-10092,CVE-2016-10093,CVE-2016-10094,CVE-2016-6223,CVE-2017-12944,CVE-2018-19210
Sources used:
SUSE Linux Enterprise Software Development Kit 12-SP4 (src):    tiff-4.0.9-44.30.1
SUSE Linux Enterprise Software Development Kit 12-SP3 (src):    tiff-4.0.9-44.30.1
SUSE Linux Enterprise Server 12-SP4 (src):    tiff-4.0.9-44.30.1
SUSE Linux Enterprise Server 12-SP3 (src):    tiff-4.0.9-44.30.1
SUSE Linux Enterprise Desktop 12-SP4 (src):    tiff-4.0.9-44.30.1
SUSE Linux Enterprise Desktop 12-SP3 (src):    tiff-4.0.9-44.30.1

openSUSE-SU-2018:4256-1: An update that fixes 6 vulnerabilities is now available.

Category: security (moderate)
Bug References: 1017693,1054594,1115717,990460
CVE References: CVE-2016-10092,CVE-2016-10093,CVE-2016-10094,CVE-2016-6223,CVE-2017-12944,CVE-2018-19210
Sources used:
openSUSE Leap 42.3 (src):    tiff-4.0.9-43.1

released