[Gs-code-review] A fix for #516530 heap crash with OTGuide.pdf .

[L. Peter Deutsch]

> > > +     penum->mask[0].depth = penum->mask[1].depth = 0;
> >
> > Please explain why this is needed.
> 
> See gximag3x.c, line 217.
> Need to fall into /* mask not supplied */ if a mask isn't supplied.

In my opinion, this is not the correct fix for this problem.

First, gsipar3x.h should say that MaskDict.BitsPerComponent may be zero in
gs_image3x_mask_t.  (This is not allowed for any other kind of image.)

Second, in gximag3x.c, the comment before check_image3x_mask is incomplete.
It should also say that this procedure sets pmcs->depth, and sets it even if
the mask is omitted.  The procedure then needs to be changed so that if the
"mask missing" test succeeds, it sets pmcs->depth = 0 before returning.

With these changes, mask[0] and mask[1] are guaranteed to be in an
acceptable state if check_image3x_mask succeeds.

>> What goes wrong when passing NULL?
> 
> GS crashes some later accessing penum->pis which is NULL.

That depends on the command line switches.  With -Z@\?, I get

Page 7

   **** File encountered an 'rangecheck' error while processing an image.
AFPL Ghostscript CVS PRE-RELEASE 7.10: ./src/ilocate.c(471): Reference to free object 0x825a2e4(12), in chunk 0x8259d78!

So the problem is not as simple as a null reference.

I will investigate this further tomorrow.

-- 

L. Peter Deutsch     |    Aladdin Enterprises   |  203 Santa Margarita Ave.
ghost at aladdin.com    |  http://www.aladdin.com  |  Menlo Park, CA 94025

	The future of software is at http://www.opensource.org