[gs-cvs] rev 8887 - trunk/gs/src

[leonardo at ghostscript.com]

Author: leonardo
Date: 2008-07-28 00:52:42 -0700 (Mon, 28 Jul 2008)
New Revision: 8887

Modified:
   trunk/gs/src/gdevprn.c
Log:
Fix (printer) : A check for buffer space overflow was dramatically overestimated.

DETAILS :

In the old code :

    new_height < max_ulong/(mem_space + (new_height * pdf14_trans_buffer_size))

Where pdf14_trans_buffer_size is (raster * NUM_PDF14_BUFFERS).

equivalent to :

    
    new_height * (mem_space + (new_height * (raster * NUM_PDF14_BUFFERS))) < max_ulong


equivalent to :

    new_height * mem_space + new_height * new_height * raster * NUM_PDF14_BUFFERS < max_ulong


So it counts new_height quadratically, but it must be the linear multiple by the raster.

Debugged with the test case of the bug 689982.
It creates a 15Mb raster, but could not switch off the banding
even with -dMaxBitmap=1000000000 .

EXPECTED DIFFERENCES :

None.


Modified: trunk/gs/src/gdevprn.c
===================================================================
--- trunk/gs/src/gdevprn.c	2008-07-28 03:32:10 UTC (rev 8886)
+++ trunk/gs/src/gdevprn.c	2008-07-28 07:52:42 UTC (rev 8887)
@@ -282,7 +282,7 @@
 	mem_space = buf_space.bits + buf_space.line_ptrs;
 	if (ppdev->page_uses_transparency) {
 	    pdf14_trans_buffer_size = (ESTIMATED_PDF14_ROW_SPACE(max(1, new_width)) >> 3);
-	    if (new_height < max_ulong/(mem_space + (new_height * pdf14_trans_buffer_size))) {
+	    if (new_height < (max_ulong - mem_space) / pdf14_trans_buffer_size) {
 		pdf14_trans_buffer_size *= new_height;
 		mem_space += pdf14_trans_buffer_size;
 	    } else {