[gs-cvs] rev 8853 - trunk/gs/src

[ray at ghostscript.com]

Author: ray
Date: 2008-07-18 13:53:54 -0700 (Fri, 18 Jul 2008)
New Revision: 8853

Modified:
   trunk/gs/src/gdevprn.c
Log:
Fix for Seg fault caused by wrap around of an unsigned long. Seen at 600 dpi
to pkmraw with Bug689369.pdf and -dBufferSpace=32000000.

DETAILS:

While the pdf14_trans_buffer_size was small enough, the mem_space calculation
would wrap around.


Modified: trunk/gs/src/gdevprn.c
===================================================================
--- trunk/gs/src/gdevprn.c	2008-07-18 13:59:05 UTC (rev 8852)
+++ trunk/gs/src/gdevprn.c	2008-07-18 20:53:54 UTC (rev 8853)
@@ -278,9 +278,9 @@
 	size_ok = ppdev->printer_procs.buf_procs.size_buf_device
 	    (&buf_space, pdev, NULL, pdev->height, false) >= 0;
 	if (ppdev->page_uses_transparency) {
-	    if (new_height < max_ulong/(ESTIMATED_PDF14_ROW_SPACE(max(1, new_width)) >> 3))
-		pdf14_trans_buffer_size = new_height
-		    * (ESTIMATED_PDF14_ROW_SPACE(new_width) >> 3);
+	    pdf14_trans_buffer_size = (ESTIMATED_PDF14_ROW_SPACE(max(1, new_width)) >> 3);
+	    if (new_height < max_ulong/(mem_space + (new_height * pdf14_trans_buffer_size)))
+		pdf14_trans_buffer_size *= new_height;
 	    else {
 		size_ok = 0;
 		pdf14_trans_buffer_size = 0;