[gs-cvs] rev 7865 - trunk/gs/toolbin/tests
thomasd at ghostscript.com
thomasd at ghostscript.com
Thu Apr 19 08:04:51 PDT 2007
Author: thomasd
Date: 2007-04-19 08:04:51 -0700 (Thu, 19 Apr 2007)
New Revision: 7865
Added:
trunk/gs/toolbin/tests/README
trunk/gs/toolbin/tests/make_testdb.py
Removed:
trunk/gs/toolbin/tests/make_testdb
Log:
Integrate make_testdb.py with new environment,
including testing.cfg and python procedures in other scripts.
make_testdb.py supports creation of a testdata.db with checksums
generated by running gs build in HEAD/ and installed in gshead/.
Added: trunk/gs/toolbin/tests/README
===================================================================
--- trunk/gs/toolbin/tests/README 2007-04-18 00:23:49 UTC (rev 7864)
+++ trunk/gs/toolbin/tests/README 2007-04-19 15:04:51 UTC (rev 7865)
@@ -0,0 +1,149 @@
+crontab executes run_nightly
+
+run_nightly provides the correct python environment for running run_nightly.py
+ export PYTHONPATH=.:/home/regression/python:$PYTHONPATH
+
+Since we now have shared python modules in /home/regression/python,
+it needs to be added to the appropriate path.
+For example in .bashrc
+
+ export PYTHONPATH=.:/home/regression/python:$PYTHONPATH
+
+This addition is also helpful
+
+ export PATH=.:/home/regression/bin:$PATH
+
+
+To rebuild the HEAD executable without running any tests
+
+./run_nightly.py --noregression --notestregression
+
+To run regression tests without rebuilding HEAD
+./run_nightly.py --noupdate
+
+
+The structure of running nightly regressions is:
+
+run_nightly (bash script to provide python environment)
+ run_nightly.py
+ build_revision.py build the requested revision (default is HEAD)
+ run_regression.py run that revision of gs on all testfiles with all paramsets
+ test_diff.py analyze checksums: compare this run with recent and with baseline
+ get_baseline_log.py summarize recent changes to the baseline update log
+
+These script use the environment provided by gsconf.py, which reads the file testing.cfg;
+as well as using the command line.
+ run_nightly.py
+ build_revision.py
+ run_regression.py
+
+The data flow:
+
+All directories are relative to gsconf.root (usually /home/regression)
+
+build_revision.py
+ input
+ from svn gs repository
+ output
+ gs build tree, located at
+ gsroot (if revsion==HEAD) -
+ root/gs.<revision> (for specific revisions)
+ gs installation tree (only if revision==HEAD)
+ installtree
+ logs/<revision>.update*,make*,config*,
+
+
+run_regression
+ input
+ comparefiles (usually /home/regression/comparefiles) - run all "appropriate" files (tested by file extension) using gs
+ gsparamsets.py - sets of output devices for gs testing
+ output
+ dailydir/<revision>.db (usually /home/regression/regression/daily) - store db with (testfile-device,checksum)
+ logs/<revision>.regression
+ logs/<revision>.gs*
+
+<recent date> is determined by the existence of the most recent checksum database, before today
+
+test_diff <recent date>
+ input
+ baselinedb (usually /home/regression/regression/baseline/baseline.db)
+ dailydir (usually /home/regression/regression/daily)
+ <previous revision>.db
+ <revision>.db
+ output
+ text report of differences. For all testfile-device entries in this revision's checksum db,
+ if the checksum does not match the previous revision, report as a new difference
+ if the checksum does not match the baseline, report as a difference
+ logs/<revision>.differences
+
+get_baseline_log <recent date>
+ input
+ baseline_log (usually /home/regression/regression/baseline/baseline.log)
+ output
+ text report of updates to the baseline since the specified date
+
+
+Details of checksums
+
+ all checksum db are in "dailydir" (usually /home/regression/regression/daily)
+ checksums are calculated using the python md5 module.
+ these dbs are files of anydb format, with key=testfile-device, value=checksum
+ New differences result from the comparison with a recent db.
+ All differences result from the comparison with the baseline.db
+ Each day (or revision) a new checksum db is created.
+ Each day, the new checksum db is compared with the most recent day's checksum db.
+ Each day the checksum db is also compared with the baseline checksum db (baseline.db).
+ A checksum log can contain 15,000 or more entries (number of test files * number of devices in gsparamsets)
+
+ bugs:
+ checksum logs that are used for comparison are still named by date, and it should be revision only.
+ each day's log is also copied to a db named by revision (but these are not yet used)
+
+ There are tools to examine the checksum dbs
+ dump_checksum.py - normal use. Be aware: you get LOTS of output
+ dump_checksum_raw.py - for debugging
+ dump_checksum_plus.py - specialized
+ collate.py - gather results of several dbs and sort
+
+
+Details of testfiles:
+ all testfiles in the gsconf.comparefiledir are tested. Smaller subsets (in other directories) can be used for testing.
+ The normal situation:
+ For each testfile, there is a set of compressed raster files (one for each device).
+ For each raster file there is an entry in the baseline checksum db.
+ To make the raster file and baseline checksum db up-to-date, use update_baseline.py (--help for command line options)
+
+
+Under the covers:
+ gsconf.py - provide an object with information about the testing environment
+ gstestgs.py - run gs
+ gsparamsets.py - provide sets of output devices (one set for "normal" and one for "pdfwrite")
+ gstestutils.py - utilities
+ gsutil.py - utilities
+ gssum.py - calculate checksums and insert into a checksum db
+ rasterdb.py - create and get compressed raster files
+
+ These scripts do the work of running gs and adding checksums to the daily checksum db:
+
+ gscheck_all.py - run the following - creating test cases for later execution
+
+ gscheck_raster.py - normal
+ for each test file, for each device create a normal test case
+ gscheck_pdfwrite.py - pdfwrite
+ for each test file, for each device create a pdfwrite test case
+
+ gscheck_testfiles.py - test only the files in a specified list
+
+ check_* - scripts to test the other aspects of the code base (dirs, comments, docrefs, source)
+
+
+Repair:
+ regen_baseline.py
+ regenerate all checksums from the raster files (>15,000 files - takes a long time)
+
+ regen_filelist.py
+ regenerate checksums specified in a file list from the raster files (>15,000 files - takes a long time)
+
+
+Tests with larger scope than nightly:
+ run_series.py - run_nightly.py with a specified range of revisions.
\ No newline at end of file
Deleted: trunk/gs/toolbin/tests/make_testdb
===================================================================
--- trunk/gs/toolbin/tests/make_testdb 2007-04-18 00:23:49 UTC (rev 7864)
+++ trunk/gs/toolbin/tests/make_testdb 2007-04-19 15:04:51 UTC (rev 7865)
@@ -1,124 +0,0 @@
-#!/usr/bin/env python
-# -*- Mode: python -*-
-
-# Copyright (C) 2001 Artifex Software Inc.
-#
-# This software is provided AS-IS with no warranty, either express or
-# implied.
-#
-# This software is distributed under license and may not be copied,
-# modified or distributed except as expressly authorized under the terms
-# of the license contained in the file LICENSE in this distribution.
-#
-# For more information about licensing, please refer to
-# http://www.ghostscript.com/licensing/. For information on
-# commercial licensing, go to http://www.artifex.com/licensing/ or
-# contact Artifex Software, Inc., 101 Lucas Valley Road #110,
-# San Rafael, CA 94903, U.S.A., +1(415)492-9861.
-
-# $Id$
-
-#
-# make_testdb <dir>
-#
-# this script creates the gstestdb database and intializes it
-# with the md5sums of the test data files run through the
-# baseline ghostscript
-
-import gstestgs
-import gsconf
-import gssum
-import gsparamsets
-import rasterdb
-import anydbm
-import os, sys
-import string
-import gsutil
-
-def make_compare_entry(ifile, device, dpi, band):
- ofile = "%s.%s.%d.%d" % (ifile, device, dpi, band)
- print "creating entry: " + ofile + "...",
- sys.stdout.flush()
-
- gs = gstestgs.Ghostscript()
- gs.log_stdout = gsconf.log_stdout
- gs.log_stderr = gsconf.log_stderr
- gs.command = gsconf.baselinegs
- gs.infile = gsconf.comparefiledir + ifile
- gs.outfile = ofile
- gs.device = device
- gs.dpi = dpi
- gs.band = band
-
- if not gssum.exists(ofile):
- if gs.process():
- try:
- gssum.add_file(ofile)
- rasterdb.put_file(ofile)
- os.unlink(ofile)
- print "done."
- except OSError:
- print "no output produced."
- else:
- print "error."
- else:
- print "exists."
-
-def make_pdfcompare_entry(ifile, device, dpi, band):
- ofile = "%s.pdf.%s.%d.%d" % (ifile, device, dpi, band)
- print "creating entry: " + ofile + "...",
- sys.stdout.flush()
-
- if gssum.exists(ofile):
- print "exists."
- return
-
- gs = gstestgs.Ghostscript()
- gs.log_stdout = gsconf.log_stdout
- gs.log_stderr = gsconf.log_stderr
- gs.command = gsconf.baselinegs
- gs.infile = gsconf.comparefiledir + ifile
- gs.dpi = dpi
- gs.band = band
-
- # make file->PDF
-
- tfile = ofile + ".pdf"
- gs.outfile = tfile
- gs.device = 'pdfwrite'
- gs.dpi = None
-
- if not gs.process():
- print "error."
- return
-
- gs.infile = tfile
- gs.outfile = ofile
- gs.device = device
- gs.dpi = dpi
-
- if gs.process():
- try:
- gssum.add_file(ofile)
- rasterdb.put_file(ofile)
- os.unlink(tfile)
- os.unlink(ofile)
- print "done."
- except OSError:
- print "no output produced."
- else:
- print "error."
-
-
-if __name__ == "__main__":
- # create the testdatadb
- db = anydbm.open(gsconf.testdatadb, 'c')
- db.close()
-
- files = os.listdir(gsconf.comparefiledir)
- for f in files:
- if gsutil.check_extension(f):
- for params in gsparamsets.testparamsets:
- make_compare_entry(f, params.device, params.resolution, params.banding)
- for params in gsparamsets.pdftestparamsets:
- make_pdfcompare_entry(f, params.device, params.resolution, params.banding)
Added: trunk/gs/toolbin/tests/make_testdb.py
===================================================================
--- trunk/gs/toolbin/tests/make_testdb.py 2007-04-18 00:23:49 UTC (rev 7864)
+++ trunk/gs/toolbin/tests/make_testdb.py 2007-04-19 15:04:51 UTC (rev 7865)
@@ -0,0 +1,133 @@
+#!/usr/bin/env python
+# -*- Mode: python -*-
+
+# Copyright (C) 2001 Artifex Software Inc.
+#
+# This software is provided AS-IS with no warranty, either express or
+# implied.
+#
+# This software is distributed under license and may not be copied,
+# modified or distributed except as expressly authorized under the terms
+# of the license contained in the file LICENSE in this distribution.
+#
+# For more information about licensing, please refer to
+# http://www.ghostscript.com/licensing/. For information on
+# commercial licensing, go to http://www.artifex.com/licensing/ or
+# contact Artifex Software, Inc., 101 Lucas Valley Road #110,
+# San Rafael, CA 94903, U.S.A., +1(415)492-9861.
+
+# $Id: make_testdb 6300 2005-12-28 19:56:24Z giles $
+
+#
+# make_testdb <dir>
+#
+# this script creates a gstestdb database and intializes it
+# with the md5sums of the test data files run through the
+# executable last built in HEAD/ and installed in HEAD gshead/
+
+import gstestgs
+import gsconf
+import gssum
+import gsparamsets
+import rasterdb
+import anydbm
+import os, sys
+import string
+import gsutil
+
+def make_compare_entry(dbname,ifile, device, dpi, band):
+
+ ofile = "%s.%s.%d.%d" % (ifile, device, dpi, band)
+ print "creating entry: " + ofile + "...",
+ sys.stdout.flush()
+
+ gs = gstestgs.Ghostscript()
+
+ gs.gsroot = gsconf.gsroot
+ gs.log_stdout = gsconf.gs_stdout
+ gs.log_stderr = gsconf.gs_stderr
+ gs.command = gsconf.headinstallpath
+
+ gs.infile = gsconf.comparefiledir + ifile
+ gs.outfile = ofile
+ gs.device = device
+ gs.dpi = dpi
+ gs.band = band
+
+ if not gssum.exists(ofile,dbname):
+ if gs.process():
+ try:
+ gssum.add_file(ofile,dbname)
+ rasterdb.put_file(ofile)
+ os.unlink(ofile)
+ print "done."
+ except OSError:
+ print "no output produced."
+ else:
+ print "error."
+ else:
+ print "exists."
+
+def make_pdfcompare_entry(dbname,ifile, device, dpi, band):
+ ofile = "%s.pdf.%s.%d.%d" % (ifile, device, dpi, band)
+ print "creating entry: " + ofile + "...",
+ sys.stdout.flush()
+
+ if gssum.exists(ofile,dbname):
+ print "exists."
+ return
+
+ gs = gstestgs.Ghostscript()
+
+ gs.gsroot = gsconf.gsroot
+ gs.log_stdout = gsconf.gs_stdout
+ gs.log_stderr = gsconf.gs_stderr
+ gs.command = gsconf.headinstallpath
+
+ gs.infile = gsconf.comparefiledir + ifile
+ gs.dpi = dpi
+ gs.band = band
+
+ # make file->PDF
+
+ tfile = ofile + ".pdf"
+ gs.outfile = tfile
+ gs.device = 'pdfwrite'
+ gs.dpi = None
+
+ if not gs.process():
+ print "error."
+ return
+
+ gs.infile = tfile
+ gs.outfile = ofile
+ gs.device = device
+ gs.dpi = dpi
+
+ if gs.process():
+ try:
+ gssum.add_file(ofile,dbname)
+ rasterdb.put_file(ofile)
+ os.unlink(tfile)
+ os.unlink(ofile)
+ print "done."
+ except OSError:
+ print "no output produced."
+ else:
+ print "error."
+
+
+if __name__ == "__main__":
+ # create the testdatadb
+ dbname = gsconf.testdatadb
+ db = anydbm.open(dbname, 'c')
+ db.close()
+
+ files = os.listdir(gsconf.comparefiledir)
+ files.sort()
+ for f in files:
+ if gsutil.check_extension(f):
+ for params in gsparamsets.testparamsets:
+ make_compare_entry(dbname,f, params.device, params.resolution, params.banding)
+ for params in gsparamsets.pdftestparamsets:
+ make_pdfcompare_entry(dbname,f, params.device, params.resolution, params.banding)
Property changes on: trunk/gs/toolbin/tests/make_testdb.py
___________________________________________________________________
Name: svn:executable
+ *
More information about the gs-cvs
mailing list