Testing for reproducibility is an important principle behind Madagascar’s design. It works on several levels.

1. Inside a project directory (with SConstruct file that contains from rsf.proj import *), run
scons lock

to create Result files (figures) and to copy them to a different location (specified by RSFFIGS or $RSFROOT/share/madagascar/figs by default). Papers included with Madagascar (under$RSFSRC/book) have their result figures saved in a repository.
To come back and test if the results are still reproducible, run

scons test

or

scons figurename.test

This command performs an intelligent comparison of figures using Joe Dellinger’s sfvplotdiff and reports an error if the figures are different. In the case of an error, you can run

scons figurename.flip

to flip between the new version of the figure and the old version and on the screen and to compare them visually. Based on that comparison, you can either “lock” the new version with

scons figurename.lock

or debug the error that caused the difference and try to fix it.

2. To test all projects where a particular program, say sfspike, is used, run
cd $RSFSRC/book; scons sfspike.test This is useful for regression testing for changes in programs that may cause reproducibility failures. You can also run cd$RSFSRC/book; scons test

to test all projects and all Madagascar programs. By default, testing is limited to projects that use only publicly available data and less that 1 Mb of disk space. This behavior can be changes by giving all=y or size= parameters to scons test.

3. A collection of scripts developed by Jim Jennings and explained on the Automatic Testing page performs fine-grain testing with extended diagnostics.