Difference between revisions of "Package overview"

From Madagascar
Jump to navigation Jump to search
m
 
(40 intermediate revisions by 5 users not shown)
Line 1: Line 1:
 
[[Image:Fotolia_5127720_XS.jpg|right|]]
 
[[Image:Fotolia_5127720_XS.jpg|right|]]
  
Madagascar provides:
+
The mission of the Madagascar project is to provide a '''shared research environment''' for computational data analysis in geophysics and related fields.
# Standalone programs for out-of-core numerical analysis;
+
 
 +
The Madagascar environment consists of:
 +
# Standalone programs for out-of-core data analysis;
 
# Standalone programs for geophysical data processing and imaging;
 
# Standalone programs for geophysical data processing and imaging;
# A development kit for C, C++, F77, F90, Matlab and Octave;
+
# A development kit for C, C++, Java, Fortran-77, Fortran-90, Python, Matlab, and Octave;
# A framework for [http://en.wikipedia.org/wiki/Regression_testing regression testing] and reproducible numerical experiments, based on [http://www.scons.org/ SCons] and [http://en.wikipedia.org/wiki/LaTeX LaTeX];
+
# A framework for reproducible numerical experiments, based on [http://www.scons.org/ SCons];
# A collection of reproducible scientific articles also used as regression tests for the standalone programs;
+
# A framework for scientific publications, based on [http://www.scons.org/ SCons] and [http://en.wikipedia.org/wiki/LaTeX LaTeX];
# A collection of datasets used as input to reproducible numerical experiments / software tests.
+
# A collection of reproducible scientific articles also used as usage examples and regression tests for the standalone programs;
 +
# A collection of datasets used as input to reproducible numerical experiments.
 +
 
 +
This guide serves as a brief introduction of different components and shows how they all fit together.
 +
 
 +
==How to obtain Madagascar==
  
==Standalone programs==
+
See [[Download|download]] and [[Installation | installation instructions]]. Madagascar runs on Unix/Linux platforms, including MacOS X and Unix emulations under Miscrosoft Windows. Its installation requires, at a minimum, a working C compiler and Python.
Most programs act as filters on input data and can be chained through Unix pipes, i.e.:
+
 
 +
==How to find your way around Madagascar==
 +
 
 +
Start by checking the [[Reproducible_Documents|list of reproducible papers]]. If any of these papers looks close to your interests, follow the links until you find a figure with a "wrench" button under it [[Image:configure.png]]. Click on the wrench, and it will open a computational recipe used for generating the figure (the <tt>SConstruct</tt> file).
 +
<syntaxhighlight lang="python">
 +
from rsf.proj import *
 +
Flow('rose',None,
 +
    '''
 +
    math n1=629 d1=0.01 o1=0 n2=40 d2=1 o2=5
 +
    output="x2*(8+sin(6*x1+x2/10))" |
 +
    rtoc |
 +
    math output="input*exp(I*x1)"
 +
    ''')
 +
Result('rose',
 +
      'graph title=Rose screenratio=1 wantaxis=n')
 +
End()
 +
</syntaxhighlight>
 +
You can copy this recipe to your computer or simply find it already existing in the "book" subtree under the Madagascar source directory. For example, the recipe at http://www.reproducibility.org/RSF/book/rsf/rsf/sfmath.html also exists in the file <tt>RSFSRC/book/rsf/rsf/sfmath/SConstruct</tt>. After copying or locating the appropriate <tt>SConstruct</tt> file,
 +
run
 +
<pre>
 +
scons view
 +
</pre>
 +
on the command line to generate all the figures in the selected project and to display them on your screen. For example, try
 +
<pre>
 +
bash$ cd RSFSRC/book/rsf/rsf/sfmath
 +
bash$ scons view
 +
</pre>
 +
where <tt>bash$</tt> stands for the Unix prompt and <tt>RSFSRC</tt> stands for the Madagascar source directory. The output should look like
 +
<pre>
 +
scons: Reading SConscript files ...
 +
scons: done reading SConscript files.
 +
scons: Building targets ...
 +
/RSFROOT/bin/sfmath n1=629 d1=0.01 o1=0 n2=40 d2=1 o2=5 output="x2*(8+sin(6*x1+x2/10))" | /RSFROOT/bin/sfrtoc |
 +
/RSFROOT/bin/sfmath output="input*exp(I*x1)" > rose.rsf
 +
< rose.rsf /RSFROOT/bin/sfgraph title=Rose screenratio=1 wantaxis=n > Fig/rose.vpl
 +
/RSFROOT/bin/sfpen Fig/rose.vpl
 +
scons: done building targets.
 +
</pre>
 +
with a picture appearing on your screen.
 +
[[Image:rose.png|frame|center]]
 +
If there are several figures in the recipe, you can run
 +
<pre>
 +
scons figurename.view
 +
</pre>
 +
(e.g. <tt>scons rose.view</tt>) to display individual figures. To remove all files that <tt>scons view</tt> generated, run
 +
<pre>
 +
scons -c view
 +
</pre>
 +
If you want to know in advance what commands will be executed to generate the figures, try
 +
<pre>
 +
scons -n view
 +
</pre>
 +
You can output this command to a file
 +
<pre>
 +
scons -n -Q view > script.sh
 +
</pre>
 +
and use <tt>script.sh</tt> as a shell script. If you are to make modifications to the data processing recipe (changing parameters or trying new data), working with SCons is more powerful and convenient than running shell scripts.
 +
 
 +
A computational recipe puts together individual commands through Unix pipes and SCons rules. These commands act like Lego blocks for creating complex data analysis constructions. In the example above, three "blocks" are used: <tt>sfmath</tt>, <tt>sfrtoc</tt>, and <tt>sfgraph</tt>. To find out what a particular command is doing, you can follow the links from the bottom of the web page http://www.reproducibility.org/RSF/book/rsf/rsf/sfmath.html
 +
Alternatively, run the command without arguments on the command line. Running
 +
<pre>
 +
bash$ sfrtoc
 +
</pre>
 +
produces something like
 +
<pre>
 +
NAME
 +
        sfrtoc
 +
DESCRIPTION
 +
        Convert real data to complex (by adding zero imaginary part).
 +
SYNOPSIS
 +
        sfrtoc < real.rsf > cmplx.rsf
 +
COMMENTS 
 +
        See also: sfcmplx
 +
USED IN
 +
        bei/ft1/plane4
 +
        bei/ft1/autocor
 +
        bei/ft1/brad
 +
        [...]
 +
SOURCE
 +
        user/main/rtoc.c
 +
DOCUMENTATION
 +
        http://reproducibility.org/wiki/Guide_to_madagascar_programs#sfrtoc
 +
</pre>
 +
 
 +
The <b>DOCUMENTATION</b> section provides a link to a more detailed documentation on the web. The most useful part is the <b>USED IN</b> section, which points to more examples of using the program. As an exercise, change directory to <tt>RSFSRC/book/bei/ft1/plane4</tt> or any other example directory, examine the <tt>SConstruct</tt> file, and run <tt>scons view</tt>. Alternatively, look at http://www.reproducibility.org/RSF/book/bei/ft1/plane4.html
 +
 
 +
Want to find a program by keywords? Try <tt>sfdoc -k</tt>.
 +
<pre>
 +
bash$ sfdoc -k complex
 +
sfsort: Sort a float/complex vector by absolute values.
 +
sfrtoc: Convert real data to complex (by adding zero imaginary part).
 +
sfjacobi2: Find eigenvalues of a general complex matrix by Jacobi-like iteration.
 +
sfboolcmp: Element-wise boolean comparison of values. For int/float/complex data-sets.
 +
sfcmatmult: Simple matrix multiplication for complex matrices
 +
sfimag: Extract real (sfreal) or imaginary (sfimag) part of a complex dataset.
 +
sfthr: Threshold float/complex inputs given a constant/varying threshold level.
 +
sfcpef: 1-D prediction-error filter estimation from complex data
 +
sfroots: Find roots of a complex polynomial.
 +
sfreal: Extract real (sfreal) or imaginary (sfimag) part of a complex dataset.
 +
sfcmplx: Create a complex dataset from its real and imaginary parts.
 +
sfsin: Simple operations with complex sinusoids
 +
sfcdottest: Generic dot-product test for complex linear operators with adjoints
 +
sfcconjgrad: Generic conjugate-gradient solver for linear inversion with complex data
 +
</pre>
 +
 
 +
Individual components of the Madagascar environment are described in more details below.
 +
 
 +
==Madagascar components==
 +
 
 +
===Standalone programs===
 +
The list of all standalone programs is available [http://www.reproducibility.org/RSF/ online]. Most programs act as filters on input data and can be chained through Unix pipes, i.e.:
 
<bash>
 
<bash>
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf
 
</bash>
 
</bash>
 
 
This approach follows the Unix philosophy, as formulated by Doug McIlroy, the inventor of Unix pipes (Salus, 1994<ref>Salus, P. H.,  1994, A quarter-century of Unix: Addison-Wesley.</ref>):   
 
This approach follows the Unix philosophy, as formulated by Doug McIlroy, the inventor of Unix pipes (Salus, 1994<ref>Salus, P. H.,  1994, A quarter-century of Unix: Addison-Wesley.</ref>):   
 
#Write programs that do one thing and do it well.   
 
#Write programs that do one thing and do it well.   
Line 20: Line 136:
 
#Write programs to handle text streams, because that is a universal interface.  
 
#Write programs to handle text streams, because that is a universal interface.  
  
Documentation has designed in a layered fashion. Following the Unix convention, programs have brief <tt>man</tt> pages, which explain the program purpose and parameters. These pages can also be accessed by running a program without any arguments, or [http://www.reproducibility.org/RSF/ online]. A more detailed level of documentation, with basic equations, figures and examples, is provided through the wiki in the [Guide to madagascar programs | Guide to programs]. The [[Task-centric program list]] categorizes and briefly describes all components. At an even higher level of details, the programs can be seen in actual use in the [Reproducible Documents]. Search capabilities are currently provided by the <tt>sfdoc</tt> utility.  
+
Following the Unix convention, programs have brief <tt>man</tt> pages, which explain the program purpose and parameters. You can access this documentation by running a program without parameters. To search for a program by a keyword, use <tt>sfdoc -k <keyword></tt>.  
 +
 
 +
The [[guide to madagascar programs]] provides more detailed documentation for selected programs while the [[task-centric program list]] attempts to categorize them. You can see the programs in actual use in the [[Reproducible Documents]].
  
==Madagascar data formats==
+
===Data format===
  
 
For data, Madagascar uses the [[Guide to RSF file format| Regularly Sampled Format]] (RSF), which is based on the concept of hypercubes (n-D arrays, or regularly sampled functions of several variables), much like the SEPlib (its closest relative), DDS, or the regularly-sampled version of the Javaseis format (SVF). Up to 9 dimensions are supported. For 1D it is conceptually analogous to a time series, for 2D to a raster image, and for 3D to a [http://en.wikipedia.org/wiki/Voxel voxel volume].  The format (actually a [http://en.wikipedia.org/wiki/Meta meta]format) makes use of a ASCII file with metadata (information about the data), including a pointer (<tt>in=</tt> parameter) to the location of the file with the actual data values. Irregularly sampled data are currently handled as a pair of datasets, one containing data and the second containing the corresponding irregular geometry information. Programs for conversion to and from other formats such as SEG-Y and SU are provided.  
 
For data, Madagascar uses the [[Guide to RSF file format| Regularly Sampled Format]] (RSF), which is based on the concept of hypercubes (n-D arrays, or regularly sampled functions of several variables), much like the SEPlib (its closest relative), DDS, or the regularly-sampled version of the Javaseis format (SVF). Up to 9 dimensions are supported. For 1D it is conceptually analogous to a time series, for 2D to a raster image, and for 3D to a [http://en.wikipedia.org/wiki/Voxel voxel volume].  The format (actually a [http://en.wikipedia.org/wiki/Meta meta]format) makes use of a ASCII file with metadata (information about the data), including a pointer (<tt>in=</tt> parameter) to the location of the file with the actual data values. Irregularly sampled data are currently handled as a pair of datasets, one containing data and the second containing the corresponding irregular geometry information. Programs for conversion to and from other formats such as SEG-Y and SU are provided.  
Line 28: Line 146:
 
For graphics, Madagascar currently uses the Vplot vector graphics format. Converters to other graphics formats (Postscript, PNG, GIF, JPEG) are also provided.
 
For graphics, Madagascar currently uses the Vplot vector graphics format. Converters to other graphics formats (Postscript, PNG, GIF, JPEG) are also provided.
  
==Reproducible documents==
+
===Reproducible documents===
  
 
A reproducible document consists of LaTeX source combined with SCons rules required to fully build the documents. These rules are expressed in terms of SCons extensions that are provided as part of Madagascar.  
 
A reproducible document consists of LaTeX source combined with SCons rules required to fully build the documents. These rules are expressed in terms of SCons extensions that are provided as part of Madagascar.  
Line 34: Line 152:
 
This is the key to the reproducibility aspect of Madagascar. An introduction to reproducible Madagascar documents is at [[Reproducible_computational_experiments_using_SCons]] .
 
This is the key to the reproducibility aspect of Madagascar. An introduction to reproducible Madagascar documents is at [[Reproducible_computational_experiments_using_SCons]] .
  
==Madagascar Display: Vplot==
+
===Vplot graphics===
  
 
In contrast to most other Madagascar Components, graphics components produce Vplot data as output.
 
In contrast to most other Madagascar Components, graphics components produce Vplot data as output.
Line 44: Line 162:
 
filters it, and saves the result
 
filters it, and saves the result
  
<bash>
+
<syntaxhighlight lang="bash">
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf  
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf  
</bash>
+
</syntaxhighlight>
  
 
In this more elaborate case, the final output is passed to a graphics program and plotted.
 
In this more elaborate case, the final output is passed to a graphics program and plotted.
  
<bash>
+
<syntaxhighlight lang="bash">
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 | sfcontour | xtpen
 
< data.rsf sfwindow n1=100 | sfbandpass fhi=60 | sfcontour | xtpen
</bash>
+
</syntaxhighlight>
  
More extensive examples are seen at [[Guide to madagascar programs]] . The novice reader should probably read the material below before proceeding to that page.
+
More extensive examples are seen in [[Guide to madagascar programs]] . The novice reader should probably read the material below before proceeding to that page.
  
==Reproducibility and Project Management==
+
===Reproducibility and Project Management===
  
 
Madagascar uses and extends [http://www.scons.org/ SCons], an open-source  
 
Madagascar uses and extends [http://www.scons.org/ SCons], an open-source  
Line 67: Line 185:
 
SCons builds based on those rules. Components regarded as "up-to-date" are not rebuilt.  
 
SCons builds based on those rules. Components regarded as "up-to-date" are not rebuilt.  
  
SCons allows user-contributed Builders (meta-rule categories) and Madagascar uses this capability extensively.  
+
SCons allows for user-contributed Builders (meta-rule categories) and Madagascar uses this capability extensively.  
 
The idea is that building an output file based on a workflow chain is very much analogous to building a  
 
The idea is that building an output file based on a workflow chain is very much analogous to building a  
 
software package based on a software tool chain. The calculation is seen simply as a build with dependencies.  
 
software package based on a software tool chain. The calculation is seen simply as a build with dependencies.  
Line 82: Line 200:
 
cover a variety of data processing and imaging tasks described in the books. Figures and recipes serve  
 
cover a variety of data processing and imaging tasks described in the books. Figures and recipes serve  
 
dual purpose with respect to Madagascar maintenance. They provide demos for introducing new users to the  
 
dual purpose with respect to Madagascar maintenance. They provide demos for introducing new users to the  
functionality of the package and, at the same time, regression tests for assuring the system stability  
+
functionality of the package and, at the same time, [[Automatic Testing|regression tests]] for assuring the system stability  
under change.  
+
under change.
  
==How it All Comes Together==
+
==Madagascar Trivia==
  
Here is an example code, described in detail on the [[Reproducible computational experiments using SCons | SCons]] page.
+
===Why the Name "Madagascar"?===
  
<python>
+
[http://ahay.org/blog/2006/04/19/madagascar/ Whimsy, really]. It seems easier to remember than the previous name "RSF", and it provides us interesting [http://ahay.org/blog/2008/10/21/ahay/ mascots].
from rsfproj import *
 
 
# Download the input data file
 
Fetch('lena.img','imgs')
 
 
# Create RSF header
 
Flow('lena.hdr','lena.img',
 
    'echo n1=512 n2=513 in=$SOURCE data_format=native_uchar',
 
    stdin=0)
 
 
# Convert to floating point and window out first trace
 
Flow('lena','lena.hdr','dd type=float | window f2=1')
 
 
# Display
 
Result('lena',
 
      '''
 
      sfgrey title="Hello, World!" transp=n color=b bias=128
 
      clip=100 screenratio=1
 
      ''')
 
 
# Wrap up
 
End()
 
</python>
 
  
==Getting Madagascar==
+
===License===
  
Madagascar runs on Unix/Linux platforms, including MacOS X and Unix emulations under Miscrosoft Windows. Its installation requires, at a minimum, a working C compiler and Python. Most users will also want an X-Window system on their desktop. See [[Download|download]] and
+
The Madagascar package is released in an open-source form under the standard [http://www.gnu.org/copyleft/gpl.html GNU GPL] license. In simple  
[[Installation | installation instructions]].
 
 
 
==License==
 
 
 
The Madagascar package is released in an open-source form under the standard GNU GPL license. In simple  
 
 
words, there are no restrictions on the use of the software (including copying, modifying, selling, etc.)  
 
words, there are no restrictions on the use of the software (including copying, modifying, selling, etc.)  
 
However, there are restrictions on the software redistribution intended to prevent the package from losing  
 
However, there are restrictions on the software redistribution intended to prevent the package from losing  
its open-source status. Users are encourages to submit their modifications back to the original distribution  
+
its open-source status. Users are encouraged to [[Contributing new programs to Madagascar|submit their modifications]] back to the original distribution for the benefit of the whole [[Package_overview#Community|community]].
to the benefit of the whole user community.
 
 
 
==Why the Name "Madagascar"?==
 
 
 
[http://www.reproducibility.org/rsflog/index.php?/archives/67-Madagascar.html Whimsy, really]. It seems easier to remember than the previous name "RSF", and it provides us lots of interesting [http://www.reproducibility.org/rsflog/index.php?/archives/178-Ahay!.html mascots].
 
  
==Madagascar Community==
+
===Community===
  
Madagascar seeks to become an open and active open source community. Active mailing lists are maintained and annual meetings take place. See  
+
Madagascar seeks to be an open and active community. Mailing lists are maintained, and annual meetings take place. See  
 
* [[Conferences]]
 
* [[Conferences]]
* [http://www.ahay.org/rsflog Development blog]
+
* [http://ahay.org/blog/ Development blog]
 
* [https://lists.sourceforge.net/lists/listinfo/rsf-user RSF-user mailing list]  
 
* [https://lists.sourceforge.net/lists/listinfo/rsf-user RSF-user mailing list]  
 
* [https://lists.sourceforge.net/lists/listinfo/rsf-devel RSF-devel mailing list]  
 
* [https://lists.sourceforge.net/lists/listinfo/rsf-devel RSF-devel mailing list]  
 +
* [http://www.linkedin.com/e/vgh/1847746 LinkedIn group]
 
Your participation is welcome.
 
Your participation is welcome.
  
==History==  
+
===History===
 +
 
 +
Madagascar was first publicly presented at the [[Conferences#Vienna_2006_.28EAGE.29|EAGE Workshop]] in Vienna in June 2006. The work on the package (previously named RSF) was started by Sergey Fomel in 2003. Since then, many people have contributed to it. See [https://github.com/ahay/src/blob/master/AUTHORS.txt AUTHORS.txt] for an incomplete list.
  
In the present form, the Madagascar package, while being completely written from scratch, borrows ideas from the design of [http://sepwww.stanford.edu/doku.php?id=sep:software:seplib SEPlib], a publicly available software package, maintained by Bob Clapp at the Stanford Exploration Project. Generations of SEP students and researchers contributed to SEPlib. Most important contributions came from Rob Clayton, Jon Claerbout, Dave Hale, Stew Levin, Rick Ottolini, Joe Dellinger, Steve Cole, Dave Nichols, Martin Karrenbach, Biondo Biondi, and Bob Clapp.
+
While being written mostly from scratch, Madagascar borrows ideas from the design of [http://sepwww.stanford.ed/doku.php?id=sep:software:seplib SEPlib], a package maintained by Bob Clapp at the Stanford Exploration Project (SEP). Generations of SEP students and researchers contributed to SEPlib. Most important contributions came from Rob Clayton, Jon Claerbout, Dave Hale, Stew Levin, Rick Ottolini, Joe Dellinger, Steve Cole, Dave Nichols, Martin Karrenbach, Biondo Biondi, and Bob Clapp.
  
Madagascar also borrows ideas from [http://timna.mines.edu/cwpcodes/ Seismic Unix] (SU), a package maintained by John Stockwell at the Center for Wave Phenomenon at the Colorado School of Mines (Stockwell, 1997<ref>Stockwell, J. W.,  1997, Free software in education: A case study of CWP/SU: Seismic Unix: The Leading Edge, '''16''', 1045--1049.</ref>;Stockwell, 1999<ref>--------, 1999, The CWP/SU: Seismic Un*x package: Computers and  Geosciences, '''25''', 415--419.</ref>). Main contributors to SU included Einar Kjartansson, Shuki Ronen, Jack Cohen, Chris Liner, Dave Hale, and John Stockwell. SU is open-source software (distributed with BSD-style license) starting with release 40 (April 10, 2007).
+
Madagascar also borrows ideas from [http://timna.mines.edu/cwpcodes/ Seismic Unix] (SU), an open-source package maintained by John Stockwell at the Center for Wave Phenomena (CWP) at the Colorado School of Mines (Stockwell, 1997<ref>Stockwell, J. W.,  1997, Free software in education: A case study of CWP/SU: Seismic Unix: The Leading Edge, '''16''', 1045--1049.</ref>;Stockwell, 1999<ref>--------, 1999, The CWP/SU: Seismic Un*x package: Computers and  Geosciences, '''25''', 415--419.</ref>). Main contributors to SU included Einar Kjartansson, Shuki Ronen, Jack Cohen, Chris Liner, Dave Hale, and John Stockwell. SU adopted an open-source BSD-style license starting with release 40 (April 10, 2007).
  
==References==
 
 
<references/>
 
<references/>
 +
 +
===Estimated Cost===
 +
 +
The Project Cost Calculator on Madagascar's [https://www.openhub.net/p/m8r Open Hub (formerly Ohloh) metrics page] shows the estimated cost of the project about $ 19,945,109 using the Basic COCOMO Model by April 20, 2020.

Latest revision as of 03:36, 23 August 2020

Fotolia 5127720 XS.jpg

The mission of the Madagascar project is to provide a shared research environment for computational data analysis in geophysics and related fields.

The Madagascar environment consists of:

  1. Standalone programs for out-of-core data analysis;
  2. Standalone programs for geophysical data processing and imaging;
  3. A development kit for C, C++, Java, Fortran-77, Fortran-90, Python, Matlab, and Octave;
  4. A framework for reproducible numerical experiments, based on SCons;
  5. A framework for scientific publications, based on SCons and LaTeX;
  6. A collection of reproducible scientific articles also used as usage examples and regression tests for the standalone programs;
  7. A collection of datasets used as input to reproducible numerical experiments.

This guide serves as a brief introduction of different components and shows how they all fit together.

How to obtain Madagascar

See download and installation instructions. Madagascar runs on Unix/Linux platforms, including MacOS X and Unix emulations under Miscrosoft Windows. Its installation requires, at a minimum, a working C compiler and Python.

How to find your way around Madagascar

Start by checking the list of reproducible papers. If any of these papers looks close to your interests, follow the links until you find a figure with a "wrench" button under it Configure.png. Click on the wrench, and it will open a computational recipe used for generating the figure (the SConstruct file).

from rsf.proj import *
Flow('rose',None,
     '''
     math n1=629 d1=0.01 o1=0 n2=40 d2=1 o2=5 
     output="x2*(8+sin(6*x1+x2/10))" |
     rtoc |
     math output="input*exp(I*x1)"
     ''')
Result('rose',
       'graph title=Rose screenratio=1 wantaxis=n')
End()

You can copy this recipe to your computer or simply find it already existing in the "book" subtree under the Madagascar source directory. For example, the recipe at http://www.reproducibility.org/RSF/book/rsf/rsf/sfmath.html also exists in the file RSFSRC/book/rsf/rsf/sfmath/SConstruct. After copying or locating the appropriate SConstruct file, run

scons view

on the command line to generate all the figures in the selected project and to display them on your screen. For example, try

bash$ cd RSFSRC/book/rsf/rsf/sfmath
bash$ scons view

where bash$ stands for the Unix prompt and RSFSRC stands for the Madagascar source directory. The output should look like

scons: Reading SConscript files ...
scons: done reading SConscript files.
scons: Building targets ...
/RSFROOT/bin/sfmath n1=629 d1=0.01 o1=0 n2=40 d2=1 o2=5 output="x2*(8+sin(6*x1+x2/10))" | /RSFROOT/bin/sfrtoc | 
/RSFROOT/bin/sfmath output="input*exp(I*x1)" > rose.rsf
< rose.rsf /RSFROOT/bin/sfgraph title=Rose screenratio=1 wantaxis=n > Fig/rose.vpl
/RSFROOT/bin/sfpen Fig/rose.vpl
scons: done building targets.

with a picture appearing on your screen.

Rose.png

If there are several figures in the recipe, you can run

scons figurename.view

(e.g. scons rose.view) to display individual figures. To remove all files that scons view generated, run

scons -c view

If you want to know in advance what commands will be executed to generate the figures, try

scons -n view

You can output this command to a file

scons -n -Q view > script.sh

and use script.sh as a shell script. If you are to make modifications to the data processing recipe (changing parameters or trying new data), working with SCons is more powerful and convenient than running shell scripts.

A computational recipe puts together individual commands through Unix pipes and SCons rules. These commands act like Lego blocks for creating complex data analysis constructions. In the example above, three "blocks" are used: sfmath, sfrtoc, and sfgraph. To find out what a particular command is doing, you can follow the links from the bottom of the web page http://www.reproducibility.org/RSF/book/rsf/rsf/sfmath.html Alternatively, run the command without arguments on the command line. Running

bash$ sfrtoc

produces something like

NAME
        sfrtoc
DESCRIPTION
        Convert real data to complex (by adding zero imaginary part).
SYNOPSIS
        sfrtoc < real.rsf > cmplx.rsf
COMMENTS   
        See also: sfcmplx
USED IN
        bei/ft1/plane4
        bei/ft1/autocor
        bei/ft1/brad
        [...]
SOURCE
        user/main/rtoc.c
DOCUMENTATION
        http://reproducibility.org/wiki/Guide_to_madagascar_programs#sfrtoc

The DOCUMENTATION section provides a link to a more detailed documentation on the web. The most useful part is the USED IN section, which points to more examples of using the program. As an exercise, change directory to RSFSRC/book/bei/ft1/plane4 or any other example directory, examine the SConstruct file, and run scons view. Alternatively, look at http://www.reproducibility.org/RSF/book/bei/ft1/plane4.html

Want to find a program by keywords? Try sfdoc -k.

bash$ sfdoc -k complex
sfsort: Sort a float/complex vector by absolute values.
sfrtoc: Convert real data to complex (by adding zero imaginary part).
sfjacobi2: Find eigenvalues of a general complex matrix by Jacobi-like iteration. 
sfboolcmp: Element-wise boolean comparison of values. For int/float/complex data-sets.
sfcmatmult: Simple matrix multiplication for complex matrices 
sfimag: Extract real (sfreal) or imaginary (sfimag) part of a complex dataset. 
sfthr: Threshold float/complex inputs given a constant/varying threshold level.
sfcpef: 1-D prediction-error filter estimation from complex data 
sfroots: Find roots of a complex polynomial. 
sfreal: Extract real (sfreal) or imaginary (sfimag) part of a complex dataset. 
sfcmplx: Create a complex dataset from its real and imaginary parts.
sfsin: Simple operations with complex sinusoids 
sfcdottest: Generic dot-product test for complex linear operators with adjoints 
sfcconjgrad: Generic conjugate-gradient solver for linear inversion with complex data

Individual components of the Madagascar environment are described in more details below.

Madagascar components

Standalone programs

The list of all standalone programs is available online. Most programs act as filters on input data and can be chained through Unix pipes, i.e.: <bash> < data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf </bash> This approach follows the Unix philosophy, as formulated by Doug McIlroy, the inventor of Unix pipes (Salus, 1994[1]):

  1. Write programs that do one thing and do it well.
  2. Write programs to work together.
  3. Write programs to handle text streams, because that is a universal interface.

Following the Unix convention, programs have brief man pages, which explain the program purpose and parameters. You can access this documentation by running a program without parameters. To search for a program by a keyword, use sfdoc -k <keyword>.

The guide to madagascar programs provides more detailed documentation for selected programs while the task-centric program list attempts to categorize them. You can see the programs in actual use in the Reproducible Documents.

Data format

For data, Madagascar uses the Regularly Sampled Format (RSF), which is based on the concept of hypercubes (n-D arrays, or regularly sampled functions of several variables), much like the SEPlib (its closest relative), DDS, or the regularly-sampled version of the Javaseis format (SVF). Up to 9 dimensions are supported. For 1D it is conceptually analogous to a time series, for 2D to a raster image, and for 3D to a voxel volume. The format (actually a metaformat) makes use of a ASCII file with metadata (information about the data), including a pointer (in= parameter) to the location of the file with the actual data values. Irregularly sampled data are currently handled as a pair of datasets, one containing data and the second containing the corresponding irregular geometry information. Programs for conversion to and from other formats such as SEG-Y and SU are provided.

For graphics, Madagascar currently uses the Vplot vector graphics format. Converters to other graphics formats (Postscript, PNG, GIF, JPEG) are also provided.

Reproducible documents

A reproducible document consists of LaTeX source combined with SCons rules required to fully build the documents. These rules are expressed in terms of SCons extensions that are provided as part of Madagascar.

This is the key to the reproducibility aspect of Madagascar. An introduction to reproducible Madagascar documents is at Reproducible_computational_experiments_using_SCons .

Vplot graphics

In contrast to most other Madagascar Components, graphics components produce Vplot data as output.

Vplot is a device independent graphics format that allows both vector and raster elements (as such, it is comparable to Postscript). Vplot files are interpreted by a number of output devices. Its typical usage is for a visual display in X-windows. A list of them is provided on the wiki.

Here is an example of a Madagascar pipe. In this case it takes a subsection of a file, low-pass filters it, and saves the result

< data.rsf sfwindow n1=100 | sfbandpass fhi=60 > data2.rsf

In this more elaborate case, the final output is passed to a graphics program and plotted.

< data.rsf sfwindow n1=100 | sfbandpass fhi=60 | sfcontour | xtpen

More extensive examples are seen in Guide to madagascar programs . The novice reader should probably read the material below before proceeding to that page.

Reproducibility and Project Management

Madagascar uses and extends SCons, an open-source software construction package, to document and maintain data processing flows. Documented projects become computational recipes that can be easily exchanged among Madagascar users.

SCons is a rule-based package in Python typically used as a build system analogous to make. Familiarity with any build system will be helpful in understanding SCons. SCons statements, as python statements, are invoked in the sequence they are written, but as such they only define rules. The rules are invoked in accordance with a dependency graph which SCons builds based on those rules. Components regarded as "up-to-date" are not rebuilt.

SCons allows for user-contributed Builders (meta-rule categories) and Madagascar uses this capability extensively. The idea is that building an output file based on a workflow chain is very much analogous to building a software package based on a software tool chain. The calculation is seen simply as a build with dependencies. This is a considerable benefit in developing alternative workflows using a given dataset. The system maintains an awareness of already completed calculations. Without user intervention, redundant calculations are avoided.

Madagascar calculations are thus expressed as SCons scripts (SConstruct files). SCons extensions follow SCons conventions in beginning with an uppercase letter. The most common Madagascar extensions are Flow(), Result(), and End(). A Flow() invocation wraps Madagascar computational components. Result() is a version of Flow() with a graphical output. Finally an End() actually invokes the default rules for multiple results.

Finally, Madagascar enables a collection of reproducible documents, organized in living books. Each reproducible book contains a collection of Madagascar recipes (SConstruct files) used to generate book figures. The recipes cover a variety of data processing and imaging tasks described in the books. Figures and recipes serve dual purpose with respect to Madagascar maintenance. They provide demos for introducing new users to the functionality of the package and, at the same time, regression tests for assuring the system stability under change.

Madagascar Trivia

Why the Name "Madagascar"?

Whimsy, really. It seems easier to remember than the previous name "RSF", and it provides us interesting mascots.

License

The Madagascar package is released in an open-source form under the standard GNU GPL license. In simple words, there are no restrictions on the use of the software (including copying, modifying, selling, etc.) However, there are restrictions on the software redistribution intended to prevent the package from losing its open-source status. Users are encouraged to submit their modifications back to the original distribution for the benefit of the whole community.

Community

Madagascar seeks to be an open and active community. Mailing lists are maintained, and annual meetings take place. See

Your participation is welcome.

History

Madagascar was first publicly presented at the EAGE Workshop in Vienna in June 2006. The work on the package (previously named RSF) was started by Sergey Fomel in 2003. Since then, many people have contributed to it. See AUTHORS.txt for an incomplete list.

While being written mostly from scratch, Madagascar borrows ideas from the design of SEPlib, a package maintained by Bob Clapp at the Stanford Exploration Project (SEP). Generations of SEP students and researchers contributed to SEPlib. Most important contributions came from Rob Clayton, Jon Claerbout, Dave Hale, Stew Levin, Rick Ottolini, Joe Dellinger, Steve Cole, Dave Nichols, Martin Karrenbach, Biondo Biondi, and Bob Clapp.

Madagascar also borrows ideas from Seismic Unix (SU), an open-source package maintained by John Stockwell at the Center for Wave Phenomena (CWP) at the Colorado School of Mines (Stockwell, 1997[2];Stockwell, 1999[3]). Main contributors to SU included Einar Kjartansson, Shuki Ronen, Jack Cohen, Chris Liner, Dave Hale, and John Stockwell. SU adopted an open-source BSD-style license starting with release 40 (April 10, 2007).

  1. Salus, P. H., 1994, A quarter-century of Unix: Addison-Wesley.
  2. Stockwell, J. W., 1997, Free software in education: A case study of CWP/SU: Seismic Unix: The Leading Edge, 16, 1045--1049.
  3. --------, 1999, The CWP/SU: Seismic Un*x package: Computers and Geosciences, 25, 415--419.

Estimated Cost

The Project Cost Calculator on Madagascar's Open Hub (formerly Ohloh) metrics page shows the estimated cost of the project about $ 19,945,109 using the Basic COCOMO Model by April 20, 2020.