Difference between revisions of "Tutorial - Users 1 - Command line interface"

From Madagascar
Jump to navigation Jump to search
 
(14 intermediate revisions by 2 users not shown)
Line 1: Line 1:
Madagascar is designed to be used from the command line.  Programmers create Madagascar programs (prefixed with the sf designation) that read and write Madagascar files.  These programs are designed to be as general as possible, each one operates on any dataset provided in RSF format, provided you also provide the correct parameters and have the right type of data for the program.  Thus, each Madagascar program can be used in even ways that the original developer may not have foreseen, which makes your life easier as a user!
+
Madagascar was designed initially to be used from the command line.  Programmers create Madagascar programs (prefixed with the sf designation) that read and write Madagascar files.  These programs are designed to be as general as possible, each one operates on any dataset in RSF format, provided you also supply the correct parameters and have the right type of data for the program.  Thus, each Madagascar program can be used in different ways that the original developer may not have foreseen, which makes your life easier as a user!
  
 
= Command line interface =  
 
= Command line interface =  
Madagascar programs follow the standard UNIX conventions for reading and writing RSF files to and from STDIN and STDOUT.  This is also the same convention that SeismicUnix uses, so for those of you who are SU fans, this should be an easy conversion.   
+
Madagascar programs follow the standard UNIX conventions for reading and writing RSF files to and from STDIN and STDOUT.  This is also the convention that [http://www.cwp.mines.edu/cwpcodes/ Seismic Unix] uses, so for those of you who are SU fans, this should be an easy conversion.   
 
 
For example: the program sfattr allows us to get attributes about an RSF file (mean, standard deviation, min, max, etc.)  To get the attributes for a pre-computed file, we might use:
 
  
 +
For example: the program [[Guide_to_madagascar_programs#sfattr|sfattr]] allows us to get attributes about an RSF file (mean, standard deviation, min, max, etc.)  To get the attributes for a pre-computed file, we might use
 +
<bash>
 
sfattr < junk.rsf  
 
sfattr < junk.rsf  
 
+
</bash>
 
which outputs:
 
which outputs:
 
<code>
 
<code>
Line 20: Line 20:
 
   total samples = 64000  
 
   total samples = 64000  
 
</code>
 
</code>
For an example of reading and writing an RSF file I'm going to demonstrate the use of sfwindow which is a program that allows us to select portions of an RSF file.  When sfwindow is used without any additional parameters, we are able to make a copy of a file with a different filename.  For example:
+
For an example of reading and writing an RSF file I'm going to demonstrate the use of [[Guide_to_madagascar_programs#sfwindow|sfwindow]] which is a program that allows us to select portions of an RSF file.  When <tt>sfwindow</tt> is used without any additional parameters, we are able to make a copy of a file with a different filename.  For example:
 
+
<bash>
 
sfwindow < junk.rsf > junk2.rsf
 
sfwindow < junk.rsf > junk2.rsf
 +
</bash>
 +
gives us two files, junk.rsf and junk2.rsf which are identical but not the same file. If your intention is simply to copy a file, you can also use [[Guide_to_madagascar_programs#sfcp|sfcp]].
  
gives us two files, junk.rsf and junk2.rsf which are identical but not the same file.
+
In addition to specifying files to read in and out on the command line we can specify the parameters for each program that are necessary for it to run, or to produce the desired result.  The general format for specifying parameters on the command line is '''key=val''', where '''key''' is the name of the parameter that you want to set, and '''val''' is the value of the parameter.  There are four (4) different types of values that are acceptable:  int, float, boolean, or string.  Going back to the window program, we can specify the number of traces or pieces of the file that we want to keep like:
 
 
In addition to specifying files to read in and out on the command line we can specify the parameters for each program that are necessary for it to run, or to produce the desired result.  The general format for specifying parameters on the command line is key=val, where key is the name of the parameter that you want to set, and val is the value of the parameter.  There are four (4) different types of values that are acceptable:  int, float, boolean, or string.  Going back to the window program, we can specify the number of traces or pieces of the file that we want to keep like:
 
  
 +
<bash>
 
sfwindow < junk.rsf n1=10 > junk-win.rsf
 
sfwindow < junk.rsf n1=10 > junk-win.rsf
 +
</bash>
  
 
= Self-documentation =
 
= Self-documentation =
Line 34: Line 36:
 
Of course, we can specify as many parameters as we'd like on the command line.  To figure out which parameters are needed for a specific program, just type the name of the program with no input files our output files on the command line as follows:
 
Of course, we can specify as many parameters as we'd like on the command line.  To figure out which parameters are needed for a specific program, just type the name of the program with no input files our output files on the command line as follows:
  
 +
<bash>
 
sfwindow
 
sfwindow
 +
</bash>
  
 
which brings up the self-documentation, which looks something like the following:
 
which brings up the self-documentation, which looks something like the following:
  
 +
<pre>
 
NAME
 
NAME
 
         sfwindow
 
         sfwindow
Line 63: Line 68:
 
         bei/fld/yc
 
         bei/fld/yc
 
         bei/ft1/autocor
 
         bei/ft1/autocor
 +
</pre>
  
The self-documentation tells us the function of the program, as well as the parameters that are available to be specified.  The parameter format is type - name=default value [options] and then a short description of the parameter.  File parameters request a name of a file.  For example:  file=junk.rsf
+
The self-documentation tells us the function of the program, as well as the parameters that are available to be specified.  The parameter format is type - name=default value [options] and then a short description of the parameter.  File parameters request a name of a file.  For example:  <tt>file=junk.rsf</tt>
  
 
= Piping =
 
= Piping =
  
Sometimes we want to chain multiple commands together without writing intermediate Madagascar RSF files in the process.  Fortunately, we can do this using another standard UNIX construct, Pipes.  Pipes allow us to connect the standard output from one Madagascar program to the standard input to another program without first writing to a file.  For example we could do the following without pipes:
+
Sometimes we want to chain multiple commands together without writing intermediate Madagascar RSF files in the process.  Fortunately, we can do this using another standard UNIX construct, [http://en.wikipedia.org/wiki/Pipeline_(Unix) Pipes].  Pipes allow us to connect the standard output from one Madagascar program to the standard input to another program without first writing to a file.  For example we could do the following without pipes:
 +
<bash>
 +
  sfwindow < junk.rsf > junk-win.rsf
 +
  sftransp < junk-win.rsf > junk2.rsf
 +
</bash>
 +
Or we could do the equivalent using pipes on one line:
 +
<bash>
 +
  sfwindow < junk.rsf | sftransp > junk2.rsf
 +
</bash>
 +
Pipes simply make these statements more compact, and allow us to reduce the number of files that we need to save to disk.  If you're doing something many times over, then pipes will make your life significantly easier.  I highly recommend that you use pipes!  Also, you should know that you can chain as many commands together as necessary using Pipes.
  
sfwindow < junk.rsf > junk-win.rsf
+
= Interacting with files =
sftransp < junk-win.rsf > junk2.rsf
 
  
Or we could do the equivalent using pipes on one line:
+
Ultimately though, 95% of your time using Madagascar on the command line will be to inspect and view files that are output by your scripts.  Some of the key commands that are used to interact with files on the command line are:
 +
 
 +
'''sfin - get header information'''
 +
 
 +
[[Guide_to_madagascar_programs#sfin|SFIN]] is one of the most used program on the command line, because most often we simply need to check the dimensionality of our files to make sure that we have them in the correct order.
 +
 
 +
<bash>
 +
sfin junk.rsf
 +
 
 +
junk.rsf:
 +
    in="/var/tmp/junk.rsf@"
 +
    esize=4 type=float form=native
 +
    n1=100        d1=0.004      o1=0          label1="Time" unit1="s"
 +
    n2=34          d2=0.1        o2=0          label2="Distance" unit2="km"
 +
3400 elements 13600 bytes
 +
</bash>
 +
 
 +
'''sfattr - get file attributes'''
 +
 
 +
[[Guide_to_madagascar_programs#sfattr|SFATTR]] is also commonly used from the command line to check files for correct values.  Most often, we use <tt>sfattr</tt> to ensure that files are not filled with zeros, or with NaN's after a long computation, or to make sure that the values are reasonable.  sfattr can be used to obtain basic statistics about the files as well. 
 +
 
 +
<bash>
 +
sfattr < junk.rsf
 +
 
 +
*******************************************
 +
    rms =            1
 +
    mean =            1
 +
  2-norm =      58.3095
 +
variance =            0
 +
std dev =            0
 +
    max =            1 at 1 1
 +
    min =            1 at 1 1
 +
nonzero samples = 3400
 +
  total samples = 3400
 +
*******************************************
 +
</bash>
 +
 
 +
'''sfwindow - window out specific portions of a file to be used elsewhere '''
 +
 
 +
[[Guide_to_madagascar_programs#sfwindow|SFWINDOW]] is one of the most versatile programs to use.  Typically, it is used to select subsets of the data contained in an RSF file for computation elsewhere.  Typically, you specify the data subset you want to keep using, the n, j, and f parameters which specify indices in the arrays to keep.  For example if we want to keep the 15th-30th  time samples for each distance in junk.rsf, we might use the following command:
 +
 
 +
<bash>
 +
sfwindow < junk.rsf f1=15 n1=15 j1=1 > junk-win.rsf
 +
</bash>
 +
 
 +
'''sftransp - transpose the dataset axes'''
  
sfwindow < junk.rsf | sftransp > junk2.rsf
+
[[Guide_to_madagascar_programs#sftransp|SFTRANSP]] is also very versatile, and applied to reorder RSF files for other programs to be used. For example:
  
Pipes simply make these statements more compact, and allow us to reduce the number of files that we need to save to disk. If you're doing something many times over, then pipes will make your life significantly easier. I highly recommend that you use pipes!  Also, you should know that you can chain as many commands together as necessary using Pipes. 
+
<bash>
 +
sftransp < junk.rsf plane=12 > junk-transposed.rsf
 +
</bash>
  
Note: Madagascar uses a trick to make sure that you can send as much data as you want through the pipes because a conventional UNIX pipe has a maximum amount of data that can be sent through it.  To avoid this limitation, Madagascar writes the data to temporary files before sending it through a pipe.
+
swaps the first and second axes, so that now the first axis is distance and the second axis is time.
  
= Commonly used programs =
+
See [[Guide_to_madagascar_programs|Guide to Madagascar programs]] for a description of other common programs.
  
Here's a short list of commonly used programs, their usage:
+
= Scripting =
  
sfwindow - window out portions of a dataset
+
While you could use Madagascar solely from the command line, you'll probably find that it is far too tedious to do so.  Especially, when you want to change the value of a parameter and then rerun the same task.  To avoid some of that nastiness, you could use shell scripting to automate portions of your processing flow, but shell scripts quickly become cumbersome and are not optimal scripts (shell language is ugly in comparison with other scripting languages).  Fortunately, Madagascar comes with batteries included and supports another scripting language available for you to use called [http://www.scons.org/ SCons].  SCons is incredibly powerful as we will see in later tutorials, so please don't rush out and start writing shell scripts.  Read the next few tutorials then decide whether you still want to do so.
sftransp    - transpose the axes of a dataset
 
sfin          - see the shape of a file (axes)
 
sfattr          - common attributes of files
 
sfspike      - create files
 
sfmath      - basic mathematical operations (add subtract multiply)
 
sfadd        - like sfmath but easier to use
 
sfgrey        - raster plots
 
sfgraph      - line plots
 
sfwiggle    - wiggle plots
 
  
For more information see [[Guide_to_madagascar_programs | Program guide]].
 
  
= Interacting with files =
+
''' Back to [[Tutorial]]'''

Latest revision as of 15:32, 9 May 2011

Madagascar was designed initially to be used from the command line. Programmers create Madagascar programs (prefixed with the sf designation) that read and write Madagascar files. These programs are designed to be as general as possible, each one operates on any dataset in RSF format, provided you also supply the correct parameters and have the right type of data for the program. Thus, each Madagascar program can be used in different ways that the original developer may not have foreseen, which makes your life easier as a user!

Command line interface

Madagascar programs follow the standard UNIX conventions for reading and writing RSF files to and from STDIN and STDOUT. This is also the convention that Seismic Unix uses, so for those of you who are SU fans, this should be an easy conversion.

For example: the program sfattr allows us to get attributes about an RSF file (mean, standard deviation, min, max, etc.) To get the attributes for a pre-computed file, we might use <bash> sfattr < junk.rsf </bash> which outputs:

    rms =       1.41316 
   mean =      0.999667 
 2-norm =       357.503 
 variance =      0.997693 
std dev =      0.998846 
    max =       5.05567 at 36 8 27 
    min =      -3.59936 at 18 9 6 
 nonzero samples = 64000 
 total samples = 64000 

For an example of reading and writing an RSF file I'm going to demonstrate the use of sfwindow which is a program that allows us to select portions of an RSF file. When sfwindow is used without any additional parameters, we are able to make a copy of a file with a different filename. For example: <bash> sfwindow < junk.rsf > junk2.rsf </bash> gives us two files, junk.rsf and junk2.rsf which are identical but not the same file. If your intention is simply to copy a file, you can also use sfcp.

In addition to specifying files to read in and out on the command line we can specify the parameters for each program that are necessary for it to run, or to produce the desired result. The general format for specifying parameters on the command line is key=val, where key is the name of the parameter that you want to set, and val is the value of the parameter. There are four (4) different types of values that are acceptable: int, float, boolean, or string. Going back to the window program, we can specify the number of traces or pieces of the file that we want to keep like:

<bash> sfwindow < junk.rsf n1=10 > junk-win.rsf </bash>

Self-documentation

Of course, we can specify as many parameters as we'd like on the command line. To figure out which parameters are needed for a specific program, just type the name of the program with no input files our output files on the command line as follows:

<bash> sfwindow </bash>

which brings up the self-documentation, which looks something like the following:

NAME
        sfwindow
DESCRIPTION
        Window a portion of a dataset. 
SYNOPSIS
        sfwindow < in.rsf > out.rsf verb=n squeeze=y j#=(1,...) d#=(d1,d2,...) f#=(0,...) min#=(o1,o2,,...) n#=(0,...) max#=(o1+(n1-1)*d1,o2+(n1-1)*d2,,...)
PARAMETERS
        float   d#=(d1,d2,...)  sampling in #-th dimension 
        largeint f#=(0,...)     window start in #-th dimension 
        int     j#=(1,...)      jump in #-th dimension 
        float   max#=(o1+(n1-1)*d1,o2+(n1-1)*d2,,...)   maximum in #-th dimension 
        float   min#=(o1,o2,,...)       minimum in #-th dimension 
        largeint n#=(0,...)     window size in #-th dimension 
        bool    squeeze=y [y/n] if y, squeeze dimensions equal to 1 to the end 
        bool    verb=n [y/n]    Verbosity flag
USED IN
        bei/dpmv/krchdmo
        bei/dpmv/matt
        bei/dwnc/sigmoid
        bei/fdm/kjartjac
        bei/fld/cube
        bei/fld/shotmovie
        bei/fld/synmarine
        bei/fld/yc
        bei/ft1/autocor

The self-documentation tells us the function of the program, as well as the parameters that are available to be specified. The parameter format is type - name=default value [options] and then a short description of the parameter. File parameters request a name of a file. For example: file=junk.rsf

Piping

Sometimes we want to chain multiple commands together without writing intermediate Madagascar RSF files in the process. Fortunately, we can do this using another standard UNIX construct, Pipes. Pipes allow us to connect the standard output from one Madagascar program to the standard input to another program without first writing to a file. For example we could do the following without pipes: <bash>

 sfwindow < junk.rsf > junk-win.rsf
 sftransp < junk-win.rsf > junk2.rsf

</bash> Or we could do the equivalent using pipes on one line: <bash>

 sfwindow < junk.rsf | sftransp > junk2.rsf

</bash> Pipes simply make these statements more compact, and allow us to reduce the number of files that we need to save to disk. If you're doing something many times over, then pipes will make your life significantly easier. I highly recommend that you use pipes! Also, you should know that you can chain as many commands together as necessary using Pipes.

Interacting with files

Ultimately though, 95% of your time using Madagascar on the command line will be to inspect and view files that are output by your scripts. Some of the key commands that are used to interact with files on the command line are:

sfin - get header information

SFIN is one of the most used program on the command line, because most often we simply need to check the dimensionality of our files to make sure that we have them in the correct order.

<bash> sfin junk.rsf

junk.rsf:

   in="/var/tmp/junk.rsf@"
   esize=4 type=float form=native 
   n1=100         d1=0.004       o1=0          label1="Time" unit1="s" 
   n2=34          d2=0.1         o2=0          label2="Distance" unit2="km" 

3400 elements 13600 bytes </bash>

sfattr - get file attributes

SFATTR is also commonly used from the command line to check files for correct values. Most often, we use sfattr to ensure that files are not filled with zeros, or with NaN's after a long computation, or to make sure that the values are reasonable. sfattr can be used to obtain basic statistics about the files as well.

<bash> sfattr < junk.rsf

    rms =             1 
   mean =             1 
 2-norm =       58.3095 

variance = 0

std dev =             0 
    max =             1 at 1 1 
    min =             1 at 1 1 

nonzero samples = 3400

 total samples = 3400 

</bash>

sfwindow - window out specific portions of a file to be used elsewhere

SFWINDOW is one of the most versatile programs to use. Typically, it is used to select subsets of the data contained in an RSF file for computation elsewhere. Typically, you specify the data subset you want to keep using, the n, j, and f parameters which specify indices in the arrays to keep. For example if we want to keep the 15th-30th time samples for each distance in junk.rsf, we might use the following command:

<bash> sfwindow < junk.rsf f1=15 n1=15 j1=1 > junk-win.rsf </bash>

sftransp - transpose the dataset axes

SFTRANSP is also very versatile, and applied to reorder RSF files for other programs to be used. For example:

<bash> sftransp < junk.rsf plane=12 > junk-transposed.rsf </bash>

swaps the first and second axes, so that now the first axis is distance and the second axis is time.

See Guide to Madagascar programs for a description of other common programs.

Scripting

While you could use Madagascar solely from the command line, you'll probably find that it is far too tedious to do so. Especially, when you want to change the value of a parameter and then rerun the same task. To avoid some of that nastiness, you could use shell scripting to automate portions of your processing flow, but shell scripts quickly become cumbersome and are not optimal scripts (shell language is ugly in comparison with other scripting languages). Fortunately, Madagascar comes with batteries included and supports another scripting language available for you to use called SCons. SCons is incredibly powerful as we will see in later tutorials, so please don't rush out and start writing shell scripts. Read the next few tutorials then decide whether you still want to do so.


Back to Tutorial