|
|
| (4 intermediate revisions by the same user not shown) |
| Line 1: |
Line 1: |
| − | ==Madagascar 2008 Implementation Workshop: Towards full automation and better robustness==
| |
| − | [[Image:Fotolia_800161_XS.jpg|right|]]
| |
| | [[Image:M8rsprint2.jpg|right|]] | | [[Image:M8rsprint2.jpg|right|]] |
| − | '''UNDER CONSTRUCTION. This page is being changed from an invitation into a description of the meeting.'''
| |
| | | | |
| | The workshop took place over the Memorial Day weekend, from Fri May 23 to Mon May 26, at the [http://www.mines.edu/academic/geophysics/prospective_students_maps.html Geophysics Department] of [http://www.mines.edu Colorado School of Mines], in [http://en.wikipedia.org/wiki/Golden,_Colorado Golden, Colorado]. | | The workshop took place over the Memorial Day weekend, from Fri May 23 to Mon May 26, at the [http://www.mines.edu/academic/geophysics/prospective_students_maps.html Geophysics Department] of [http://www.mines.edu Colorado School of Mines], in [http://en.wikipedia.org/wiki/Golden,_Colorado Golden, Colorado]. |
| | | | |
| − | participants worked together on implementing features in Madagascar.
| + | ==Participants== |
| | + | From left to right, in the picture: |
| | + | #Paul Sava (CSM) |
| | + | #Joe Dellinger (BP) |
| | + | #Sean Ross-Ross (UBC) |
| | + | #Jim McClean (PGS) |
| | + | #Sam Brown (U of Utah) |
| | + | #Ioan Vlad (StatoilHydro) |
| | + | #Yang Liu (UT Austin) |
| | + | #Michael Tobis (UT Austin) |
| | + | #Sjoerd de Ridder (Stanford) |
| | + | #Paul Murray (UT Austin) |
| | + | #Jules Browaeys (UT Austin) |
| | | | |
| | + | Not in the picture: Peter Traynin (ExxonMobil) |
| | + | |
| | + | ==Summary== |
| | + | The intended "deliverables" of this meeting were: |
| | | | |
| − | The "deliverables" of this meeting are:
| |
| | * A stronger geophysical open-source community | | * A stronger geophysical open-source community |
| | * A short-term Madagascar road map, created by discussing, extending, and prioritizing the feature request list | | * A short-term Madagascar road map, created by discussing, extending, and prioritizing the feature request list |
| | * A real move towards version 1.0 | | * A real move towards version 1.0 |
| | | | |
| − | ===Pool of suggested features (to be moved to Feature requests/another page)===
| + | The community was indeed strengthened by participants getting to know each other. |
| − | The m8r features under consideration can be grouped into several categories, which are listed below. The grouping attempts to describe the software equivalent of [http://en.wikipedia.org/wiki/Maslow's_hierarchy_of_needs Maslow's pyramid of needs]. First the basics (reproducibility, I/O, parallelization, graphics) should be consolidated, then efforts should move up to numerical tools like solvers, FFTs and transposes, then up to widely-used geophysical algorithms. Interested parties are invited to brainstorm below! | + | |
| | + | The second goal was not only fulfilled, but far exceeded. Not one road map was created, but as many road maps as participants. |
| | | | |
| − | # '''Features that provide functionality that is needed in order to have a minimalistic fully-automated m8r project setup'''
| + | The real move to version 1.0 was done by Joe, who completed <tt>vplotdiff</tt>, allowing automatic testing. Joe also played a crucial role in clarifying and solving the IP issues created by Madagascar containing some code from 1994 versions of SEPlib and vplot. In the correspondence that followed, the copyright holders and Stanford University's Office of Technology Licensing have agreed to release the code under GPL (for the portions of SEPlib used) and LGPL (for vplot). |
| − | ## Vplot diffs: These would allow the m8r project to fulfill one of its main goals – having fully automatic regression tests. See [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ## rsfbook completion, also from the [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ## Moving the wiki to Dreamhost so that it actually functions as a wiki again
| |
| − | ## Automatically sync-ing the Wiki "Guide to programs" with the self-doc;
| |
| − | ## Saving a static copy of the wiki so that consulting important parts of the documentation does not require a centralized, brittle software stack (local internet connectivity + remote web server + php + SQL + Mediawiki subject to constant spam attacks) to be running at that very moment. That would also allow splitting the "Guide to Programs" into one-page-per-program files, to be concatenated together into one searchable page in the static copy of the wiki, and for user-contributed parts to be included in the HTML self-doc;
| |
| − | # '''Features that make m8r more user-friendly'''
| |
| − | ## A sane configuration script for the tex2pdf reproducible paper engine, that pulls a complete list of missing dependencies and has a way of extracting them from TeX Live (or instructing the user to do so). Right now the huge size of TeX Live (>1Gb) and its fast pace of advancement precludes its installation on legacy systems (i.e. RHEL 2,3,4). The goal is to make the pdf paper generating mechanism as easy to install as the rest of m8r, i.e. configuration either fails with a helpful error message or everything installs fine and it Just Works.
| |
| − | ## sfdatadoc – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ## Go through all the programs to make sure that there are no undocumented or unclearly specified parameters. Also look at inconsistencies (when parameters with similar meaning get different names in different programs). Establish and implement coding conventions.
| |
| − | ## Binary packages – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker] (''Note 1: there has been [http://osdir.com/ml/programming.tools.scons.devel/2006-06/msg00021.html noise] since 2006 on SCons developer mail lists about providing a SCons command that would automatically create RPMs and maybe debs as well. Anyone know the implementation status of that? Note 2: Packages are typically split into a "package core" providing executables and help for them and a "package-devel" providing a [http://en.wikipedia.org/wiki/Software_development_kit Software development kit] consisting of development libraries, headers and other include files, API documentation, etc. Doing things this way would require deep changes of the current Madagascar build mechanisms'')
| |
| − | ## man pages – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ## Decide how to ensure that the [[Task-centric program list]] is always sync-ed with the source code. (keywords picked up by rsfdoc?)
| |
| − | ## Create "Migrant's dictionary" for newcomers from other packages, by using ''their'' task-centric pages such as [http://sepwww.stanford.edu/oldsep/cliner/files/suhelp/suhelp.html this one] to show which m8r programs correspond to which programs from their "home country"
| |
| − | ## TKSU-based GUI – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker] (''Note: the Madagascar plugin for OpendTect is also [http://www2.opendtect.org/wiki/madagascar/ on track]'')
| |
| − | # '''Features that extend m8r's capabilities'''
| |
| − | ## M8r-based programming
| |
| − | ### Java API (more details on the [[GSOC2008]] page)
| |
| − | ### Extending the Python interface (more details on the [[GSOC2008]] page)
| |
| − | ### Finishing the Octave interface
| |
| − | ### A tool to convert "normal" rsf to a "transfer-ready" rsf that has the binary converted to portable XDR, then gzipp-ed, the number of bytes of the compressed file and the MD5sum of the binary and of the compressed binary in the ASCII header, and this is shown by a keyword like form=xdr.gz . I checked with various types and sizes of data and even in the worst-case scenario, a factor of 2 in compression is attained. This would allow safe, fully-automatic data transfers.
| |
| − | ### Introduce a sane way to control the optimization level for all languages in a m8r build, using a single flag. A researcher using Madagascar should not be an expert in compiler usage, he should just set the level. For example:
| |
| − | ###* Level A, have all warnings and debugging info turned on. Link to rsflib version compiled similarly.
| |
| − | ###* Level B, compile with -O2 and link against rsflib version compiled similarly.
| |
| − | ###* Level C, same as level B, and also detect if compiler has interprocedural optimization abilities. If yes, compile everything with appropriate flags. If compiler cannot deal with optimizing across multiple files, have Python feed to compiler concatenated source code files that make a single program instead of "include" statements.
| |
| − | ###* Level D, same as level C, and also have a special SCons "Optiflow" rule type, in which programs to be run are actually recompiled with parameters hardwired into them, so that optimizations such as dead branch elimination and loop unrolling are feasible. Use SCons to perform compilations in parallel. A fully-optimized compilation of most of m8r for a single flow may seem like overkill, but is necessary in the case of huge datasets that take weeks and months to process.
| |
| − | ###* Level E, same as level D, but compile with -O3 and also, where available, use special alternative versions of the codes optimized by hand and with less error checking.
| |
| − | ## Graphics
| |
| − | ### Bezier curves in vplot – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### xtpen antialiasing – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### bargraph – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### graph3 completion – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ## Geophysical/numerical tools
| |
| − | ### Harlan's CG – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### conjgrad extensions – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### kirmod in layered media – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| − | ### minidds – see [http://sourceforge.net/tracker/?group_id=162909&atid=825648 Feature request tracker]
| |
| | | | |
| − | [[Image:M8rmap.png]]
| + | There were other achievements as well. Bugs in vplot were fixed, documentation was created, and the source code tree was partly redesigned. |
The workshop took place over the Memorial Day weekend, from Fri May 23 to Mon May 26, at the Geophysics Department of Colorado School of Mines, in Golden, Colorado.
Participants
From left to right, in the picture:
- Paul Sava (CSM)
- Joe Dellinger (BP)
- Sean Ross-Ross (UBC)
- Jim McClean (PGS)
- Sam Brown (U of Utah)
- Ioan Vlad (StatoilHydro)
- Yang Liu (UT Austin)
- Michael Tobis (UT Austin)
- Sjoerd de Ridder (Stanford)
- Paul Murray (UT Austin)
- Jules Browaeys (UT Austin)
Not in the picture: Peter Traynin (ExxonMobil)
Summary
The intended "deliverables" of this meeting were:
- A stronger geophysical open-source community
- A short-term Madagascar road map, created by discussing, extending, and prioritizing the feature request list
- A real move towards version 1.0
The community was indeed strengthened by participants getting to know each other.
The second goal was not only fulfilled, but far exceeded. Not one road map was created, but as many road maps as participants.
The real move to version 1.0 was done by Joe, who completed vplotdiff, allowing automatic testing. Joe also played a crucial role in clarifying and solving the IP issues created by Madagascar containing some code from 1994 versions of SEPlib and vplot. In the correspondence that followed, the copyright holders and Stanford University's Office of Technology Licensing have agreed to release the code under GPL (for the portions of SEPlib used) and LGPL (for vplot).
There were other achievements as well. Bugs in vplot were fixed, documentation was created, and the source code tree was partly redesigned.
