Reproducible Research, a manifest-like paper by a number of authors from different scientific disciplines, is published by Computing in Science and Engineering.

Progress in computational science is often hampered by researchers’ inability to independently reproduce or verify published results. Attendees at a roundtable at Yale Law School formulated a set of steps that scientists, funding agencies, and journals might take to improve the situation. We describe those steps here, along with a proposal for best practices using currently available options and some long-term goals for the development of new tools and standards.

Most of the time, when we talk about reproducibility in computational sciences, we assume that of numerical results. We expect computational experiments to produce identical results in different execution environments for same input data.
But it is not the case all the time – quite often, the goal of a research endeavor is to design a faster algorithm. Then, the result of the experiment is performance information and demonstration of a speedup over existing algorithms for finding a solution for the same problem. Speedup, just like numerical results, should be reproducible in different execution environments.
Sid-Ahmed-Ali Touati, Julien Worms, and Sebastien Briais of INRIA published an excellent work on methodology of reproducible speedup tests.
A part of the introduction from their paper is worth quoting on this blog:

Known hints for making a research result non reproducible
Hard natural sciences such as physics, chemistry and biology impose strict experimental methodologies and rigorous statistical measures in order to guarantee the reproducibility of the results with a measured confidence (probability of error/success). The reproducibility of the experimental results in our community of program optimisation is a weak point. Given a research article, it is in practice impossible or too difficult to reproduce the published performance. If the results are not reproducible, the benefit of publishing becomes limited. We note below some hints that make a research article non-reproducible:

• Non using of precise scientific languages such as mathematics. Ideally, mathematics must always be preferred to describe ideas, if possible, with an accessible difficulty.
• Non available software, non released software, non communicated precise data.
• Not providing formal algorithms or protocols make impossible to reproduce exactly the ideas.
• Hide many experimental details.
• Usage of deprecated machines, deprecated OS, exotic environment, etc.
• Doing wrong statistics with the collected data.

Part of the non-reproducibility (and not all) of the published experiments is explained by the fact that the observed speedups are sometimes rare events. It means that they are far from what we could observe if we redo the experiments multiple times. Even if we take an ideal situation where we use exactly the original experimental machines and software, it is sometimes difficult to reproduce exactly the same performance numbers again and again, experience after experience. Since some published performances numbers represent exceptional events, we believe that if a computer scientist succeeds in reproducing the performance numbers of his colleagues (with a reasonable error ratio), it would be equivalent to what rigorous probabilists and statisticians call a surprise. We argue that it is better to have a lower speedup that can be reproduced in practice, than a rare speedup that can be remarked by accident.

Read the full document for a thorough explanation of how to avoid creating non-reproducible and erroneous speedup tests by using proper scientific techniques.

After returning back from the NSF Workshop Archiving Experiments to Raise Scientific Standards, here are some thoughts on reproducible research. Thanks to Dan Gezelter, Dennis Shasha, and others for inspiring discussions.

First of all, it is important to point out that reproducibility is not the goal in itself. There are many situations in which strict computational reproducibility is not achievable. The goal is an exposure of the scientific communication to a skeptic inquiry. A mathematical proof is an example of a scientific communication, which is constructed as a dialogue with a skeptic: someone who might say “What if your conclusions are not true?” Step by step, a mathematical proof is designed to convince the skeptic that the conclusion (a theorem) has to be true. As for computational results, even the simplest skeptic inquiry “What if there is a bug in your code?” cannot be answered unless the software code and every computational step that led to the published result are open for inspection.

If you attend a mathematical conference, you can notice that mathematicians do not usually go through every step in the proof to present a theorem, it is enough to sketch the main idea of the proof. However, the audience understands that the detailed proof should be available in the published work, otherwise the theorem cannot be accepted. Similarly, in a presentation of a computational work, one can simply show results of a computational experiment. However, such results cannot be accepted as scientific unless the full computation is disclosed for a skeptic inquiry. As stated by Dave Donoho (who paraphrased Jon Claerbout),

An article about computational science in a scientific publication is not the scholarship itself, it is merely advertising of the scholarship. The actual scholarship is the complete software development environment and the complete set of instructions which generated the figures.

If you don’t want to disclose the details of your computation, then the work that you do is not science. As for reproducibility, there seems to be different degrees of it:

1. Replicability: the ability to reproduce the computation as published
2. Reusability: the ability to apply a similar algorithm to a different problem or different input data
3. Reconfigurability: the ability to obtain a similar result when the parameters of the experiment are perturbed deliberately

Some algorithms are perfectly replicable but of limited use, because they are too sensitive to the choice of parameters to be reusable or reconfigurable. Nevertheless, such algorithms deserve a place in the scientific body of knowledge, because they may lead to a discovery or invention of more robust algorithms.

Those who read Italian may enjoy the philosophical article on open software and reproducible research by Alessandro Frigeri and Gisella Speranza: “Eppur si muove” Software libero e ricerca riproducibile Eppur si muove (and yet it moves) are the words attributed to Galileo Galilei, the father of modern science.

A article published in Guardian argues for software openness in science.

.…if you are publishing research articles that use computer programs, if you want to claim that you are engaging in science, the programs are in your possession and you will not release them then I would not regard you as a scientist; I would also regard any papers based on the software as null and void.

For an emprical survey on why scientists do and do not share their software, see The Scientific Method in Practice: Reproducibility in the Computational Sciences by Victoria Stodden.

The Madagascar project has been registered with Ohloh in order to get free code metrics. Visit the Madagascar code analysis page on Ohloh to find out many things, such as: what types of software licenses besides GPL 2+ are in Madagascar; how many non-blank, non-comment lines are written in each programming language; and to see an evolution of the total number of lines over time! Apparently we have more than 152 000 lines of C. Awesome.
Especially nice and encouraging is the graph with the steady growth of number of source code lines over time. According to Ohloh: “It’s generally a good sign to see sustained, constant activity over a long period of time. This means that people are continually updating it (fixing bugs and/or improving features), and that the project has staying power.” The official Ohloh assessment about Madagascar is: “Mature, well-established codebase; Large, active development team”.
Ohloh also lets enthusiastic users vote for the package, so that popular packages become even more visible! Vote today so that we get even more users and developers, even more bug reports, documentation and contributions, and overall a more useful package!

Madagascar is mentioned in Reproducible Research in Signal Processing – What, why, and how, a new paper by Patrick Vandewalle, Jelena Kovacevic, and Martin Vetterli, which appears in the latest issue of the IEEE Signal Processing Magazine and makes a strong case for reproducible research.

Madagascar is all about users helping each other and advancing open technology for the benefit of everyone. Because of the network effect, all users benefit from an increase in their ranks, through more bug reports, more user-contributed documentation, more platforms configured and more developers emerging. I see two ways of increasing the community that can be exploited through concerted action. I list them below and welcome opinions and suggestions in the “Comments” section.
The first avenue for concerted action is targeted more towards small users. It consists in conducting a “Google bombing”. Let me explain what that is.
There are basically three ways in which users get to Madagascar’s website: (1) Word of mouth; (2) following a link from a website, and (3) typing a search phrase in a search engine, and finding Madagascar highly ranked in the search results. It is this last one (3) which can be improved. The number of phrases that a user would search for a package like Madagascar is relatively small: “open source seismic software”, “free seismic software”, maybe with some attribute like “processing” before “seismic”, or “package” instead of “software”. Madagascar users are a pretty web-literate bunch, and many of us have websites. If we decide on a search phrase that users are more likely to search on, we can improve Madagascar’s ranking for that search by using that phrase in links from our websites to Madagascar! Technical details on Wikipedia. Suggestions of which search phrases users are more likely to search are welcome.
Google’s bots are smart! It is important that the link be made from a page that contains actual content (especially content related to Madagascar, i.e. geophysical/reproducibility/software dMadagascar Enterpriseevelopment). It may be tempting to make the link invisible and place it on a lot of pages unrelated to that topic on one’s website, but this can cause Google to downgrade the linker itself for spamming. The domain name of the linker matters. Websites of well-regarded institutions with educated, selected members and no commercial interests can propel the ranking to the top of Google. A single link from a page with a reputable “.edu” subdomain can propel a search result to first place. It is a good idea to put the link on a page that itself is being a target of honest links from other pages, especially popular, high-content-quality pages on other domains. If possible, the linker should be as close to the root of his domain name as possible: A page buried 7 levels deep in a website is not even indexed. Also, the Madagascar has 3 URL aliases, only one should be chosen.
The second currently unexploited avenue for increasing the number of users is geared towards the industry. One of the classic reasons why companies are reluctant to embrace open-source solutions is that many times the open-source projects do not provide an “interface” for the checks that companies use to verify whether suppliers / business partners are reliable. The “interface” components relevant to Madagascar are:
(1) From the lawyers: “Who are these guys?” A supplier or a business partner must be incorporated in some legal form that states the degree of liability of the people involved, who they are, whether they can represent the project, etc. That is why for example the Apache Software Foundation is a 501(c)(3) nonprofit corporation. A good idea is to get the creator of the Madagascar logo to trademark and license it for free to basically anybody who is not seeking to harm the project in any way (suing, spamming, etc). I am not sure if an already-existing name like Madagascar can be trademarked. Those entirely made-up can certainly be trademarked – as Linux is. I should have thought about that when there was the discussion about choosing a name 🙁
(2) From IT: “Will their software break our systems?” Corporate IT has to deal with hundreds of installation requests daily, and cannot manually audit all codes for security and stability. That is why the Linux Standard Base, and all enterprise-class OS vendors, provide certifications for applications.
(3) From the resource manager: “I need some verification that an employee has had proper Madagascar training, so I can mark him in the SAP database as being available for projects needing Madagascar-trained people”. The solution is an online paid certification system for users and developers. It should be cheap enough that people can become a “Certified Madagascar Operator/Developer” on their own, but expensive enough that it is taken seriously by companies paying it for their employees. The study material is: everything in the wiki and in the Madagascar distribution, including the books. The study group is the users mailing list and the forum. The Operator exam consists of giving the applicant a temporary Subversion user account and an input data set and asking him to solve a given geophysical problem using a flow in a SConstruct and to produce a short reproducible paper about his solution.
The fees can be used towards the time of those preparing the exam (grading is easy: the paper has to be reproducible!) and for other Madagascar purposes, i.e. LSB Certification ( $1000 for one release on one architecture + $250 per additional architecture or additional release – see schedule). Having a fee of $250 ensures one certified user one gets one additional architecture-release certified 🙂 A reduced rate could be offered to students, so that they can put their Madagascar experience in their CV. For a Developer certificate, one would have to have: (1) passed the user exam; (2) added meaningful GPL-ed capabilities to the Madagascar core that show he understands the way its SCons system functions; (3) contributed a GPL-ed program of moderate complexity with a reproducible paper and user guide/implementation notes.
(4) From the project manager: “It’s nice you guys have a two-day school once a year, but employee X needs to start working as a user on project Y in 1 month, and anyway it’s a headache to get him a visa for the country the school is held in”. Well, that’s why I said “online course”.
(5) From the procurement manager who is contemplating paying some huge amount for the purchase of a commercial geophysical package with a framework for developing plug-ins, and is told of the existence of a free alternative: “Who provides support?” Answer: “Get the Developer Certificate to one of your employees, or hire somebody with such a certificate, and he will”. That’s long-term thinking; to start competing with the commercial systems, Madagascar still needs a few features like a 3-D data format, a parallelization system, and GUI tools for tasks that actually need it, like picking velocities.
So, what do you think?
Nick

OpenScience.org has a lot of useful information about scientific open-source projects. Vote for Madagascar.