Reproducibility

Reproducibility refers to the ability to obtain consistent results when using the same data, methods, and analyses. Ensuring that our work is reproducible is essential—not only so that our findings can be verified by ourselves and by external researchers, but also to facilitate transparency and credibility. A reproducible workflow also makes it easier to repurpose parts of the code for other projects, saving time and reducing the likelihood of introducing errors.

Here we discuss two ways to enhance the reproducibility of the analysis:

  1. organizing the code base in a logical sequence of scripts, and
  2. package management.

1 Project script structure

For reproducibility, it is important to break down the project workflow into a logical sequence of scripts and clearly document the order in which they should be executed. We recommend the following practices:

1.1 Split a code base into multiple scripts

Breaking a project into a series of scripts makes it easier to test, maintain, and reuse code compared to working in a single monolithic file. While there is no strict rule for how to divide code, the following are good practices:

  • Focus on one substantial responsibility per script.
    Each script should perform a clearly defined task, such as pre-processing a large dataset, creating geographical overlays, merging supplementary datasets, generating descriptive statistics, or running a specific analysis.

  • Separate time-consuming steps.
    If a procedure is computationally expensive, place it in its own script and save the output as an intermediate dataset for later steps. This avoids unnecessarily re-running lengthy processes, such as complex geospatial calculations or aggregating large datasets.

  • Isolate key intermediate outputs.
    If a block of code produces an output that will be used in multiple subsequent steps, place it in a dedicated script and save the output for reuse.

  • Extract reusable code blocks.
    If a section of code—such as a function—is repeated across multiple scripts, centralize it (along with the required libraries) in a single location. The code block or function can then be sourced or imported from other scripts. For example:

    • In R: source("script_name.R")
    • In Python: import module_name
    • In Stata: do filename.do

1.2 Number scripts and script folders

Script names can begin with a two-digit number (e.g., 00_, 01_, 02_) to encode execution order. When a folder is sorted by name, the scripts appear in the intended run sequence. For steps that can run in parallel, append letters (e.g., 01a_, 01b_, 01c_).

Apply the same scheme to folders (e.g., 00_setup, 01_processing, 02_analysis) so dependencies are clear—for example, cleaning steps run before analysis.

An example of this structure is provided below.

Example Script Folder

scripts/
|--00_tools/                          # Shared helpers, not part of pipeline
|   |--filters.R                      # Reusable filter functions
|   |--data_io.R                      # Reusable input/output wrapper functions
|
|--01_processing/                     # Data prep and integration
|   |--01_clean.R                     # Clean raw data
|   |--02_aggregate.R                 # Aggregate to analysis unit
|   |--03_merge_external.R            # Merge with supplementary datasets
|
|--02_analysis/                       # Analysis and outputs
|   |--01a_summary_stats.R            # Summary statistics
|   |--01b_map.R                      # Descriptive spatial visualizations
|   |--01c_time_series_figure.R       # Descriptive time series plots
|   |--02_main_regressions.R          # Main statistical analysis
|   |--03_robustness_checks.R         # Alternative specifications

1.3 Create a run_all script

A run_all script executes all other scripts in the project in the correct order. This script documents the full workflow and can be used to reproduce the project’s complete set of outputs, including intermediate data products and final results, starting from the raw data.

When using script numbering together with a run_all script, it is important to keep both up to date whenever new scripts are added or existing scripts are reorganized.

If the project uses global parameters referenced across multiple scripts, place them in a dedicated configuration script and load that script at the beginning of the run_all file.

Example run_all.R Script

# This script runs all the code in my project, from scratch

# Prepare analysis data
source("scripts/01_processing/01_clean.R")                    
source("scripts/01_processing/02_aggregate.R")       
source("scripts/01_processing/03_merge_external.R")

# Descriptive stats and figures
source("scripts/02_analysis/01a_summary_stats.R")                    
source("scripts/02_analysis/01b_map.R")       
source("scripts/02_analysis/01c_time_series_figure.R")

# Analysis
source("scripts/02_analysis/02_main_regressions.R")
source("scripts/02_analysis/03_robustness_checks.R")

2 Package management

An analysis using the same code and data may not run identically across different machines if the machines have different operating systems, application versions, or package versions installed. Therefore, package management is a critical component of reproducibility.

Virtual environments provide a way to isolate the software dependencies of a project, such as package versions and system libraries, so that they are consistent over time and independent of other projects on the same machine. Maintaining separate environments for different projects prevents conflicts between dependencies, allows older analyses to remain reproducible even as packages evolve, and supports collaborative work by ensuring that all contributors run the code under the same computational conditions.

Three common tools for managing computational environments are renv, conda and Docker.

  • renv (for R) creates project-specific environments that record and restore the exact package versions used in an analysis. This ensures that results can be reproduced even if packages have been updated globally.

  • conda (for Python, R, and other languages) manages project-specific environments by explicitly specifying package versions and system dependencies. conda helps ensure that analyses run consistently across machines and over time, particularly for workflows that rely on compiled libraries or mixed-language dependencies.

  • Docker goes further by packaging the entire computational environment (including the operating system and versions of R, Python, Julia, and all associated packages) into a self-contained container that can be run on any machine.

At this time, the working group has not fully tested the functionality of these tools. The resources listed below are provided for reference. We would love to hear from users who have implemented any of these tools in RFF’s computing environment. Please get in touch with us to share your experience.

renv

conda

Docker