RIA tools Workflows

The FACE-IT RIA workflow was developed to streamline the multi-model crop model simulations performed for AgMIP Regional Integrated Assessments. It includes applications for translating data to crop model formats (QuadUI), running crop models (DSSAT 4.5 Plus and APSIM 7.5 Plus), and producing standardized outputs and plots for analysis and view (Stanardplot_ria, Plot Climate Scenarios Anomaly, Plot AcmoOutput Relationships).



Apps


QuadUI

The QuadUI app is used for translating the user's uploaded data into AgMIP Crop Experiment (ACE) format using JSON (JavaScript Object Notation) data structures.


NOTE: for AgMIP users who are familiar with the QuadUI desktop utility, this app does not perform the translation from ACE to crop model formats. Those translations are done as part of the crop model apps, e.g., DSSAT 4.5 Plus and APSIM 7.5 Plus.

    Input Data Types: Depends on user choice of input_type selection, the inputs might have different data types as follow


 Input_type csv_zip (default) aceb
 Survey Data zip archive of csv tables of survey data *.ACEB (AgMIP Crop Experiment Binary data)
 Supplemental  Data zip archive which could contains supplementary data, include cultivar, weather, soil and etc. The internal format could be Survey data CSV files; DSSAT files; APSIM cultivar XML files; *AgMIP weather files. *.zip (Same to the csv zip mode)
 DOMESeparate Field Overlay and Seasonal Strategy files in zip_csv format Single file in *.DOME format
 Linkage *.ALNK *.ALNK 

    Processes: QuadUI processes the survey, management, weather and soil, and DOME data into ACE format for input to multiple crop model apps.



    Output Data Types:

  •     Json (JSON)
  •     Zip archive of crop cultivar information (zip)
    • The zip archive of cultivar has a fixed internal folder structure, The rule is that, for any model, there will be a folder called [model name]_specific in the archive root (e.g., dssat_specific) and all the model required files (e.g., cultivar files) will be located under that folder. The folders are allowed to be missing if there is no files for that model.

    Options

  • Input_type: there are two options for selecting data input type: csv_zip and aceb:
    • Option csv_zip indicates the app accepts zipped csv file format as inputs for Survey data (zip file), Field Overlay DOME (zip file), Seasonal Strategy DOME (zip file), and Linkage between field and DOME (csv) (a).   
    • Option aceb denotes users need to input Survey aceb (aceb file), DOME file (DOME) and Linkage between field and DOME (csv) (b).
  • Supplemental Data: 
    • When users have supplemental data files, such as weather or calibrated cultivar files, click the Add new Supplement button (c) at right panel. This will enable another entry in QuadUI (option csv_zip) (a) allowing users input such data. Users can also use model defaulted cultivars or customized cultivars in panel c.
  • Cultivar_source:
    • Option Customized: application will scan the input data and pick up the cultivar files from the model-specific folder structure (same structure as mentioned above under Output Data Types) and generate a cultivar zip archive as output.
    • Option System default: application will ignore the cultivar files in the input data and no zip archive will be generated. It is important to notice that, once you choose to use the system default option for cultivar source, you should not connect the cultivar output to any other Apps in your workflow, otherwise it will cause system error and crash.


    Links:



DSSAT 4.5 plus

The popular DSSAT family of crop models, can be used to simulate the impact of environment and management on crops. The DSSAT 4.5 app uses version 4.5.1.23 of the DSSAT Cropping Systems Model. The DSSAT 4.5 plus include the functionality of the DSSAT model plus data translation from the AgMIP harmonized ACE (JSON) format to DSSAT input formats, and data translation from DSSAT output formats to the harmonized AgMIP Crop Model Ouptut (ACMO) format.

    Input Data Types:

  • Harmonized ACE data format (JSON)
  • Cultivar File Package
    • Created when customized cultivar source is chosen (default)
    • Hidden when system default for cultivar source is chosen
    • User could provide manually created cultivar zip archive by following the internal folder structure rule mentioned above.

    Processes: users can run DSSAT either standalone or incorporated into the workflow


    Output Data Types: Model Input and output files are just for reference purpose, not need to link to other Apps in a typical RIA workflow.

  • AcmoOutput csv
  • Zip archive of DSSAT input files
  • Zip archive of DSSAT output files

    Options:

  • Cultivar_source: similar to the one existed in the QuadUI App. Default is customized.

    Links:  For additional information on DSSAT see dssat.net.


APSIM 7.5 Plus

APSIM (Agricultural Production Systems Simulator) is a farming systems model that simulates the effects of environmental variables and management decisions on crop yield, profits and ecological outcomes. The APSIM 7.5 Plus includes the functionalities of APSIM plus data translation from the AgMIP harmonized ACE (JSON) format to APSIM input formats, and data translation from APSIM output formats to the harmonized AgMIP Crop Model Ouptut (ACMO) format

    Input Data Types:

  • Harmonized ACE data format (JSON)
  • Cultivar File Package
    • Created when customized cultivar source is chosen (default)
    • Hidden when system default for cultivar source is chosen
    • User could provide manually created cultivar zip archive by following the internal folder structure rule mentioned above.

    Processes: users can run APSIM either standalone or incorporated into the workflow


    Output Data Types: Model Input and output files are just for reference purpose, not need to link to other Apps in a typical RIA workflow.

  • AcmoOutput csv
  • Zip archive of APSIM input files
  • Zip archive of APSIM output files.

    Options:

  • Cultivar_source: similar to the one described in the QuadUI App. Default is customized.

    Links: For more information about APSIM see www.apsim.info.


Standardplots_ria

The Standardplots_ria app contains R scripts that generate box and CDF plots for single or multiple Acmo output files. Users can interactively define the type, name, variables, and colors of the plot.

    Input Data Types: Acmo out files with extension of .csv

    Processes: users can use the app standalone or incorporate it into the workflow. When run it separately, click Analyze Data (left top) -> Standardplots_ria (in the ria section at left panel) -> select input data, name the output image, and define other plots characteristics (right panel). When incorporate into the workflow, once the app in the central workflow window, click the app and make interactive input at the right panel.

    Output Data Types: image file with extension of png, as well as an acmo output file (.csv) that combined all acmo outputs genernated during the simulation and selected as the inputs of standardplots_ria

    Options: users need to define type (Boxplot or CDF), title, legend name, and colors of the plots.



Plot Climate Scenarios

Plot Climate Scenarios is a standalone R script to plot climate anomalies for single or multiple climate file inputs.

    Input Data Types: Currently the app accepts compressed file (.aceb and .zip) as input, in which it contains one or multiple weather csv files. Users can indentify how many input files they want to use to generate the climate anomaly. Inputs must include at least one future climate and one baseline climate.

    Processes: Click Analyze Data (left top) -> Plot Climate Scenarios -> Choose climate variable that want to plot -> Select plot type (the relative or absolute value) -> Choose input date type (csv.aceb or cvs.zip) -> Select future climate file and historical or baseline climate files -> Execute

    Output Data Types: image file with .png extension

    Options: When using the app, users must identify one input compress file including the future climate and one associated compress file including the observed or historical climate.


Plot AcmoOutput Relationship

Plot AcmoOutput Relationship is a standalone R script to explore the correlation between different Acmo output variables.

    Input Data Types: Single acmo file generated from Acmoplot, with csv extension, or combining acmo files from multiple crop model simulation.

    Processes: Click Analyze Data (left top) -> Plot AcmoOutput Relatinship-> Choose X and Y variables -> Choose grouping method (the plot will divided into separate plot based on the group variable) -> Execute

    Output Data Types: image file with .png extension

    Options: Users can choose different group methods to aggregate the output.



Datatypes

RIA uses following datatypes: zip, gzip, aceb, dome, alnk, and csv, which defined in datatype config files.

  • zip – standard zip archive. .ZIP is an archive file format that supports lossless data compression. A .ZIP file may contain one or more files or folders that have been compressed.
  • aceb – The aceb format is a gzipped JSON object containing AgMIP Crop Experiment (ACE) data format. It was designed to provide a means of storing the highly variable types of data (for example, field experiment or survey data) associated with AgMIP crop modeling exercises in flexbile and efficient schema. For the FACE-IT RIA workflow, aceb files are used for the “Survey Data”.
  • dome – Data Overlay for Multi-model Export (DOME) files are used to provide data which are needed for crop model simulations, but were not recorded in the survey data. A DOME is referred to as a “field overlay” or "seasonal strategy" file, depending on the use of the supplemental data.
  • alnk –  A csv format file containing the link information which associates each entry in the survey data with the appropriate field overlay and seasonal strategy DOMEs.
  • csv – Comma-separated-values file stores tabular data (numbers and text) in plain text form, which can be generated from Microsoft Excel or Access. ACMO files are also in csv format.



The Workflow

1.     Plan your work flow and data that will be used.  In a case of our demonstration project, the Nioro Senegal Regional Integrated Assessment (, there are five sets of simulations, including historical, current, future, current plus adaptation, and future plus adaptation). The future invovles six climate scenarios (observed, IEFA, IIFA, IKFA, IOFA, and IRFA). Therefore, based on the data and simulation objective, it’s better to orginze a simulation plan like Table 1

Table 1. List of files used in demo workflow for Nioro, Senegal maize analysis.

File names

Historical

Current

Future

Current + Adaptation

Future + Adaptation

Survey_data-Nioro-MAZ.zip          

X

X

X

Field_Overlay-Nioro-MAZ.zip        

X

X

X

Weather-Nioro-0XFX.zip

X

X

X

Linkage-Nioro-MAZ-historical.csv   

X

Seasonal_strategy-Nioro-MAZ-0XFX.zip

X

Linkage-Nioro-MAZ-0XFX.csv

X

Weather -Nioro-IxFA.zip

X

X

Seasonal_strategy-Nioro-MAZ-IxFA.zip

X

Linkage-Nioro-MAZ-IxFA.csv

X

Survey_data-Nioro-MAZ-Ax.zip       

X

X

Field_Overlay-Nioro-MAZ-Ax.zip

X

X

Seasonal_strategy-Nioro-MAZ-0XFX-Ax.zip

X

Linkage-Nioro-MAZ 0XFX-Ax.csv

X

Seasonal_strategy-Nioro-MAZ-IxFA-Ax.zip

X

Linkage-Nioro-MAZ-IxFA-Ax.csv

X

Notes:

For current climate conditions, CLIM_ID is represented by “0XFX”.

For future climate scenarios, CLIM_ID is represented generically as “IxFA”, where the “x” represents the GCM used for the analysis.

Red highlight indicates files that are repeated for multiple Climate scenarios or GCMs.

Adaptation scenarios are referred to as Ax, where “Ax” represents the ID of the adaptation package.

2.     Upload/extract data. Uploaded data will be shown in the history panel on the right side of the screen.

a.   Upload data from the AgMIP Data Interchange. From the Tools Menu, click Get Data -> AgMIP Browser, right click the point that you select, then download the requested data and save them in your history panel for later use.

b.   Upload users’ data. Make sure first the data is in the correct format; currently the RIA workflow only accepts csv and zip files, or aceb, dome, and alnk files as input. Click Get Data -> Upload File, and then upload the local data to the server.

3.     Create a workflow, save the workflow and run it.

a.      Usually the RIA workflow uses the following sequence:QuadUI->DSSAT4.5 Plus (or APSIM 7.5 Plus) -> Standardplots_ria or Plot AcmoOutput Relationship

4.     Before running the workflow, input data must be selected from the current history for the QuadUI by selecting data using the drop-down menus associated with each app. You may have to switch the current history to access the correct data.

5.     Run the workflow.

6.     Riew the results after the simulation being done.




Additional documentation of the RIA workflow can be found in these videos:


Comments