Tools for Systematic Sensitivity Analysis

Programming Tools for Systematic
Sensitivity Analysis of GAMS Models

Thomas F. Rutherford*

May 2007

* This web page documents a computational tool which has been developed within the project "Indicators and Quantitative Tools for Improving the Impact Assessment Process for Sustainability" (I.Q. Tools), 6th Framework Programme of the European Commission, Contract SSP1-CT-2003-502078, Thematic Priority 8: Policy Oriented Research.

Installation

Here is a zip file containing the solution management and report generation programs, qs.gms and qr.gms. I recommend that you store original versions of these files in your GAMS libinclude directory. You can subsequently copy these files into individual model project directories for use. It is helpful to have separate versions for each model so that you can make modifications as needed.
Here is a small scale application of these tools to a static general equilibrium model.
Here is a sample electricity market application with an associated paper and talk.
Here is a sample CGE application addressing issues related to the competitive effects of climate policy. Here is an associated paper and a related talk.

Introduction

GAMS is a modeling language designed for use by specialized analysts in a wide range of disciplines. GAMS was invented by an Austrian physicist, Alex Meeraus, for use by economists at the World Bank during the 1970s. The fundamental strngth of GAMS lies in the ease with which mathematically-defined models can be formulated and solved. GAMS itself does not provide model solutions; these tasks are delegated to a large number of subsystems, specialized algorithms for particular classes of mathematical programs. GAMS provides an intermediate set of services by translating an analyst's description of a mathematical model into a format understood by an appropriate solver.

GAMS has been designed primarily to meet the needs of the modeller rather than the needs of the modeller's client. A client for a mathematical model typically has data, a conceptual framework and a set of policy-related questions. The client contracts the modeller to formulate and analyze a model (or several models) which address the policy questions in which he is interested. The deliverables for a modeling contract typically include an operational GAMS program and a written report summarizing the mathematical details of the model and the model results. Results are typically in the form of tables or figures which compare model outcomes for alternative policies and a common set of assumptions. It would also be common practice to deliver a sensitivity analysis of model results for a particular policy, under a range of alternative parameter assumptions. For simplicity, we can name these "policy" and "sensitivity" analyses.

A model-based analysis often involves iterations between the client and the modeller in which the specific set of policy instruments and underlying parameter assumptions are defined and evaluated.

Figure 1: Model-Based Policy Analysis

In Figure 1 the client suggests a set of policy inputs, t, which may be evaluated with the model. He also may suggest a set of assumptions for behavioral parameters, ξ, such as elasticities of substitution or long-run growth rates. The modeller takes these suggestions, revises her GAMS program accordingly, and then generates a set of tables and figures, T(t,ξ) and F(t,ξ).

It has been my experiece that clients are often disappointed with the amount of specialized knowledge required to assess a model. A written report provides only a glimpse into what has been learned in the total modelling exercise. If the modeller has good writing skills but poor programming skills, the client may very easily be mislead about the true nature of what can be learned on the basis of the calculations. On the other hand, if the modeller is a good programmer but a poor writer, the client may have a very informative model but at the same time have no idea of the policy implications of the results.

QS.GMS -- A Program for Enumerating Scenario Solutions

Invocation
QS.GMS is a GAMS program which can be used to solve a model for a sequence of alternative input values. Here is a typical invocation of the program from a DOS batch file:
```
GAMS QS --p1="epsilon / 5, 10, 20/" --p2="eta /5,10,20/" --p3="etaL /5,10,20/" 

del output\*.lst
del output\*.gdx

call output\RUNGAMS.BAT
```
QS.GMS produces the batch file RUNGAMS.BAT which executes GAMS solutions for all combintations of three input parameters, representing a total of 27 different scenarios. This file appears as follows:
```
@echo off

if not exist output\nul mkdir output\

title Solving case 1 out of 27
echo set q_inputs/"5"."5"."5"/;  >output\1.gms
echo $set epsilon "5" >>output\1.gms
echo $set eta "5" >>output\1.gms
echo $set etaL "5" >>output\1.gms
echo $include model >>output\1.gms
gams output\1.gms gdx=output\1 o=output\1.lst

...
```
The ECHO command is used to generate scenario files in which input values are assigned and a set is declared which "labels" the resulting GDX file with the input assumptions. For example, program 1.gms appears as:
```
set q_inputs/"5"."5"."5"/;  
$set epsilon "5" 
$set eta "5" 
$set etaL "5" 
$include model 
```
Instrument a GAMS Model with Environment Variable Inputs
In order to use QS.GMS, the modeller first produces a GAMS application in which inputs (behavioral parameters, ξ, and policy parameters, t shown in Figure 1.) enter as environment variables. Default values for these inputs can be declared at the top of the GAMS program using statements such as:
```
$if not set epsilon $set epsilon 10
$if not set eta     $set eta  10
$if not set etaL    $set etaL 10
```
Through the use of environment variables, inputs may be either numeric or text. Furthermore, a model may have many such inputs, corresponding to the many dimensions in which sensitivity anlysis may be conducted. In any given study of a model, only a selected set of inputs may be varied.
Environment variables are treated a string substitutions in GAMS programs. Hence, if the inputs shown in this example may be referenced in the GAMS program as follows:
```
*	Environment variables can appear in declarations:

parameter	etaL	Elasticity of transformation for land /%etaL%/;

*	Environment variables can be used in assignments:

eta("tex") = %eta%;
epsilon("tex") = %epsilon%;
```
The Output Directory
A QS run produces a subdirectory with a collection of GMS, LST and GDX files corresponding to different sets of model scenarios. These results files can be merged to produce reports. Input assumptions associated with each run are stored in set Q_INPUTS which is written to the output GDX file. This set provides a convenient means of relabelling model output for report generation.

QR.GMS -- A Program for Producing Scenario Comparisons in Excel PivotTables

Invocation

QR.GMS is a GAMS program which can be used to merge results from a set of scenarios solved with QS.GMS. Here is a typical application of the program from a DOS batch file:


copy pivotreporttemplate.xls output\pivotreport.xls

gams qr --item=hev    --domain=""		o=output\hev.lst	xlsfile=output\pivotreport
gams qr --item=quant  --domain="s,item"		o=output\quant.lst	xlsfile=output\pivotreport
gams qr --item=qpct   --domain="s,item"		o=output\qpct.lst	xlsfile=output\pivotreport
gams qr --item=gprice --domain="s,item"		o=output\gprice.lst	xlsfile=output\pivotreport
gams qr --item=fprice --domain="factor,item"	o=output\fprice.lst	xlsfile=output\pivotreport

Program output
One invocation of QR.GMS is required for each item to be written to an Excel file. In small scale models, one Excel file could contain multiple report worksheets. In larger models, it may advisable to use one workbook for each pivot report, as these may consume large amounts of data. The invocation of QR.GMS includes the name of the parameter to be written and the identifiers to be used as headings for each of the parameter dimensions. For example, the fprice parameter generated by the last line in the batch file shown above is identified here as a two-dimensional parameter with domain factor,item. The domain domensions are concatenated with the input identifiers when the parameter is written to the Excel file:
PivotTables and PivotCharts
PivotTables and PivotCharts are in my view the most powerful and appealing features in Excel. PivotTables permit rapid analysis of large amounts of data using tables and charts which are formatted independently from the original data layout. PivotTables and Charts are interactive, thereby allowing the user to explore large amounts of data.
When you use QR.GMS to transfer data to Excel, it is formatted as a flat list with the expectation that these data will be used in Pivot Table and Pivot Chart reports.
Here is the PivotTable which is linked to the fprice parameter:

A Pivot Chart provides an interactive framework for formatting graphs which compare results across different scenarios. Here is the PivotChart which is produced by parameter hev:

A concise introduction is provided here. (More links are provided here.)
Many books have been written to introduce these Excel programming tools. One example is the O'Reilly book Excel 2003: The Missing Manual by Matthew MacDonald. Chapter 20 describes PivotTables and PivotCharts. (Numerous features of PivotTables are only accessible via the programmatic interface (VBA), hence books such as Writing Excel Macros by Steven Roman (O'Reilly) may also be useful.)

GAMS Source Code for QS.GMS

$title	QTool Solution Program  (QS.GMS)

$if not set model $set model model
$if not set outdir $set outdir output\

$call 'if not exist %outdir%\nul mkdir %outdir%'

$if not set p1 $abort "Input parameter P1 must be defined as an environment variable." 

SETS
$if set p1	p1	%p1%
$if set p2	p2	%p2%
$if set p3	p3	%p3%
$if set p4	p4	%p4%
$if set p5	p5	%p5%
$if set p6	p6	%p6%
;


file kbat /%outdir%\rungams.bat/; kbat.lw=0; kbat.nd=0; kbat.nw=0; kbat.tw=0; put kbat,'@echo off'/'if not exist %outdir%\nul mkdir %outdir%'/;
file kinc /%outdir%\qr.inc/;
scalar id /0/, nrun;

*	Branch to the appropriate loop depending on the number of items:

$if not defined p2 $goto 1
$if not defined p3 $goto 2
$if not defined p4 $goto 3
$if not defined p5 $goto 4
$if not defined p6 $goto 5
$goto 6

$label 1
$if defined p3	$abort 'Did not expect p3 to be defined if p2 is not defined.'
$if defined p4	$abort 'Did not expect p4 to be defined if p2 is not defined.'
$if defined p5	$abort 'Did not expect p5 to be defined if p2 is not defined.'
$if defined p6	$abort 'Did not expect p6 to be defined if p2 is not defined.'

nrun = card(p1);
loop(p1,
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts/;
$exit

$label 2
$if defined p4	$abort 'Did not expect p4 to be defined if p3 is not defined.'
$if defined p5	$abort 'Did not expect p5 to be defined if p3 is not defined.'
$if defined p6	$abort 'Did not expect p6 to be defined if p3 is not defined.'

nrun = card(p1)*card(p2);
loop((p1,p2),
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"."',p2.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p2.ts,' "',p2.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts,',',p2.ts/;
$exit

$label 3
$if defined p5	$abort 'Did not expect p5 to be defined if p4 is not defined.'
$if defined p6	$abort 'Did not expect p6 to be defined if p4 is not defined.'

nrun = card(p1)*card(p2)*card(p3);
loop((p1,p2,p3),
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"."',p2.tl,'"."',p3.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p2.ts,' "',p2.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p3.ts,' "',p3.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts,',',p2.ts,',',p3.ts/;
$exit

$label 4
$if defined p6	$abort 'Did not expect p6 to be defined if p5 is not defined.'

nrun = card(p1)*card(p2)*card(p3)*card(p4);
loop((p1,p2,p3,p4),
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"."',p2.tl,'"."',p3.tl,'"."',p4.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p2.ts,' "',p2.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p3.ts,' "',p3.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p4.ts,' "',p4.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts,',',p2.ts,',',p3.ts,',',p4.ts/;
$exit

$label 5
nrun = card(p1)*card(p2)*card(p3)*card(p4)*card(p5);
loop((p1,p2,p3,p4,p5),
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"."',p2.tl,'"."',p3.tl,'"."',p4.tl,'"."',p5.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p2.ts,' "',p2.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p3.ts,' "',p3.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p4.ts,' "',p4.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p5.ts,' "',p5.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts,',',p2.ts,',',p3.ts,',',p4.ts,',',p5.ts/;
$exit

$label 6
nrun = card(p1)*card(p2)*card(p3)*card(p4)*card(p5)*card(p6);
loop((p1,p2,p3,p4,p5,p6),
	id = id + 1;
	put 'title Solving case ',id,' out of ',nrun/;
	put 'echo set q_inputs/"',p1.tl,'"."',p2.tl,'"."',p3.tl,'"."',p4.tl,'"."',p5.tl,'"."',p6.tl,'"/;  >%outdir%\',id,'.gms'/;
	put 'echo $set ',p1.ts,' "',p1.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p2.ts,' "',p2.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p3.ts,' "',p3.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p4.ts,' "',p4.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p5.ts,' "',p5.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $set ',p6.ts,' "',p6.tl,'" >>%outdir%\',id,'.gms'/;
	put 'echo $include %model% >>%outdir%\',id,'.gms'/;
	put 'gams %outdir%\',id,'.gms gdx=%outdir%\',id,' o=%outdir%\',id,'.lst'/;
);
putclose;
putclose kinc, '$setglobal inputs ',p1.ts,',',p2.ts,',',p3.ts,',',p4.ts,',',p5.ts,',',p6.ts/;
$exit

GAMS Source Code for QR.GMS Code

$title	QTool Report Program (QR.GMS)

$if not set item   $abort 'Expected to have an item defined.'
$if not set domain $abort 'Expected to have the domain defined.'

$if not set outdir $set outdir output
$if not set xlsfile $set xlsfile %outdir%\pivotreport


$call 'gdxmerge %outdir%\*.gdx id=%item%,q_inputs';

set	q_r(*)	Runs;

$gdxin merged.gdx
$load q_r=Merged_set_1

$if exist "%outdir%\qr.inc" $include "%outdir%\qr.inc"
alias (*,%inputs%);

$if "%domain%"=="" $goto nodomain

$label domain
alias (*,%domain%);
set	header /%inputs%,%domain%,value/;
parameter	pt_item(*,%domain%)		Item to be pivoted;

set		inputs(*,%inputs%)	Correspondence of runs to input assignments;

$load pt_item=%item% inputs=q_inputs
display pt_item;

parameter	p_item;
loop(q_r,	loop(inputs(q_r,%inputs%),
		p_item(%inputs%,%domain%) = pt_item(q_r,%domain%);
));
execute_unload '%outdir%\pivotdata.gdx',p_item=%item%,header;
$goto gdxxrw

$label nodomain
set	header /%inputs%,value/;
parameter	pt_items(*)		Item to be pivoted;
set		inputs(*,%inputs%)	Correspondence of runs to input assignments;

$load pt_items=%item% inputs=q_inputs
display pt_items;

parameter	p_items;
loop(q_r,	loop(inputs(q_r,%inputs%),
		p_items(%inputs%) = pt_items(q_r);
));
execute_unload '%outdir%\pivotdata.gdx',p_items=%item%,header;

$label gdxxrw
$onecho	>%outdir%\gdxxrw.rsp
set=header rng="%item%!a1" values=nodata cdim=1 rdim=0
par=%item% rng="%item%!a2" cdim=0 
$offecho
execute 'gdxxrw i="%outdir%\pivotdata.gdx" o="%xlsfile%.xls" @%outdir%\gdxxrw.rsp';
execute 'del %outdir%\pivotdata.gdx';

Programming Tools for Systematic Sensitivity Analysis of GAMS Models

Thomas F. Rutherford*

May 2007

Installation

Introduction

QS.GMS -- A Program for Enumerating Scenario Solutions

Invocation

Instrument a GAMS Model with Environment Variable Inputs

The Output Directory

QR.GMS -- A Program for Producing Scenario Comparisons in Excel PivotTables

Invocation

Program output

PivotTables and PivotCharts

GAMS Source Code for QS.GMS

GAMS Source Code for QR.GMS Code

Programming Tools for Systematic
Sensitivity Analysis of GAMS Models