GTAP Social Accounting Matrix (SAM) Extraction Programme

This programme sorts GTAP data into a global SAM structure.

 This version
        output is in the form of both a GDX file and a HAR file

        the default setting switches off the DISPLAY statements to reduce
        the size of the lst file - search for "displ" to change this setting

        does not adjust for the law of one price
        has the GTAP price system
        has GTAP style Regional and Private Households
        runs with the full GTAP database version 8 (2007)
             i.e., BaseData.har
                   Default.har
                   gsdvol.har
                   co2.har
                   TStrade.har
        There is no code to make adjustments for minor rounding 'errors' in
        the GTAP database - these may need to be addressed in some models
        that work to double precision.

        In addition to the SAM the GDX file contains all the data supplied
        with GTAP v8 (2007), e.g., elasticities, time series trade data,
        energy volumes, CO2 emissions, additional data on taxes, etc.

 All versions of this programme together with the documentation are
 supplied in the form of a USER MODEL LIBRARY. Unzip the archive into
 a directory and from GAMSIDE select
         FILE > Model Library > Open User Model Library
 and select the library index file (gtaplib.glb) and then select the
 version of the programme that matches the version of GTAP with which you
 are working.

 The GTAP version supported by programmes in the library are
           GTAP v5.3
           GTAP v5.4
           GTAP v6.0 pre 5
           GTAP v6.0
           GTAP v7.0 release candidate 4
           GTAP v7.0
           GTAP v7.1
           GTAP v8.0 2007 & 2004

 For GTAP versions 6 and 7rc4 the programme extracts nearly all the data
    supplied by GTAP.
 For GTAP version 7 & 7.1 the programme extracts ALL the data supplied by GTAP.
 For GTAP version 8 (2004 & 2007) the programme extracts ALL the data
    supplied by GTAP.

$offtext
*############## GENERAL INFORMATION #############################
$ontext

This programme was developed by Scott McDonald

 Scott McDonald
 Dept of Accounting, Finance & Economics
 Oxford Brookes University
 Wheatley Campus
 Oxford, OX33 1HX, UK
 email:
    smcdonald@brookes.ac.uk

Acknowledgements:
    Terrie Walmsley provided invaluable helpful advice about the GTAP model
        and GEMPACK.
    Helpful comments were provided by the referees for GTAP Tech Paper 22.
    Vitaly Kharitonov identified an error in the code.
    Karen Thierfelder for the GLOBE model that keeps this effort going.
    Mark Horridge and GTAP for including the SAM representation of the
        GTAP database in GTAPAGG.
    The GTAP database team whose efforts make it possible for the global
        CGE community to realise our models.

This version works with GTAP version 8.0 2007 & 2004.

This version last checked 10th April 2012

Documentation for the SAM representation of GTAP data is in

Scott McDonald and Karen Thierfelder,
'Deriving a Global Social Accounting Matrix from GTAP version 5 Data',
Global Trade Analysis Project Technical Paper 22.
Center for Global Trade Analysis: Purdue University. (www.gtap.org)

Copyright (c) 2012 Scott McDonald

    This program is free software; you can redistribute it and/or modify
    it under the terms of the GNU General Public License as published by
    the Free Software Foundation; either version 2 of the License, or
    (at your option) any later version.

    This program is distributed in the hope that it will be useful,
    but WITHOUT ANY WARRANTY; without even the implied warranty of
    MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
    GNU General Public License for more details.

    You should have received a copy of the GNU General Public License
    along with this program; if not, write to the Free Software
    Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA  02111-1307  USA

A copy of the GNU GPL, gpl.txt, is included with the programme files.

Do NOT remove this notice if you redistribute this programme

If you use this programme you should acknowledge the source and not claim
authorship. Altered versions should be clearly distinguished from the original.

$offtext
*############## USER GUIDE v7 ######################################
$ontext

SOFTWARE REQUIREMENTS

  1) This programme is known to work with version 23.7 of GAMS 250.
     It is believed to work with all versions of GAMS 250 since
     distribution 22.6 in Dec 2007. It has not been tested for earlier
     versions of GDX, i.e., before v7

  2) This programme does not require a full licensed version of GAMS.

  3) Note that GAMS has used several formats for GDX. In some circumstances
     the version of GDX can create problems for users, e.g., when a
     GDX file is created using one version of GAMS and sent to a user with
     a different version of GAMS.
     With GAMS 22.8 the default format for gdx files is V7 uncompressed.
     V7 formatted files were introduced with version 22.6 of GAMS;
     V6 formatted files were introduced with version 22.3 of GAMS.
        Users with older licenced versions of GAMS may choose to use a
        recent demo version of GAMS resolve such problems.

  4) Recent versions of GAMS include the GDX2HAR and HAR2GDX utilities.
     If your version of GAMS does not include GDX2HAR and HAR2GDX it is
     necessary to get copies of the GDX2HAR and HAR2GDX utilities developed
     by Horridge and Rutherford (revised Nov 2003).
            http://debreu.colorado.edu/gdxhar
                   OR
            http://www.monash.edu.au/policy/ftp/gpextra/gdxhar.zip
       There are three files that need to be copied to the GAMS directory
            gdx2har.exe
            har2gdx.exe
            gdxiomh.dll
     However since this programme runs with the demo version of GAMS it may
     be easier to get a demo copy of GAMS from the GAMS website.

  5) It is also useful to have a copy of GDXVIEWER. This is standard in
     recent versions of GAMS.
     NB: for GAMS versions 22.6 and 22.7 the cube facility in GDXViewer
         does not seem to work correctly.
     Users of versions of GAMS that do not include GDXViewer should note that
     GDXViewer was one of the WTOOLS utilities developed by
     Erwin Kalvelagen. (GDXVIEWER allows the resultant Global SAM to be
     viewed on screen in a simple user friendly environment). The WTOOLS
     utilities were available from
            http://www.gams.com/~erwin/interface/interface.html#1.31_WTOOLS

DATA REQUIREMENTS

  1) No GTAP data are provided. The user must have a licensed version
     of the GTAP database complete with the GTAPAgg (or FLEXAgg) programme.

  2) All the sets and set mapping required to use the GAMS programmes for
     specific versions of the GTAP database are provided from
     an Excel workbook.
         GTAP version 5.3               - data_gtp_53.xls
         GTAP version 5.4               - data_gtp_54.xls
         GTAP version 6.0 pre release 5 - data_gtp_60_p5.xls
         GTAP version 6.0               - data_gtp_60.xls
         GTAP version 7.0rc4            - data_gtp_70_rc4.xls
         GTAP version 7.0               - data_gtp_70.xls
         GTAP version 7.1               - data_gtp_71.xls
         GTAP version 8.0               - data_gtp_804.xls
         GTAP version 8.0               - data_gtp_807.xls

     An additional Excel workbook is also supplied that provides a simple
     semi-automated method for producing new set and mapping configurations.

     Step 11 in the User Instructions below provides brief guidance for
     working with other - more recent or older - versions of the GTAP
     database.

USER INSTRUCTIONS
   (These instructions assume the user is working in a Windows XP/Vista/7
    environment using GAMSIDE and Microsoft Excel 2003/2007)

  1) The programme files are made available in a WinZip archive - gtapsam.zip.

  2) The programme files are organised so that they can be used as a
     GAMS User Model Library (see the GAMS documentation for information
     on using User Model Libraries). These instructions assume that files
     in gtapsim.zip have been extracted to a directory - gsamlib.

  3) Create a directory for the programme. The path to this directory
     should follow standard/old DOS conventions - no directory with more
     than an 8 character name and no spaces in the names in the path.

     (This is probably no longer necessary but I am old fashioned)

     Create a project file (*.gpr) in this directory.

  4) To copy the programme files to the project directory.
     In GAMSIDE with the project file controlling paths do the following
     a) Locate the User Model Library
          File>Model Library>Open User Model Library and then select the
          file gtablib.glb in the directory gsamlib
     b) Double click on the NAME of the version of the programme that
        corresponds to the version of the GTAP database with which you are
        working. ALL the files needed to run the programme other than the
        GTAP database files will be copied to the directory containing the
        project file.
        NB: A brief description of the programme is provided in the window
            at the bottom of the User Model Library window.

  5) Read and study the GAMS programme.

  6) Create the database for the programme
     a) Use GTAPAgg to create an aggregation with 1:1 mappings for
        sectors, regions and factors.
     b) Extract the files Basedata.har, default.prm, gsdvole.har and
        tstrade.har from the WinZip archive created by GTAPAgg
        and copy to the project directory.
     c) NB: GTAP have labeled a *.har file as a *.prm file. Thus
        default.prm should be renamed default.har for HAR2GDX.
           IF default.har does NOT exist the programme fails on a Display
           statement for elastexist.

  7) Check that programme compiles by using SHIFT+F9.
        If the steps above have been followed there should be no problems.
        a) Some users can have problems with the GDXXRW facility
           that generates the file data_in.gdx from the Excel file. These
           problems mostly seem to originate with a corruption of the
           'LAYOUT' worksheet and result in an error message saying the
           file data_in.gdx does not exist. To overcome this type of problem
              i)  check that gdxxrw.exe exists in the GAMS directory
              ii) copy the contents of the worksheet 'LAYOUT2' and replace
                  the worksheet 'LAYOUT' using Paste Special
        b) If an error message saying that the file database.gdx does not
           exist arises check that the files har2gdx.exe, gdx2har.exe and
           gdxiomh.dll exist in the GAMS programme directory.

  8) Run the programme. The output to the working directory should be as
     follows:
     a) A GDX file for checking aspects of the SAM
          gtptots.gdx
          gtpmxmin.gdx
          trade_data_CHK.gdx
     b) Results files
          i)    ALL the GTAP data to GDX
                    gtpdatav804.gdx
     c) Additional results files are available. These need the user to
        remove the characters that comment out the unload commmands. (This
        reduces the amount of potentially unuseful output.)
          i)    ONLY A Global SAM in GDX format (section 13.2)
                    gtpsam804.gdx
          ii)   A Global SAM in HAR format (section 13.3)
                    gsam804.har

     NB: For version 8 of the GTAP database the directory will be in the
         order of 1.4 GB with the Global SAM in HAR format accounting for
         some 590 MB, the lst file (with all display statements) some 350 MB,
         and the ALL the GTAP data in GDX some 72 MB.
         A file - clean_gtp.bat - that removes surplus files is included:
         this is a batch file, i.e., to run double click on the file in
         Windows Explorer.
         This leaves all the GDX file with ALL the GTAP data, and all data
         and programme files needed to run the programme.
         (This clears out about 1.1 GB of files.)

  9) Exports fob and Imports fob calculated directly do not always equate -
     indirect and direct methods of calculating imports are included
     search for TRADECAL to control the choice of method. All the checks
     labelled TRADECHK should equal ZERO. The programme aborts with an
     error message if the trade accounts do not balance.

  10)GAMS code for aggregating the Global SAM programme is provided
     in the User Model Library. Two 'methods' are provided:
     10.1) A simple GAMS programme with an Excel workbook to assist
           generating the mapping accounts.
     10.2) A VBA programme that generates the mapping accounts and runs
           a GAMS programme.

  11)These notes relate to situations where a user wishes to work with
     another version of the GTAP database. It is assumed this will be a later
     version.
     A new workbook - data_gtp_**.xls - will need to be compiled.
     a) Using the file gtp_sets_v80.xls as a template
        i)   Get the 'new' sector, region, factor and margin codes and
             descriptions from the GTAP database and copy them into the
             worksheet 'Set data from GTAPAgg'
        ii)  Check that the worksheets
             'Set creation regions/sectors/factor/margins' have all updated
             correctly - NB the number of entries in each group of
             codes is the critical point.
        iii) Compile the new sets in the worksheet 'Compiling Sets'
     b) Using the file data_gtp_80.xls as a template
        i)   Generate a revise worksheet 'Sets' - using copy and paste
             SPECIAL from the new file gtp_sets_V**.xls.
        ii)  Generate a revise worksheet 'Maps' - using copy and paste
             from the new worksheet 'Sets'.
        iii) The worksheet 'Layout' should NOT need to be changed unless
             new sets or maps are created.
     c) Update the programme file to read the new sets and maps.

     d) Some general suggestions
        i)   Spaces in the cells of the file data_gtp_**.xls are a
             problem - to GAMS leading and trailing spaces are characters.
        ii)  If the orderings in the mapping sets are not consistent the
             mappings implemented will be wrong.
        iii) 'Corruption' of the worksheet 'Layout' will cause the sets and
             maps not to be read in.
        iv)  The codes used for account names in the GTAP database may
             have changed.
        v)   'Big' initial 'errors' (say entries in the first DIFFTOT that
             are greater than 1) are likely to be due to an error in the
             sets and maps files.
        vi)  'Small' initial 'errors' (say less that 0.01) are more likely
             to rounding issues due to the degree of precision in the
             GTAP database.
        vii) When adjusting the programme to remove 'errors' due to
             the degree precision in the database, it is best by far to
             work through the accounts as they appear in the SAM, i.e.,
             start with the import commodity accounts, then domestic
             commodities, etc.