Difference between revisions of "Staging Area Generator"

Revision as of 14:53, 20 June 2014

The IMSMA Staging Area Generator is a tool that creates a staging area from an IMSMA^NG database. It is thus part of the IMSMA^NG tool set, but not included in the IMSMA^NG installation package. IMSMA Staging Area Generator thus needs to be installed separately. Please contact your GICHD IM advisor in order to get the latest installation package.

The Staging Area Generator can only be used with IMSMANG 6.0 and higher.

Basic Principles

The Staging Area Generator takes an IMSMA^NG database as input and outputs a staging area database, i.e. a flattened version of the IMSMA database, containing only approved data. The structure of the staging area and a details on the data it contains can be found on the IMSMA Staging Area page.

Where to install and use it

In terms of the technical architecture for IMSMA and the staging area, there are the following options:

• The staging area can be generated on the same server and database server as IMSMA. In this case, the tool is run locally and makes only local connections to the source and target databases.
• The staging area can be generated on another server and database server than IMSMA. For example, there could be a dedicated reporting server, in order not to overload the server on which data entry in IMSMA is done. In this case, the Staging Area Generator has to be run on the target server.

Installing the IMSMA Staging Area Generator

Prior to installing the Staging Area Generator

1. To install the staging area generator (SAG) double-click on the etl.exe file.
2. The graphical installer then starts. On the first screen, choose the language you would like to use for the installation and click OK.
   
3. Navigate the Installation panels of the graphical installer as described below:
   1. On the Welcome screen, click Next.
   2. On the Infromation screen, click Next.
   3. We strongly recommend to use the default installation path for the Staging Area Generator. Click Next.
   4. If the target folder (C:\IMSMAETLTool) does not already exist a message will be shown, click OK.
   5. In the packages selection panel, click Next.
   6. Wait for the installation progress bar to finish, then click Next.
   7. When the Setup Shortcuts panel displays, click Next.
   8. When the installation has completed successfully, click Done.

After the installation, the following menu entry will be visible in the Start menu → All Programs → IMSMA:

Using the IMSMA Staging Area Generator

To start the Staging Area Generator, click on the Staging Area Generator Icon in the Start menu → All Programs → IMSMA. This will open the following graphical user interface.

select *
from spatial_ref_sys

Finally, click on Start to start the staging area generation. In the next window, click on Start once more:

A log with all the steps is written in this window as the generation process goes on. This might take some time, depending on the size of the source IMSMA database. The range is from very few minutes up to two hours for very big databases. Once the generation is finished, the newly generated staging area database can be explored with a database query tool such as pgAdmin3 or Navicat, and reporting tools can be connected to it.

Scheduling the Staging Area creation

The above described generation of a staging area needs human interaction. However, once all the options are fixed, there might be the requirement to re-generate the staging area regularly, in order to have an up-to-date reporting database. It is possible to automatize the staging area generation by following these steps:

1. In the C:\IMSMAETLTool folder, create a .bat script named for example runStagingArea.bat with the following content:

   cd C:\IMSMAETLTool
   call bin\runnogui.bat
   

2. Edit the properties file C:\IMSMAETLTool\conf\config.PROPERTIES by specifying the required values for all the parameters (they correspond to the ones from the graphical user interface described above). If a parameter should not be used, e.g. script_file, then comment it by placing a # at the beginning of the line:

   conversion_language=en
   gazetteer_sequence_level=0
   #script_file=scripfile.sql
   

3. Schedule a task in an operating system's task scheduler. For example, using Windows 7, follow those steps:
   1. Go to Control Panel → Administrative Tools → Task Scheduler
   2. In the Task Scheduler interface, click on Create Basic Task...
   3. In the Create Basic Task Wizard, provide a name, e.g. Staging Area Generation and an optional description. Click Next.
   4. In the next window, specify the required recurrence (daily, weekly, etc.) and click Next.
   5. Next, refine recurrence criteria (e.g. time of the day) and click Next.
   6. In the next window, select Start a Program and click Next.
   7. Browser to the previously created script, e.g. C:\IMSMAETLTool\runStagingArea.bat. Click Next and then Finish.

The staging area generation should now start at the specified time.

It is advised to plan the generation time carefully, i.e. best during the night when no one is accessing and entering data into IMSMA^NG, as this might lead to inconsistencies. Ideally, the IMSMA server should be stopped before the staging area generation and then restarted. It also needs to be assured that no one (and no automatic reporting process) is accessing the staging area database at the time it has to be re-created. As long as there are open connections to the database, it cannot be dropped and re-created. Contact your GICHD IM advisor in case of questions.