Structure of the IMSMA Staging Area

From IMSMA Wiki
Jump to: navigation, search

Flattening principles

Note.jpg This section requires basic knowledge of the IMSMANG database table structure.
Where to find which data in the IMSMA staging area
Type of attribute Where? Example
Standard text, numeric, date and time attributes Stored directly in the main object table, similar to IMSMANG The HAZARD (Land) attribute hazard_localid is stored in a column also called hazard_localid in the HAZARD table of the staging area.
Standard single-selects Stored directly in the main object table (whereas in IMSMANG the main object table only contains a GUID reference to the IMSMAENUM table) The Land attribute status is stored in a column called status_enum in the HAZARD table of the staging area. It contains the translated value of the status base value, e.g. Open or Closed in English or the equivalent values in another language that has been specified at staging area generation.
Standard multi-selects Stored directly in the main object table as comma-separated list of values AND in a normalized way in the <OBJECT>_STD_MULTISELECT table (in IMSMANG the values for a multi select have to be looked up in the <OBJECT>_HAS_IMSMAENUM and IMSMAENUM tables) The Land object has a multi-select attribute called Marking (Marking Method). In the staging area, the HAZARD table has an attribute marking_method with a comma-separated list of values, for example 'Official Signs, Local Signs'. Additionally, the table HAZARD_STD_MULTISELECT can be joined with the HAZARD table in order to get to the same values as rows. Reusing the above example, there will be two rows, one with 'Official Signs' and one with 'Local Signs' as values (or the translated values if any translations are available).
CDF text, numeric, date and time attributes and CDF single-selects Stored directly in the main object table (as opposed to IMSMANG where a lookup has to be made through the <OBJECT>_HAS_CDFVALUE, CDFVALUE and CUSTOMDEFINEDFIELD tables) Each CDF will be turned into a column in the staging area. For example, a CDF called My Hazard CDF defined on Land will result in a column named my_hazard_cdf in the HAZARD table of the staging area.
Note.jpg Note that non-standard characters in CDF names, such as blanks, are replaced by underscores in the column names
CDF multi-selects Stored directly in the main object table as comma-separated list of values AND in a normalized way in the <OBJECT>_CDF_MULTISELECT table (in IMSMANG the values for a multi select have to be looked up in the <OBJECT>_HAS_CDFVALUE, CDFVALUE and CUSTOMDEFINEDFIELD tables) Let's assume that the Land object has a multi-select CDF attribute called My Land Multi-Select. In the staging area, this will result in a column in the HAZARD table called my_land_multi_select with a comma-separated list of values, for example 'Value1, Value2'. Additionally, the table HAZARD_CDF_MULTISELECT can be joined with the HAZARD table in order to get to the same values as rows. Reusing the above example, there will be two rows, one with 'Value1' and one with 'Value2' as values.
Location data The guid, localid and name of the location an object is assigned to are directly stored in the main object table. The HAZARD table in the staging area has the columns location_guid, location_localid and location_name with the values of the location that each object is assigned to.
Organisation data The guid, localid and name of the organisation defined on an object are directly stored in the main object table. The HAZARD table in the staging area has the columns org_guid, org_localid and org_name with the values of the organisation defined for each object. If CDFs of type organisation have been created in IMSMA, these will also be available in the staging area.
Place data The localid and name of the place linked to an object are directly stored in the main object table. The HAZARD table in the staging are has the columns ammunition_storage_localid and ammunition_storage_name that refer to a place object.
Classification data (Country structure, Assistance Classification, Cause Classification and Needs Assessment Classification) The guid, localid and name of classifications (country structure (gazetteer), assistance classification, cause classification and needs assessment classification) associated to an object are directly stored in the main object table. Since a classification can have several levels, there is a placeholder for each level, up to the maximum number of levels. In the HAZARD table in the staging area, there are the following columns regarding the country structure classification (gazetteer): gazetteer_guid, gazetteer_level1_localid, gazetteer_level1_name, gazetteer_level2_localid, gazetteer_level2_name, ..., gazetteer_level7_localid, gazetteer_level7_name. There are seven placeholders because the country structure in IMSMANG can have up to seven levels.

Database model

A simplified IMSMA staging area database model for the Land (Hazard) object is depicted below. For all the other main objects, the model looks very similar.

Staging Area Model Hazard.png

For more on column names in the Staging area, see Name Rules Staging Area, and for post-processing, see Post Processing SQL Scripts.

The IMSMANG database is described here. The columns names are not always identical in the Staging area database but very similar. Some of the main item changed names in 6.0 but table names did not change:

  • Land = hazard
  • Activity = hazreduc
  • Education = mre
  • QM = qa

Special table

Table name Description
ETLHISTORY This table is the only one that does never get dropped from the staging area, but persists as the database is re-generated again and again (see Staging Area Generator). It contains information about all the staging area generations on a specific database (i.e. for example when a staging area database with the same name is generated regularly, e.g. once per day or week). It has two columns:
  • etlruntime: this is the timestamp of the start of the generation
  • runtime: this is the time in minutes that the generation took

Having x rows in this table means that the database has been generated x times, and allows to keep a history of the individual runs. It also allows to determine how old the information in the staging area is, by looking at latest timestamp.