|
<< Click to Display Table of Contents >> Navigation: Modules > Asset Management > Consolidation |
Consolidation is the task of taking all the records currently in the IMAC Staging table and consolidating them into the main repository. The Staging table is where raw data sourced by the various catch points resides. Consolidation takes that data, applies business rules, and merges it into the main asset tables.
Since consolidating records can be a potentially massive job (thousands of records at a time), the consolidation engine runs as a process on the MIE. The process monitors the MIE Job Queue. When it sees a job in the queue in “SUBMITTED” status, it picks it up, changes the status to “PROCESSING”, and begins to process the job. The service monitors the queue in a very tight loop. So that a processing job that gets kicked off does not take the attention of the whole server (making other jobs wait in the queue until it is done), each consolidation job is run as a separate thread. That way, there can be as many consolidation jobs running in parallel as the server can support, which is based on its horsepower and whatever else it’s being asked to do.
A user on the MIE interactively places a consolidate job in the queue by using the MIE's Timed Operations capability. The status is set to “SUBMITTED”. The server engine picks up the job and sets the status to “PROCESSING”. When complete, the server engine sets the status to “COMPLETE”. The MIE picks up all jobs in status complete and sends a notification email to the user who submitted the job, changing the status to “NOTIFIED”.
The source is the staging table, which is a very wide table with many columns (FLAT). The destination is comprised of the MIE’s Asset Management data schema, namely the following tables: ATdataAssets, dataContacts, dataLocations, ATdataMasterProduct, valATOrganization.
There are many rules enforced:
1. MIE table definition rules – the MIE maintains several settings for each table. Those settings include:
i. InsertOnly – records can only be inserted into the table, no updates
ii. Table cannot be added to – no records may be added to the table through automated feeds
iii. Log changes to existing records
iv. Blank table first before loading
The consolidation routine honors the InsertOnly flag, the CannotAdd records flag, and the log changes flag. The blank first flag is not honored.
2. MIE trusted catch point fields – the MIE allows various CMDB-DD data elements to be assigned to a catch point, designating that catch point as the most trusted source of information for those data elements. Therefore, every time a record is processed from the staging table, the catch point that sourced the record is used to compare to the most trusted catch point for each field (for those that are defined as trusted). If a field is assigned to a most trusted catch point, then only that catch point can update that field. The only time a different catch point can update that field is when the most trusted catch point has not yet sourced data to that particular record.
3. Add HW or SW – the MIE allows various catch points to be designated as being able to source new hardware or software items in the master products table. For instance, the ASN processor can be defined on the MIE to be able to source new H/W models. This is basically the same as saying the catch point can perform INSERTS on the Master Products table. The Add new HW and Add new SW flags on a catch point OVERRIDE the MIE’s table settings for the Master Products table. That is, if the MIE table settings allow inserts on the Master Products table, yet the current catch point’s settings do not allow adding h/w or s/w, no inserts will be allowed.
4. Auto-adjust for length differences – the MIE will truncate values coming from the staging table that are too long to fit in the main repository destination table/field. Every time a truncation occurs, the MIE will log a warning.
5. Set default status - the MIE will automatically add a status value when so configured if the catch point does not explicitly set the status.
The MIE defines three ways to determine a record’s uniqueness. 1) by its primary key value, 2) by it’s “report” field (pseudo key), and 3) by a defined “composite key”. These definitions are maintained in the MIE’s table definitions (RDCnet, Tables). The MIE uses non-intelligent keys for all tables (meaning a 32 bit integer value that has no correlation to any of the information held in the record). The “report” field is one of the fields in the table that contains a user-editable value that is guaranteed to be unique. An example would be the Asset Tag number in the assets table. A composite key is defined on the MIE as one or more fields that, when taken together, can identify a record uniquely. Manufacturer + Product Description can determine uniqueness in the Master Products table, for instance. The consolidation routine uses two of the three (since the primary key is not known to the staging table), in the following order: “Report” field is used first, then the composite key. The MIE processes “child” tables first for each staging record, saving the main asset table record for last. That way, the foreign key values of all the associated child records are known before processing the parent (asset) record. This allows all the foreign key values to be applied when the parent record is saved. When processing records, the MIE must be able to determine if the record already exists (in which case it will be updated), or if a new record must be inserted. For child tables, this is even more important so that the parent record can be associated with the proper child record. The rule enforced by the MIE on child tables is that if a catch point sources insufficient information to form a proper unique lookup, then the child table update will be skipped and no child record will be associated with the parent. As an example, let’s say a catch point is able to report the asset’s manufacturer as “Dell”, but does not report the product description. Under the uniqueness rules for the Master Products table (as defined on the MIE), both Manufacturer and Product Description must be provided in order to find a unique record. If “Dell” is the only value presented, there are many “Dell” records that would match, and the parent would likely be associated with the wrong “Dell” record. Therefore, the catch point MUST provide sufficient information to form a unique look-up into the child table before the MIE will trust that it can locate the correct child record and make the proper association. Otherwise, any association that previously existed will be maintained.
The MIE supports a location hierarchy that involves 6 location tables: dataLocations, dataLocationBuilding, dataBuildingFloor, dataFloorRoom, dataRoomRack, dataRackPosition. Some catch points may enforce the hierarchy and some may not. The staging table supports using the hierarchy, and it supports capturing site, building ,floor, room, rack, and rack position data as uncorrelated data. However, during consolidation, if sufficient information is presented by the staging table record to build some or all of the hierarchy chain – starting at the top – then the consolidation routine will build the hierarchy and will update the foreign keys in the main repository table with the values of the 6 tables that participate in the hierarchy. In addition, the text values will also be populated.
The consolidation routine will honor the warning level setting that is set through the Settings/Admin menu option. The default is 4 (highest). At warning level 4, there are potentially a large number of warnings as every time a catch point sources insufficient data to uniquely identify a record, a level 4 warning is logged.
For every table that is defined as requiring the logging of change, the consolidation routine will log to the change log every field of every record that is changed. The table, primary key, name of field, old value, and new value data elements are logged. This maintains complete traceability as to how data elements were changed. It also allows a future roll-back capability.
Warnings and errors, along with the number of inserts, updates, field-level changes, and the time (in seconds) that it took to run the job are logged to the job log. The entry in the job queue that kicked off the job is linked to the job log entry. There is potentially quite a bit of information logged to the job log.
When a consolidate job is run, only records in the staging table that have no MIESTATUS value or have an MIESTATUS value other than “Consolidated” are included. If any errors occur during consolidating the record, the MIESTATUS field will be set to “Exception”. Otherwise, the MIESTATUS field will be set to “Consolidated”.
Each catch point that sources data to each record in the ATdataAssets table is logged in the ATdataAssetsXCatchPoints table. If a catch point has sourced data before, its “Updated” field will be updated with the current date/time. Otherwise, a new entry will be added to show that a catch point sourced data to a record.