Importing - wvulibraries/mfcs GitHub Wiki

Importing data into MFCS from another system.

MFCS was designed with easy importing in mind. To import data, you follow the same steps, and use the same methods, as MFCS does internally.

Importing into MFCS from an external source should be considered destructive. Back up the database before importing. While the importing is happening MFCS should be set to read-only mode. The data should be verified on a development instance of MFCS before running in production. The data should be verified in production before MFCS is returned to a write/normal state. Once MFCS is returned to normal production and data is entered you will be unable to revert to the saved copy of the database and any import errors will need to be corrected on the production database.

General Steps

  1. Create new Object Form in MFCS
  2. Create any supporting metadata forms in MFCS, be sure they are properly linked to the Object form.
  3. Back up the MFCS database before beginning
  4. Importing scripts should
    • be set to run from the command line only.
    • be run from inside the MFCS directory
    • Log any errors.
    • Be run with MFCS in read only mode
    • removed when exporting is finished
  5. Import scripts will
    • read the data from a location
    • create subject headings
    • create objects
    • copy original digital material to the MFCS archival location

NOTE: The exact process will vary for each collection that is added to the system.

MFCS methods utilized

Remembering that there are 2 types of Objects in MFCS: Objects and Metadata Objects, and that Metadata Objects are used for subject headings, we add both types of objects using the objects::add() method.

The objects::add() method only needs to take two parameters to create an object, the formId and the submissionArray.

The formID is the MFCS identifier for the form. It appears in the query string when viewing the form.

The submissionArray is an associative array where each key=>value pair is a field in the form. key is the Field Name on the field, and value is what will be saved for that field in the record.

Example usage:

$form_id = 1;
$submission_array = array();
$submission_array['title'] = "My Title";
if (objects::add($form_id, $submission_array) === false) {
  // alert/monitoring here
}

This will add a new object to form 1 with the only information being the Field title => "My Title" ... this could be an example of a simple subject heading.

An example of a subject heading that uses linked data may be:

$form_id = 1;
$submission_array = array();
$submission_array['title'] = "My Title";
$submission_array['LinkedDataDomain'] = 234; // this is an ID number to the linked domain table
$submission_array['LinkedDataUri'] = "/some/uri:1234";
if (objects::add($form_id, $submission_array) === false) {
  // alert/monitoring here
}

Importing Subject Headings

When importing subject headings we need to make sure we don't enter duplicates. If the subject heads for an import will be going into Metadata forms that are exclusive to this Object Form the process becomes simpler. All subject heads can be loaded into an associative array:

$subject_headings = array();

foreach($heading_from_old_system as $heading=>$heading_metadata) {
  $subject_headings[$heading] => array('linkedDataDomain' => $heading_metadata['linked_data_domain'], 
                                       'linkedDataUri' => $heading_metadata['linked_data_uri']);
} 

You will need to do something similar to that for each type of subject heading (creator personal name, topical subject, etc ...)

If the subject headings will be merged into exiting metadata form's, you first need to get all the objects for a form, then compare and merge from the old system. If you didn't normalize your data in the old system you need to take special care to correct common spelling, capitilization, and punctuation errors. Also, be sure to trim white space.

To get all the objects for a Metadata form from MFCS use objects::getAllObjectsForForm($formID); This will return an array of objects.

String conversions

Any string conversations and data manipulation should be done in the migration scripts.

Example:

function convertString($string) {
  $string = preg_replace('/%Oitalic%/',   '<em>',      $string);
  $string = preg_replace('/%Citalic%/',   '</em>',     $string);
  $string = preg_replace('/%Obold%/',     '<strong>',  $string);
  $string = preg_replace('/%Cbold%/',     '</strong>', $string);
  $string = preg_replace('/%underline%/', '<u>',       $string);
  $string = preg_replace('/%underline%/', '</u>',      $string);
  $string = preg_replace('/%link url="(.+?)"%(.+?)%\/link%/', '<a href="$1"><u>$2</u></a>', $string);
  return $string;
}

Loading data

If loading directly from a database, use this strategy for connecting to multiple databases with EngineApi 3.x:

$engine->dbConnect("server","database.server.com"); // be sure that remote permissions are set properly
$engine->dbConnect("username","database-user");
$engine->dbConnect("password",'DatabasePassword');
$remoteDB = $engine->dbConnect("database","DatabaseName",FALSE);
 
// Reset the values for the local database
$engine->openDB = NULL;
$engine->dbConnect("server","localhost");
$engine->dbConnect("username","MFCS-DB-Username");
$engine->dbConnect("password",'MFCS-DB-Password');
$engine->dbConnect("database","mfcs",TRUE);

// You can now use $remoteDB as a normal database object, like $openDB, throughout the script
$remoteDB->escape();
$remoteDB->query();
⚠️ **GitHub.com Fallback** ⚠️