Data Sources - greenriver/hmis-warehouse GitHub Wiki

Data is imported into the warehouse through Data Sources. The list of Data Sources and their affiliated organizations, projects, import status, and import extensions are located in the main menu and visible to users with the applicable assigned permissions and data access.

HMIS Data Sources

HMIS Data Sources are imported into the warehouse through an HMIS CSV file, often originating from an HMIS installation. Data Sources can be configured for individual CoCs, multiple HMIS installations within a CoC, and Statewide installations.

The table of existing HMIS Data Sources is visible by clicking Data Sources in the Main Menu.

Table elements include the following:

  • Name - Name provided for the source. This may be the specific name of an organization, CoC, or HMIS implementation.
  • Short Name - An abbreviated name of the data source. The short name is displayed alongside enrollment details in client profiles and on some reports.
  • Client Records - The number of client records included in the data source.
  • Project Count - The number of projects included in the data source.
  • Visible in Window- A flag on data sources that control visibility of client records for specific users depending on permission settings. Check with your local administrator to see if this feature is utilized locally.
  • Authoritative - Indicates if the Data Source is marked as an authoritative source, meaning data uploads are not permitted and data needs to be entered manually.
  • Last Import Run - The last date and time an import was added to the data source. Imports that are still processing will display the number of enrollments remaining to process.
  • Delete button - Represented by an X-mark allows the user to delete the data source and all associated data.

Data Source List Image of Data Source Table

HMIS Data Source Details

The HMIS Data Source Details includes information about the data uploaded into the warehouse. Details can be viewed by clicking on the data source name in the data source table. On this page, the user can make edits to uploaded information; upload HUD HMIS zips; automate uploads; adjust importer extensions; and edit organizations and projects.

  • Data Source ID - The unique identifier for each data source within the warehouse.
  • Last Import Run - See Data HMIS Data Source for more information.
  • Date Span - Represents the span of time covered by all of the data in the data source, not the chosen date range for a given import.
  • Uploads - A link to the upload table which lists the history of uploads. See HMIS Upload Table for more information.
  • HMIS Export SourceID - A SourceID is usually a unique text label that is configured within HMIS and included in the Export.csv file (part of the HUD CSV zip file download). This is used to confirm uploads are made to the appropriate data source. If the SourceID is entered, the system will prevent the upload of any .zip file that does not have a matching id. This is a safeguard against an administrative user accidentally uploading an export into the wrong data source.
  • Visible in the Window into the Warehouse - See Data HMIS Data Source for more information.
  • Does the Personal ID column contain a UUID - This is applicable for uploads that use a UUID as the Personal ID. If marked as Yes, the UUID will add dashes to the Personal ID when displayed, converting 5011A79B04E34BB99D5A002A3E079C61 into 5011A79B-04E3-4BB9-9D5A-002A3E079C61.
  • Import Paused - Indicates if the import is marked as paused for the data source. If paused, automated imports will not occur. Manual uploads will still be processed.
  • Exposed with ROI - Indicates if the data source should be exposed when a client has a consent (ROI) on file. If unchecked, only users with the data source assigned directly to them will be able to see the source client details.
  • Service Scanning Enabled - The warehouse can be used to track bed-nights directly, if enabled, this data source can be used to collect very basic service information. Historically, this has been used to bridge time where a CoC might not have access to their HMIS.
  • Viewable by the following users, regardless of visibility window - Indicates who is able to view the data source.

Actions available on the data source are listed below. Availability of actions will vary by warehouse installation.

  • Edit Data Source - Allows the user to make updates to the data source.
  • Upload HUD HMIS Zip - Area where manual uploads are made.
  • Automate HMIs CSV Loads - Area where automated uploads are configured. Communities will probably need to work with their HMIS vendors and Green River to set-up automated uploads.
  • Configure ETO API - This configuration is used to attempt connections to the ETO API and to transform returned client demographic and Touch Point information into usable data in the warehouse. This is only applicable for ETO HMIS installations and will require access to the ETO API with appropriate credentials.
  • Show Importer Extensions - A list of editable features of the importer. Updates to the extensions will only be applied to data uploaded after the update is made, and will not apply to historic data loads. See Importer Extensions for a full list of available extensions.
  • Show Custom Imports - If your HMIS has non-HMIS data that needs to be imported to the warehouse, this may provide insight into the status of those non-HMIS data imports. You will need to work with Green River to have those setup.

Green River Data Source

Importer Extensions

Importer extensions are applied to imports as a way to fix consistent issues. Updates to importer extensions will only impact data moving forward, not historic imports. Extensions are applied to specific pieces of the data. Ex. Project - Prepend the HMIS Project ID to the ProjectName field. If the data falls outside the regular import timeframe, consider re-importing information that encompasses the timeframe for that specific data.

Importer Extensions Example of Importer Extensions

HMIS Upload Table

The HMIS Upload Table is found under Uploads on the HMIS Data Source Details page. Each upload is listed as an individual row in the table and includes the following:

  • Upload HUD CSV - The location where uploads can be manually added. See Manual Data Uploads for more information on uploading data.
  • File Name - The name of the file.
  • Uploaded By - The individual uploading the data. Automated uploads will be listed as System User.
  • Uploaded On - The day and time the upload was made.
  • Processing Started - The day and time the data began processing.
  • Completed In - The amount of time; and the start and end for the data to be uploaded. Files are marked as Processing when the data is still being uploaded and Failed when the file failed to upload and be processed.

Uploads Example of Upload Table

Import Log

Import logs include details related to individual uploads of HMIS files into the warehouse. They can be found by clicking the hyperlink on the desired file in the Upload Table. Information in the log includes:

  • Date/Time - Imported On, Loaded, Imported, and Completed In.
  • Files Found - Option to Download the Original file and the Download Error Report
  • IDs - Loader Log ID, Upload ID, Import Log ID, and Importer Log ID. (These are internal ids used to track down issues with specific imports.)
  • Loader Summary - Result of reading the file and breakdown of what data was actually uploaded.
  • Importer Summary - Result of processing the data and identifies if there are any associated issues. The warehouse attempts to bring in any data it can from the HMIS CSVs, it will call out data that is malformed, but sufficiently complete that it can import it as validation flags. At times it will need to reject a row that is malformed or missing critical data, in these instances, you will see counts in the error flags and errors columns.
    • Validation Flags - Validation flags are added when the data was imported but did not meet the HMIS CSV specification.
    • Error Flags - Error flags expose data that the warehouse was unable to import. Generally this indicates a mis-match of type or missing required data.
    • Errors - Errors are reserved for data that is so malformed the warehouse was unable to process the row at all.

Importer Log Example of Import Log Details

Loader Summary Example of Loader Summary

Importer Summary Example of Importer Summary

Ad-Hoc Data Sources

Ad-HoC Data Sources are used to identify overlap between external data sets and clients in the warehouse. Individual clients uploaded through Ad-Hoc sources can be matched to other clients in the warehouse. If a match is found, a link will be generated to the associated record.

Data must be formatted with the correct headers (First Name, Middle Name, Last Name SSN, and DOB) to be successfully uploaded to the Warehouse. A template is available within the individual ad-hoc data source page.

Uploads

Adding an HMIS Data Source

Data sources are usually configured for each sending HMIS. Often this equates to one data source per CoC, however, if there are multiple HMIS installations within a CoC, each HMIS should be configured as a separate data source. If one HMIS installation supports multiple CoCs, only one data source needs to be configured, but it may make setting up permissions easier to configure each CoC as a data source.

  1. Navigate to the Data Sources from the main menu.
  2. Click the + Add Data Source button.
  3. Complete information related to the data source.
    • Name - The name of the source HMIS System. Ex. Sunshine CoC
    • Short Name - The name that will display on client enrollments and reports. Ex. MI-801.
  4. Click Create Data Source.

Uploading HMIS Data

After the HMIS Data Source is created, the user can upload the HUD HMIS exported zip file. Manual uploads are completed by a user in the warehouse at any desired frequency.

  1. Download a HUD HMIS zip file from the applicable HMIS installation. The export range is generally made for the 12 months prior to the current date. Ex. If a user runs the export on 4/1/2023, the export should be from 4/1/2022 - 4/1/2023.
  2. Navigate to the desired Data Source from the main menu.
  3. Click on the **Upload HUD HMIS ZIP **button.
  4. Choose the HUD HMIS zip file to upload.

Ulpload screen

  1. Click Upload Zip.

Once a file has been uploaded, the system will begin processing and looking for possible errors. Updates to client records will not be available immediately in the warehouse as some data is cached during an update process, which is scheduled to run nightly.

Automated Data Uploads

Data uploads can be automated between HMIS and the warehouse. To set this up, reach out to Green River, we’ll need to work with your HMIS vendors to get data flowing. To view details of existing automated uploads navigate to the desired data source and click Automate HMIS CSV Loads.

Adding an Ad-Hoc Data Source

  1. Navigate to the Ad-Hoc Data Sources from the main menu.
  2. Click the + Add Ad-Hoc Data Source button.
  3. Complete information related to the data source.
    • Name - The name of the source HMIS System. Ex. Sunshine CoC
    • Short Name - The name that will display on client enrollments and reports. Ex. MI-801.
  4. Click Create Ad-Hoc Data Source

Uploading Non-HMIS Data

  1. Navigate to the desired Ad-Hoc Data Source from the main menu.
  2. Click on + Upload Batch button.
  3. Choose the file using the template.
  4. Click Upload.

Organizations and Projects

Organization Details

Organizations are visible by Data Source in the Warehouse. Each organization has an associated list of projects. Contacts can be assigned to organizations and will be used in some reports.

Organizations and Projects

Project Details

Projects are grouped together by Organization and are visible by Data Source. The project table includes a list of projects, Project Type, Tracking Method, and Confidentiality Status. You can also edit project details, and view the full HMIS data as seen in the most-recent import.

The HMIS button on the project table displays details for the imported data:

  • Warehouse Value - Represents the final data after merging the post-processed data with the existing warehouse record.
  • Post-Processed Value - Shows the record after passing through the import and cleaning process.
  • CSV Value - Is the record as imported directly from the HMIS CSV import.

Each project in the table is linked to project details. The fields included in the project details are the Project Descriptor Data Elements found in the HMIS Data Standards. These include, Project Information, Continuum of Care Information, Funding Sources, and Bed and Unit Inventory Information. The list of clients enrolled in the project is also listed with their entry and exit date for users with appropriate permission.

Project Details Example of Project Details

Project Details - Additional Example of Project Details

Marking Organizations and Projects as Confidential

Users with the appropriate permissions are allowed to mark organizations and projects as confidential. Organizations marked as confidential, marks all projects within the organization as confidential, including projects added to the organization in the future. If checked, when exporting HMIS CSVs and choosing the confidential option, the organization's name will be obfuscated. All projects within the organization will also be marked as confidential.

Projects marked as confidential will be displayed as confidential on a client’s profile enrollment information. It also prevents client information in the project from being pulled into reports for users who are not set up to view confidential information.

If marked as confidential, the project name will be replaced with "Confidential Project" within individual client pages. Users with the appropriate permissions will still see the project name.

Mark Organizations as Confidential:

  1. Navigate to the Organization within the Data Source.
  2. Click Edit Organization.
  3. Checkmark Is this organization confidential?
  4. Click Save

Mark Projects as Confidential:

  1. Navigate to the Project within the Data Source.
  2. Click the edit Pencil.
  3. Checkmark Confidential?
  4. Click Save

Editing a Project

A project can be edited to override or add information received from a file upload. Funding Sources, Bed and Unit Inventory, and Project CoCs can also be manually added to a project or overridden.

Edit a project, funder, inventory, or project CoC:

  1. Navigate to the Project within the Data Source.
  2. Click the appropriate Edit button .
  3. Update applicable fields.
  4. Changes made to these fields are considered overrides and will persist between uploads until removed.

Add Funding Sources, Inventory, or Project CoC:

  1. Navigate to the Project within the Data Source.
  2. Click Add Manual Funder, Add Manual Inventory, or Add Manual Project CoC Record.
  3. Update applicable fields.
  4. Click add.

Deleting a Project

Projects and the associated enrollment data can be removed from the warehouse.

  1. Navigate to the Project within the Data Source.
  2. Click the Delete Project button.
  3. In the sending HMIS, prevent the project from being exported, otherwise the project will reappear after the next upload.

Delete Project Example of Delete Project button