SRS - TriageCapacityPlanning/Triage GitHub Wiki
Last revised: March 31st, 2021
All tables discussing progress refer to the PUC table above them
Sam Cymbaluk - 400138646
Shawn Esseltine - 400145888
Daniel Rubinstein - 400111892
Nikhail Singh - 400069771
Kristie Tom - 400018028
- Table of Contents:
- 1. Introduction
- 2. Product Overview
- 3. Requirements
- 4. Verification
- 5. Deprecated Work
- 6. Future Work
- 7. Appendixes
The purpose of this document is to provide a detailed description of the Triage Capacity Planning System. It will explain: the motivation behind the system, the features and interface of the system, how the system will work, the constraints on the system and its design, and the verification methods that will be used to test the system. This document is intended both as a proposal to be approved by the McMaster Children’s Hospital and as a reference to the development team when they are developing the first version of this system.
Triage refers to the sorting of patients (as in an emergency room) according to the urgency of their need for care.[5] The idea is to treat the most urgent patients first because that maximizes health outcomes for everyone. For example, if you had a patient with a heart attack and a patient with the flu, triaging these two patients would tell you to treat the heart attack first. McMaster Children's Hospital currently triages its patients into one of six classes: 1. Immediate, 2. Urgent, 3. Semi-urgent 4. Standard. 5. Forwarded. 6. Declined. A lower number indicates a higher urgency with which a patient needs to be seen. However, it is also important to consider that if left untreated, the ailments of patients with a lower triage class can get worse so it is still important to see these patients in a good time frame. Patients are scheduled based on their triage class however currently the Hospital is just guessing how many slots it needs for each type of patient and if more patients with a higher triage class come in than expected other patients are pushed down the schedule leading to delays and a long waiting queue.
The “Triage Capacity Planning System” (“Triage”) will be a data-driven application for the McMaster Children’s Hospital. The system is designed to eliminate the guesswork involved in scheduling patients by estimating the amount of urgent, semi-urgent and standard patients that will arrive in a given time period to determine how many slots are needed to see those patients within the required amount of time. Currently wait times for urgent and semi-urgent patients are longer than target wait times, potentially causing increased health risks in patients. Long-term, the system will also help track the hospital’s progress in satisfying the new guidelines set by the CPSO[1] which outline the amount of time specialists have to accept referrals and provide a waiting time estimate, and decrease the overall backlog of patients.
The system will use historical data on the triage class and arrival times of patients in order to generate a distribution for the number of arrivals of each triage class. It will then use that distribution to simulate the hospital many times to determine the number of times slots that need to be reserved each week for each type of patient. The system is designed to be able to dynamically adjust the number of required slots based on new data while still remaining flexible and easy to use. The system will facilitate the input of data by users, the setting of parameters for the output, and the visualization of the recommended number of slots. It also contains a relational database for storing the data. These system functionalities are discussed wholly in the subsequent sections.
| Term | Definition |
| Triage | the assignment of degrees of urgency to wounds or illnesses to decide the order of treatment of a large number of patients or casualties. |
| patient | a person receiving or registered to receive medical treatment. |
| CPSO | College of Physicians and Surgeons of Ontario |
| Clinic | An establishment where patients are admitted for special study and treatment by a group of health care professionals practicing together.[3] |
| Wait times | The length of time a patient must wait before being seen by the necessary doctor |
| Referral | Doctors from clinics around Hamilton refer patients to the Specialists at the McMaster Children’s Hospital |
| Triage Class | Categorization established by the McMaster Children’s Hospital for triage purposes defined as follows:
|
| PIPEDA | The Personal Information Protection and Electronic Documents Act |
| UI | User Interface |
| Data module | The module that stores data. Also referred to as the datastore module. |
| Confidence interval (CI) | A confidence interval specifies the range in which a random value appears a given proportion of the time.
For example, if we say that the average number of patients a week is 10 with a 95% CI of 2. That means that 95% of the weeks, the number of patients will be between 8 and 12 inclusive. |
[3]https://medical-dictionary.thefreedictionary.com/clinic
[4]https://machinelearningmastery.com/k-fold-cross-validation/
[5]https://www.merriam-webster.com/dictionary/triage
The remainder of this document contains three chapters and appendices. Chapter two, Product Overview provides an overview of the system and its functionality, describing the system’s origin, functionality, constraints, and users. It also lays the groundwork for the technical requirements that will be detailed later on.
Chapter three, Requirements provides detailed technical specifications for the system including all of its interfaces. It is written primarily with developers in mind and as such describes the design and functionality of the system in technical terms.
Chapter four, Verification provides the verification approaches and methods planned to qualify the software and ensure that it meets the requirements set out in this document.
It should be noted that we are working with an external client to derive the requirements and so there are significant delays between emails and meetings due to their schedule. Therefore in some cases we are awaiting confirmation on some emails.
The system is a standalone application which will replace the existing method of determining time slot reservations.The system is standalone in the sense that it will not be integrating with any pre-existing hospital software. It will also replace the existing method of manually guessing the number of slots needed so our system is not going to be replacing any software or hardware currently in use by the hospital.
The system contains four core components: an interface consisting of a front-end interface for user input and a back-end API for communicating with the rest of the system, a utility service that allows the various parts of the system to work together, a prediction module consisting of a prediction model and a simulation module, and a data module. For further information see figure 1. Part of this system is external, and the other part internal.
The external system can be described in the following example: suppose a user wants to see how many slots are needed to service 95% of the expected critical patients within 2 weeks. The user interface will take user input of 95 and 2, send it via an HTTP request to the API which invokes a utility service. The utility service forwards the request to the simulation module which then returns the final output of the number of slots available. The API returns this result.
Secondly, the internal division of the system can be described as follows: the model takes the data from the datastore module and analyzes it to come up with aggregate information which is then used by the simulation module to create a final prediction for the number of time slots needed which it sends to the user interface via the API. Lastly, the data module is used to store the data inputted by the user so that it can later be used by the model to self-train.
This section will outline the functions the system will perform, organized by component.
Figure 1: Context Diagram
Brief description: the user interface allows users to interact with the system, handling input and output.
List of functions:
- Input necessary data
- Select parameters for the final result
- required confidence level
- time interval boundary (days vs weeks vs months)
- number of intervals
- service percent for each type of patient (urgent, semi-urgent, standard)
- how soon each type of patient needs to be seen
- starting queue of patients (optional)
- Ask for a result based on the inputted data
- View the final prediction with error bars
- view statistics on how different amounts of slots would impact the confidence that you could meet patient demand
Brief description: the Interface API accepts the input from the user, and sends it to the data module. It also accepts the final result from the model and transforms it so that it can be presented to the user via the UI.
List of functions:
- Transform user input into a format usable by the utility service
- Send the parameters the user selected to the utility service
- Receive a final result from the model
- Transform the final result into a method that can be displayed by the UI
- Send the transformed result to the UI
Brief description: enables API functionality, interfaces with simulation module and data module, and scheduling ML model training sessions.
List of functions:
- Automatically schedule ML Training sessions
- Call the simulation module to produce schedules given the parameters provided from the API /prediction. Return the results of the simulation back to the API.
- Provide access to managers for clinic patient data
- Provide access to managers for clinic ML models
Brief description: the module receives data and uses it to perform calculations and analyses to produce a final schedule that satisfies the parameters set by the user.
List of functions: The API/controller module should invoke the simulation and thus set up all the parameters, and pre-massage the data.
- Receive data from the API/controller
- Generate minimum expected slot allocation schedules for a given type of triage class.
- Send the expected of slots needed schedule, the level of confidence, and the CI margin of error
Brief description: the Data Module stores all of the data both inputted by the user and historical data for use by the prediction model.
List of functions:
- Insert new data
- Delete outdated/improper data
- Store hundreds/thousands of rows of data
Brief description: the ML Training Module is initialized with a clinic ID and training hyperparameters to produce a trained weights file that is written to the Data Module.
List of functions:
- Perform training session and write resulting weights file to the data module
This subsection should provide a general description of any other items that will limit the developer’s options. These may include:
- Interfaces to users, other applications or hardware.
- Quality of service constraints.
- Standards compliance.
- Constraints around design or implementation.
The accuracy and certainty of the final prediction generated by the model is limited by the amount of data we can give the model and by the quality of that data. If the data is of low-quality (missing data points, not detailed enough, not representative of the trends the system is analayzing etc.) then the system will not be able to generate accurate predictions no matter how much data we provide. The accuracy and certainty of the system will be limited by the amount of good data we can get. The more good data the better and there is a minimum threshold of good data that is required for the model to generate a result that the McMaster Children’s Hospital could comfortably trust.
The amount of data that can be given to the model is constrained by how much access to data McMaster Children’s Hospital and the associated clinics can provide. This is limited by how well they have been tracking this data on their own as well as how much access they can legally provide which is limited by The Personal Information Protection and Electronic Document Act.(PIPEDA [2]) PIPEDA protects the privacy of patient information including limiting access to patient data which is needed for the system to work.
The overall system design must meet the requirements set by the McMaster Children’s Hospital’s IT team in order for it to be deployed to the hospital and maintained by their staff. The project and its development will also need to comply with all of the requirements set by the 4ZP6 course.
As a part of this project we will be receiving patient data from the Hamilton Integraged Research Ethics Board (HIREB) and so we must comply with their rules and guidelines in order to ensure continued acess to the data.
There are two types of users who will interact with the system directly; doctors and hospital administrators, and hospital IT staff and developers. Hospital patients are also a stakeholder because even though they will not directly interact with the system they will be affected by the outcome. Lastly, because this project is being developed for the CS 4ZP6 course, the course administrators, Dr. Carette and the 4ZP6 teaching Assistants (TAs) are also stakeholders. These groups will interact with the system and the project in very different ways discussed below.
Doctors and hospital administrators will primarily be handling the input of new data into the system and will be the ones requesting predictions from it. They do not have technical expertise with respect to the system so they would not be expected or required to access the API, model or data module directly. Rather they would only be interacting with the web app’s UI to perform data entry, modify parameters, and request time slot predictions. We estimate that they will use the system weekly to generate a prediction and weekly/monthly to do data input depending on who is doing the entry and their schedule.
The hospital IT staff and/or developers will be doing maintenance and modifications to the system. This includes editing or removing data from the data module, adjusting the model to change how it generates its result, and fixing any errors or issues that might occur. They have technical expertise and thus would have full access to every component including the source code. They will use the system on an ad-hoc basis depending on how often changes are needed or requested by the doctors and hospital administrators.
While hospital patients will never interact with the system or even be aware of the system it will affect them. Better scheduling will reduce patient wait-times and make it more likely that patients are seen on time. This will lead to better health outcomes for the patients.
Dr. Carette is the professor for and chief administrator of the Computer Science 4ZP6: Capstone Project course. All project deliverables required by the course have deadlines designated by him. He will also be doing the grading of the final deliverable.
The two teaching assistants for this course, Ethan Chan and Brendan Fallon will both assist Dr. Carette in administering the course. They will provide feedback and guidance on the project as well as mark many of the deliverables.
Our first assumption is that the hospital servers will be able to run the application. This includes the servers having enough capacity to store the necessary data and host the user interface, and the server having enough processing capacity so that our model can process all of the data.
Another assumption is that the system will be supplied with enough data so that the predictions generated by the model will be usable with a high degree of confidence (see 4.2.3 Reserved Slot Predictions for more detail). While the system is constrained by the amount of data it can access and all of the data we need is subject to PIPEDA we assume that there will still be enough anonymized data so that we can generate a trustworthy prediction.
Related to the assumption on the amount of data another assumption is that the data used by the system will be valid, meaning that the data accurately represents what we think it represents and captures the trends that we think it captures. The system will be able to clean and process the data which will improve the quality however if the underlying data is not valid then our system will not be able to generate useful predictions because it will be basing those assumptions on data that does not capture what the system thinks it does.
One dependency is that the system depends on the programming language and software packages that we are using for development. Changes to these packages may make certain functions of the system no longer work properly and if a package we use becomes obsolete we will need to use a new one. These software dependencies will need to be managed to ensure that we are not depending on unreliable or unstable software and that we are updating these dependencies in a timely manner.
For deployment of the system to be successful we will need to prepare a user instruction manual, a demo on how to use the system, and maintenance notes on how to modify and upkeep the model and data module. This will be done via github READMEs.
The user instruction manual will provide a detailed walkthrough of how the user interface works. It will detail how to do proper data input, how to select parameters and what those parameters might affect, and how to obtain and understand the result generated by the model.
To go along with the instruction manual we will also prepare a video demonstration providing step by step instructions on the process of working with the system.
Lastly, maintenance notes will be essential for the hospital IT team so that they modify the system, fix any issues or make any updates. These notes will help them understand how to access error logs, make changes to the source code, and understand how the model works.
Each requirement is apportioned according to what component it belongs to in the specific requirements detailed in the next section.
Automating data entry and integration with other hospital systems may be apportioned to future versions. Future versions may also increase the scope of the system to cover other departments in the hospital or other hospitals outside of the McMaster Children’s Hospital.
The users of the externally-facing UI are hospital medical staff excluding IT. We will have the following pages defined by the following UI components:
All pages will have the following features
| Feature | Description |
| Navbar |
|
With this navbar and collapsible hamburger menu, and the fact that there will not be any nested routes it should be really easy to navigate backwards. Using these common UI elements will add to the ease-of-use and ease-of-learning. Any further questions can be seen in the Help section of the hamburger menu.
The prediction and admin page allow users to configure the prediction parameters
- A form that allows user to enter the number and duration of intervals that they want predictions over.
- The results in text of the number of urgent patients seen, the configuration options selected, and the number of appointment slots needed. We will also display the confidence interval to show the range in which we expect the required slots to be within, as well as an info button to describe what it means and how it was calculated. By default this section is hidden, and after filling out the form this section will be populated.
Allows for configuration of more technical parameters
- A form which contains configurable options: X% service threshold; Y weeks that the prediction is made over. It has a button to save these configurations. When the make prediction button is clicked. There is one for each triage class.
- An option to upload historical data
The user info is accessible in the navbar. It displays the clinic name and user name.
The help page will list common FAQs on how to use the product, what to do if an error occurs or if the server is down, etc. This makes the product much easier to use.
- GPUs will be used for model training, but not a part of the deployed product
Components:
- Server host
- Data training
- Web client host
- Database host
Encapsulates the predictive model with a REST API via flask. It communicates with both the web client and the database server
| Communicates with | Web client |
| Nature of communication | HTTP. HTTPS is not needed since it is on an internal network. |
| Responds with (/) | A default health page for the API |
| Responds with (/calculate) | Number of slots for patients needed given X% service level of urgent patients within A time frame Y% service level of semi-urgent patients within B time frame, Z% of standard patients within C time frame |
| Communicates with | Database |
| Nature of communication | Asynchronous inter-process communication |
| Requests | A query result for a table in the database |
| Receives | The response to the query, or the PostgresQL error |
Libraries used:
- Flask - 1.1.2
- Psycopg2 - 2.8.6
- Swagger.io - 2.1.7
Used to train a predictive model it will use the following libraries:
- Tensorflow - r2.4
- Keras - 2.4.0
- Numpy - 1.2.0
- Pandas - 1.2.3
- Matplotlib - 3.4.0
Displays appointment slot predictions to hospital medical staff.
| Communicates with | Server |
| Nature of communication | HTTP. HTTPS is not needed since it is on an internal network. |
| Requests | Prediction given X% service level of urgent patients within A time frame, Y% service level of semi-urgent patients within B time frame, Z% of standard patients within C time frame |
| Receives | Total number of slots for all patients, or an error. |
Libraries
- Typescript - 4.2.3
- TsLint - 6.1.3
- Angular - 11.1.3
- Node - 15.11.0
Store hospital clinic information including arrivals, triage classes for those arrivals and dates for when patients arrived. It communicates with:
| Communicates with | Server |
| Nature of communication | Asynchronous inter-process communication |
| Receives | A query to a table in the database |
| Responds with | The response to the query, or the PostgresQL error |
| ID: INT-1 | Type: Functional Requirements (Web App Module) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
A form for user inputs taking in a configurable X% service level of urgent patients within A time frame, Y% service level of semi-urgent patients within B time frame, Z% of standard patients within C time frame |
|
Rationale: |
According to the client, ideally over 95% of urgent patients should be seen within 2 weeks, but it should be configurable if the healthcare worker wants an alternative deadline. |
|
Fit Criterion: |
A user-friendly form. On error return 500 server error. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Prediction Module |
| Code Triage-Frontend |
| ID: INT-2 | Type: Functional Requirements (Web App Module) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
The form has a send button. On send, a request is sent to the API server for a prediction on how many slots for urgent patients are needed. The data must be received by storing the API call in a variable that is returned by some function. |
|
Rationale: |
This is needed so that the front-end can call a black-box oracle which will suggest the number of totalslots. (The rationale as to why the call is needed can also be answered by the benefits of microservicing.) |
|
Fit Criterion: |
It returns the number of slots needed, and the confidence interval in a JSON-able format. On error return 500 server error. |
|
Priority: |
High |
| Final Code Submision |
|---|
| Design: see get prediction results under the Prediction Module Interface |
| Code Triage-Frontend/get-prediction |
| ID: INT-3 | Type: Functional Requirements (Web App Module) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Displaying the needed appointment slots and its CI. On error, display a pop-up indicating that there is an error in calculation due to the error response message sent from the API. |
|
Rationale: |
This is needed to fulfill the user-requirements. |
|
Fit Criterion: |
It must be legible. On error, display a pop-up indicating that it could not be calculated due the reason indicated in the API’s error emssage. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Prediction Module |
| Code Triage-Frontend |
| ID: INT-4 | Type: Functional Requirements (Backend API) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create a default route with path “/<baseurl>/” that returns the basic information of the endpoint |
|
Rationale: |
It is standard API practice |
|
Fit Criterion: |
Does not need authentication, and returns the endpoint name. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Rest API Spec Version 1 |
| Code Triage-Backend/api/triage_api.py |
| ID: INT-5 | Type: Functional Requirements (Backend API) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create the baseroute with a version i.e. “/v1” |
|
Rationale: |
API versioning helps with release notes |
|
Fit Criterion: |
Must be “v1” |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: REST API Spec Version v1 |
| Code Triage-Backend/api/triage_api.py |
| ID: INT-6 | Type: Functional Requirements (Backend API) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create a path that when called as a HTTP GET and passed the following parameters: X% service level of urgent patients within A time frame, Y% service level of semi-urgent patients within B time frame, Z% of standard patients within C timeline the API invokes a subroutine that returns the number of slots needed for the urgent patients |
|
Rationale: |
Customer requirement |
|
Fit Criterion: |
GET, takes at least 2 parameters, returns a number in a JSON response. On error return 500 internal server error. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see get v1/classes under REST API SPEC Version v1 |
| Code Triage-Backend/api/resources/update_triage_classes.py |
| ID: INT-6 | Type: Functional Requirements (Backend API) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create a subroutine invoked by the API that returns the number of slots needed and some other stats by invoking the simulation model for each triage classs to return the minimum total number of slots for the coresponding triage class. Therefore if there are n triage classes, take as parameters the following: X1%, ..., Xn% of patients (for each triage class) at least should be serviced within Y1,..., Yn time units respectively, a confidence level, the arrival schedule. Calculate n slot allocation schedules each satisfying at least Xi% patients are seen within Yi time units. |
|
Rationale: |
Customer requirement |
|
Fit Criterion: |
Takes at least following parameters: X1,...,Xn; Y1,...,Yn; the number of appointment slots available at the clinic; and the prediction length. The prediction length is passed to the simulation model which returns the expected influx of patients according to their triage class. On error, throw and log error message. Otherwise, maximize the number of patients seen while meeting the constraints given by the floor constraint Xi and the exact constraints Yj and the number of slots available, else implement a greedy algorithm favouring urgent patient criteria satisfaction. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see GET v1/predict under REST API Spec version v1 |
| Code Triage-Backend/api/resources/predict.py |
| ID: INT-7 | Type: Functional Requirements (Backend API) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create a path that when called as a HTTP POST and passed a CSV file in the body it invokes a subroutine to load the CSV into the database |
|
Rationale: |
User requirement |
|
Fit Criterion: |
POST request that takes the csv and loads it to a subroutine |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see GET v1/upload/past-appointments under REST API Spec version v1 |
| Code Triage-Backend/api/resources/upload.py |
| ID: DAT-1 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
Create a subroutine that is invoked by the API to load a CSV into the database |
|
Rationale: |
Will reduce manual labour of loading data to the training model |
|
Fit Criterion: |
Successfully writes all relevant fields of the CSV fields to the database in the same table |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: see PUT v1/upload/past-appointments Under REST API Version v1 |
| Code Triage-Backend/api/resources/upload.py |
| ID: DAT-2 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Data from the datastore module is injected into the prediction model |
|
Rationale: |
The prediction model needs to use data to make the predictions |
|
Fit Criterion: |
Data is able to be injected into the prediction model in a format that is compatible with the model |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: ML Training Module |
| Code Triage-ML-Training/data |
| ID: DAT-3 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Data from the data module is injected into the prediction module |
|
Rationale: |
The training module needs to use data to perform prediction and validation |
|
Fit Criterion: |
Data is able to be injected into the training prediction in a format compatible with the training process |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: See ML Training Module and Database Module |
| Code Triage-ML-Training/data |
| ID: DAT-4 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Storage of anonymized patient data |
|
Rationale: |
The patient data used for prediction must be available |
|
Fit Criterion: |
The data is available in the data module or through a HHS provided API |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see Database Module |
| Code Triage-Backend/data/build_tables.sql |
| ID: DAT-5 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Storage of weights for trained prediction models |
|
Rationale: |
Easily accessible model weights will aid in simple deployments |
|
Fit Criterion: |
Trained model weights are accessible in the data module |
|
Priority: |
Low |
| Final Code Submission |
|---|
| Design: see Models under Database Module |
| Code Triage-ML-Training/triage_ml/train_radius_variance.py |
| ID: DAT-6 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Storage of program logs |
|
Rationale: |
Storing logs enables better debugging when errors occur |
|
Fit Criterion: |
Logs for all Triage services for the past 14 days are accessible in the data module |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: Program logs found in docker container used for deployment. |
| Code Triage-Backend/Dockerfile |
| ID: DAT-7 | Type: Functional Requirements (Datastore Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Storage of client information |
|
Rationale: |
Information about the client clinics is needed to organize our system |
|
Fit Criterion: |
Data about the clinics is available in the data module |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see Clinic under Database Module |
| Code Triage-Backend/data/build_tables.sql |
| ID: MOD-1 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Prediction module can take in data from data module |
|
Rationale: |
The training process relies on data stored in the data module |
|
Fit Criterion: |
The training data must be injected in a format compatible with the training process |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see Secrets under ML Training Module |
| Code Triage-ML-Training/data |
| ID: MOD-2 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Prediction module can perform training (likely SGD) to converge on meaningful results |
|
Rationale: |
The training process must produce useful results |
|
Fit Criterion: |
The model converges on a non-trivial local minimum |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: ML Training Module |
| Code Triage-ML-Training/triage_ml/train_radius_variance.py |
| ID: MOD-3 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Prediction module can write the model weights to disk |
|
Rationale: |
The trained model must be stored for use in the model module |
|
Fit Criterion: |
The trained model can be written and loaded from disk |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see database module and ML Training Module |
| Code Triage-ML-Training/triage_ml/train_radius_variance.py |
| ID: MOD-4 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Kristie |
|
Description: |
The prediction model can be validated using test data |
|
Rationale: |
The test data is used to validate the model |
|
Fit Criterion: |
The data must be received from the datastore module and be in a format compatible with the validation process |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: see database module and ML Training Module |
| Code Triage-ML-Training/TriageValidation.ipynb |
| ID: MOD-5 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
The validation data is used to test the performance of the trained model |
|
Rationale: |
The results of training must be measured to ensure convergence has occurred |
|
Fit Criterion: |
A numeric score of the model is produced, allowing comparisons between versions of the model |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: see ML Training Module |
| Code Triage-ML-Training/TriageValidation.ipynb |
| ID: MOD-7 | Type: Functional Requirements (Prediction Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Data is used to produce a prediction by feeding it into the model |
|
Rationale: |
The prediction is used to estimate the projected volume of patients |
|
Fit Criterion: |
Data is able to be sent to the prediction model module and a prediction is returned |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see database module and ML Training Module |
| Code Triage-ML-Training/train.py |
| ID: SIM-1 | Type: Functional Requirements (Simulation Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
Predicted arrivals from the prediction model module is used to predict the total number of expected slots needed slots needed to address those arrivals per triage class |
|
Rationale: |
This result is used by doctors to create a safe schedule |
|
Fit Criterion: |
The prediction is computed |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Simulation Module |
| Code Prediction Module |
| ID: SIM-2 | Type: Functional Requirements (Simulation Module) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
The simulation module can be backtested on old data and an accuracy can be computed |
|
Rationale: |
It is important for the accuracy of the results to be verifiable |
|
Fit Criterion: |
The proportion of times the predicted reserve slots are sufficient for the actual patient inflow should be returned |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: Simulation Module |
| Code Prediction Module Note: the design of the sim itself allows for backtesting |
| ID: PRF-1 | Type: Non-Functional Requirements (Performance) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
The web application is able to return results relatively quickly. |
|
Rationale: |
Users may require quick schedule estimations to make on the spot decisions. |
|
Fit Criterion: |
Users are able to retrieve their results within 10 seconds. |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: N/A |
| Code using our current front end we can get a prediction in under 10 seconds |
| ID: SEC-1 | Type: Non-Functional Requirements (Security) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
Data is stored securely, such that patient privacy is upheld. |
|
Rationale: |
Patient information could become publicly available failing to meet the PIPEDA compliance |
|
Fit Criterion: |
Ensure that data storage follows PIPEDA compliance guidelines. |
|
Priority: |
Very High |
| Final Code Submission |
|---|
| Design: The database schema shown under Database Module has been confirmed by the hospital to be PIPEDA compliant |
| Code: build_tables |
| ID: SEC-2 | Type: Non-Functional Requirements (Security) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
Data is cautiously utilized such that patient privacy is upheld. |
|
Rationale: |
Patient information could become publicly available failing to meet the PIPEDA compliance |
|
Fit Criterion: |
Ensure that data usage follows PIPEDA compliance guidelines. |
|
Priority: |
Very High |
| Final Code Submission |
|---|
| Design: The database schema shown under Database Module and its usage for prediction has been confirmed by hospital staff to conform to all necessary standards. |
| Code: N/A this requirement is met by the overall design of the project not by any particular piece of code |
| ID: SEC-3 | Type: Non-Functional Requirements (Security) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
The web application is only accessible by trusted users, which is assumed to be hospital staff. |
|
Rationale: |
Users that are not authorized to access the project services should not be able to utilize the web interface. |
|
Fit Criterion: |
The web application is not accessible to users that do not work for the hospital (Hamilton Health Sciences) |
|
Priority: |
Very High |
| Final Code Submission |
|---|
| Design: see GET v1/auth under REST API Spec version v1 |
| Code Triage-Backend/api/resources/auth.py and Triage-Backend/api/resources/AuthResource.py |
| ID: SEC-4 | Type: Non-Functional Requirements (Security) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
The API is only accessible by client applications. |
|
Rationale: |
Unauthorized users should not be able to access the API to communicate with the model. |
|
Fit Criterion: |
The API is not accessible to users that do not work for the hospital (Hamilton Health Sciences) |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Deployment through docker containers ensures that only the frontend container can communicate with the API container. (for containerization information see Triage-Backend/Dockerfile) |
| Code: see Triage-Backend/Dockerfile |
| ID: REL-1 | Type: Non-Functional Requirements (Reliability) |
| Related PUC: N/A | Originator: Sam |
|
Description: |
All model predictions shown to the end users must be quantified with a confidence interval. |
|
Rationale: |
HHS needs to know how accurate the predictions are in order to justify their use for scheduling patients |
|
Fit Criterion: |
An accuracy measure is always displayed to the user alongside any predictions |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: Prediction Module |
| Code Triage-Frontend |
| ID: AVA-1 | Type: Non-Functional Requirements (Availability) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
Backups of model weights should be stored for recovery purposes. |
|
Rationale: |
Training the model should be reversible in case of unexpected circumstances. |
|
Fit Criterion: |
The model training is reversible in case of emergency. |
|
Priority: |
High |
| Final Code Submission |
|---|
| Design: see GET v1/upload/models under REST API Spec version v1 |
| Code Triage-Backend/api/resources/auth.py |
| ID: AVA-2 | Type: Non-Functional Requirements (Availability) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
Detailed logs of usage and error occurrence must be stored. |
|
Rationale: |
Tracing errors to investigate downtime causes should be possible. |
|
Fit Criterion: |
Logs can be used to identify availability difficulties. |
|
Priority: |
Medium |
| Final Code Submission |
|---|
| Design: Program logs found in docker container used for deployment. |
| Code Triage-Backend/Dockerfile |
| ID: AVA-3 | Type: Non-Functional Requirements (Availability) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
Services should be monitored with automated restart functionalities. |
|
Rationale: |
Without these features, the system would require constant manual monitoring resulting in longer downtime periods. |
|
Fit Criterion: |
The project is able to monitor and maintain uptime automatically. |
|
Priority: |
Low |
| Final Code Submission |
|---|
| This is based on project deployment not on the code or the design |
| ID: CMP-1 | Type: Non-Functional Requirements (Compliance) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
All usage/storage of data must follow PIPEDA data standards. |
|
Rationale: |
This is a strict legal requirement. |
|
Fit Criterion: |
All data is stored, protected, and analyzed in a manner compliant with PIPEDA guidelines. |
|
Priority: |
Very High |
| Final Code Submission |
|---|
| Design: The database schema shown under Database Module and its usage for prediction has been confirmed by hospital staff to conform to all necessary standards. |
| Code: N/A this requirement is met by the overall design of the project not by any particular piece of code |
| ID: CMP-1 | Type: Non-Functional Requirements (Compliance) |
| Related PUC: N/A | Originator: Nikhail |
|
Description: |
All development, documentation, and course deliverables must comply with the guidelines set by the 4ZP6 course. |
|
Rationale: |
This is a requirement for the 4ZP6 course for which this project is being developed. |
|
Fit Criterion: |
All course deliverables, documentation, and development is compliant with the standards and requirements set by the 4ZP6 course. |
|
Priority: |
Very High |
| Final Code Submission |
|---|
| N/A |
Installation of the client application will be a fairly streamlined process given that users will access services through a web interface. This web application must be hosted on hospital servers along with the API that allows it to communicate with the simulation model module to make predictions given the users input data. Once these two sub-modules are successfully installed, trusted users, such as doctors and hospital staff, within the network will be able to access the web application through their computer browsers.
The simulation model module as well as any tools for automating the process of training and validating the model will also be hosted on hospital owned servers. This module will only require an initial setup.
All data and data manipulation services will be hosted on hospital owned servers. Depending on the preferences of the IT department at the McMaster Children’s Hospital, data will either be stored within a dedicated database for the project, or will be pulled from a view of hospital database tables exposing only the necessary data for training / validating the module. This module will only require an initial setup.
We therefore will have every repository manually creating a Dockerfile which can be installed on the corresponding servers. Therefore it is expected of users to have Docker installed on their environment. The entire setup will be documented in GitHub README.md files.
Given that users access the services of this project through a web application, distribution of the product will be accomplished through sharing the various user interface endpoints with doctors after the software has been installed and deployed on the hospital network. We will guarantee support for the latest Google Chrome.
By ensuring separation of concerns through modularity (see the description of our 3 major modules and their sub-modules above), the project should have a clear structure allowing for easier maintainability.
Utilizing CI/CD tools to build automated testing (unit and integration) and deployment pipelines would further increase the ease of developing and maintaining the project. Developers will be able to modify, add, and deploy changes to the project with ease while ensuring the correctness and reliability of the software.
Lastly, by developing clear and concise documentation outlining not only features of the project but also best practices and code standards (ex. Code review practices, testing requirements, etc.) the team should ensure that development methods are formally established to promote efficient modification and maintenance of the codebase.
By ensuring clear modularity, we ensure that various aspects of the project are reusable for similar use cases.
The web application connects to a backend API that runs model prediction and analyzes or modifies the results to display them to the user. This communication will be well defined in order to allow for the web application to be easily modified to connect to any API that follows the communication design.
The data and model modules will communicate to provide the model with data for training and validation. This infrastructure can similarly be reused with various datasets and models given that their communication conforms to the standards set out by these modules.
Given that the schema of data used for training the model and as inputs for the web application will be formalized, the application will be reusable for any clinic that is able to provide this information.
Portability of the application can be broken down into two categories, front end portability and back end portability. For front end portability, the application should be portable to any machine with a modern compatible web browser and appropriate security permissions by default. This will ensure the application is portable to any hospital staff using the application. This web application should also be easily deployable to any server that is ready to accept a modern front end framework such as angular. The backend of the application may require specific computing resources such as computing power not easily portable to other platforms other than the selected deployment service. However, the application will be OS-agnostic since it will run on Docker.
| Factor | Description | Cost |
| Software Licensing | All software tools used will be free. | None |
| Development Time | 8 Months (5 developers) working at a rate of $0/hour. $0 total development cost. | 8 Months of Development Time
$0 |
| Factor | Description | Cost |
| Software Licensing | Given that all software tools are free, no licensing must be renewed. We will use the standard open source Apache 2.0 license | None |
| Application / Data Monitoring | The application and data will be hosted and monitored from tools on the hospital servers mitigating all costs. | None |
| Development Time | Dependant on maintenance requirements (i.e. debugging, increasing functionality, etc.) | None |
| Factor | Details | Cost |
| Application / Data Hosting | All application modules and data will be hosted on hospital servers mitigating all costs. | None |
| Infrastructure Monitoring | Monitoring of the infrastructure itself will be the responsibility of the IT department of the hospital. | None |
| Due Date | Milestone |
| October 2, 2020 | Problem Statement |
| October 19, 2020 | Requirements Document (SRS) |
| October 30, 2020 | Prototype 1 |
| November 13, 2020 | Design |
| February 1, 2021* | Prototype 2 |
| March 1, 2021* | Internal and External Testing |
| April 1, 2021* | Final Project Submission |
| April 1, 2021* | Internal Project Demo |
| April 5, 2021 | Faculty of Engineering Capstone Day Materials |
*Subject to 4ZP6 schedule changes
As a health care application, Triage may be subject to a verification process by the McMaster Children’s Hospital to ensure correct behaviour and handling of patient data. The exact scope of a hospital audit still remains to be determined and is tentative on the Hospital’s confirmation. We are actively working with an IT specialist at Hamilton Health Sciences to ensure we are meeting their guidelines and requirements. As part of our planning process to meet deadlines, this overhead is considered.
Automated testing automatically ensures the correctness of software in defined situations. Triage will include “smoke tests” to quickly ensure the software is functioning normally after code changes. It will also include unit and integration testing in areas of high complexity or where bugs are uncovered during development. These tests will be focused primarily on our backend API, models, and simulations.
Manual testing is an important part of development and will be the main method used to verify the Triage web interface. Part of the manual testing process involves the end user. While developers can take steps to ensure the product matches the agreed upon requirements, it is impossible to specify requirements perfectly, making the solicitation of feedback from clients an important part of the development process. By letting clients run through the Triage critical user journeys, we can gauge the usability of the product and take steps to adapt it to work best for them.
The proportion of patients being triaged into each category for a given clinic is critical in determining the number of slots to reserve for the high priority categories. For this reason, we will produce a value with a 95% confidence interval. This interval will be determined statistically.
The expected volume of patients in the future will be predicted using historical volumes. To verify the results, we will use k-fold cross validation [4] to estimate the accuracy of the predictions on new data. The results of this verification do not provide an absolute measure of accuracy; they provide a relative accuracy we can use to compare different instances of our predicted models.
The reserved slot predictions are created by using the triage result and patient volume predictions. These will be combined with the current wait list in a simulation that can be used to backtest the predictions on historical data. This produces an exact proportion of time the predicted reversed slots are sufficient for the actual patients.
- From 3.1.1.1, a user settings page: "This would be to enable colour modes, font adjustments for readability, and user administration." This is covered under the Admin Page. Particular fonts and colours are no longer a requirement from the user.
From 3.3.1: the requirement below is deprecated because the client wanted to be able to control when the model is retrained.
| ID: PRF-2 | Type: Non-Functional Requirements (Performance) |
| Related PUC: N/A | Originator: Daniel/Shawn |
|
Description: |
The model is able to train on data as it becomes available. |
|
Rationale: |
Scheduling decisions should be made taking into account recent historical data. |
|
Fit Criterion: |
Model training should automatically run on new data as it becomes available. |
|
Priority: |
Medium |
- Originally from section 3.1.1.1, a button to clear the form and the page contents. This will be displayed only when (2) is displayed
- Originally from 3.1.1.1, if the server is down then the form should not be loaded and instead an error page should be displayed. Therefore, when clicking the prediction page, a non-blocking asynchronous call should be made to check if the server is alive. If it is down, the prediction page should be replaced with an error screen instructing the user to contact the Hospital IT staff, and a hyperlink to the FAQ page. This will help the ease-of-use in the event of the asynchronous requests not working.