Questions and Answers

• We are working with a linked dataset containing multiple hospital indicators of financial performance (annual revenue, costs, etc), operational infrastructure (number of physicians, staff in the ICU, etc), among other characteristics of the hospital. Our goal is to develop a score that will predict whether a hospital will close in the following year or not. Is this score an index or a scale? Why?

  The difference is as follows:

  1. in a scale, the construct dominates the response of the items. for example, depression dominates the answers to questions such as "did you feel sad this past week?" This means that the arrows go from the construct to the item. Analysis methods include EFA and CFA
  2. in an index, the items are a measurement that contributes to the index, and so the arrows go from the item to the construct. methods include PCA (principal component analysis), although you could use EFA/CFA. PCA does not assume an error in the measurement of each item.

  In my original question, it would an index since things like number of physicians and others are components of this score -- the arrows go from the item to the index

• What does alpha spending mean? Why is it used? Why did the p value change and what will be the implications of that change when assessing whether the intervention is or not significantly associated with an outcome? Alpha spending is a function to decrease p values depending on the number of interim analyses, meaning that the higher the number of interim analysis event, the lower the p value you will have to have in order to call something a statistically significant event. In other words, it will be more difficult to call one treatment different (better) than another. The reason for decreasing the value of p is that the more interim analyses one makes, the higher the chance of having spurious significant p values.
  The alpha spending function, or how you calculate the decrease in the p value, is related to the percentage of the sample size that has completed the trial at the time of interim analysis (or the people who have full information about the outcomes at the point of the interim analysis), which is the so-called information fraction. This information fraction is then taken into account when calculating the probability of an analysis rejecting the null hypothesis (i.e., having a significant p value) when the null hypothesis is true. Alpha Spending Function approach.

• Which strategic factors will be important in your decision in deciding whether a proposal has a fair chance of winning?
  Some of the strategic factors are the participation of important institutions in the Consortium, participation of top researchers (with recent publications at high-impact journals), and alignment of the proposal with the announcement.

• What are the three criteria used in PubMed search to find the core idea for a proposal?
  Search on high-impact journals (using our filter), search for meta-analyses, reviews, and systematic reviews. These types of studies usually have a section on gaps in the literature that we can use to identify what has not been done yet, articles published in the past year. For instance:

  1. You were asked in a recent meeting with two research groups from Greece and Italy, where they were brainstorming about ideas on how to use mammography data (they are specialized in computer vision) in an innovative proposal to the European Commission. The proposal is medium size for European standards (6 million euros split across the multiple participating sites), so it has to be truly innovative. One of the researchers said that a challenge with mammography is that it is used everywhere (it's recommended once/year or once every two years, depending on age and other factors) and that it is tough to come up with a new idea -- here is what he did: He ran a PubMed search combining (a) the IEO as an institution since I needed to hook it up to them (they are our clients), (b) cancer screening (since that was the topic of the funding announcement) and (c) some big-name journals (a subset of our usual filter, where I only picked stuff like Nature, Science, NEJM); 2. sorted it out by dates because I needed to come up with something recent; 3. started going through and found the perfect paper, since it combined mammography with -omics material from pap smear cells, making it a perfect fit for the proposal -- of importance, I had a general idea of what they wanted.

• Is it possible to use the HCUP database to evaluate the distribution of CABG vs. PCI procedures around the country?
  Yes, it is possible through a linkage with the AHA survey, which has zips for every hospital where the procedures were performed. AHA also has thousands of variables about each hospital, which is interesting. With that said, the researcher should be warned that he/she will have to get AHA separately from the HCUP dataset.

• What is your first question to ask a researcher at the beginning of a study?
  What are the three mock conclusions you would like to state at the end of your study. Imagine presenting your results at the top conference of your specialty, and your last slide will summarize the three things you learned from the study. What would these three sentences say?

  1. A researcher wants to use the N3C dataset to evaluate complications after robotic hernia procedures. What is your first question?-- the answer is the same.

• What are the differences between charges and costs?
  Charge is cost + profit (also called margin for non-profit hospitals), while cost is a hospital's amount to provide a given service or product. Costs are more important in most circumstances since profit margins vary across hospitals from a research perspective. The amount the patient pays is not a cost; it's a charge. Also, hospital charges are not necessarily four times the cost, as the cost to charge ratio varies depending on the service/product as well as the hospital

• What does variable direct cost mean?
  Variable direct cost has two components:

  • Variable means that it varies depending on the amount of care being provided. For example, the cost of the building is part of the total cost (and patients/insurers pay for it), but it doesn't vary depending on the amount of care that is being provided.
  • Direct means that it is directly applied to the case. For example, health insurance paid to nurses is an indirect cost, which patients and insurers also pay for, but it's not directly related to their care. Costs directly related to care are part of the variable direct cost, but that's not the whole definition.

  Any datasets you would recommend?
  Probably the most important dataset with the direct variable cost is CMS claims, which has detailed information on not only direct variable costs, but also splits those costs per category (devices, nurses, drugs, etc.) and is specific for each hospital for the entire US. Camila, SASD is part of a large family of datasets called HCUP, and it does have cost, but not variable direct costs.

• Below are the mock conclusions of a certain article we received recently. Which methods would you apply to this article based on the mock conclusions? 1- The underlying etiology for pediatric benign esophageal strictures changes with age. 2- Treatment for pediatric esophageal strictures is likely stepwise, with dilations and injections occurring multiple times prior to surgical intervention. 3- Outcomes for esophageal strictures are generally good with a high rate of oral nutrition.

  1. It is likely that the researcher who formulated these mock conclusionslacks extensive research experience, so, I would probably send over some use cases as the design evolves.
  2. The answer to all three mock conclusions in their current form is exploratory analysis, and maybe some simple inferential tests for mock conclusion #1, with #3 potentially being addressed through propensity scores
  3. Despite the numerous predictor variables (e.g. age, injections, oral nutrition) influencing the outcome that make stepwise regression a logical choice, stepwise regression is somewhat problematic - check this https://journalofbigdata.springeropen.com/articles/10.1186/s40537-018-0143-6

• How are death status and dates ascertained in CDW?
  Ascertaining Veterans' Vital Status: VA Data Sources for Mortality Ascertainment and Cause of Death answers this question.

• Modern causal evaluation is based on three main principles, one of them being called exchangeability. What is exchangeability?
  Exchangeability, in other words, means there should be no unmeasured confounding factors. In causal evaluation, it assumes equivalent distribution of the independent predictors of the outcome between the treated and the untreated groups. The two groups are exchangeable with respect to an outcome measure because if the subject had remained untreated in the treated group, he/she would have experienced the same average outcomes as the untreated and vice versa. The lack of exchangeability may lead to or amplify the bias.

• A group of researchers sent a dataset with several items (questions represented as variables) that they believe represented constructs (also called domains, factors, or latent variables). Each cluster of items was clustered around a given domain, for example, five items for leadership, seven items for job satisfaction, etc. They then posed a question regarding the impact of one construct on another. But even before getting to the issue of evaluating that question, what would be your sequence of analysis for this project? The usual sequence is data management, imputation, correlation plots (heatmaps), scree plots, EFA, CFA. at that point we would be ready for SEM to evaluate the association among latent variables. However, if the goal is to use just the score on some other type of model, we might proceed to scoring (congeneric or using item loads as weight vs. summation) and possibly validation and reliability (the latter usually through Cronbach's alpha).

• We talked about exchangeability as a causal principle in the past. But there are also the positivity and consistency requirements. What are they?

  1. Positivity: In a causal study, there should be a probability greater than zero (a positive probability) of being assigned to each treatment level. For example, we are doing a study comparing two types of pre-anesthesia evaluation across hospitals. But in some hospitals, one of the types of evaluation doesn't exist. Therefore, a patient attending that type of hospital has a zero probability of getting that intervention. As a result, positivity is not present, and you can't make a comparison. To make the comparison, you would have to restrict the sample to hospitals that do both types of assessment.
  2. Consistency: Consistency means that the observed outcome for every treated individual equals their outcome if they had received treatment and that the observed outcome for every untreated individual equals their outcome if they had remained untreated. For example, there is a difference in how we treated COVID in early 2020 and now. We didn't have vaccinated patients then, and some people tried all kinds of crazy medications (like Trump and Bolsonaro advocating for hydroxychloroquine sulfate). So, a question asking if the treatment for COVID works doesn't make sense from a causal perspective since there is no consistency in the intervention because the intervention changed a lot during that period.

  • Is it possibleto use the HCUP database to evaluate the distribution CABG vs PCI procedures around the country? Yes, it possible through a linkage with the AHA survey, which has zips for each and every hospital where the procedures were performed. AHA also has thousands of variables about each hospital, which is interesting. With that said, the researcher should be warned that he/she will have to get AHA separately from the HCUP dataset.

  • what is the difference between a classical psychometric and the latent variable approach? Classical Test Theory (CTT) was designed to quantify measurement error and to address related problems such as correcting observed dependencies between variables (e.g., correlations) for the attenuation due to measurement errors. True score and measurement error variables are fundamental concepts in CTT. These concepts are defined as specific conditional expectations and its residual, respectively. The true score and error variables are assumed in CTT models, which allows the theoretical parameters (such as true score variance and error variance) to be identified from the variances and covariances of the observable measurements (test score variables).

  CTT has several weaknesses that have led to the development of alternative test scores. First, the concept of reliability is dependent on the group used to develop the test. If the group has a wide range of skill or abilities, then the reliability will be higher than if the group has a small range of skill or abilities. Apart from that, the common notion that the standard error is the same for test takers of all abilities is frequently inaccurate. CTT also fails to account for observed test score distributions with floor and ceiling effects, in which a considerable number of test takers fall on one side of the test score range.

  These problems of CTT are partly due to some haziness in the theory, in which the population to be sampled is usually not considered in any detail in the theory. The problems are also due to the fact that most data collecting is done on a random sample from any demographic, let alone one that's relevant to the test under investigation. In practice, convenience samples are employed; these are samples that meet certain criteria set by the investigator but are otherwise taken as they are available to the researcher.

  The latent variable models differ significantly from those used in classical test theory. In latent variable models, one creates a formal framework that correlates test scores to the predicted attribute, deduces empirical implications, and assesses the model's adequacy by looking at the goodness of fit with empirical data. By selecting an appropriate item response function, the model links expected item responses to a latent variable. This function formulates a regression of the item score on a latent variable. The expected item response may be interpreted in two ways: as a true score (following a stochastic subject interpretation), or as a subpopulation mean (following a repeated sampling interpretation). The model is best interpreted in a realistic manner. Thus, latent variable theory goes beyond classical test theory in that it attempts to construct a hypothesis about the data-generating mechanism in which the attribute is explicitly represented as a latent variable

  • The UNOS dataset has a variable about the last time a patient was contacted, whether the patient is alive or dead, and also whether the patient was lost to follow-up. The clinical researcher wants to run a time to event analysis, and is asking which variables are necessary for that analysis. What would you say? you only need two of them: whether patients had or didn't have the event, and when they were last contacted. The reality is that for the purposes of a time to event analysis it wouldn't matter whether they are marked as lost to follow-up because that will be implicit in the previous two variables

  • Pros and cons of using polynomial regression vs generalized additive models to measure dose-response Polynomial regression offers a more flexible fitting of the curve to the data than linear regression. The major drawback of a fitted curve is that it is obtained by global training, and ideally, adapting locally within subregions would be preferred, but that can be fixed using step functions or splines. Another disadvantage is that it may be overfitted depending on how much you increase the polynomial degree (Runge phenomenon—oscillations at the edges of an interval).

  Moreover, if the association is more complex, with a high number of potential predictors, or if the association is highly non-linear, it is preferable to use other methods such as GAM. GAM's advantage is that it is much more flexible and generalizable because we can have a different smooth function fj for each Xj. This comes with higher computational complexity and a high propensity for overfitting due to over-smoothing. That can be fixed by reducing "wiggliness" by using a smoothing parameter.

  Polynomial regression would be the best fit for dose-response analysis, mainly because the data has a lower complexity level, so GAM analysis could be over-smoothed.

  • During the design of a project, which aimed at comparing patients who underwent non-surgical (i.e., antibiotics) vs. surgical treatments, all patients who underwent non-surgical treatment and all patients who underwent surgical treatment were selected, regardless of undergoing antibiotics treatment first. What is the problem with this selection, keeping in mind the intention-to-treat principle? How should this selection have been made? Under ITT (see article, study participants are analyzed as members of the treatment group to which they were randomized regardless of their adherence to or whether they received the intended treatment. So, the problem was analyzing patients who first received antibiotics and later surgery as surgical patients. To solve this, patients who underwent surgical treatment after an antibiotic treatment must be kept in the "non-surgical" group, which was the first treatment.

  Note that eliminating study participants who were randomized but not treated or moving participants between treatment groups according to the treatment they received would violate the ITT principle.

Additional information

Mock Conclusion

One thing that tends to help with large database analyses is to "start from the end." For example, if you were to talk about the results of a project at the main conference for Pediatric Surgery, which three conclusions would you have in your final slide? Don't worry about them being right or wrong -- we will check that with the data -- but I would only focus on what you would like to say to make the conclusions both novel and clinically relevant. Having those mock conclusions will help us work backward and build the analysis strategy to test whether those results might or not hold.

high impact journals

("Proceedings of the National Academy of Sciences of the United States of America" [Journal] OR "Science (New York, N.Y.)" [Journal] OR "Health affairs (Project Hope)" [Journal] OR "J Am Med Assoc" [Journal] OR "JAMA" [Journal] OR "Med Care" [Journal] OR "J Clin Epidemiol" [Journal] OR "Am J Epidemiol" [Journal] OR "Epidemiology" [Journal] OR "Health Serv Res" [Journal] OR "Int J Epidemiol" [Journal] OR "Lancet" [journal] OR "J Am Coll Cardiol" [journal] OR "Br Med J" [journal] OR "Ann Intern Med" [Journal] OR "JAMA internal medicine" [Journal] OR "JAMA dermatology" [Journal] OR "JAMA dermatology" [Journal] OR "jama ophthalmology" [Journal] OR "JAMA surgery" [Journal] OR "JAMA neurology" [Journal] OR "JAMA pediatrics" [Journal] OR "JAMA otolaryngology-- head & neck surgery" [Journal] OR "JAMA oncology" [Journal] OR "JAMA psychiatry" [Journal] OR "JAMA cardiology" [Journal] OR "JAMA facial plastic surgery" [Journal] OR "Nature" [Journal] OR "Nature communications" [Journal] OR "Nature medicine" [Journal] OR "Nature biotechnology" [Journal] OR "Nature genetics" [Journal] OR "Nature neuroscience" [Journal] OR "Nature cell biology" [Journal] OR "Nature immunology" [Journal] OR "Nature materials" [Journal] OR "Nature methods" [Journal] OR "Nature reviews. Drug discovery" [Journal] OR "Nature protocols" [Journal] OR "Nature reviews. Microbiology" [Journal] OR "Nature: New biology" [Journal] OR "The Lancet. Oncology" [Journal] OR "Lancet (London, England)" [Journal] OR "The Lancet. Infectious diseases" [Journal] OR "The Lancet. Neurology" [Journal] OR "The Lancet. Respiratory medicine"[Journal] OR "The New England Journal of Medicine"[Journal] OR "JAMA network open"[Journal] OR "bmj"[Journal] OR "annals of internal medicine"[Journal] OR "the american journal of medicine"[Journal] OR "jco clinical cancer informatics"[Journal] OR "jco glob oncol"[Journal] OR "jco global oncology"[Journal] OR "jco oncol pract"[Journal] OR "jco clin cancer inform"[Journal] OR "PLoS medicine"[Journal] OR "The Journal of pediatrics"[Journal] OR "jci insight"[Journal] OR "Journal of internal medicine"[Journal] OR "journal of general internal medicine"[Journal] OR "the journal of adolescent health official publication of the society for adolescent medicine"[Journal] OR "European Heart Journal"[Journal] OR "ca a cancer journal for clinicians"[Journal] OR "bulletin of the world health organization"[Journal] OR "PloS one"[Journal] OR "American Journal of Preventive Medicine"[Journal] OR "Pediatrics"[Journal] OR "Circulation"[Journal] OR "mayo clinic proceedings"[Journal] OR "The American Journal of Gastroenterology"[Journal] OR "canadian medical association journal"[Journal] OR "annual review of public health"[Journal] OR "Academic medicine : journal of the Association of American Medical Colleges"[Journal] OR "The American journal of psychiatry"[Journal] OR "Annals of surgery"[Journal] OR "American Journal of Respiratory and Critical Care Medicine"[Journal] OR "Health services research"[Journal] OR "American journal of surgery"[Journal] OR "health affairs"[Journal] OR "bmc medicine"[Journal] OR "clinical medicine london england"[Journal] OR "journal of the american college of cardiology"[Journal] OR "american journal of public health"[Journal] OR "medical care"[Journal] OR "journal of aging and health"[Journal] OR "nature reviews immunology"[Journal] OR "epidemiologic reviews"[Journal] OR "nature reviews disease primers"[Journal] OR "the medical journal of australia"[Journal] OR "american journal of epidemiology"[Journal] OR "annals of family medicine"[Journal] OR "international journal of general medicine"[Journal] OR "proceedings of the national academy of sciences of the united states of america"[Journal])