ReaderBench Model 1b Variable Importance - shmercer/writeAlizer GitHub Wiki

Ensemble Weightings and Metric Importance

ReaderBench Model 1b

This model used ReaderBench scores from 7 min narrative writing samples ("I once had a magic pencil and ...") from 131 students in the winter of Grades 2-5 (Mercer et al., 2019) to predict holistic writing quality on the samples (elo ratings calculated from paired comparisons).

Highly correlated ReaderBench metrics (r > |.90|) were excluded during pre-processing (see section on Scoring Model Development for more details).

Mercer, S. H., Keller-Margulis, M. A., Faith, E. L., Reid, E. K., & Ochs, S. (2019). The potential for automated text evaluation to improve the technical adequacy of written expression curriculum-based measurement. Learning Disability Quarterly, 42, 117-128. https://doi.org/10.1177/0731948718803296

Algorithm Weightings in Ensemble

Abbreviations:

  • all = ensemble model
  • gbm = stochastic gradient boosted trees
  • pls = partial least squares regression
  • svm = support vector machines
  • enet = elastic net regression
  • rf = random forest regression
  • mars = bagged multivariate adaptive regression splines
  • cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept gbm pls svm enet rf mars cube
-2.0039 0.3112 0.1353 0.2667 -0.0102 0.1234 0.0268 0.222

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

ReaderBench metric names can be found here. Some metric names are abbreviated more to improve the display of the table.

Metric all gbm pls svm enet rf mars cube
WdEnt 11.34 23.56 2.43 1.65 0.74 3.11 31.83 12.9
AvgDepsBl_det 6.98 12.06 2.13 1.17 2.37 2.1 9.48 12.41
AvgPronounBl 3.71 2.92 1.9 0.99 0 1.27 7.58 10.22
AvgUnqVerbBl 3.43 6.53 2.29 1.36 1.89 2.17 0 3.65
AvgDepsBl_nsubj 2.97 5.88 2.3 1.43 1.14 2.11 0 2.19
AvgUnqPrepositionBl 2.78 4.75 2.01 1.04 1.59 1.86 0 3.65
LxcDiv 2.6 4 2.33 1.44 0.22 1.92 0 3.16
WdDiffWdStem 1.99 0.84 0.97 0.5 0.82 0.63 0 7.3
AvgBlScore 1.56 2.98 1.74 1.22 1.18 1.81 0 0
AvgDepsBl_punct 1.52 2.67 1.81 0.85 0.54 1.23 0 0.97
AvgUnqNoundBl 1.48 0.02 1.81 0.84 0.31 1.09 11.62 2.68
AggPronSen_third_person 1.37 0.36 0.55 0.21 0.28 0.67 18.65 2.19
TCorefChainDoc 1.3 0.78 1.88 0.91 0.67 1.7 0 2.19
RdbltyFlesch 1.3 2.18 0.2 0.44 1.65 0.98 0 2.19
AvgPronBl_first_person 1.19 0 1.69 0.74 1.34 0.66 0 3.65
AvgSenBl 1.16 2.23 1.8 0.84 0.27 0.97 0 0
AvgDepsSen_advcl 1.14 0.06 0.01 0.49 0.72 0.76 0 4.62
AvgSenBlCoh_LeackChod 1.12 1.42 1.49 0.61 0.42 1.35 0 1.22
AvgSenBlCoh_LDA 1.07 0.19 0.94 0.65 1.67 0.94 15.45 0.49
LexChainMaxSp 0.99 1.34 1.69 0.74 2.27 1.78 0 0
AvgDepsBl_mark 0.94 0.2 1.21 0.38 1.95 0.86 0 2.68
CharEnt 0.86 0.48 1.61 0.8 0.84 0.82 0 1.22
AvgWdLen 0.85 0.9 1.12 0.76 0.69 0.73 0 0.97
AvgChainSpan 0.79 0.68 1.73 1.07 1.32 1.08 0 0
AvgAOESen_InfPointPoly 0.73 0.22 0.35 0.23 0.12 0.39 0 2.68
AvgDepsSen_det 0.72 0.92 0.1 0.35 0.83 0.57 0 1.46
AvgDepsSen_dobj 0.69 0.6 0.53 0.4 0.94 0.52 0 1.46
AvgUnqPronounBl 0.66 0.14 1.72 0.76 0.12 1.13 0 0.49
AvgConnBl_temp_conns 0.65 1.22 1.12 0.33 1.18 0.69 0 0
AvgDepsSen_compound 0.64 0.99 0.55 0.18 1.46 1.18 0 0.49
AvgSenAdjCoh_Path 0.64 0.78 1.29 0.71 0.77 0.79 0 0
WdDiffLemmaStem 0.63 1.33 0.19 0.58 0.14 0.74 0 0
RdbltyDaleChall 0.63 1.19 1.14 0.4 0.66 0.48 0 0
WdMaxDpthHypernymTree 0.59 1.07 0.69 0.6 0.39 0.51 0 0
LexChainAvgSpan 0.57 0.28 0.88 0.7 1.27 0.59 3.91 0
SenStdDevWd 0.54 0.55 0.97 0.56 0.03 0.68 0 0.24
AvgDepsSen_amod 0.54 0.18 0.25 0.34 0.58 0.73 0 1.46
AvgDepsBl_dobj 0.54 0.02 1.67 0.72 2.01 0.84 0 0.24
AvgDepsSen_dep 0.52 0.87 0.47 0.4 0.18 1.08 0 0
AvgDepsBl_nmod 0.52 0.05 1.74 0.78 0.19 0.95 0 0
FrqRhythmId 0.51 0.56 1.17 0.42 0.91 0.9 0 0
AvgDepsBl_advcl 0.5 0.1 1.08 0.3 1.09 0.56 0 0.97
AvgCorefChain 0.5 0.3 1.13 0.49 1.28 0.45 0 0.49
AvgDepsSen_nmod 0.49 0.06 0.34 0.5 0.22 0.58 0 1.22
AvgConnSen_addition 0.49 0.1 0.62 0.49 0.44 0.68 0 0.97
AvgAdjectiveBl 0.48 0.23 1.51 0.59 0.56 0.7 0 0
WdLettStdDev 0.46 0.17 1.22 0.74 0.3 0.72 0 0
AvgPronBl_indefinite 0.44 0.23 1.25 0.4 0.87 1.02 0 0
AvgConnBl_addition 0.44 0.63 0.99 0.25 0.74 0.66 0 0
AvgDepsSen_mark 0.44 0.16 0.1 0.39 1.46 0.73 0 0.97
LangRhythmCoeff 0.44 0.71 0.54 0.34 0.04 0.82 0 0
AvgVoice 0.43 0.26 1.31 0.44 0.83 0.69 0 0
AvgUnqAdverbBl 0.43 0.03 1.52 0.6 0.67 0.76 0 0
AvgAOABl_Shock 0.43 0.25 1.21 0.44 0.69 0.88 0 0
AvgBlLen 0.41 0 0 1.69 0 0 0 0
AvgPronBl_third_person 0.41 0.2 1.12 0.32 0.64 0.33 0 0.49
AvgUnqWdBl 0.41 0 0 1.7 0 0 0 0
Content.words 0.4 0 0 1.67 0 0 0 0
AvgWdBl 0.4 0 0 1.67 0 0 0 0
Words 0.39 0 0 1.6 0 0 0 0
AvgAOEBl_InfPointPoly 0.39 0.14 0.34 0.24 0.81 0.48 0 0.97
TCorefChainBigSpan 0.38 0 1.06 0.29 1.22 1.08 1.47 0
AvgDepsBl_cop 0.37 0 1.4 0.51 1.32 0.58 0 0
AvgConnSen_semi_coords 0.37 0.06 0.12 0.29 0.84 0.63 0 0.97
AvgVerbBl 0.37 0 0 1.55 0 0 0 0
AvgDepsSen_xcomp 0.35 0.38 0.05 0.6 1.2 0.67 0 0
AvgRhythmUnitStreesSyll 0.34 0.4 0.2 0.33 1.14 0.93 0 0
AvgAOEBl_InvLinRegSlo 0.34 0.7 0.09 0.23 0.67 0.59 0 0
AvgNmdEntSen 0.34 0.3 0.21 0.4 0.48 1.1 0 0
AvgConnBl_coord_connects 0.34 0 1.25 0.4 1.85 0.63 0 0
AvgAOABl_Bird 0.34 0.25 0.44 0.41 1.53 0.92 0 0
AvgDepsBl_xcomp 0.33 0.18 1.15 0.34 0 0.45 0 0
AvgAdverbSen 0.33 0.27 0.2 0.4 2.52 0.97 0 0
AvgConnBl_reas_purp 0.33 0 0.96 0.24 1.1 0.45 0 0.49
AvgAdverbBl 0.33 0.04 1.29 0.43 0.75 0.5 0 0
TActCorefChainWd 0.33 0.62 0.39 0.29 0.22 0.34 0 0
AvgAOABl_Bristol 0.32 0.11 0.75 0.33 0.23 1 0 0
AvgAOEBl_IndexPolyFAT.3 0.32 0.58 0 0.44 0.44 0.43 0 0
AvgDepsBl_aux 0.32 0 0.52 0.07 0.8 0.39 0 0.97
AvgConnSen_logical_conns 0.31 0.13 0.7 0.47 0.7 0.55 0 0
AvgDepsSen_punct 0.31 0.05 0.9 0.55 0.75 0.45 0 0
AvgCommaBl 0.3 0 1.16 0.35 0.23 0.69 0 0
AvgDepsSen_aux 0.29 0.02 0.45 0.36 0.43 1.18 0 0
AvgNounBl 0.29 0 0 1.2 0 0 0 0
WdPolysemyCnt 0.29 0.56 0.15 0.14 0.03 0.68 0 0
AvgConnSen_reas_purp 0.29 0.02 0.16 0.26 1.13 0.85 0 0.49
AvgDepsBl_compound 0.29 0.02 0.12 0 1.6 0.57 0 0.97
AvgDepsBl_amod 0.28 0 1.1 0.32 0.78 0.53 0 0
AvgAOASen_Bird 0.28 0.51 0.15 0.23 1.13 0.46 0 0
WdPathCntHypernymTree 0.28 0.06 0.7 0.51 0.06 0.52 0 0
AvgDepsBl_acl 0.27 0.03 1 0.26 0.7 0.59 0 0
AvgDepsBl_ccomp 0.27 0 0.9 0.21 0.92 0.86 0 0
AvgAOASen_Shock 0.27 0.02 0.53 0.5 0.19 0.66 0 0
AggPronSen_indefinite 0.27 0.07 0.04 0.61 0.75 0.79 0 0
AvgDepsSen_cop 0.27 0.16 0.03 0.55 0.5 0.75 0 0
AvgConnBl_simp_subords 0.27 0.06 1.07 0.3 0.95 0.41 0 0
AvgConnBl_semi_coords 0.26 0 0.68 0.12 0.22 0.42 0 0.49
AvgAOESen_IndexAbThr.0.3. 0.26 0.1 0.23 0.53 0.02 0.64 0 0
AvgAOEBl_IndexAbThr.0.3. 0.26 0.21 0 0.61 0.28 0.43 0 0
WdSylCnt 0.25 0.15 0.39 0.48 0.97 0.3 0 0
AvgDepsBl_neg 0.25 0.03 0.86 0.19 0.09 0.84 0 0
AvgConnBl_contrasts 0.23 0 0.73 0.14 0.65 0.85 0 0
AvgNounSen 0.23 0.05 0.42 0.24 0.23 0.89 0 0
AvgDepsSen_ccomp 0.23 0.18 0.07 0.29 0.47 0.83 0 0
AvgConnBl_logical_conns 0.23 0.16 0.87 0.19 0.05 0.27 0 0
AvgConnSen_simp_subords 0.23 0.1 0.05 0.54 0 0.54 0 0
AvgAdjectiveSen 0.23 0.1 0.21 0.34 0.85 0.79 0 0
AvgConnBl_oppositions 0.23 0 0.79 0.16 0.75 0.8 0 0
AvgInferenceDistChain 0.23 0.1 0.42 0.26 0.53 0.76 0 0
AvgConnBl_order 0.22 0 0.9 0.21 2.32 0.29 0 0
AvgDepsBl_conj 0.22 0 0.79 0.16 1.65 0.57 0 0
AvgPrepositionBl 0.22 0 0 0.9 0 0 0 0
AvgSenScore 0.22 0.05 0.14 0.33 0.33 0.9 0 0
AvgDepsBl_case 0.22 0 0 0.9 0 0 0 0
AvgAOESen_IndexPolyFAT.3 0.22 0.07 0.26 0.42 0.35 0.55 0 0
AvgAOABl_Cortese 0.22 0.05 0.35 0.32 1.05 0.67 0 0
AvgConnSen_oppositions 0.21 0.03 0.03 0.35 1.28 0.84 0 0
AvgIntraBlCoh_word2vec 0.21 0 0 0.88 0 0 0 0
AvgRhythmUnits 0.21 0 0.34 0.42 1.08 0.53 0 0
AvgNounNmdEntBl 0.2 0.14 0.52 0.07 2.7 0.46 0 0
Sentences 0.2 0 0 0.84 0 0 0 0
AvgAOABl_Kuperman 0.2 0.13 0.06 0.32 0.93 0.64 0 0
AvgIntraBlCoh_LDA 0.2 0 0 0.84 0 0 0 0
AvgNmdEntBl 0.2 0.03 0.82 0.18 1.51 0.35 0 0
AvgConnSen_order 0.19 0.11 0.13 0.37 0.55 0.41 0 0
AvgConnSen_temp_conns 0.19 0.27 0.12 0 1.47 0.77 0 0
LxcSoph 0.19 0.05 0.18 0.34 0.45 0.61 0 0
LangRhythmDiameter 0.19 0 0.3 0.02 1.45 0.37 0 0.49
AvgIntraBlCoh_LSA 0.19 0 0 0.81 0 0 0 0
AvgIntraBlCoh_Path 0.19 0 0 0.81 0 0 0 0
AvgAOASen_Kuperman 0.18 0.28 0.06 0.17 0.06 0.45 0 0
AvgDepsBl_nsubjpass 0.18 0.04 0.66 0.11 0.38 0.53 0 0
AvgAOASen_Bristol 0.18 0.13 0.18 0.25 0.36 0.55 0 0
AvgSenAdjCoh_LDA 0.17 0 0 0.69 0 0 0 0
AvgSenBlCoh_Path 0.17 0 0 0.69 0 0 0 0
AvgSenAdjCoh_LeackChod 0.17 0 0 0.7 0 0 0 0
AvgIntraBlCoh_LeackChod 0.17 0 0 0.71 0 0 0 0
AvgSenAdjCoh_word2vec 0.17 0 0 0.71 0 0 0 0
AvgIntraBlCoh_WuPalmer 0.17 0 0 0.72 0 0 0 0
SenAsson 0.16 0 0.47 0.06 0.6 0.7 0 0
AvgDepsBl_nummod 0.16 0 0.47 0.06 0.07 0.75 0 0
AvgSenBlCoh_word2vec 0.16 0 0 0.66 0 0 0 0
AvgSenAdjCoh_WuPalmer 0.16 0 0 0.66 0 0 0 0
SenStDevUnqWd 0.15 0 0 0.61 0 0 0 0
AvgAOEDoc_IndexAbThr.0.3. 0.15 0 0 0.61 0 0 0 0
AvgUnqWdSen 0.15 0 0 0.62 0 0 0 0
SynSoph 0.15 0 0 0.63 0 0 0 0
AvgUnqAdjectiveBl 0.15 0 0 0.63 0 0 0 0
AvgSenAdjCoh_LSA 0.15 0 0 0.63 0 0 0 0
AvgWdSen 0.14 0 0 0.56 0 0 0 0
AvgSenLen 0.14 0 0 0.56 0 0 0 0
AvgConnBl_sentence_link 0.14 0 0 0.57 0 0 0 0
WdAvgDpthHypernymTree 0.14 0 0 0.58 0 0 0 0
AvgSenBlCoh_LSA 0.14 0 0 0.58 0 0 0 0
AvgSenBlCoh_WuPalmer 0.14 0 0 0.6 0 0 0 0
AvgDepsSen_mwe 0.13 0 0.31 0.03 0.32 0.7 0 0
AvgDepsBl_advmod 0.13 0 0 0.54 0 0 0 0
AvgDepsSen_advmod 0.13 0 0 0.54 0 0 0 0
AvgConnSen_contrasts 0.13 0 0.14 0.22 0.06 0.55 0 0
AvgDepsSen_nsubj 0.12 0 0 0.48 0 0 0 0
SenScoreStDev 0.12 0 0 0.49 0 0 0 0
AvgVerbSen 0.12 0 0 0.49 0 0 0 0
AvgSenStressedSyll 0.12 0 0 0.5 0 0 0 0
AvgBlVoiceCoOcc 0.12 0 0 0.51 0 0 0 0
AvgAOADoc_Shock 0.11 0 0 0.44 0 0 0 0
AvgAOEDoc_IndexPolyFAT.3 0.11 0 0 0.44 0 0 0 0
AvgSemDep 0.11 0 0 0.44 0 0 0 0
AvgDepsSen_case 0.11 0 0 0.46 0 0 0 0
AvgRhythmUnitSyll 0.1 0 0 0.4 0 0 0 0
AvgSenSyll 0.1 0 0 0.4 0 0 0 0
AvgAOADoc_Bird 0.1 0 0 0.41 0 0 0 0
AvgDepsSen_acl 0.1 0.06 0.2 0.01 0.6 0.43 0 0
RdbltyKincaid 0.1 0 0 0.42 0 0 0 0
AvgAOASen_Cortese 0.1 0.12 0.06 0.25 0.22 0 0 0
AvgConnBl_conjunctions 0.09 0 0 0.36 0 0 0 0
AvgPronounSen 0.09 0 0 0.37 0 0 0 0
RdbltyFog 0.09 0 0 0.37 0 0 0 0
AvgAOADoc_Cortese 0.08 0 0 0.32 0 0 0 0
AvgAOADoc_Kuperman 0.08 0 0 0.32 0 0 0 0
AvgPrepositionSen 0.08 0 0 0.32 0 0 0 0
AvgDepsBl_dep 0.08 0.04 0.25 0.02 0.15 0.29 0 0
AvgDepsBl_cc 0.08 0 0 0.33 0 0 0 0
AvgAOADoc_Bristol 0.08 0 0 0.33 0 0 0 0
AvgConnSen_sentence_link 0.08 0 0 0.33 0 0 0 0
AvgDepsSen_neg 0.08 0 0.13 0 0.14 0.58 0 0
AvgDepsSen_conj 0.08 0 0 0.34 0 0 0 0
AvgDepsBl_mwe 0.08 0 0.43 0.05 0.03 0.18 0 0
AvgConnSen_coord_connects 0.07 0 0 0.29 0 0 0 0
AvgConnSen_coord_conjs 0.07 0 0 0.29 0 0 0 0
AvgAOEDoc_InvAverage 0.07 0 0 0.29 0 0 0 0
AvgAOEBl_InvAverage 0.07 0 0 0.29 0 0 0 0
AvgAOESen_InvAverage 0.07 0 0 0.3 0 0 0 0
AvgDepsSen_cc 0.07 0 0 0.31 0 0 0 0
AvgConnSen_conjunctions 0.07 0 0 0.31 0 0 0 0
AvgAOEDoc_InfPointPoly 0.06 0 0 0.24 0 0 0 0
AvgUnqNmdEntBl 0.06 0 0 0.24 0 0 0 0
LangRhythmId 0.06 0.03 0.04 0 0.33 0.37 0 0
AvgAOESen_InvLinRegSlo 0.05 0 0 0.22 0 0 0 0
AvgAOEDoc_InvLinRegSlo 0.05 0 0 0.23 0 0 0 0
AvgDepsBl_auxpass 0.04 0 0 0.18 0 0 0 0
AvgConnBl_coord_conjs 0.03 0 0 0.12 0 0 0 0