ReaderBench Model 1f Variable Importance - shmercer/writeAlizer GitHub Wiki

Relative Importance of Each Metric

ReaderBench Model 1f

This model used Coh-Metrix scores from 7 min narrative writing samples (I once had a magic pencil and ...) from 124 students in the spring of Grades 2-5 (Mercer et al., 2019) to predict holistic writing quality on the samples (elo ratings calculated from paired comparisons).

Principal component scores were generated during data pre-processing (see section on Scoring Model Development for more details).

Mercer, S. H., Keller-Margulis, M. A., Faith, E. L., Reid, E. K., & Ochs, S. (2019). The potential for automated text evaluation to improve the technical adequacy of written expression curriculum-based measurement. Learning Disability Quarterly, 42, 117-128. https://doi.org/10.1177/0731948718803296

Algorithm Weightings in Ensemble

Abbreviations:

  • all = ensemble model
  • gbm = stochastic gradient boosted trees
  • pls = partial least squares regression
  • svm = support vector machines
  • enet = elastic net regression
  • rf = random forest regression
  • mars = bagged multivariate adaptive regression splines
  • cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept gbm pls svm enet rf mars cube
-9.2262 0.1219 0.7713 0.1603 -0.3706 -0.0217 0.3129 0.0233

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

PC1 = scores on 1st principal component extracted, ...

Note: Importance is unavailable for support vector machines when PCA-based pre-processing is used (so all values for svm are 0).

Metric all gbm pls svm enet rf mars cube
PC2 31.31 56.99 35.09 0 11.67 23.86 35.8 21.83
PC1 11.79 6.72 17.18 0 3.69 5.61 8.89 21.83
PC4 10.38 7.69 9.11 0 8.29 5.24 16.85 12.66
PC44 4.45 0.7 0.97 0 7.79 1.41 10.78 3.71
PC9 4.36 2.62 4.85 0 7.31 1.81 0 10.92
PC27 3.4 0.68 0.79 0 1.88 1.75 12.84 0
PC14 3.37 1.1 3.45 0 6.67 1.95 0 7.21
PC43 2.78 1.23 1.19 0 9.29 1.77 0 0
PC6 2.51 0.56 3.24 0 3.33 1.2 0.08 9.17
PC16 2.38 0.82 1.17 0 1.74 1.2 6.85 0
PC15 2.31 1.41 2.25 0 4.31 1.89 0 10.92
PC5 2.22 0.82 2.97 0 2.79 2.34 0.2 1.75
PC28 1.47 0.78 1.17 0 3.69 1.94 0 0
PC10 1.46 1.85 1.73 0 2.08 1.98 0 0
PC11 1.33 1.77 1.55 0 1.91 2.26 0 0
PC37 1.29 0.57 0.8 0 3.79 1.57 0 0
PC12 1.19 0.4 1.5 0 1.87 2.16 0 0
PC34 1.08 0.19 0.77 0 3.06 0.68 0 0
PC31 1.05 0.52 0.79 0 2.74 2.09 0 0
PC30 0.9 0.05 0.21 0 0 0.29 4.02 0
PC19 0.84 1.4 0.92 0 1.29 0.05 0 0
PC23 0.8 0.38 0.83 0 1.59 1.37 0 0
PC3 0.77 0.92 0.36 0 0 2.84 2.68 0
PC40 0.75 0.22 0.5 0 2.14 1.16 0 0
PC21 0.74 0.02 0.7 0 0.87 1.03 1 0
PC45 0.73 0.12 0.43 0 2.23 1.46 0 0
PC25 0.65 0.57 0.67 0 1.22 1.65 0 0
PC18 0.43 0.14 0.62 0 0.45 2.08 0 0
PC20 0.38 0.49 0.55 0 0.3 1.36 0 0
PC38 0.37 0.35 0.35 0 0.72 1.69 0 0
PC35 0.37 0.16 0.4 0 0.7 0 0 0
PC32 0.27 0.8 0.34 0 0.17 1.33 0 0
PC17 0.24 0.08 0.47 0 0 1.41 0 0
PC39 0.23 0.46 0.29 0 0.25 1.31 0 0
PC13 0.23 1.06 0.32 0 0 2.26 0 0
PC22 0.21 0.42 0.36 0 0 1.86 0 0
PC41 0.2 0.42 0.27 0 0.16 1.77 0 0
PC7 0.18 0.41 0.3 0 0 1.37 0 0
PC26 0.17 1.46 0.12 0 0 2.05 0 0
PC24 0.12 1.74 0 0 0 1.36 0 0
PC42 0.1 0.32 0.15 0 0 1.77 0 0
PC8 0.07 0.36 0.06 0 0 1.67 0 0
PC33 0.06 0.12 0.08 0 0 1.34 0 0
PC36 0.06 0 0.11 0 0 0.72 0 0
PC29 0.02 0.13 0.02 0 0 2.08 0 0

Proportion of Variance by Varimax Rotated Component (RC)

Due to space limitations, loadings for only the first five principal components are displayed. The full PCA results are available here

RC1 RC2 RC3 RC4 RC5
SS loadings 49.09 28.90 15.04 12.78 8.00
Proportion Var 0.24 0.14 0.07 0.06 0.04
Cumulative Var 0.24 0.39 0.46 0.53 0.57
Proportion Explained 0.43 0.25 0.13 0.11 0.07
Cumulative Proportion 0.43 0.69 0.82 0.93 1.00

Varimax Rotated Loadings

Abbreviated ReaderBench metric names can be found here.

Metric RC1 RC2 RC3 RC4 RC5
Sentences -0.62 0.59 -0.01 -0.09 0.19
Words 0.13 0.87 -0.07 0.37 0.19
Content.words 0.02 0.85 0.01 0.36 0.07
RdbltyFlesch -0.83 -0.11 -0.08 0.04 0.04
RdbltyFog 0.9 0.08 0.01 -0.01 0
RdbltyKincaid 0.89 0.07 0.02 0 -0.01
RdbltyDaleChall 0.47 -0.12 0.41 -0.23 0.28
AvgBlLen -0.01 0.88 0.03 0.29 -0.02
AvgCommaBl -0.18 0.28 -0.06 -0.18 0.05
AvgSenLen 0.94 0.15 0.04 0 -0.04
AvgSenBl -0.62 0.59 -0.01 -0.09 0.19
AvgUnqWdBl 0.02 0.89 0.06 0.22 0.02
AvgUnqWdSen 0.95 0.12 0.03 0.01 -0.04
AvgWdLen -0.12 0.48 0.04 -0.15 -0.47
AvgWdBl 0.02 0.85 0.01 0.36 0.07
AvgWdSen 0.95 0.1 0.02 0.01 0.03
CharEnt -0.18 0.59 -0.07 -0.04 0.21
SenStDevUnqWd -0.35 0.23 0.05 0.67 0.07
SenStdDevWd -0.29 0.21 0.03 0.67 0.08
WdEnt -0.01 0.88 0 0.12 0.16
WdLettStdDev -0.04 0.56 0.08 0.02 -0.16
LxcDiv 0.08 0.81 0.05 0.16 -0.06
LxcSoph 0.56 0.08 0.09 -0.08 -0.48
SynSoph 0.89 0.25 0.04 0.2 0.03
AvgNounBl 0.13 0.55 0.22 0.28 0.52
AvgPronounBl 0.04 0.71 -0.19 0.37 0.11
AvgVerbBl 0.04 0.84 -0.17 0.36 0.02
AvgAdverbBl 0.2 0.61 -0.01 0.25 -0.12
AvgAdjectiveBl -0.08 0.65 0.05 0.06 0.04
AvgPrepositionBl 0.11 0.79 -0.1 0.18 0.08
AvgNounSen 0.89 -0.04 0.07 -0.01 0.26
AvgPronounSen 0.9 -0.01 -0.07 0.03 0.02
AvgVerbSen 0.97 0.08 -0.05 0 -0.04
AvgAdverbSen 0.8 0.17 0.04 0.09 -0.15
AvgAdjectiveSen 0.81 0 0 -0.16 -0.05
AvgPrepositionSen 0.9 0.17 0 0 0.02
AvgUnqNoundBl 0.13 0.52 0.2 0.16 0.48
AvgUnqPronounBl -0.01 0.54 -0.15 0.18 0.09
AvgUnqVerbBl 0.01 0.8 -0.09 0.29 0.01
AvgUnqAdverbBl 0.12 0.69 -0.01 0.1 -0.11
AvgUnqAdjectiveBl -0.07 0.65 0.04 -0.02 0.04
AvgUnqPrepositionBl 0.04 0.77 -0.11 0.17 0.05
AvgPronBl_first_person -0.02 0.47 -0.22 0.38 0.16
AvgPronBl_indefinite -0.03 0.57 -0.13 -0.04 0.14
AggPronSen_indefinite 0.74 0.09 -0.09 -0.12 0.15
AvgPronBl_third_person 0.08 0.54 -0.03 0.14 -0.03
AggPronSen_third_person 0.78 0.06 -0.02 -0.05 -0.13
AvgSemDep 0.98 0.06 -0.01 0.01 0.08
WdDiffLemmaStem -0.11 0.34 0.02 -0.16 -0.19
WdDiffWdStem -0.11 0.39 -0.03 -0.1 -0.42
WdMaxDpthHypernymTree 0.02 -0.21 0.13 0.1 0.23
WdAvgDpthHypernymTree 0.02 -0.22 0.09 0.1 0.19
WdPathCntHypernymTree 0.01 -0.23 0.09 0.04 0.27
WdPolysemyCnt -0.04 -0.04 -0.4 0.25 0.07
WdSylCnt -0.11 0.5 0.32 -0.08 -0.3
AvgAOADoc_Shock -0.06 0.48 -0.01 0.02 -0.15
AvgAOABl_Shock -0.06 0.48 -0.01 0.02 -0.15
AvgAOASen_Shock 0.49 0.17 -0.02 0.08 -0.39
AvgAOADoc_Cortese -0.09 -0.14 0.49 -0.16 -0.23
AvgAOABl_Cortese -0.09 -0.14 0.49 -0.16 -0.23
AvgAOASen_Cortese 0.25 -0.19 0.35 0.03 -0.39
AvgAOADoc_Kuperman -0.1 -0.04 0.64 -0.1 -0.12
AvgAOABl_Kuperman -0.1 -0.04 0.64 -0.1 -0.12
AvgAOASen_Kuperman 0.11 -0.05 0.62 0 -0.32
AvgAOADoc_Bird -0.01 0.07 0.58 0.1 -0.3
AvgAOABl_Bird -0.01 0.07 0.58 0.1 -0.3
AvgAOASen_Bird 0.35 -0.02 0.26 0.18 -0.53
AvgAOADoc_Bristol -0.07 0.14 0.5 0.02 -0.31
AvgAOABl_Bristol -0.07 0.14 0.5 0.02 -0.31
AvgAOASen_Bristol 0.41 -0.05 0.1 0.01 -0.51
AvgAOEDoc_IndexPolyFAT.3 -0.04 -0.1 0.89 0.02 0.22
AvgAOEBl_IndexPolyFAT.3 -0.04 -0.1 0.89 0.02 0.22
AvgAOESen_IndexPolyFAT.3 0.06 -0.03 0.83 0.04 0.08
AvgAOEDoc_InvLinRegSlo -0.05 -0.2 0.84 -0.02 0.3
AvgAOEBl_InvLinRegSlo -0.05 -0.2 0.84 -0.02 0.3
AvgAOESen_InvLinRegSlo 0.17 -0.15 0.69 0.07 -0.03
AvgAOEDoc_InfPointPoly -0.03 -0.05 0.85 -0.11 0.22
AvgAOEBl_InfPointPoly -0.03 -0.05 0.85 -0.11 0.22
AvgAOESen_InfPointPoly 0.21 -0.05 0.7 0.01 -0.14
AvgAOEDoc_InvAverage -0.04 -0.08 0.88 -0.13 0.22
AvgAOEBl_InvAverage -0.04 -0.08 0.88 -0.13 0.22
AvgAOESen_InvAverage 0.2 -0.06 0.73 -0.01 -0.13
AvgAOEDoc_IndexAbThr.0.3. -0.07 -0.07 0.81 0.06 0.23
AvgAOEBl_IndexAbThr.0.3. -0.07 -0.07 0.81 0.06 0.23
AvgAOESen_IndexAbThr.0.3. -0.03 0.02 0.78 0.04 0.18
AvgNmdEntBl 0.06 0.31 0.06 0.04 0.62
AvgNounNmdEntBl 0.04 0.16 0.2 0 0.67
AvgUnqNmdEntBl 0.03 0.34 0.06 0.03 0.63
AvgNmdEntSen 0.57 -0.06 0.08 0 0.43
TCorefChainDoc 0.02 0.51 -0.04 0.33 0.25
AvgCorefChain 0 0.44 -0.14 0.18 0.03
AvgChainSpan 0.17 0.66 -0.09 0.15 -0.05
AvgInferenceDistChain 0.26 0.29 -0.06 0.06 0.15
TActCorefChainWd -0.09 -0.34 -0.03 -0.01 0.03
TCorefChainBigSpan 0.21 0.34 -0.02 0.16 0.02
AvgConnBl_addition 0.45 0.25 -0.03 0.6 0.11
AvgConnSen_addition 0.85 -0.02 0.02 0.17 0.09
AvgConnBl_conjunctions 0.48 0.32 -0.03 0.43 0.19
AvgConnSen_conjunctions 0.91 -0.02 0.02 0.05 0.11
AvgConnBl_contrasts 0.16 0.51 0.12 -0.15 0
AvgConnSen_contrasts 0.67 0.22 0.05 -0.24 -0.1
AvgConnBl_coord_conjs 0.11 0.28 -0.06 0.38 -0.23
AvgConnSen_coord_conjs 0.5 0.11 0.01 0.15 -0.25
AvgConnBl_coord_connects 0.47 0.41 -0.03 0.52 0.09
AvgConnSen_coord_connects 0.94 0.01 0.02 0.08 0.05
AvgConnBl_logical_conns 0.44 0.24 -0.04 0.5 0.17
AvgConnSen_logical_conns 0.86 -0.04 0.03 0.12 0.1
AvgConnBl_oppositions 0.17 0.49 0.07 -0.22 0.04
AvgConnSen_oppositions 0.66 0.19 0.03 -0.29 -0.05
AvgConnBl_order 0.04 0.16 -0.02 0.45 -0.06
AvgConnSen_order 0.4 -0.09 0.01 0.28 0.01
AvgConnBl_reas_purp 0.07 0.32 -0.03 0.49 -0.16
AvgConnSen_reas_purp 0.58 0.04 0.03 0.25 -0.17
AvgConnBl_semi_coords 0.11 0.28 -0.06 0.38 -0.23
AvgConnSen_semi_coords 0.5 0.11 0.01 0.15 -0.25
AvgConnBl_sentence_link 0.39 0.51 -0.1 0.58 0.07
AvgConnSen_sentence_link 0.95 0 0 0.11 0.06
AvgConnBl_simp_subords -0.13 0.52 -0.14 -0.04 0.01
AvgConnSen_simp_subords 0.52 0.16 -0.03 -0.1 -0.02
AvgConnBl_temp_conns -0.1 0.34 -0.21 0.22 0.04
AvgConnSen_temp_conns 0.33 0 -0.12 0.07 0.06
LexChainAvgSpan -0.04 0.29 -0.21 0.47 0.17
LexChainMaxSp 0.04 0.61 -0.1 0.45 0.02
AvgBlScore 0.22 0.56 -0.18 0.53 0.08
AvgSenScore 0.93 0.05 -0.06 0.06 0.07
SenScoreStDev -0.39 0.27 -0.02 0.67 0.07
AvgIntraBlCoh_LeackChod -0.71 0.35 0.01 0.54 0.08
AvgSenAdjCoh_LeackChod -0.69 0.33 0.02 0.56 0.07
AvgSenBlCoh_LeackChod 0.76 -0.5 -0.07 -0.05 -0.17
AvgIntraBlCoh_WuPalmer -0.71 0.35 0.01 0.53 0.08
AvgSenAdjCoh_WuPalmer -0.7 0.33 0.01 0.55 0.08
AvgSenBlCoh_WuPalmer 0.76 -0.51 -0.08 -0.07 -0.17
AvgIntraBlCoh_Path -0.72 0.34 0 0.53 0.09
AvgSenAdjCoh_Path -0.7 0.32 0 0.55 0.08
AvgSenBlCoh_Path 0.78 -0.52 -0.07 -0.17 -0.14
AvgIntraBlCoh_LSA -0.7 0.35 -0.01 0.55 0.09
AvgSenAdjCoh_LSA -0.68 0.34 -0.01 0.57 0.09
AvgSenBlCoh_LSA 0.76 -0.49 -0.13 -0.04 -0.15
AvgIntraBlCoh_LDA -0.7 0.37 -0.02 0.53 0.08
AvgSenAdjCoh_LDA -0.68 0.35 -0.02 0.55 0.07
AvgSenBlCoh_LDA 0.71 -0.39 -0.2 0.02 -0.19
AvgIntraBlCoh_word2vec -0.69 0.35 0.01 0.55 0.08
AvgSenAdjCoh_word2vec -0.69 0.33 0 0.56 0.09
AvgSenBlCoh_word2vec 0.76 -0.53 -0.09 -0.07 -0.13
AvgBlVoiceCoOcc -0.05 0.47 -0.14 0.51 0.08
AvgVoice -0.05 0.46 -0.15 0.54 0.08
AvgSenSyll 0.98 0.06 0.01 -0.01 0.06
AvgSenStressedSyll 0.96 0.08 0.03 0.01 0.06
AvgRhythmUnits 0.18 0 -0.08 -0.18 0.01
AvgRhythmUnitSyll 0.81 -0.04 0.02 0.09 0.09
AvgRhythmUnitStreesSyll 0.8 -0.03 0.04 0.11 0.1
LangRhythmCoeff 0.27 -0.25 -0.03 0.24 0.22
LangRhythmId -0.19 0.11 0.02 -0.04 0.21
FrqRhythmId 0.68 -0.44 -0.04 -0.07 -0.11
LangRhythmDiameter 0.34 0.07 0 0.11 0.35
SenAsson -0.05 0.28 -0.05 0.17 -0.09
AvgDepsBl_acl -0.14 0.18 -0.12 0.02 -0.13
AvgDepsSen_acl 0.12 -0.23 -0.17 -0.14 -0.17
AvgDepsBl_advcl 0.14 0.49 -0.08 0.25 -0.2
AvgDepsSen_advcl 0.66 0.18 0.02 0.03 -0.23
AvgDepsBl_advmod 0.14 0.61 -0.03 0.33 -0.12
AvgDepsSen_advmod 0.79 0.15 0.01 0.15 -0.12
AvgDepsBl_amod -0.1 0.58 0.09 0.11 0.15
AvgDepsSen_amod 0.7 0 0.02 -0.07 0.09
AvgDepsBl_aux 0.16 0.54 -0.15 0.06 -0.03
AvgDepsSen_aux 0.76 0.21 -0.09 -0.06 -0.08
AvgDepsBl_auxpass 0.04 0.4 -0.05 0.2 -0.08
AvgDepsBl_case 0.08 0.67 -0.08 0.16 0.28
AvgDepsSen_case 0.84 0.1 -0.03 -0.01 0.17
AvgDepsBl_cc 0.49 0.35 0 0.43 0.19
AvgDepsSen_cc 0.92 0 0.03 0.05 0.09
AvgDepsBl_ccomp 0.2 0.28 -0.25 0.31 0.11
AvgDepsSen_ccomp 0.76 -0.05 -0.15 -0.02 0.11
AvgDepsBl_compound -0.02 -0.06 0.5 0.1 0.4
AvgDepsSen_compound 0.37 -0.22 0.24 0.02 0.28
AvgDepsBl_conj 0.6 0.31 0.02 0.37 0.15
AvgDepsSen_conj 0.88 0.02 0.02 0.03 0.08
AvgDepsBl_cop -0.05 0.49 -0.09 -0.11 -0.03
AvgDepsSen_cop 0.64 0.02 -0.13 -0.16 -0.15
AvgDepsBl_dep 0.38 0.28 0.03 0.25 0.25
AvgDepsSen_dep 0.73 0.01 0.05 0.02 0.18
AvgDepsBl_det 0.14 0.58 0.03 0.31 0.22
AvgDepsSen_det 0.76 0.11 -0.01 0.03 0.19
AvgDepsBl_dobj 0.14 0.56 -0.02 0.47 0.08
AvgDepsSen_dobj 0.9 -0.03 0 0.06 0.06
AvgDepsBl_mark 0.09 0.6 -0.1 0.25 -0.23
AvgDepsSen_mark 0.75 0.1 -0.03 0 -0.24
AvgDepsBl_mwe 0.14 0.34 0.04 0.06 0.15
AvgDepsSen_mwe 0.45 0.15 0.08 0.04 0.14
AvgDepsBl_neg 0 0.38 0.01 -0.11 0.02
AvgDepsSen_neg 0.39 0.05 0.05 -0.21 -0.07
AvgDepsBl_nmod 0.09 0.62 -0.1 0.16 0.3
AvgDepsSen_nmod 0.83 0.08 -0.03 -0.03 0.2
AvgDepsBl_nsubj 0.07 0.79 -0.16 0.34 0.15
AvgDepsSen_nsubj 0.97 0.03 -0.04 0 0.04
AvgDepsBl_nsubjpass 0.05 0.34 -0.04 0.2 -0.04
AvgDepsBl_nummod 0.02 0.2 -0.15 0.06 0.24
AvgDepsBl_punct -0.53 0.62 -0.01 -0.15 0.13
AvgDepsSen_punct -0.11 0.31 0 -0.06 -0.15
AvgDepsBl_xcomp 0.03 0.41 0.03 0.34 -0.22
AvgDepsSen_xcomp 0.71 0.07 0.07 0.06 -0.24