ReaderBench Model 1d Variable Importance - shmercer/writeAlizer GitHub Wiki

Relative Importance of Each Metric

ReaderBench Model 1d

This model used ReaderBench scores from 7 min narrative writing samples ("I once had a magic pencil and ...") from 136 students in the fall of Grades 2-5 (Mercer et al., 2019) to predict holistic writing quality on the samples (elo ratings calculated from paired comparisons).

Principal component scores were generated during data pre-processing (see section on Scoring Model Development for more details).

Mercer, S. H., Keller-Margulis, M. A., Faith, E. L., Reid, E. K., & Ochs, S. (2019). The potential for automated text evaluation to improve the technical adequacy of written expression curriculum-based measurement. Learning Disability Quarterly, 42, 117-128. https://doi.org/10.1177/0731948718803296

Algorithm Weightings in Ensemble

Abbreviations:

  • all = ensemble model
  • gbm = stochastic gradient boosted trees
  • pls = partial least squares regression
  • svm = support vector machines
  • enet = elastic net regression
  • rf = random forest regression
  • mars = bagged multivariate adaptive regression splines
  • cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept gbm pls svm enet rf mars cube
-8.4195 0.0406 0.8127 0.0694 -0.0509 0.1058 0.0038 0.0448

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

PC1 = scores on 1st principal component extracted, ...

Note: Importance is unavailable for support vector machines when PCA-based pre-processing is used (so all values for svm are 0).

Metric all gbm pls svm enet rf mars cube
PC2 50.2 61.92 52.85 0 12.57 36.4 46.85 23.81
PC1 8.03 1.95 9.25 0 1.38 5.78 0.27 8.39
PC3 7.38 1.56 8.43 0 3.57 5.47 0 8.39
PC5 5.42 3.46 5.74 0 5.67 3.71 0 13.83
PC4 2 1.81 2.27 0 1.61 1.06 0 0
PC24 1.9 0.9 1.48 0 6.31 1.73 16.76 1.59
PC14 1.7 1.4 1.56 0 3.22 1.85 3.63 3.85
PC8 1.47 0.85 1.45 0 1.88 2.26 0 1.59
PC30 1.47 1.67 1.1 0 6.24 2.51 0 6.58
PC6 1.09 0.73 0.95 0 0.78 2.84 0 0
PC31 1.09 1.01 0.92 0 5.49 0.03 0 9.98
PC7 1.03 0.17 1.17 0 1.27 0.95 0 0
PC17 1.02 1.53 0.62 0 1.24 3.12 0 4.31
PC43 0.94 1.87 0.58 0 5.88 1.5 0 4.76
PC33 0.87 0.6 0.88 0 5.75 0.44 0 0
PC32 0.82 0.21 0.6 0 3.29 2.36 0 1.59
PC39 0.81 1.14 0.54 0 4.22 1.87 1.81 0
PC27 0.77 0.48 0.66 0 2.74 0.62 0 4.99
PC13 0.73 1.28 0.73 0 1.02 0.55 0 0
PC34 0.7 0.46 0.2 0 0.43 2.05 13.64 0
PC38 0.67 0.36 0.55 0 4.08 0.86 0 2.72
PC23 0.64 0.5 0.72 0 2.45 0.1 0 0
PC19 0.6 0.9 0.56 0 1.28 0.84 0 0
PC16 0.6 0.67 0.22 0 0 2.25 6.99 0
PC45 0.6 1.92 0.07 0 0 1.77 10.06 0.91
PC20 0.56 0 0.53 0 1.3 1.26 0 0
PC28 0.54 1.55 0.39 0 1.29 0.94 0 0.45
PC40 0.54 0.77 0.52 0 4.01 0.04 0 0.45
PC18 0.53 0.36 0.47 0 0.92 1.27 0 0
PC11 0.51 0.87 0.55 0 0.48 0.18 0 0
PC22 0.49 0.35 0.42 0 0.99 1.2 0 0
PC36 0.46 0.95 0.09 0 0 3.14 0 0
PC12 0.45 0.22 0.46 0 0.33 0.68 0 0
PC29 0.44 0.95 0.29 0 0.79 1.36 0 0
PC41 0.43 0.27 0.35 0 2.57 0.79 0 0
PC42 0.4 0.32 0.31 0 2.25 1.03 0 0
PC9 0.37 0.34 0.4 0 0.13 0.33 0 0
PC44 0.29 0.31 0.22 0 1.53 0.68 0 0
PC46 0.28 0.91 0.05 0 0 1.65 0 0.45
PC15 0.26 0.87 0.14 0 0 0.59 0 1.36
PC26 0.25 0.16 0.29 0 0.59 0 0 0
PC35 0.23 0.02 0.19 0 0.43 0.7 0 0
PC37 0.13 0.09 0.13 0 0.03 0.22 0 0
PC10 0.12 0.4 0 0 0 0.99 0 0
PC25 0.08 0.29 0.08 0 0 0.03 0 0
PC21 0.06 0.64 0.03 0 0 0.01 0 0

Proportion of Variance by Varimax Rotated Component (RC)

Due to space limitations, loadings for only the first five principal components are displayed. The full PCA results are available here

Variable RC1 RC2 RC3 RC5 RC4
SS loadings 44.56 31.07 19.29 10.07 9.38
Proportion Var 0.22 0.15 0.10 0.05 0.05
Cumulative Var 0.22 0.38 0.47 0.52 0.57
Proportion Explained 0.39 0.27 0.17 0.09 0.08
Cumulative Proportion 0.39 0.66 0.83 0.92 1.00

Varimax Rotated Loadings

Abbreviated ReaderBench metric names can be found here.

Metric RC1 RC2 RC3 RC5 RC4
Sentences -0.589 0.59 -0.023 -0.072 -0.157
Words 0.086 0.947 0.097 0.203 0.039
Content.words -0.006 0.908 0.091 0.119 0.153
RdbltyFlesch -0.891 -0.159 -0.034 -0.06 0.033
RdbltyFog 0.95 0.117 0.07 0.127 -0.022
RdbltyKincaid 0.948 0.116 0.044 0.128 -0.02
RdbltyDaleChall 0.4 -0.276 -0.098 0.136 -0.503
AvgBlLen -0.041 0.903 0.095 0.055 0.156
AvgCommaBl -0.1 0.309 0.05 -0.227 0.006
AvgSenLen 0.91 0.198 0.058 0.044 0.17
AvgSenBl -0.589 0.59 -0.023 -0.072 -0.157
AvgUnqWdBl -0.015 0.901 0.103 0.081 0.118
AvgUnqWdSen 0.922 0.161 0.089 0.096 0.134
AvgWdLen -0.089 0.407 0.562 -0.271 0.362
AvgWdBl -0.006 0.908 0.091 0.119 0.153
AvgWdSen 0.931 0.147 0.057 0.086 0.135
CharEnt -0.006 0.489 0.287 -0.198 0.059
SenStDevUnqWd -0.429 0.328 0.099 0.075 0.453
SenStdDevWd -0.354 0.326 0.076 0.112 0.472
WdEnt 0.01 0.886 0.21 0.066 0.085
WdLettStdDev -0.158 0.401 0.362 -0.257 0.224
LxcDiv 0.065 0.791 0.172 0.047 0.204
LxcSoph 0.334 0.207 0.6 -0.068 0.407
SynSoph 0.825 0.348 0.092 0.118 0.259
AvgNounBl -0.017 0.705 0.149 0.361 0.031
AvgPronounBl 0.159 0.77 0.042 0.026 -0.004
AvgVerbBl 0.102 0.911 0.04 0.051 0.058
AvgAdverbBl 0.043 0.697 0.012 -0.126 -0.103
AvgAdjectiveBl -0.011 0.517 0.115 0.028 0.031
AvgPrepositionBl 0.078 0.786 0.06 0.154 -0.004
AvgNounSen 0.834 -0.018 0.109 0.357 -0.019
AvgPronounSen 0.928 0.03 0.053 0.017 -0.023
AvgVerbSen 0.957 0.1 0.031 0.013 0.042
AvgAdverbSen 0.701 0.22 -0.042 -0.147 -0.05
AvgAdjectiveSen 0.699 0.007 0.137 0.004 -0.055
AvgPrepositionSen 0.803 0.212 0.018 0.139 -0.013
AvgUnqNoundBl -0.014 0.632 0.13 0.431 -0.023
AvgUnqPronounBl -0.007 0.533 0.088 0.074 0.058
AvgUnqVerbBl 0.122 0.858 0.047 0.038 0.043
AvgUnqAdverbBl -0.007 0.719 0.011 -0.17 -0.142
AvgUnqAdjectiveBl -0.025 0.523 0.104 -0.022 0.022
AvgUnqPrepositionBl 0.054 0.813 0.061 0.075 -0.009
AvgPronBl_first_person 0.168 0.637 0.017 0.058 -0.022
AvgPronBl_indefinite 0.011 0.577 0.018 0.033 0.138
AggPronSen_indefinite 0.677 0.159 -0.005 0.004 0.112
AvgPronBl_third_person 0.131 0.424 0.057 -0.018 0.019
AggPronSen_third_person 0.782 -0.078 0.043 -0.033 0.007
AvgSemDep 0.967 0.07 0.088 0.181 0.001
WdDiffLemmaStem -0.016 0.289 0.104 0.007 -0.106
WdDiffWdStem 0.041 0.483 0.031 -0.313 0.132
WdMaxDpthHypernymTree -0.08 0.083 0.356 0.059 0.349
WdAvgDpthHypernymTree -0.064 0.073 0.369 0.063 0.349
WdPathCntHypernymTree -0.066 0.123 0.217 -0.019 0.464
WdPolysemyCnt 0 -0.045 -0.037 0.192 0.244
WdSylCnt -0.126 0.215 0.592 -0.117 0.108
AvgAOADoc_Shock -0.084 0.426 0.396 0.15 0.04
AvgAOABl_Shock -0.084 0.426 0.396 0.15 0.04
AvgAOASen_Shock 0.289 0.196 0.419 0.105 0.111
AvgAOADoc_Cortese 0.011 0.056 0.743 -0.088 0.221
AvgAOABl_Cortese 0.011 0.056 0.743 -0.088 0.221
AvgAOASen_Cortese 0.261 0.168 0.575 0.045 0.211
AvgAOADoc_Kuperman -0.016 0.083 0.785 0.109 -0.07
AvgAOABl_Kuperman -0.016 0.083 0.785 0.109 -0.07
AvgAOASen_Kuperman 0.1 0.18 0.742 0.068 -0.023
AvgAOADoc_Bird 0.011 0.057 0.76 -0.004 0.135
AvgAOABl_Bird 0.011 0.057 0.76 -0.004 0.135
AvgAOASen_Bird 0.262 0.14 0.554 0.029 0.173
AvgAOADoc_Bristol 0.018 0.308 0.513 0.022 0.201
AvgAOABl_Bristol 0.018 0.308 0.513 0.022 0.201
AvgAOASen_Bristol 0.413 0.102 0.453 0.113 0.185
AvgAOEDoc_IndexPolyFitAbThr.0.3. -0.049 0.027 0.591 0.176 -0.597
AvgAOEBl_IndexPolyFitAbThr.0.3. -0.049 0.027 0.591 0.176 -0.597
AvgAOESen_IndexPolyFitAbThr.0.3. 0.006 0.133 0.617 0.152 -0.495
AvgAOEDoc_InverseLinearRegressionSlope -0.082 0.015 0.857 0.089 -0.285
AvgAOEBl_InverseLinearRegressionSlope -0.082 0.015 0.857 0.089 -0.285
AvgAOESen_InverseLinearRegressionSlope 0.077 0.163 0.795 0.054 -0.145
AvgAOEDoc_InflectionPointPolynomial -0.099 0.06 0.848 -0.017 -0.214
AvgAOEBl_InflectionPointPolynomial -0.099 0.06 0.848 -0.017 -0.214
AvgAOESen_InflectionPointPolynomial 0.064 0.198 0.8 -0.021 -0.076
AvgAOEDoc_InverseAverage -0.102 0.053 0.857 -0.014 -0.243
AvgAOEBl_InverseAverage -0.102 0.053 0.857 -0.014 -0.243
AvgAOESen_InverseAverage 0.06 0.191 0.808 -0.021 -0.097
AvgAOEDoc_IndexAboveThreshold.0.3. -0.093 0.045 0.476 0.179 -0.636
AvgAOEBl_IndexAboveThreshold.0.3. -0.093 0.045 0.476 0.179 -0.636
AvgAOESen_IndexAboveThreshold.0.3. -0.071 0.107 0.491 0.174 -0.556
AvgNmdEntBl 0.057 0.52 -0.052 0.139 0.032
AvgNounNmdEntBl 0.088 0.363 -0.01 0.198 -0.015
AvgUnqNmdEntBl 0.119 0.537 -0.058 0.14 -0.003
AvgNmdEntSen 0.752 0.106 -0.084 0.167 0.06
TCorefChainDoc -0.03 0.621 0.189 0.003 -0.062
AvgCorefChain 0.143 0.47 0.111 0.016 0.079
AvgChainSpan 0.088 0.729 0.038 -0.003 0.128
AvgInferenceDistChain 0.245 0.306 0.046 0.001 -0.012
TActCorefChainWd -0.092 -0.329 0.268 -0.225 -0.102
TCorefChainBigSpan 0.108 0.426 0.243 -0.098 0.002
AvgConnBl_addition 0.067 0.309 0.032 0.777 0.055
AvgConnSen_addition 0.658 -0.166 0.072 0.622 -0.077
AvgConnBl_conjunctions 0.168 0.362 0.061 0.775 -0.017
AvgConnSen_conjunctions 0.72 -0.147 0.097 0.579 -0.152
AvgConnBl_contrasts 0.076 0.444 0.114 0.035 -0.118
AvgConnSen_contrasts 0.512 0.196 0.125 -0.059 -0.149
AvgConnBl_coordinating_conjuncts 0.381 0.45 -0.054 0.141 0.092
AvgConnSen_coordinating_conjuncts 0.72 0.205 -0.012 -0.003 -0.013
AvgConnBl_coordinating_connectives 0.255 0.506 0.035 0.7 0.003
AvgConnSen_coordinating_connectives 0.818 -0.034 0.071 0.48 -0.119
AvgConnBl_logical_connectors 0.186 0.317 0.046 0.76 -0.008
AvgConnSen_logical_connectors 0.693 -0.154 0.069 0.563 -0.121
AvgConnBl_oppositions 0.096 0.391 0.08 0.048 -0.132
AvgConnSen_oppositions 0.494 0.14 0.106 -0.083 -0.187
AvgConnBl_order -0.115 0.255 -0.013 0.188 0.176
AvgConnSen_order 0.315 0.084 -0.013 0.115 0.215
AvgConnBl_reason_and_purpose 0.204 0.536 -0.017 0.194 0.163
AvgConnSen_reason_and_purpose 0.71 0.226 0.002 0.034 0.064
AvgConnBl_semi_coordinators 0.381 0.45 -0.054 0.141 0.092
AvgConnSen_semi_coordinators 0.72 0.205 -0.012 -0.003 -0.013
AvgConnBl_sentence_linking 0.241 0.603 0.014 0.606 0.063
AvgConnSen_sentence_linking 0.862 0.027 0.048 0.411 -0.059
AvgConnBl_simple_subordinators 0.081 0.52 0.062 -0.082 -0.029
AvgConnSen_simple_subordinators 0.681 0.165 -0.053 -0.039 0.023
AvgConnBl_temporal_connectors 0.121 0.382 -0.081 -0.07 0.114
AvgConnSen_temporal_connectors 0.637 0.153 -0.183 -0.036 0.106
LexChainAvgSpan 0.128 0.407 0.308 0.073 0.415
LexChainMaxSp -0.007 0.663 0.02 0.171 0.207
AvgBlScore 0.148 0.801 0.043 0.165 0.285
AvgSenScore 0.909 0.12 0.044 0.055 0.165
SenScoreStDev -0.466 0.332 0.071 0.09 0.501
AvgIntraBlCoh_LeackockChodorow -0.741 0.302 0.152 0.008 0.471
AvgSenAdjCoh_LeackockChodorow -0.735 0.263 0.166 -0.002 0.488
AvgSenBlCoh_LeackockChodorow 0.434 -0.139 0.696 0.053 0.24
AvgIntraBlCoh_WuPalmer -0.748 0.296 0.152 0.001 0.465
AvgSenAdjCoh_WuPalmer -0.743 0.259 0.167 -0.01 0.482
AvgSenBlCoh_WuPalmer 0.443 -0.145 0.694 0.049 0.226
AvgIntraBlCoh_Path -0.749 0.277 0.154 -0.007 0.46
AvgSenAdjCoh_Path -0.744 0.234 0.175 -0.021 0.478
AvgSenBlCoh_Path 0.528 -0.219 0.642 0.044 0.155
AvgIntraBlCoh_LSA -0.739 0.338 0.148 0.005 0.446
AvgSenAdjCoh_LSA -0.729 0.295 0.166 -0.011 0.473
AvgSenBlCoh_LSA 0.598 -0.209 0.547 0.09 0.148
AvgIntraBlCoh_LDA -0.738 0.345 0.151 -0.004 0.449
AvgSenAdjCoh_LDA -0.718 0.324 0.158 -0.003 0.473
AvgSenBlCoh_LDA 0.513 -0.066 0.598 0.08 0.202
AvgIntraBlCoh_word2vec -0.743 0.31 0.158 -0.026 0.457
AvgSenAdjCoh_word2vec -0.734 0.273 0.179 -0.046 0.48
AvgSenBlCoh_word2vec 0.581 -0.259 0.577 0.048 0.165
AvgBlVoiceCoOcc 0.083 0.49 -0.117 0.351 0.258
AvgVoice 0.086 0.506 -0.117 0.355 0.247
AvgSenSyll 0.973 0.081 0.076 0.138 -0.008
AvgSenStressedSyll 0.939 0.141 0.049 0.106 0.117
AvgRhythmUnits 0.315 0.184 -0.003 -0.262 0.043
AvgRhythmUnitSyll 0.807 -0.027 0.082 0.284 -0.045
AvgRhythmUnitStreesSyll 0.809 0.024 0.053 0.235 0.073
LangRhythmCoeff 0.24 0.048 -0.011 0.129 -0.121
LangRhythmId -0.173 -0.023 -0.06 0.028 -0.387
FrqRhythmId 0.657 -0.336 -0.016 0.096 0.056
LangRhythmDiameter 0.373 0.113 0.117 0.197 -0.139
SenAsson -0.051 0.221 0.079 0.066 0.065
AvgDepsBl_acl 0.092 0.23 0.074 0.206 0.093
AvgDepsSen_acl 0.485 0.081 0.031 0.343 0.108
AvgDepsBl_advcl 0.394 0.519 -0.037 0.042 0.175
AvgDepsSen_advcl 0.768 0.219 -0.041 0.009 0.065
AvgDepsBl_advmod 0.141 0.71 -0.023 -0.136 -0.045
AvgDepsSen_advmod 0.753 0.193 -0.054 -0.132 -0.006
AvgDepsBl_amod -0.092 0.476 0.104 0.038 0.13
AvgDepsSen_amod 0.563 0.072 0.117 -0.001 0.112
AvgDepsBl_aux -0.02 0.402 0.09 -0.282 0.005
AvgDepsSen_aux 0.621 0.072 0.112 -0.18 -0.071
AvgDepsBl_auxpass -0.079 0.424 0.001 -0.052 -0.166
AvgDepsBl_case -0.121 0.772 0.094 0.169 -0.072
AvgDepsSen_case 0.747 0.175 0.093 0.178 -0.024
AvgDepsBl_cc 0.203 0.37 0.063 0.746 -0.039
AvgDepsSen_cc 0.73 -0.143 0.088 0.553 -0.149
AvgDepsBl_ccomp 0.383 0.496 0.036 -0.022 0.173
AvgDepsSen_ccomp 0.781 0.073 0.031 -0.041 0.09
AvgDepsBl_compound 0.104 0.149 0.038 0.366 -0.006
AvgDepsSen_compound 0.587 -0.062 0.073 0.412 -0.064
AvgDepsBl_conj 0.31 0.297 0.122 0.722 -0.045
AvgDepsSen_conj 0.731 -0.134 0.134 0.536 -0.162
AvgDepsBl_cop -0.001 0.419 0.055 -0.12 -0.002
AvgDepsSen_cop 0.583 0.061 0.018 -0.107 0.003
AvgDepsBl_dep 0.234 0.123 0.104 0.47 -0.059
AvgDepsSen_dep 0.527 -0.187 0.15 0.494 -0.174
AvgDepsBl_det -0.182 0.549 0.13 0.222 0.071
AvgDepsSen_det 0.576 -0.06 0.152 0.415 -0.023
AvgDepsBl_dobj 0.119 0.64 0.078 0.196 0.13
AvgDepsSen_dobj 0.856 -0.002 0.047 0.128 0.028
AvgDepsBl_mark 0.367 0.535 0.01 0.087 0.139
AvgDepsSen_mark 0.795 0.152 -0.063 0.085 0.073
AvgDepsBl_mwe -0.01 0.262 0.021 0.142 0.008
AvgDepsSen_mwe 0.383 0.097 0.025 0.044 0.031
AvgDepsBl_neg 0.197 0.361 0.053 -0.181 -0.132
AvgDepsSen_neg 0.549 0.073 -0.024 -0.17 -0.086
AvgDepsBl_nmod -0.134 0.727 0.094 0.237 -0.052
AvgDepsSen_nmod 0.661 0.15 0.086 0.281 -0.02
AvgDepsBl_nsubj 0.141 0.88 0.062 0.041 0.08
AvgDepsSen_nsubj 0.969 0.02 0.051 0.018 0.003
AvgDepsBl_nsubjpass -0.015 0.417 -0.005 -0.065 -0.158
AvgDepsBl_nummod 0.132 0.492 -0.071 -0.009 0.13
AvgDepsBl_punct -0.483 0.644 -0.017 -0.108 -0.086
AvgDepsSen_punct -0.192 0.389 0.063 -0.232 0.191
AvgDepsBl_xcomp 0.053 0.501 0.03 0.186 0.068
AvgDepsSen_xcomp 0.666 0.075 0.061 0.196 0.024