Ensemble Weightings and Metric Importance

ReaderBench Model 3exp

This model used ReaderBench scores from 15 min expository writing samples from 200 students in Grades 2-5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons).

Highly correlated ReaderBench metrics (r > |.90|) were excluded during pre-processing (see section on Scoring Model Development for more details).

Algorithm Weightings in Ensemble

Abbreviations:

• overall = ensemble model
• pls = partial least squares regression
• gbm = stochastic gradient boosted trees
• svm = support vector machines
• enet = elastic net regression
• rf = random forest regression
• mars = bagged multivariate adaptive regression splines
• cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept	rf	mars	gbm	svm	enet	cube
-0.0156	0.0826	0.3112	0.0319	0.1360	0.3306	0.1259

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

Metric	overall	rf	mars	gbm	svm	enet	cube
Content.words	20.83	5.13	35.84	48.17	3.43	17.49	14.66
RB.AvgWdLen	4.5	1.29	10.64	1.06	0.42	1.6	4.32
RB.AvgDepsBl_compound	4.11	0.71	8.06	0.36	0.06	3.36	3.92
RB.AvgConnBl_order	3.63	0.6	6.17	0.15	0.21	4.43	1.81
RB.SenStdDevWd	3.56	1.02	5.2	0.29	1.18	4.38	2.41
RB.LangRhythmId	3.46	0.96	10.64	0.29	0.26	0	0.7
RB.TCorefChainDoc	3.34	1.93	0	4.43	2.48	7.01	3.51
RB.WdEnt	3.21	2.23	0	1.79	2.35	6.08	5.52
RB.AggPronSen_first_person	2.93	0.99	8.76	0.77	0.12	0	1.1
Sentences	2.9	1.37	0	2.38	1.47	6.99	2.01
RB.AvgSenAdjCoh_Path	2.68	1.05	0	1.28	1.09	5.93	3.92
RB.CAF	2.51	1.33	0	1.24	1.75	5.87	1.81
RB.AvgPronBl_third_person	2.49	0.76	7.52	0.04	0.79	0	0.2
RB.AvgBlScore	2.27	2.1	4.33	1.67	2.38	0	3.31
RB.AvgPronBl_second_person	2.03	1.17	0	0.64	0.95	4.73	2.01
RB.LangRhythmDiameter	1.92	0.73	2.84	0.28	0.15	2.24	1.91
RB.TActCorefChainWd	1.4	0.76	0	0.47	1.04	2.97	1.81
RB.TCorefChainBigSpan	1.31	0.44	0	1.17	1.63	2.75	1
RB.AvgUnqAdjectiveBl	1.06	1.01	0	0.23	1.7	1.95	0.9
RB.WdDiffWdStem	0.99	1.06	0	1.8	0.52	2.17	0.6
RB.AvgDepsSen_nmod	0.94	0.95	0	0.52	1.14	1.67	1.2
RB.AvgDepsBl_expl	0.89	0.79	0	0.42	0.42	1.86	1.2
RB.RdbltyDaleChall	0.86	1.25	0	0.91	0.78	1.01	2.41
RB.AvgAOEBl_InflectionPointPolynomial	0.77	0.72	0	0.27	0.7	1.85	0.1
RB.AvgConnBl_temporal_connectors	0.76	0.91	0	0.04	0.5	1.37	1.41
RB.AvgPronBl_indefinite	0.75	2.03	0	5.56	1.56	0	1.61
RB.SynDiv	0.71	0.61	0	0.28	1.04	1.39	0.5
RB.LxcDiv	0.69	1.51	0	1.57	2.14	0	1.91
RB.AvgAOASen_Bristol	0.66	0.56	0	0.14	0.31	1.56	0.5
RB.AvgDepsBl_root	0.65	0.09	0	0.04	0.04	1.95	0
RB.AvgDepsBl_nsubj	0.62	1.9	0	0.88	2.19	0	1.2
RB.AvgPronounBl	0.59	1.54	0	0.25	1.94	0	1.61
RB.AvgPrepositionBl	0.59	1.37	0	1.41	2.07	0	1.31
RB.AvgUnqNoundBl	0.49	0.83	0	0.41	1.02	0	2.21
RB.AvgDepsBl_parataxis	0.47	0.53	0	0.01	0.15	1.26	0
RB.LangRhythmCoeff	0.44	0.58	0	0.33	0.4	0.93	0.2
RB.AvgUnqPrepositionBl	0.43	0.94	0	0.2	2.05	0	0.6
RB.AvgAOASen_Bird	0.43	0.63	0	0.4	0.79	0.63	0.5
RB.WdSylCnt	0.42	0.96	0	0.54	0.18	0.5	1.1
RB.AvgDepsBl_nmod	0.42	1.01	0	0.52	1.59	0	0.9
RB.AvgChainSpan	0.41	1.04	0	0.4	1.58	0	0.8
RB.AvgDepsBl_nummod	0.41	0.7	0	0.01	0.21	1	0
RB.AvgDepsSen_expl	0.4	0.41	0	0.23	0.06	1.1	0
RB.AvgPronBl_first_person	0.39	0.71	0	0.51	0.5	0.25	1.41
RB.AvgUnqVerbBl	0.38	0.91	0	0.06	1.71	0	0.6
RB.AvgDepsBl_aux	0.37	0.59	0	0.19	0.93	0.4	0.5
RB.AvgAdverbBl	0.33	0.6	0	0.11	1.31	0	0.8
RB.AvgDepsBl_punct	0.33	1.26	0	0.3	1.17	0	0.5
RB.AvgNounSen	0.33	0.99	0	0.05	0.22	0	1.81
RB.LxcSoph	0.32	0.79	0	0.3	0.75	0	1.2
RB.CharEnt	0.31	0.49	0	1.05	1.09	0.13	0.4
RB.AvgDepsSen_cop	0.31	0.86	0	0.55	0.55	0	1.2
RB.AvgDepsBl_mark	0.31	1.04	0	0.56	1.59	0	0
RB.AvgSenBlCoh_LDA	0.3	0.82	0	0.16	1.15	0	0.6
RB.RdbltyFlesch	0.29	0.47	0	0.19	0.17	0	1.81
RB.AvgCorefChain	0.28	0.76	0	0.2	1.05	0	0.6
RB.AvgDepsBl_dobj	0.28	0.92	0	0.09	1.36	0	0.2
RB.AvgDepsBl_cop	0.27	0.59	0	0.07	0.97	0	0.7
RB.AvgDepsBl_det	0.27	0.92	0	0.09	1.36	0	0.1
RB.AvgDepsSen_mark	0.27	0.68	0	0.19	1.12	0	0.5
RB.AvgDepsBl_amod	0.26	0.58	0	0.27	1.23	0	0.3
RB.AvgDepsBl_mwe	0.25	0.8	0	0.09	0.61	0.3	0
RB.AvgUnqAdverbBl	0.25	0.6	0	0.03	1.39	0	0.1
RB.AvgPrepositionSen	0.24	0.44	0	0.16	0.91	0	0.6
RB.AvgConnBl_simple_subordinators	0.23	0.76	0	0.05	1.22	0	0
RB.AvgAOASen_Kuperman	0.23	0.53	0	0.51	0.39	0.2	0.4
RB.AvgDepsSen_compound	0.23	1.22	0	0.33	0.33	0	0.6
RB.AvgDepsBl_ccomp	0.22	0.51	0	0.05	0.54	0.2	0.3
RB.AvgUnqPronounBl	0.22	0.46	0	0	1.33	0	0
RB.FrqRhythmId	0.22	0.94	0	0.3	0.68	0.06	0.2
RB.AggPronSen_indefinite	0.22	0.76	0	0.37	0.93	0	0.2
RB.AvgDepsSen_dobj	0.21	0.98	0	0.1	0.49	0	0.5
RB.AggPronSen_second_person	0.2	0.81	0	0.23	0.64	0	0.3
RB.AvgAOADoc_Shock	0.2	0.98	0	0.42	0.82	0	0
RB.AvgConnSen_semi_coordinators	0.19	0.59	0	0.29	0	0.38	0.1
RB.AvgConnBl_addition	0.18	0.7	0	0.23	0.65	0	0.2
RB.AvgRhythmUnitStreesSyll	0.18	0.89	0	0.17	0.47	0	0.3
RB.AvgDepsSen_ccomp	0.18	0.31	0	0.22	0.94	0	0.2
RB.AvgAdverbSen	0.17	0.38	0	0.06	0.99	0	0
RB.AvgCommaSen	0.17	0.62	0	0.25	0.8	0	0
RB.AvgAOEDoc_IndexAboveThreshold.0.3.	0.17	0.72	0	0.12	0.36	0	0.5
RB.AvgConnBl_contrasts	0.17	0.46	0	0.08	0.82	0	0.2
RB.AvgConnSen_simple_subordinators	0.16	0.44	0	0.13	0.88	0	0
RB.AvgConnBl_reason_and_purpose	0.16	0.73	0	0.14	0.62	0	0.1
RB.AvgAOADoc_Bird	0.16	0.79	0	0.14	0.68	0	0
RB.AvgDepsSen_amod	0.16	0.29	0	0.25	0.5	0	0.5
RB.AvgConnBl_oppositions	0.16	0.65	0	0.05	0.6	0.02	0.2
RB.AvgAOABl_Kuperman	0.15	0.11	0	0.18	0.45	0	0.6
RB.AvgDepsSen_xcomp	0.15	0.63	0	0.06	0.73	0	0
RB.AvgPronounSen	0.14	0.62	0	0.03	0.26	0	0.4
RB.AvgDepsBl_advcl	0.14	0.21	0	0.02	0.89	0	0
RB.AvgInferenceDistChain	0.14	0.56	0	0.2	0.45	0	0.2
RB.AvgNounNmdEntBl	0.14	0.49	0	0.87	0.55	0	0
RB.AggPronSen_third_person	0.14	0.65	0	0.14	0.64	0	0
RB.WdLettStdDev	0.14	0.65	0	0.18	0.63	0	0
RB.AvgConnSen_addition	0.13	0.47	0	0.23	0.63	0	0
RB.AvgNmdEntSen	0.13	0.18	0	0.36	0.81	0	0
RB.WdDiffLemmaStem	0.12	0.71	0	0.26	0.29	0	0.1
RB.AvgDepsSen_aux	0.12	0.4	0	0.03	0.64	0	0
RB.AvgCommaBl	0.12	0.66	0	0.04	0.4	0	0.1
RB.AvgAOASen_Shock	0.12	0.28	0	0.05	0.73	0	0
RB.AvgDepsBl_acl	0.12	0.47	0	0.13	0.6	0	0
RB.AvgAOABl_Cortese	0.12	0.28	0	0.1	0.64	0	0.1
RB.AvgDepsSen_advcl	0.12	0.46	0	0.25	0.59	0	0
RB.AvgDepsBl_xcomp	0.12	0.23	0	0.09	0.78	0	0
RB.AvgConnSen_temporal_connectors	0.11	0.73	0	0.06	0.09	0.06	0.1
RB.AvgAOESen_InflectionPointPolynomial	0.11	0.28	0	0.11	0.52	0	0.1
RB.AvgDepsSen_dep	0.11	0.49	0	0.17	0.38	0	0.1
RB.AvgAOASen_Cortese	0.11	0.22	0	0.17	0.66	0	0
RB.AvgDepsSen_det	0.11	0.14	0	0.12	0.54	0	0.2
RB.AvgConnSen_reason_and_purpose	0.11	0.39	0	0.12	0.58	0	0
RB.AvgAOABl_Bristol	0.1	0.45	0	0.15	0.37	0	0.1
RB.AvgDepsBl_iobj	0.09	0.74	0	0.21	0.17	0	0
RB.AvgDepsSen_mwe	0.09	0.64	0	0.44	0.21	0	0
RB.AvgConnSen_order	0.08	0.69	0	0.67	0.01	0	0
RB.AvgConnSen_oppositions	0.08	0.63	0	0.11	0.09	0	0.1
RB.AvgConnBl_disjunctions	0.08	0.5	0	0	0.32	0	0
RB.AvgConnSen_contrasts	0.07	0.6	0	0.11	0.11	0	0
RB.AvgDepsBl_auxpass	0.07	0.56	0	0.01	0.17	0	0
RB.AvgDepsSen_neg	0.07	0.47	0	0.31	0	0	0.2
RB.AvgConnBl_conditions	0.07	0.49	0	0.11	0.23	0	0
RB.AvgDepsBl_neg	0.06	0.19	0	0.03	0.23	0	0.1
RB.AvgPronBl_interrogative	0.06	0.54	0	0.04	0.14	0	0
RB.SenAsson	0.05	0.25	0	0	0.23	0	0
RB.AvgConnSen_disjunctions	0.05	0.55	0	0.03	0.05	0	0
RB.AvgConnBl_semi_coordinators	0.04	0.16	0	0.1	0.16	0	0
RB.AvgDepsSen_nummod	0.04	0.46	0	0.03	0	0	0
RB.AvgDepsSen_acl	0.04	0.46	0	0.01	0.01	0	0
RB.AvgDepsBl_csubj	0.04	0.4	0	0.01	0.05	0	0
RB.AvgDepsBl_nsubjpass	0.04	0.25	0	0	0.16	0	0
RB.AvgDepsBl_appos	0.04	0.51	0	0	0.02	0	0
RB.AvgDepsBl_dep	0.02	0	0	0.08	0.15	0	0
RB.SenAllit	0.02	0.3	0	0	0	0	0