Ensemble Weightings and Metric Importance

ReaderBench Model 3per

This model used ReaderBench scores from 15 min persuasive writing samples from 202 students in Grades 2-5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons).

Highly correlated ReaderBench metrics (r > |.90|) were excluded during pre-processing (see section on Scoring Model Development for more details).

Algorithm Weightings in Ensemble

Abbreviations:

• overall = ensemble model
• pls = partial least squares regression
• svm = support vector machines
• enet = elastic net regression
• rf = random forest regression
• mars = bagged multivariate adaptive regression splines
• cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept	pls	mars	gbm	svm	enet	cube
-0.0141	0.0326	0.2043	0.2331	0.1507	0.3202	0.0801

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

Metric	overall	pls	mars	gbm	svm	enet	cube
RB.WdEnt	9.44	1.97	0	16.45	2.58	14.62	8.38
RB.AvgPrepositionBl	8.44	1.96	20.57	11.55	2.48	2.8	4.83
Sentences	6.71	1.67	19.13	4.09	1.41	4.14	5.01
RB.AvgBlScore	5.39	2	0	15.75	2.73	2.76	5.92
RB.CAF	4.59	1.59	19.13	1.27	1.43	0	2.73
RB.AvgSenScore	3.72	0.51	8.2	0.15	0.47	5.74	2
RB.TCorefChainDoc	3.36	1.97	0	5.74	2.16	4.69	2.46
RB.AvgWdLen	2.49	1.32	0	4.98	1.13	2.84	3.28
RB.AvgAOADoc_Shock	2.39	1.4	8.2	1.65	1.21	0.2	1.18
RB.AvgPronBl_indefinite	2.34	1.76	0	3.91	2.01	2.57	3.73
RB.RdbltyDaleChall	2.32	0.79	0	1.32	0.68	5.11	3.73
RB.AvgDepsBl_compound	2.28	0.23	7.3	0.29	0.02	1.88	2
RB.AvgUnqNoundBl	2.11	0.75	1.47	0.3	1.41	4.17	2.64
RB.AvgConnBl_simple_subordinators	1.75	1.76	0	3.08	1.98	2.11	0.46
RB.AvgAOESen_InflectionPointPolynomial	1.61	0.53	5.45	0.22	0.77	0.99	0.36
RB.AvgPronBl_interrogative	1.23	0.61	0	0.53	0.13	2.92	2
RB.AvgDepsBl_nsubj	1.12	1.87	0	1.87	2.47	0	3.37
RB.AvgDepsBl_mark	1.1	1.63	0	0.89	1.7	1.41	1.91
RB.AvgNmdEntSen	1.07	0.07	0	0.25	0.41	2.79	0.91
RB.AvgDepsBl_amod	1.06	0.82	0	0.33	0.41	2.72	0.64
RB.AvgCorefChain	1.04	0.95	0	0.08	1.1	2.38	1
RB.AvgDepsSen_advmod	1.01	0.09	0	0.22	0.22	2.68	1
RB.AvgPronBl_first_person	0.96	0.7	2.97	0.06	0.27	0.75	0.64
RB.LangRhythmCoeff	0.95	0.77	0	1.43	0.81	1.35	0.73
RB.AvgAOABl_Bird	0.93	0.47	3.59	0.46	0.63	0	0
RB.AvgDepsSen_aux	0.92	0	4	0.21	0.18	0	0.55
RB.AvgSenAdjCoh_Path	0.87	1.19	0	2.13	1.26	0.33	0.73
RB.AvgDepsBl_det	0.83	1.51	0	0.54	1.48	1.21	0.73
RB.AvgConnSen_oppositions	0.8	0.24	0	0.41	0.01	2.1	0.46
RB.AvgAOASen_Shock	0.8	0.62	0	0.13	1.02	1.67	0.91
RB.LxcDiv	0.8	1.45	0	1.85	1.4	0	1.55
RB.AvgUnqPronounBl	0.77	1.68	0	0.64	1.73	0.63	1.46
RB.AvgAOADoc_Cortese	0.69	0.02	0	0.3	0.59	1.5	0.73
RB.AvgUnqAdjectiveBl	0.69	1.13	0	0.03	0.81	1.58	0.36
RB.AvgDepsBl_nsubjpass	0.69	0.48	0	0.02	0.16	2.05	0.09
RB.AvgAOASen_Bird	0.64	0.52	0	0.31	0.39	1.46	0.46
RB.AvgDepsBl_cop	0.63	1.09	0	0.1	0.8	1.3	0.55
RB.TCorefChainBigSpan	0.6	1.53	0	0.34	1.38	0.62	1
RB.AvgChainSpan	0.59	1.42	0	1	1.73	0	0.73
RB.AvgDepsBl_aux	0.59	1.42	0	0.59	1.36	0.5	0.64
RB.AvgUnqPrepositionBl	0.58	1.83	0	0.66	2.15	0	0.64
RB.AggPronSen_second_person	0.54	0.32	0	0.04	0.55	1.28	0.46
RB.SynDiv	0.51	1.15	0	0.48	1.14	0.52	0.46
RB.CharEnt	0.49	1.19	0	1.21	1.21	0	0
RB.AvgAOASen_Bristol	0.48	0.35	0	0.13	0.36	1.1	0.46
RB.AvgDepsBl_punct	0.47	1.51	0	0.72	1.39	0	0.64
RB.AvgDepsBl_nmod	0.46	1.6	0	0.48	1.72	0	0.55
RB.AvgUnqVerbBl	0.43	1.51	0	0.41	1.47	0	0.91
RB.WdDiffLemmaStem	0.42	0.86	0	0.43	0.9	0.48	0.18
RB.AvgDepsSen_mark	0.42	0.28	0	0.1	0.29	0.68	1.73
RB.WdDiffWdStem	0.42	0.67	0	0.31	0.66	0.71	0.18
RB.AvgPronounBl	0.41	1.67	0	0.06	1.67	0	1.18
RB.AvgAOASen_Cortese	0.41	0.08	0	0.15	0.3	0.88	0.73
RB.AvgConnBl_temporal_connectors	0.41	0.71	0	0.02	0.38	1.05	0
RB.AvgRhythmUnitStreesSyll	0.38	0.09	0	0.12	0.17	0.87	0.73
RB.LxcSoph	0.37	0.75	0	0.68	0.65	0	1.18
RB.AvgDepsBl_ccomp	0.34	1.38	0	0.08	1.26	0.12	0.64
RB.AvgDepsSen_neg	0.34	0.23	0	0.09	0.52	0.74	0
RB.AvgPronBl_third_person	0.34	1.34	0	0.39	1.16	0	0.46
RB.AvgDepsBl_root	0.33	0.09	0	0.06	0	1	0
RB.TActCorefChainWd	0.33	0.36	0	0.36	0.81	0.14	0.91
RB.WdSylCnt	0.3	0.76	0	0.54	0.7	0	0.64
RB.AvgUnqAdverbBl	0.29	1.34	0	0.09	1.2	0	0.64
RB.AvgDepsSen_punct	0.27	0.44	0	0.09	0.16	0.68	0
RB.AvgNmdEntBl	0.26	1.25	0	0.11	1.05	0	0.55
RB.AvgConnBl_addition	0.25	1.13	0	0.16	0.9	0	0.55
RB.AvgDepsSen_compound	0.25	0.5	0	0.19	0.56	0	1.37
RB.AggPronSen_indefinite	0.25	0.42	0	0.24	0.9	0	0.64
RB.AvgDepsBl_dobj	0.25	1.37	0	0.02	1.15	0	0.46
RB.AvgConnBl_order	0.24	0.56	0	0.03	0.21	0.59	0
RB.AvgAOADoc_Bristol	0.24	0.8	0	0.28	0.89	0	0.27
RB.SenStdDevWd	0.24	0.98	0	0.18	1.06	0	0.18
RB.FrqRhythmId	0.23	1.1	0	0.02	0.72	0.21	0.18
RB.AvgDepsBl_advmod	0.23	1.21	0	0.12	0.96	0	0.27
RB.AvgDepsBl_advcl	0.23	1.36	0	0.01	1.26	0	0
RB.AvgAdverbBl	0.23	1.25	0	0.1	1.01	0	0.27
RB.AvgConnBl_logical_connectors	0.22	1.14	0	0.2	0.89	0	0.09
RB.AvgConnBl_semi_coordinators	0.21	0.2	0	0.02	0.03	0.56	0.18
RB.AvgPronounSen	0.21	0.33	0	0.15	0.72	0	0.73
RB.AvgUnqNmdEntBl	0.21	1	0	0.18	0.65	0	0.55
RB.AvgConnSen_simple_subordinators	0.2	0.46	0	0.17	0.74	0	0.46
RB.AvgSenBlCoh_LDA	0.2	0.59	0	0.06	0.91	0.01	0.36
RB.AvgConnBl_reason_and_purpose	0.2	1.2	0	0.09	0.96	0	0
RB.AvgDepsSen_amod	0.2	0.18	0	0.18	0.62	0	0.82
RB.AvgInferenceDistChain	0.19	0.33	0	0.23	0.81	0	0.09
RB.AvgAOESen_IndexPolynomialFitAboveThreshold.0.3.	0.19	0.68	0	0.32	0.65	0	0
RB.AvgSenBlCoh_LSA	0.19	0.97	0	0.09	0.86	0	0.18
RB.AvgAOEDoc_InverseAverage	0.18	0.62	0	0.17	0.82	0	0
RB.AvgAOEBl_IndexAboveThreshold.0.3.	0.18	0.59	0	0.17	0.71	0.06	0
RB.SenAllit	0.18	0.53	0	0.01	0.19	0.43	0
RB.AvgDepsSen_dep	0.18	0.19	0	0.2	0.42	0	0.91
RB.AvgDepsBl_nummod	0.17	0.66	0	0.11	0.32	0.24	0
RB.AvgDepsSen_det	0.16	0.21	0	0.08	0.89	0	0
RB.AvgDepsBl_conj	0.16	1.02	0	0.11	0.63	0	0.09
RB.AvgDepsSen_ccomp	0.16	0.3	0	0.15	0.55	0	0.46
RB.AvgConnSen_addition	0.15	0.01	0	0.11	0.52	0	0.55
RB.AvgDepsSen_acl	0.15	0.15	0	0.13	0.01	0.36	0
RB.AvgDepsBl_xcomp	0.14	0.89	0	0.04	0.56	0	0.18
RB.AvgPronBl_second_person	0.14	0.91	0	0.05	0.55	0	0.18
RB.AvgAOABl_Kuperman	0.14	0.08	0	0.23	0.45	0	0.18
RB.AvgNounSen	0.14	0.2	0	0.03	0.22	0	1.18
RB.AvgConnBl_contrasts	0.14	1.03	0	0.05	0.67	0	0
RB.WdLettStdDev	0.13	0.6	0	0.19	0.39	0	0.09
RB.AvgDepsBl_neg	0.13	0.3	0	0.03	0.05	0.33	0
RB.AvgDepsSen_xcomp	0.13	0.06	0	0.05	0.58	0	0.36
RB.AvgDepsSen_advcl	0.13	0.14	0	0.13	0.66	0	0
RB.AvgConnBl_oppositions	0.13	0.98	0	0.03	0.54	0	0.18
RB.AggPronSen_first_person	0.13	0.06	0	0.22	0.56	0	0
RB.AggPronSen_third_person	0.12	0.38	0	0.02	0.69	0	0
RB.AvgDepsSen_dobj	0.1	0.07	0	0.07	0.31	0	0.46
RB.AvgAdjectiveSen	0.1	0.04	0	0.09	0.4	0	0.27
RB.AvgDepsSen_cop	0.1	0.14	0	0.04	0.59	0	0
RB.AvgConnSen_reason_and_purpose	0.09	0.05	0	0.06	0.25	0	0.46
RB.AvgConnBl_conditions	0.09	0.72	0	0.04	0.39	0	0
RB.LangRhythmDiameter	0.09	0.29	0	0.13	0.06	0.15	0
RB.AvgDepsBl_acl	0.09	0.74	0	0.03	0.39	0	0.09
RB.AvgAOASen_Kuperman	0.08	0.09	0	0.11	0.26	0	0.18
RB.AvgConnBl_disjunctions	0.08	0.52	0	0.03	0.21	0.08	0
RB.AvgDepsSen_nmod	0.08	0.02	0	0.15	0.17	0	0.27
RB.AvgCommaBl	0.08	0.78	0	0.02	0.36	0	0
RB.AvgDepsBl_mwe	0.07	0.67	0	0.01	0.33	0	0
RB.AvgDepsBl_dep	0.07	0.64	0	0.07	0.22	0	0.09
RB.AvgConnSen_semi_coordinators	0.06	0.13	0	0.01	0.01	0.11	0.27
RB.AvgConnSen_conditions	0.05	0.01	0	0.2	0	0	0
RB.AvgConnBl_conjuncts	0.04	0.42	0	0.01	0.14	0	0
RB.LangRhythmId	0.04	0.39	0	0.04	0.1	0	0
RB.AvgDepsBl_csubj	0.03	0.39	0	0	0.09	0	0
RB.AvgDepsBl_iobj	0.03	0.29	0	0.03	0.09	0	0
RB.AvgDepsSen_nummod	0.03	0.13	0	0.09	0.01	0	0
RB.AvgDepsBl_auxpass	0.03	0.36	0	0	0.11	0	0
RB.AvgDepsBl_expl	0.03	0.29	0	0	0.09	0.03	0
RB.SenAsson	0.03	0.37	0	0.02	0.1	0	0
RB.AvgCommaSen	0.03	0.14	0	0.05	0.02	0	0.18
RB.AvgDepsSen_csubj	0.01	0.04	0	0.02	0	0	0
RB.AvgConnSen_disjunctions	0.01	0.07	0	0.03	0.01	0	0
RB.AvgDepsBl_parataxis	0.01	0.2	0	0	0.03	0	0
RB.AvgConnBl_complex_subordinators	0	0.06	0	0	0.01	0	0
RB.AvgConnSen_temporal_connectors	0	0.01	0	0.01	0	0	0