Ensemble Weightings and Metric Importance

ReaderBench Model 3narr

This model used ReaderBench scores from 15 min narrative writing samples from 202 students in Grades 2-5 to predict holistic writing quality on the samples (theta scores calculated from paired comparisons).

Highly correlated ReaderBench metrics (r > |.90|) were excluded during pre-processing (see section on Scoring Model Development for more details).

Algorithm Weightings in Ensemble

Abbreviations:

• overall = ensemble model
• pls = partial least squares regression
• gbm = stochastic gradient boosted trees
• svm = support vector machines
• enet = elastic net regression
• rf = random forest regression
• mars = bagged multivariate adaptive regression splines
• cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept	pls	rf	mars	gbm	svm	enet	cube
0.0000	0.1419	0.0945	0.3143	0.0729	0.0816	0.1792	0.1538

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

Metric	overall	pls	rf	mars	gbm	svm	enet	cube
Content.words	13.76	2	2.22	32.41	16.24	2.38	4.95	8.73
RB.AvgWdLen	7.5	0.81	1.06	21.05	1.14	1.01	1.38	3.55
RB.AvgDepsBl_compound	4.66	0.89	1.01	11.15	2.17	0.45	2.45	3.09
RB.WdEnt	4.59	1.7	2.06	7.03	4.52	1.75	4.27	5.73
RB.LangRhythmId	3.17	0.69	0.35	8.26	0.06	0.21	2.85	0.18
RB.RdbltyDaleChall	3.15	1.27	1.24	4.63	2.86	0.95	4.05	3.27
RB.AvgUnqWdBl	2.75	1.63	1.92	4.02	0.46	1.82	0	6.45
RB.LxcDiv	2.67	1.87	1.89	0	11.82	2.04	3.58	4.27
Sentences	2.62	1.71	2.13	0	5.6	1.52	4.67	5.91
RB.TCorefChainDoc	2.35	1.99	1.91	0	5.33	1.97	5.29	3.09
RB.AvgAOADoc_Cortese	1.8	0.21	0.37	3.53	0.76	0.8	2.06	1.36
RB.CAF	1.78	1.83	1.52	0	1.84	1.87	3.65	3.27
RB.AvgNounNmdEntBl	1.6	0.53	0.45	4.63	0.21	0.19	0	0.36
RB.AvgDepsBl_nsubjpass	1.42	0.81	0.27	3.29	0.2	0.41	1.23	0.18
RB.AvgDepsBl_aux	1.21	1.48	1.75	0	2.12	1.53	1.3	2.36
RB.TActCorefChainWd	1.06	0.33	0.95	0	1.19	1.17	2.65	2
RB.AvgDepsBl_nsubj	1.05	1.67	1.78	0	2.94	1.85	0	2.09
RB.AvgPronounBl	0.99	1.72	1.84	0	3.59	2.06	0	1.18
RB.AvgUnqNoundBl	0.93	0.71	0.59	0	0.37	0.65	2.75	1.55
RB.TCorefChainBigSpan	0.89	1.61	1.19	0	0.16	1.33	2.58	0
RB.AvgAOESen_InflectionPointPolynomial	0.87	0.97	1.18	0	0.57	1.09	2.33	0.73
RB.AvgBlScore	0.83	1.46	1.28	0	0.92	1.62	0	2.18
RB.AvgConnBl_addition	0.81	0.99	0.83	0	0.77	0.66	1.36	1.73
RB.AvgChainSpan	0.81	1.52	1.28	0	2.39	1.7	0	1.27
RB.AvgPrepositionBl	0.79	1.51	1.03	0	0.76	1.6	0.95	1
RB.AvgUnqPrepositionBl	0.76	1.48	1.09	0	0.48	1.55	0.8	1.09
RB.SenStdDevWd	0.74	0.86	1.25	0	1.41	1.28	1.45	0.36
RB.AvgAOADoc_Shock	0.72	0.86	0.94	0	1.36	1.15	0.8	1.27
RB.AvgDepsBl_punct	0.68	1.26	1.34	0	1	1	0.35	1.18
RB.AvgCorefChain	0.68	1.19	1.03	0	0.34	1.28	1.1	0.73
RB.AvgNmdEntSen	0.67	0.27	0.42	0	0.17	0.72	2.16	1
RB.AvgPronBl_indefinite	0.65	1.4	1.66	0	1.98	1.31	0.13	0.27
RB.AvgDepsBl_det	0.65	1.18	0.96	0	0.43	0.97	0.92	0.91
RB.AvgDepsBl_dobj	0.65	1.4	0.87	0	0.37	1.26	0	1.73
RB.SynDiv	0.6	0.64	0.7	0	0.42	0.71	1.55	0.64
RB.AvgAOEBl_InflectionPointPolynomial	0.6	0.88	0.95	0	2.01	1.2	0	1.09
RB.FrqRhythmId	0.59	1.12	1.27	0	0.11	0.77	1.31	0.18
RB.LangRhythmDiameter	0.58	0.18	0.35	0	0.17	0.01	2.28	0.82
RB.AvgDepsBl_expl	0.58	0.64	0.28	0	0.35	0.2	1.79	0.82
RB.CharEnt	0.57	1.37	1.01	0	0.53	1.35	0.86	0
RB.AvgNounSen	0.55	0.74	0.91	0	0.31	0.46	1.46	0.36
RB.AvgDepsBl_amod	0.55	0.92	0.33	0	0.1	0.55	1.08	1.09
RB.AvgUnqVerbBl	0.54	1.56	1.01	0	0.48	1.48	0.16	0.36
RB.AvgUnqPronounBl	0.54	1.6	0.94	0	1.5	1.7	0	0
RB.AvgPronBl_first_person	0.53	1.35	0.7	0	0.34	1.23	0.61	0.36
RB.AvgConnBl_sentence_linking	0.53	1.45	1.07	0	0.41	1.41	0	0.64
RB.LxcSoph	0.52	0.32	0.77	0	0.46	0.6	0.1	2.09
RB.AvgAOEBl_IndexPolynomialFitAboveThreshold.0.3.	0.5	0.94	1.01	0	0.41	1.09	0.73	0.27
RB.AvgRhythmUnitStreesSyll	0.49	0.65	0.67	0	0.45	0.43	0.75	1
RB.AvgDepsBl_mark	0.47	1.36	0.95	0	0.08	1.19	0	0.64
RB.AvgDepsBl_nmod	0.47	1.27	0.79	0	0.43	1.1	0	0.73
RB.WdDiffLemmaStem	0.45	0.64	0.88	0	0.65	1.02	0.74	0.18
RB.AvgDepsBl_conj	0.45	0.95	0.52	0	0.28	0.64	0.41	0.91
RB.AvgAOABl_Bird	0.44	0.56	0.39	0	0.58	0.57	0.96	0.55
RB.AvgPronBl_third_person	0.43	1.36	1.14	0	0.33	1.26	0	0.09
RB.AvgDepsBl_ccomp	0.43	0.93	0.86	0	0.04	0.47	1.06	0
RB.AggPronSen_third_person	0.43	0.52	0.51	0	0.11	1.01	1.15	0.18
RB.AvgDepsSen_punct	0.43	0.38	0.62	0	0.19	0.94	0.81	0.64
RB.AvgConnBl_simple_subordinators	0.42	1.31	1.02	0	0.73	1.08	0	0.09
RB.AvgConnSen_simple_subordinators	0.41	0.15	0.52	0	0.09	0.52	1.54	0.18
RB.AvgSenBlCoh_LDA	0.4	0.74	0.95	0	0.2	1.21	0	0.73
RB.AvgDepsBl_xcomp	0.4	1.15	0.82	0	0.37	0.88	0.45	0
RB.AvgCommaBl	0.4	0.72	0.45	0	0.05	0.39	0.96	0.45
RB.AvgAOASen_Shock	0.39	0.4	0.74	0	0.45	0.9	0.79	0.18
RB.AvgSenBlCoh_word2vec	0.36	1.11	0.78	0	0.18	1.03	0	0.27
RB.WdLettStdDev	0.34	0.72	0.63	0	0.39	0.7	0.46	0.18
RB.AvgConnBl_temporal_connectors	0.34	1.03	0.92	0	0.02	0.77	0.1	0.27
RB.AvgDepsBl_acl	0.34	0.58	0.44	0	0.07	0.2	1.19	0
RB.LangRhythmCoeff	0.33	0.7	0.59	0	0.25	0.66	0.61	0
RB.WdSylCnt	0.33	0.38	0.9	0	0.27	0.79	0.26	0.45
RB.AvgDepsBl_auxpass	0.33	0.86	0.55	0	0.01	0.5	0.7	0
RB.AvgConnBl_oppositions	0.33	0.85	0.49	0	0	0.42	0.63	0.18
RB.AvgAdverbBl	0.32	1.1	0.72	0	0.11	0.83	0	0.18
RB.AvgConnBl_order	0.32	0.73	0.27	0	0.01	0.29	1	0
RB.AvgAOABl_Bristol	0.31	0.75	0.29	0	0.7	0.77	0.39	0
RB.AvgDepsSen_nmod	0.31	0.06	0.43	0	0.11	0.34	0.45	1
RB.AvgPronounSen	0.31	0.36	0.7	0	0.05	0.55	0	1
RB.AvgIntraBlCoh_Path	0.3	1.14	0.3	0	0.16	0.97	0	0.18
RB.AvgAOABl_Kuperman	0.3	0.51	0.51	0	0.71	0.59	0.1	0.45
RB.AvgDepsSen_nsubj	0.3	0.05	0.83	0	0.04	0.49	0	1.18
RB.AvgDepsSen_aux	0.3	0.17	0.62	0	0.21	0.49	0.68	0.36
RB.AvgInferenceDistChain	0.29	0.8	0.71	0	0.19	0.74	0.25	0
RB.AvgConnBl_conditions	0.29	0.9	0.45	0	0.15	0.49	0.44	0
RB.AvgDepsBl_cop	0.28	1.07	0.53	0	0.08	0.7	0	0.18
RB.RdbltyFlesch	0.28	0.49	0.88	0	0.38	0.54	0	0.45
RB.AvgConnSen_temporal_connectors	0.28	0.28	0.82	0	0.17	0.05	0.75	0.18
RB.AvgUnqAdjectiveBl	0.27	1.2	0.24	0	0.01	0.97	0.03	0
RB.AvgDepsBl_advcl	0.27	1.18	0.37	0	0.08	0.9	0.01	0
RB.AvgDepsSen_advcl	0.27	0.16	0.53	0	0.12	0.69	0.66	0.18
RB.AggPronSen_indefinite	0.25	0.42	0.84	0	0.26	1.17	0.03	0
RB.WdDiffWdStem	0.25	0.65	0.56	0	0.42	0.81	0.13	0
RB.AvgDepsBl_neg	0.24	0.45	0.08	0	0.02	0.12	0.9	0
RB.AvgDepsBl_nummod	0.23	0.45	0.11	0	0	0.12	0.89	0
RB.AvgDepsBl_mwe	0.22	0.29	0.46	0	0	0.06	0.76	0
RB.AvgDepsSen_amod	0.22	0.3	0.64	0	0.32	0.72	0.22	0
RB.AvgAOASen_Bird	0.21	0.32	0.74	0	0.27	0.42	0.25	0
RB.AvgPrepositionSen	0.21	0.07	0.66	0	0.05	0.35	0	0.73
RB.AvgConnBl_contrasts	0.21	1.04	0.19	0	0.01	0.64	0	0
RB.AvgAOASen_Kuperman	0.21	0.5	0.2	0	0.88	0.48	0	0.18
RB.AvgDepsSen_xcomp	0.21	0.19	0.71	0	0.05	1.01	0.23	0
RB.AvgDepsBl_root	0.2	0.04	0.29	0	0.02	0	0.99	0
RB.AvgDepsSen_cop	0.2	0.06	0.49	0	0.28	0.36	0.6	0
RB.AvgConnSen_reason_and_purpose	0.19	0.14	0.21	0	0.11	0.61	0.53	0
RB.AvgDepsSen_conj	0.19	0.14	0.56	0	0.02	0.42	0	0.55
RB.AvgDepsSen_dobj	0.19	0.06	0.6	0	0.23	0.38	0	0.55
RB.AvgDepsSen_dep	0.19	0.49	0.57	0	0.29	0.65	0	0
RB.AvgAdverbSen	0.19	0	0.72	0	0.05	0.87	0	0.36
RB.AvgSenLen	0.18	0.06	0.76	0	0.12	0.29	0	0.45
RB.AvgPronBl_second_person	0.18	0.7	0.62	0	0.01	0.3	0	0
RB.AvgConnBl_disjunctions	0.18	0.73	0.25	0	0.01	0.35	0.16	0
RB.AvgConnBl_reason_and_purpose	0.18	0.64	0.26	0	0.03	0.26	0.28	0
RB.AggPronSen_second_person	0.18	0.32	0.47	0	0.01	0.08	0.53	0
RB.AvgConnBl_semi_coordinators	0.16	0.35	0.31	0	0.01	0.09	0.43	0
RB.AvgPronBl_interrogative	0.16	0.7	0.35	0	0.01	0.28	0.07	0
RB.AvgAOASen_Bristol	0.15	0.41	0.39	0	0.14	0.58	0	0
RB.AvgDepsSen_ccomp	0.15	0.18	0.59	0	0.11	0.47	0	0.18
RB.AvgDepsBl_iobj	0.15	0.62	0.33	0	0.01	0.29	0	0.09
RB.AvgDepsSen_det	0.15	0.21	0.32	0	0.24	0.07	0.12	0.36
RB.AvgConnSen_addition	0.13	0.11	0.27	0	0.75	0.46	0	0
RB.AvgDepsSen_acl	0.13	0.38	0.58	0	0.11	0.08	0	0.09
RB.AvgDepsSen_mark	0.12	0.09	0.35	0	0.04	0.36	0	0.27
RB.AvgConnSen_oppositions	0.12	0.25	0.65	0	0.08	0.05	0.13	0
RB.AvgDepsBl_dep	0.11	0.58	0.04	0	0.06	0.19	0.04	0
RB.AvgConnSen_semi_coordinators	0.11	0.22	0.65	0	0.23	0.04	0	0
RB.AvgConnBl_complex_subordinators	0.11	0.39	0.19	0	0	0.12	0.19	0
RB.AvgAOASen_Cortese	0.11	0.12	0.25	0	0.35	0.64	0	0
RB.AvgAdjectiveSen	0.1	0.09	0.4	0	0.05	0.56	0	0
RB.AvgDepsSen_iobj	0.07	0.16	0.45	0	0.02	0.02	0	0
RB.AggPronSen_interrogative	0.07	0.11	0.4	0	0.21	0.01	0	0
RB.AvgConnSen_order	0.07	0.02	0.48	0	0.13	0	0	0.09
RB.SenAsson	0.07	0.24	0.41	0	0	0.02	0	0
RB.AvgConnSen_conditions	0.06	0.07	0.49	0	0.08	0	0	0
RB.AvgDepsBl_csubj	0.06	0.02	0.31	0	0.06	0	0.17	0
RB.AvgDepsSen_neg	0.06	0.17	0.4	0	0.03	0.03	0	0
RB.AvgDepsBl_parataxis	0.05	0.24	0.12	0	0	0.04	0	0
RB.AvgDepsBl_appos	0.04	0.2	0.08	0	0	0.04	0	0
RB.AvgDepsSen_nummod	0.04	0.04	0.35	0	0.02	0	0	0
RB.AggPronSen_first_person	0.04	0.02	0.2	0	0.14	0.1	0	0
RB.AvgConnSen_disjunctions	0.03	0.11	0.13	0	0.04	0.01	0	0
RB.SenAllit	0.03	0.03	0.3	0	0	0	0	0
RB.AvgDepsSen_mwe	0.01	0.07	0	0	0	0.01	0	0