Ensemble Weightings and Metric Importance

Coh-Metrix Model 2a

This model used Coh-Metrix scores from 7 min narrative writing samples ("I once had a magic pencil and ...") from 136 students in the fall of Grades 2-5 (Mercer et al., 2019) to predict holistic writing quality on the samples (elo ratings calculated from paired comparisons).

Highly correlated Coh-Metrix metrics (r > |.90|) were excluded during pre-processing (see section on Scoring Model Development for more details).

Mercer, S. H., Keller-Margulis, M. A., Faith, E. L., Reid, E. K., & Ochs, S. (2019). The potential for automated text evaluation to improve the technical adequacy of written expression curriculum-based measurement. Learning Disability Quarterly, 42, 117-128. https://doi.org/10.1177/0731948718803296

Algorithm Weightings in Ensemble

Abbreviations:

• all = ensemble model
• pls = partial least squares regression
• rf = random forest regression
• mars = bagged multivariate adaptive regression splines
• gbm = stochastic gradient boosted trees
• svm = support vector machines
• cube = cubist regression

The table below presents the linear weightings of each algorithm for the ensemble model.

Intercept	pls	rf	mars	gbm	svm	cube
-11.0081	0.1741	0.0413	0.1875	0.2353	0.206	0.2108

Metric Importance in Each Algorithm and Ensemble

Each column sums to 100 (so values can be interpreted as % contribution to the model).

Detailed information on Coh-Metrix abbreviations and indices is available here.

Metric	overall	pls	rf	mars	gbm	svm	cube
DESWC	29.91	5.86	14.51	55.78	44.87	5.96	36.49
DESWLlt	8.67	2.98	2.69	18.79	3.07	2.39	17.89
LDMTLD	7.35	4.01	5.57	0	9.55	3.9	17.89
WRDHYPn	7.16	2.9	2.68	9.55	3.85	2.29	17.89
LDTTRa	2.84	2.01	0.97	11.14	0.26	1.12	1.05
CNCPos	1.36	0.88	1.11	4.75	0.05	1.28	0.35
DESWLsy	1.3	2.09	1.9	0	1.66	1.54	1.05
CNCTempx	1.11	0.89	1.03	0	1.63	2.48	0.35
CNCLogic	1.09	1.42	1.39	0	1.81	1.7	0.35
PCDCp	1.08	2	2.56	0	1.71	1.36	0
DESPL	1.04	3.22	2.18	0	0.04	2.15	0
DESWLltd	1.02	2.3	1.13	0	1.26	1.6	0
WRDFRQa	0.96	1.86	0.58	0	0.17	1.97	1.05
DESWLsyd	0.92	1.97	1.87	0	1.21	1.29	0
DESSLd	0.9	1.67	2	0	1.27	1.36	0
CNCTemp	0.89	0.93	1.39	0	1.08	1.89	0.35
LSAGN	0.87	2.55	1.18	0	0.22	1.8	0
LSASSpd	0.86	1.95	0.86	0	0.36	1.79	0.35
CNCADC	0.85	1.12	1.41	0	0.67	2.37	0
DRPP	0.84	2.18	2.11	0	0.57	1.4	0
PCCONNz	0.82	1.09	0.4	0	1.61	1.36	0
WRDPRO	0.82	1.77	1.44	0	0.73	1.61	0
SYNSTRUTa	0.8	0.85	3.07	0	0.58	1.4	0.7
CRFCWO1d	0.79	1.67	1.39	0	0.43	1.88	0
LSASS1d	0.78	1.55	0.48	0	0.78	1.72	0
SMCAUSwn	0.76	1.4	1.04	0	0.31	2.16	0
SYNMEDpos	0.75	1.75	0.6	0	0.77	1.36	0
SMINTEp	0.73	0.75	0.85	0	0.79	1.66	0.35
LDTTRc	0.73	1.82	0.7	0	0.72	1.23	0
CRFCWOad	0.73	1.51	1.12	0	0.38	1.81	0
WRDVERB	0.71	1.04	0.93	0	0.58	0.86	1.05
WRDFAMc	0.7	1.09	0.5	0	1.32	1.08	0
WRDHYPnv	0.69	1.9	0.91	0	0.13	1.22	0.35
WRDFRQmc	0.69	1.35	2.5	0	1.3	0.4	0
WRDCNCc	0.69	1.07	0.44	0	0.58	0.8	1.05
PCNARz	0.68	1.29	1.29	0	0.58	1.1	0.35
WRDPOLc	0.67	0.85	0.77	0	0.51	1.97	0
RDFRE	0.66	1.29	0.91	0	0.76	1.23	0
CRFNOa	0.63	0.74	1.34	0	0.56	1.67	0
PCVERBz	0.63	1.55	0.67	0	0.28	1.09	0.35
LSAGNd	0.61	1.5	0.95	0	0.24	1.38	0
SMCAUSvp	0.59	0.93	0.73	0	0.36	1.67	0
WRDADV	0.58	1.3	0.95	0	0.13	1.55	0
WRDAOAc	0.58	1.66	0.46	0	0.27	1.18	0
DRNP	0.57	1.73	1.5	0	0.27	0.82	0
WRDHYPv	0.56	0.28	1.01	0	0.83	1.47	0
DRVP	0.56	0.82	0.62	0	0.82	0.75	0.35
PCNARp	0.54	1.76	1.29	0	0	1	0
CNCCaus	0.51	1.11	0.58	0	0.11	1.46	0
CRFCWOa	0.49	0.46	0.56	0	0.37	1.57	0
SMCAUSr	0.47	1.58	1.07	0	0.47	0.29	0
SMCAUSlsa	0.46	0.24	0.86	0	0.63	1.26	0
LSASSp	0.46	0.79	1.01	0	0.15	1.31	0
SMINTEr	0.45	1.83	0.9	0	0.13	0.45	0
DRAP	0.44	0.46	1.01	0	0.41	1.17	0
DRNEG	0.44	0.76	0.43	0	0.09	1.41	0
WRDNOUN	0.43	0.56	0.96	0	0.75	0.66	0
WRDFRQc	0.43	0.54	0.67	0	0.48	1.04	0
WRDPRP3s	0.43	0.76	0.84	0	0.74	0.56	0
CRFANPa	0.41	0.76	1.36	0	0.18	0.96	0
SYNLE	0.4	1.13	0.95	0	0.1	0.77	0
PCTEMPp	0.4	1.05	0.22	0	0.55	0.47	0
WRDADJ	0.39	0.99	0.94	0	0.2	0.74	0
RDL2	0.39	0.31	1.06	0	0.39	1.09	0
WRDIMGc	0.37	0.14	0.74	0	0.3	0.93	0.35
PCVERBp	0.36	0.99	1.39	0	0	0.75	0
PCCNCz	0.35	1.18	0.1	0	0.32	0.39	0
WRDMEAc	0.29	0.21	0.45	0	0.44	0.71	0
PCREFz	0.29	0.47	1.07	0	0.3	0.53	0
SMCAUSv	0.25	0.39	0.77	0	0.13	0.66	0
SYNNP	0.22	0.03	0.64	0	0.58	0.33	0
PCSYNp	0.21	0.68	1.55	0	0.15	0	0
PCCONNp	0.17	0	0.33	0	0.11	0.66	0
PCCNCp	0.16	0.57	0.78	0	0	0.16	0
PCREFp	0.15	0.06	0.74	0	0	0.58	0
WRDPRP3p	0.14	0.84	0	0	0	0.03	0