Scores and Analysis of WMT16 systems - rbawden/discomt17-pronouns GitHub Wiki
Summary of scores and difficult classes
de-en
Overall best score (Macro-R) = Turku (73.01), UKyoto (73.13), Uhelsinki (69.75)
Individual labels:
- OTHER: high P for Turku and UKyoto, high R for CUNI and UU-Stymne
- he: high R for Turku and UKyoto
- it: high P for Turku, UKyoto, Cuni, high R for Ukyoto and UHelsinki
- she: high P for Turku, high R for Ukyoto and Turku
- there: high R for CUNI
- these: N/A
- they: highest R for Ukyoto and Turku
- this: highest R for Turku
- you: highest R for Ukyoto and Uuppsala
Rarest and most difficult classes (larger weight in final score): this, there, she and he
en-de
Overall best score (Macro-R) = Turku (73.01), UKyoto (52.50)
Individual labels:
- OTHER: high P for Turku, UU-Hardmeier, high R for Uedin, UU-Hardmeier
- er: high P for UU-Stymne, and Uedin
- es: high P for Uedin and UU-Stymne, high R for UU-Cap and UU-Hardmeier
- man: high P for UU-Hardmeier and UHelsinki, high R for Turku
- sie: high P for UU-Hardmeier, Ukyoto, high R for TurkuNLP, UU-Stymne
Rarest and most difficult classes (larger weight in final score): man, er
en-fr
Overall best score (Macro-R) = Turku (65.70), UU-Stymne (63.35)
Individual labels:
- on: difficult for all systems (R)
- elle: high R for LIMSI, UU-Stymne, Ukyoto
- elles: high R for Turku
Rarest and most difficult classes (larger weight in final score): on, elle, elles
Remarks:
- Coreference chains could help the prediction of elle
fr-en
Overall best score (Macro-R) = Turku (72.03), UKyoto (65.63)
Individual labels:
- this: only UUppsala managed to predict them (and not many)
- there: high P and R for all but UU-Stymne
- she: high R for Turku and Ukyoto
Rarest and most difficult classes (larger weight in final score): this, there, she
de-en
| Precision | OTHER | he | it | she | there | these | they | this | you |
|---|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 89.41% | 78.95% | 75.40% | 64.71% | 75.00% | 0.00% | 66.30% | 66.67% | 83.96% |
| UUPPSALA | 79.81% | 78.57% | 71.54% | 42.86% | 66.67% | 0.00% | 61.70% | 0.00% | 84.11% |
| UU-Stymne | 62.86% | 70.59% | 75.25% | 70.59% | 66.67% | 0.00% | 67.44% | 25.00% | 79.35% |
| TurkuNLP | 91.03% | 78.95% | 79.80% | 84.21% | 73.68% | 0.00% | 70.45% | 17.39% | 77.19% |
| UKYOTO | 90.79% | 82.35% | 78.57% | 66.67% | 92.86% | 0.00% | 73.12% | 0.00% | 82.30% |
| CUNI | 76.52% | 79.41% | 79.63% | 22.73% | 83.33% | 0.00% | 40.35% | 0.00% | 80.85% |
| Recall | OTHER | he | it | she | there | these | they | this | you |
|---|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 73.08% | 96.77% | 83.33% | 52.38% | 75.00% | 0.00% | 78.21% | 15.38% | 83.96% |
| UUPPSALA | 79.81% | 70.97% | 77.19% | 14.29% | 75.00% | 0.00% | 74.36% | 0.00% | 84.91% |
| UU-Stymne | 84.62% | 77.42% | 66.67% | 57.14% | 37.50% | 0.00% | 74.36% | 7.69% | 68.87% |
| TurkuNLP | 68.27% | 96.77% | 69.30% | 76.19% | 87.50% | 0.00% | 79.49% | 30.77% | 83.02% |
| UKYOTO | 66.35% | 90.32% | 86.84% | 85.71% | 81.25% | 0.00% | 87.18% | 0.00% | 87.74% |
| CUNI | 84.62% | 87.10% | 75.44% | 47.62% | 93.75% | 0.00% | 58.97% | 0.00% | 35.85% |
| F1 score | OTHER | he | it | she | there | these | they | this | you |
|---|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 80.42% | 86.96% | 79.17% | 57.89% | 75.00% | 0.00% | 71.76% | 25.00% | 83.96% |
| UUPPSALA | 79.81% | 74.58% | 74.26% | 21.43% | 70.59% | 0.00% | 67.44% | 0.00% | 84.51% |
| UU-Stymne | 72.13% | 73.85% | 70.70% | 63.16% | 48.00% | 0.00% | 70.73% | 11.76% | 73.74% |
| TurkuNLP | 78.02% | 86.96% | 74.18% | 80.00% | 80.00% | 0.00% | 74.70% | 22.22% | 80.00% |
| UKYOTO | 76.67% | 86.15% | 82.50% | 75.00% | 86.67% | 0.00% | 79.53% | 0.00% | 84.93% |
| CUNI | 80.37% | 83.08% | 77.48% | 30.77% | 88.24% | 0.00% | 47.92% | 0.00% | 49.67% |
en-de
| Precision | OTHER | er | es | man | sie |
|---|---|---|---|---|---|
| UEDIN | 61.50% | 36.36% | 72.62% | 16.67% | 76.84% |
| UHELSINKI | 77.27% | 0.00% | 54.23% | 100.00% | 70.63% |
| UU-Cap | 69.17% | 0.00% | 53.53% | 0.00% | 75.27% |
| UU-Stymne | 82.30% | 57.14% | 69.61% | 33.33% | 64.56% |
| UUPPSALA | 73.13% | 14.29% | 63.33% | 33.33% | 73.11% |
| TurkuNLP | 89.69% | 10.53% | 66.67% | 63.64% | 71.62% |
| UU-Hardmeier | 85.00% | 0.00% | 61.11% | 100.00% | 81.20% |
| UKYOTO | 77.60% | 25.00% | 62.71% | 33.33% | 77.78% |
| CUNI | 54.29% | 4.00% | 45.95% | 0.00% | 44.07% |
| Recall | OTHER | er | es | man | sie |
|---|---|---|---|---|---|
| UEDIN | 85.19% | 26.67% | 60.40% | 12.50% | 58.87% |
| UHELSINKI | 62.96% | 0.00% | 76.24% | 12.50% | 71.77% |
| UU-Cap | 61.48% | 0.00% | 90.10% | 0.00% | 56.45% |
| UU-Stymne | 68.89% | 26.67% | 70.30% | 12.50% | 82.26% |
| UUPPSALA | 72.59% | 6.67% | 75.25% | 12.50% | 70.16% |
| TurkuNLP | 64.44% | 13.33% | 71.29% | 87.50% | 85.48% |
| UU-Hardmeier | 75.56% | 0.00% | 87.13% | 12.50% | 76.61% |
| UKYOTO | 71.85% | 13.33% | 73.27% | 25.00% | 79.03% |
| CUNI | 42.22% | 6.67% | 50.50% | 0.00% | 41.94% |
| F1 score | OTHER | er | es | man | sie |
|---|---|---|---|---|---|
| UEDIN | 71.43% | 30.77% | 65.95% | 14.29% | 66.67% |
| UHELSINKI | 69.39% | 0.00% | 63.37% | 22.22% | 71.20% |
| UU-Cap | 65.10% | 0.00% | 67.16% | 0.00% | 64.52% |
| UU-Stymne | 75.00% | 36.36% | 69.95% | 18.18% | 72.34% |
| UUPPSALA | 72.86% | 9.09% | 68.78% | 18.18% | 71.60% |
| TurkuNLP | 75.00% | 11.76% | 68.90% | 73.68% | 77.94% |
| UU-Hardmeier | 80.00% | 0.00% | 71.84% | 22.22% | 78.84% |
| UKYOTO | 74.62% | 17.39% | 67.58% | 28.57% | 78.40% |
| CUNI | 47.50% | 5.00% | 48.11% | 0.00% | 42.98% |
en-fr
| Precision | OTHER | ce | cela | elle | elles | il | ils | on |
|---|---|---|---|---|---|---|---|---|
| UEDIN | 76.24% | 85.29% | 62.50% | 58.82% | 21.43% | 65.15% | 72.97% | 66.67% |
| UHELSINKI | 89.29% | 90.48% | 56.00% | 29.63% | 14.29% | 59.70% | 67.47% | 50.00% |
| UUPPSALA | 86.59% | 70.11% | 48.65% | 0.00% | 20.00% | 70.97% | 50.96% | 42.86% |
| UU-Stymne | 92.11% | 95.08% | 61.54% | 44.83% | 44.44% | 66.20% | 68.89% | 54.55% |
| IDIAP | 90.00% | 51.75% | 26.32% | 25.00% | 0.00% | 30.88% | 55.74% | 100.00% |
| UU-Hardmeier | 91.57% | 84.29% | 81.82% | 35.00% | 66.67% | 63.77% | 70.10% | 44.44% |
| LIMSI | 73.08% | 89.29% | 73.33% | 45.16% | 28.57% | 58.11% | 75.36% | 50.00% |
| TurkuNLP | 88.16% | 86.76% | 48.00% | 48.00% | 40.00% | 77.08% | 73.44% | 41.67% |
| UKYOTO | 87.50% | 88.24% | 52.94% | 38.24% | 39.13% | 67.27% | 72.22% | 57.14% |
| Recall | OTHER | ce | cela | elle | elles | il | ils | on |
|---|---|---|---|---|---|---|---|---|
| UEDIN | 90.59% | 85.29% | 48.39% | 43.48% | 12.00% | 70.49% | 76.06% | 66.67% |
| UHELSINKI | 88.24% | 83.82% | 45.16% | 34.78% | 8.00% | 65.57% | 78.87% | 55.56% |
| UUPPSALA | 83.53% | 89.71% | 58.06% | 0.00% | 16.00% | 36.07% | 74.65% | 33.33% |
| UU-Stymne | 82.35% | 85.29% | 51.61% | 56.52% | 16.00% | 77.05% | 87.32% | 66.67% |
| IDIAP | 42.35% | 86.76% | 16.13% | 4.35% | 0.00% | 34.43% | 95.77% | 11.11% |
| UU-Hardmeier | 89.41% | 86.76% | 58.06% | 30.43% | 8.00% | 72.13% | 95.77% | 44.44% |
| LIMSI | 89.41% | 73.53% | 35.48% | 60.87% | 16.00% | 70.49% | 73.24% | 55.56% |
| TurkuNLP | 78.82% | 86.76% | 77.42% | 52.17% | 48.00% | 60.66% | 66.20% | 55.56% |
| UKYOTO | 82.35% | 88.24% | 58.06% | 56.52% | 36.00% | 60.66% | 73.24% | 44.44% |
| F1 score | OTHER | ce | cela | elle | elles | il | ils | on |
|---|---|---|---|---|---|---|---|---|
| UEDIN | 82.80% | 85.29% | 54.55% | 50.00% | 15.38% | 67.72% | 74.48% | 66.67% |
| UHELSINKI | 88.76% | 87.02% | 50.00% | 32.00% | 10.26% | 62.50% | 72.73% | 52.63% |
| UUPPSALA | 85.03% | 78.71% | 52.94% | 0.00% | 17.78% | 47.83% | 60.57% | 37.50% |
| UU-Stymne | 86.96% | 89.92% | 56.14% | 50.00% | 23.53% | 71.21% | 77.02% | 60.00% |
| IDIAP | 57.60% | 64.84% | 20.00% | 7.41% | 0.00% | 32.56% | 70.47% | 20.00% |
| UU-Hardmeier | 90.48% | 85.51% | 67.92% | 32.56% | 14.29% | 67.69% | 80.95% | 44.44% |
| LIMSI | 80.42% | 80.65% | 47.83% | 51.85% | 20.51% | 63.70% | 74.29% | 52.63% |
| TurkuNLP | 83.23% | 86.76% | 59.26% | 50.00% | 43.64% | 67.89% | 69.63% | 47.62% |
| UKYOTO | 84.85% | 88.24% | 55.38% | 45.61% | 37.50% | 63.79% | 72.73% | 50.00% |
fr-en
| Precision | OTHER | he | it | she | there | these | they | this |
|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 85.14% | 68.75% | 68.63% | 71.43% | 88.46% | 100.00% | 81.91% | 0.00% |
| UUPPSALA | 81.08% | 60.71% | 56.52% | 88.89% | 87.76% | 100.00% | 81.32% | 16.67% |
| TurkuNLP | 88.71% | 78.95% | 76.92% | 71.43% | 97.83% | 42.86% | 81.91% | 0.00% |
| UU-Stymne | 43.98% | 92.86% | 42.86% | 63.64% | 100.00% | 0.00% | 68.06% | 0.00% |
| UKYOTO | 87.01% | 85.71% | 71.43% | 82.35% | 90.38% | 50.00% | 86.05% | 0.00% |
| Recall | OTHER | he | it | she | there | these | they | this |
|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 73.26% | 68.75% | 61.40% | 83.33% | 95.83% | 25.00% | 96.25% | 0.00% |
| UUPPSALA | 69.77% | 53.12% | 68.42% | 44.44% | 89.58% | 50.00% | 92.50% | 33.33% |
| TurkuNLP | 63.95% | 93.75% | 70.18% | 83.33% | 93.75% | 75.00% | 96.25% | 0.00% |
| UU-Stymne | 84.88% | 40.62% | 26.32% | 38.89% | 39.58% | 0.00% | 61.25% | 0.00% |
| UKYOTO | 77.91% | 75.00% | 78.95% | 77.78% | 97.92% | 25.00% | 92.50% | 0.00% |
| F1 score | OTHER | he | it | she | there | these | they | this |
|---|---|---|---|---|---|---|---|---|
| UHELSINKI | 78.75% | 68.75% | 64.81% | 76.92% | 92.00% | 40.00% | 88.51% | 0.00% |
| UUPPSALA | 75.00% | 56.67% | 61.90% | 59.26% | 88.66% | 66.67% | 86.55% | 22.22% |
| TurkuNLP | 74.32% | 85.71% | 73.39% | 76.92% | 95.74% | 54.55% | 88.51% | 0.00% |
| UU-Stymne | 57.94% | 56.52% | 32.61% | 48.28% | 56.72% | 0.00% | 64.47% | 0.00% |
| UKYOTO | 82.21% | 80.00% | 75.00% | 80.00% | 94.00% | 33.33% | 89.16% | 0.00% |