Scoreboard - davidlabee/Graph4Air GitHub Wiki
GNN Experiment Scoreboard
This table helps track and compare results between team members for each graph construction and model configuration. Please fill in the results as you test new combinations. Metrics are displayed for internal and external validation.
Comparison to Data-Only (internal validation)
| Contributor | Graph | Model | Layers | Epochs | MAE | RMSE | Pearson R² | Hyperparameters (hu's, lr, dropout, activ.(, heads)) | #runs |
|---|---|---|---|---|---|---|---|---|---|
| Baseline | original 50m | GCN | 4 | early stopping | 4.44 | 6.54 | 0.710 | 128, 0.020, 0.4, elu | 10 |
| Baseline | original 50m | GAT | 5 | early stopping | 4.09 | 6.24 | 0.739 | 128, 0.012, 0.0, elu, 1 | 10 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| Pieter | hierarchical-4 | GCN | 3 | 200 | 5.75 | 8.10 | 0.55 | 32, 0.010, 0.0, relu | 5 |
| Pieter | hierarchical-4 | GAT | 3 | 200 | 5.54 | 7.81 | 0.58 | 32, 0.010, 0.0, relu, 1 | 5 |
| Pieter | hierarchical-8 | GCN | 3 | 200 | 5.72 | 8.07 | 0.56 | 32, 0.010, 0.0, relu | 5 |
| Pieter | hierarchical-8 | GAT | 3 | 200 | 5.52 | 7.82 | 0.58 | 32, 0.010, 0.0, relu, 1 | 5 |
| Pieter | hierarchical-16 | GCN | 3 | 200 | 5.75 | 8.12 | 0.55 | 32, 0.010, 0.0, relu | 5 |
| Pieter | hierarchical-16 | GAT | 3 | 200 | 5.63 | 7.96 | 0.57 | 32, 0.010, 0.0, relu, 1 | 5 |
| Pieter | hierarchical-32 | GCN | 3 | 200 | 5.86 | 8.25 | 0.54 | 32, 0.010, 0.0, relu | 5 |
| Pieter | hierarchical-32 | GAT | 3 | 200 | 6.08 | 8.55 | 0.50 | 32, 0.010, 0.0, relu, 1 | 5 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| David | graph augmentation optimised hyperparameters | GCN | 4 | 200 | 5.59 | 8.04 | 0.58 | 64, | 1 |
| David | graph augmentation optimised hyperparameters | GAT | 4 | 200 | 4.99 | 7.38 | 0.63 | 64 | 1 |
| David | graph augmentation optimised hyperparameters | GCN | 4 | 1000 (500) | 5.248 | 7.632 | 0.619 | 5-fold | 5 |
| David | graph augmentation optimised hyperparameters | GAT | 4 | 1000 (350) | 4.908 | 7.2 | 0.661 | 5-fold | 5 |
Comparison to Palmes Tubes (external validation)
| Contributor | Graph | Model | Layers | Epochs | MAE | RMSE | R² | Hyperparameters | #runs |
|---|---|---|---|---|---|---|---|---|---|
| Baseline | original 50m | GCN | 3 | 200 | 4.21 | 5.30 | ... | 64, 0.010, 0.5 | #runs |
| Baseline | original 50m | GAT | 3 | 200 | 3.71 | 4.81 | ... | 64, 0.010, 0.2 | #runs |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| Pieter | hierarchical-4 | GCN | 3 | 200 | 4.63 | 5.94 | 0.53 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-4 | GAT | 3 | 200 | 4.91 | 6.37 | 0.51 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-8 | GCN | 3 | 200 | 4.45 | 5.63 | 0.55 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-8 | GAT | 3 | 200 | 4.78 | 6.09 | 0.53 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-16 | GCN | 3 | 200 | 4.53 | 5.69 | 0.53 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-16 | GAT | 3 | 200 | 4.79 | 6.19 | 0.47 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-32 | GCN | 3 | 200 | 4.50 | 5.62 | 0.51 | 32, 0.010, 0.0 | 5 |
| Pieter | hierarchical-32 | GAT | 3 | 200 | 5.02 | 6.35 | 0.48 | 32, 0.010, 0.0 | 5 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| Pieter | hierarchical-4 + tuning | GCN | 4 | 200 | 4.91 | 6.55 | 0.51 | 128, 0.010, 0.2, 4 | 5 |
| Pieter | hierarchical-4 + tuning | GAT | 4 | 200 | 5.02 | 6.72 | 0.48 | 128, 0.010, 0.0, 4 | 5 |
| Pieter | hierarchical-8 + tuning | GCN | 4 | 200 | 4.31 | 5.58 | 0.54 | 128, 0.007, 0.35, 4 | 5 |
| Pieter | hierarchical-8 + tuning | GAT | 5 | 200 | 4.27 | 5.55 | 0.49 | 64, 0.016, 0.1, 5 | 5 |
| -- | -- | -- | -- | -- | -- | -- | -- | -- | -- |
| David | graph augmentation optimised hyperparameters | GCN | 4 | 200 | 4.06 | 5.25 | 0.60 | 64, | 1 |
| David | graph augmentation optimised hyperparameters | GAT | 4 | 200 | 4.24 | 5.61 | 0.55 | 64 | 1 |
| David | graph augmentation optimised hyperparameters | GCN | 4 | 1000(500) | 3.95 | 5.28 | 0.56 | 64, | 5 |
| David | graph augmentation optimised hyperparameters | GAT | 4 | 1000(350) | 4.41 | 5.98 | 0.52 | 64 | 5 |