Migrating the Community Project to a Monorepo - cortex-command-community/Cortex-Command-Community-Project GitHub Wiki

[ Note: I use the terms "Source" and "Data" here to explicitly refer to the Cortex-Command-Community-Project-Source and Cortex-Command-Community-Project-Datarepositories. Note the capitalisation of each term, which means the repos themselves. Any other mention of "source" refers to "the source of". ]

Setup

Requirements

  • Both repositories should be prepared for the migration by bringing their master branches up to date.

    I haven't tested bringing over multiple, non-master branches, so I strongly recommend that any code to be migrated is merged before starting and that all other branches are abandoned/recreated.

    This constitutes a large amount of work on its own and will require the completion and closing of all existing PRs and a review of in-progress/draft branches.

  • The creation of a new Repository in the Cortex-Command-Community organisation, to act as the new monorepo hosting the game's source code and data. While we could technically merge the Data repository into the existing Source, it would be significantly cleaner to start fresh, rather than run into issues renaming and retargeting remotes.

    This article assumes this is the case, and provides information to remedy any common issues with this clean-slate approach.

    (* Please make it something simple like Cortex-Command-Community-Project ty)

New Repository Settings

The following settings will need to be updated/reenabled on the new repository:

  • Branch Protection
  • Issues
  • Actions (General) -> Workflow permissions -> Read & Write
  • Secrets
  • Webhooks (Discord Integration)

All other settings are mostly out of my purview as I'm not responsible for managing them. Only those required for POC were tested.

The majority of the user permissions should be handled by the organisation-level permission groups already implemented.

Migrating Code and Commit History

Having created the new repository to act as the monorepo (as described above) we can follow this post as a reference to merge both repositories' code and retain their commit history.

I was more comfortable with Fork for portions of this, so I used it to add the two remotes as shown:

  • Add a new remote to the new repository

Merging the remotes

As noted in the StackOverflow post, we need to use the --allow-unrelated-histories flag when merging, and the Fork UI doesn't support this (as far as I'm aware). Opening the monorepo in the terminal we can start the Source merge using:

git merge --allow-unrelated-histories source/master

This is expected to conflict, but that can be resolved as below. This will drop all the files from the Cortex-Command-Community-Project-Source repo into the top level of the new monorepo.

We can repeat this process with the Data repo (again resolving merge conflicts) to unify the two, preserving their git history.

git merge --allow-unrelated-histories data/master

Merge Conflicts

The merge can be resolved and committed either through the command line or via Fork once it's initiated. After triggering the merge as above, I switched to Fork and resolved any merge conflicts there.

Merge conflicts occurred while merging the Data branch into the monorepo containing the Source files. They included:

  • The README.md, which for simplicity's sake I concatenated.
  • The .gitignore, which I also concatenated but could have largely been omitted given most of it was out of date.
  • The CODEOWNERS and ISSUE_TEMPLATE files in the .github/ directory that are duplicated across both repositories

Merge conflicts while merging the Source repository files only occurred if there was an existing .gitignore/README.md generated by GitHub.

GitHub Issues

Below is a template script for migrating all issues from one repo to another. This version expects it to be copied into a file (e.g. transferIssues) and invoked with two arguments, the two repositories to migrate between.

See this for a simple reference as to how to format those arguments. It's just [HOST/]OWNER/REPO but can take the full URL I believe.
e.g: cortex-command-community/Cortex-Command-Community-Project-Data

PLEASE NOTE: This script will move the issues, not duplicate them. Any issues this script migrates to a target repository will be removed from the source repository!

USAGE="Usage: $0 <source-repo> <destination-repo>"
# --- Option processing --------------------------------------------
if [ $# != 2 ] ; then
    echo $USAGE
    exit 1;
fi

gh issue list -s all -L 500 -R $1 --json number | \
    jq -r '.[] | .number' | \
    xargs -I% gh issue transfer % $2

[Must be run in the source repo, I think]

Hopefully available for download here: transferIssues.sh

Example:

./transferIssues cortex-command-community/Cortex-Command-Community-Project-Data cortex-command-community/Cortex-Command-Community-Project

To the best of my understanding, the Projects and Milestones that we have in the Source repo are stored as part of the Cortex-Command-Community org and should be fine.

Proof of Concept

See the proof of concept repo here.

Note: This incorporates both the changes described here and some features related to automated release/versioning, which are covered in a separate topic.

While packaging the two repos contents in dedicated directories was discussed, for this demo I elected to leave them as top-level files, given it would require changes to the build scripts.

Example:

.
├── Data/
│   ├── Base.rte
│   ├── Coalition.rte
│   ├── Dummy.rte
│   └── ...
└── Source/
    ├── Activities
    ├── Entities
    ├── Lua
    ├── Managers
    ├── Resources
    ├── System
    ├── external
    └── ...

It's certainly possible just not implemented in this test.

If you're interested in adding the merged files to their own directory, it's as simple as creating a new directory and switching to it before merging the Source files in the Merging the Remotes step.

The StackOverflow post linked above follows a method where each repo's contents are placed into a dedicated directory.

Sources

POC:
https://github.com/traunts/Cortex-Command-Monorepo-POC

Merging two Git Repositories:
https://stackoverflow.com/questions/1425892/how-do-you-merge-two-git-repositories

Bulk Issue Migration:
https://jloh.co/posts/bulk-migrate-issues-github-cli/
(How the gh issue CLI works)
https://cli.github.com/manual/gh_issue