FYPO local release with docker (obsolete) - pombase/fypo GitHub Wiki
FYPO release,step-by-step
NOTE: Before you continue see files too big
Go to your/path/to/pombase-git/fypo/ (wherever you have run protege, issue commands:)
| Command | Notes |
|---|---|
| Wherever you run protege | - |
| cd pombase-fypo/ | |
| % git pull | - |
| % git checkout -b [fypo-release-yyyy-mm-1dd] | creates a branch (creates and switches to temporary scratch space for editing and releasing) |
| % git branch | Reports current branch |
| Make edits in protege | |
| % cd src/ontology | - |
robot --catalog catalog-v001.xml report -i fypo-edit.owl -l true --fail-on None --base-iri http://purl.obolibrary.org/obo/FYPO_ --base-iri http://purl.obolibrary.org/obo/fypo --print 5 -o tmp/tmp-report.tsv |
(Optional, needs robot installed and accessible by your terminal). This is run by the CI pipeline, but you can run locally as well before committing, will generate a file tmp/tmp-report.tsv with errors in the ontology |
| % git add fypo-edit.owl | - |
| % git commit -m 'commit message' fypo-edit.owl | "closes #123; closes #124;" will link the commit to, and automagically close, the ticket(s) |
| - | - |
| % sh run.sh make synonyms -B | adds synonyms from .tsv file |
| % sh run.sh make fix_quality_issue -B | don't have to do this every time, but it fixes BFO:quality to PATO:quality, so do it every so often (especially if you've added "normal" or "abnormal" terms) |
| % sh run.sh make clear_obsoletes -B | only if anything has just gone obsolete |
| make release | - |
| % sh run.sh make prepare_release -B | runs release (takes a while) (minus B =report all problems)Docker might run out of memory. The following settings work in Val's computer in reasonable time: |
| If ODK fails | - |
| Fix problems in protege and re ru | |
| OR | - |
| % cd ../.. | - |
| % git status | Shows changed files |
| % git add . | - |
| % git status | - |
| % git commit -m 'yada release' | e.g. -m '2021-09-16 release' |
| % git push origin $branch | e.g. git push origin fypo-release-2021-09-16 (Until this step everything has been local directory ) wait for CI checks (can see in GitHub under "Actions" All the above edits were done in the 'edit file' terms are visible but not useable. This is a 'branch' (When CI checks are running a small brown dot is visible) origin=GitHub |
| Go to https://github.com/pombase/fypo/pulls | there should be a message "$branch had recent pushes" |
| Click "Compare & pull request" | Comments optional |
| Click "Create pull request" | (button under text box) Wait for CI checks to pass |
| Click "Squash & merge", then "Confirm squash & merge" | Wait for CI checks to pass |
| Click "Delete branch" | (deleting Git Hubs copy of the branch) |
| Return to your command line | - |
| % git checkout master | switch to master branch (local copy) master=default main branch |
| % git pull | (local copy now same as GitHub) |
| % git branch -D $branch | e.g. git branch -D fypo-release-2021-09-16 (deletes local copy of branch) |
| Go to https://github.com/pombase/fypo/releases/new | (GitHub to make public release) |
| Code -> Releases (right-hand panel) -> "Draft a new release" (button on right, near top) | - |
| Click "Choose a tag", type vyyyy-mm-dd click "Create new tag" (appears below the text box) | - |
| Add: Release title | (comments optional) |
| Click "Publish release" | (button near the bottom) |
| Return to your command line (local) | - |
| % git pull |
| % git push origin master |
| LOG ONTO OLIVER1 | - |
| % cd your/path/to/pombase-git/fypo/ | - |
| % git pull| - |
|% cd ../.. | Back to root (not necessary, IIRC this is only required if your entire release pipeline is on the same machine?) |
Troubleshooting
Viewing CI checks (if a push request fails)
- GitHub actions - go to https://github.com/pombase/fypo/actions,
- pick a workflow run, click on the TEXT NEXT TO THE little box with the x or tick (usually says "ontology_qc") (there will pe 3 links per run but all should be the same)
- expand "Run ontology QC checks". NOTE THAT THIS ONLY SHOWS 5 WRNINGS> You can see them all in src/reports
CI check failure
- If the CI checks fail, look for "ERROR" on the output page. The most common problem (so far) has been that a term has two logical definitions. This has happened when I haven't noticed that I'm "on" the wrong term, which the sodding Protege interface makes entirely too easy to do. To fix, open in Protege and remove the wrong one (and usually paste it in for the term that was supposed to get it all along). If it passes CI checks on the first push, it'll pass subsequent steps too.
Release failure
-
If the release fails, check the output (on screen as stdout and in a log). Often it's nothing to do with FYPO, but instead due to connecting to reach an external ontology -- e.g. this message:
-
File size warning (fypo #2718)
-
To undo a local change (displayed when you run "git status"): git restore some_file_that_has_been_changed
-
If you have no local changes then you can switch to the master branch with "git checkout master" (context, this was from Kim )
-
merge conflict on Oliver zero
- Look for conflict markers <<<<<< and correct the local file Select the correct part and delete everything else including the markers % git add $filename % git commit
- Conflicts that can't be fixed: git reset --hard origin/master Restart release with a new branch name
- This error was solved by updating docker
Makefile:171: warning: ignoring old recipe for target 'prepare_release'
robot --catalog catalog-v001.xml merge -i fypo-edit.owl template --template ../templates/bulk_synonyms.tsv --output ../templates/bulk_synonyms.owl
make: robot: Operation not permitted
make: *** [fypo.Makefile:65: ../templates/bulk_synonyms.owl] Error 127
- Error 18 with
curl. This is because does not manage to download one of the files. This happenned with chebi file before when downloading from EBI. You can wait and try another day after few unsuccessful attempts.
GO Term obsoletions
To check external ontology term usage (useful when GO makes terms obsolete)
- open fypo-edit.owl in Protege
- search for term of interest
- check "usage" tab for the searched-for term itself to see any terms that use it in logical defs
- can also inspect any FYPO terms that happen to match the term search
Creating logical definitions
ODK imports the "release" versions of ontologies, i.e. for GO it doesn't use the "snapshot" version. That means sometimes you have to wait to be able to add a logical definition if it uses a new GO term.
If an external ontology is out of date
sh run.sh make refresh-pato git commit (will commit all files git knows about) git push
hard reset
If you have no local changes then you can switch to the master branch with "git checkout master" (switch to master branch) git branch git checkout master git reset --hard master git status
git restore (???)
git diff git status git restore src/ontology/fypo-edit.owl git status grep FYPO:0007009 src/ontology/fypo-edit.owl (check) git checkout master git pull tail fyeco.obo
random git commands
git log (to access commit statements)
Files too big
See https://github.com/pombase/fypo/issues/4201
Github has a size limit of 100MB, some of our release artifacts are above that and can no longer be committed to GitHub. This only affects fypo.owl, fypo-full.owl and fypo-non-classified.owl so far, but we may move all release artifacts out of the repository eventually. For now, when you make a release, attach all artifacts (all the owl, json, obo, etc. files generated by the release pipeline) to the release. For this, drag them in this box during the release procedure:
If another file becomes too big in the future (you can't push that file to github anymore), you have to:
- "uncommit" the file that becomes too big (google git reset
git reset) - then remove it from the git repository (
git rm the-file). This will delete the file, so keep a copy of it elsewhere to paste it again in the repo. - then add the file to .gitignore so it's no longer tracked by git.