Updating Drupal Core - NCIOCPL/cgov-digital-platform GitHub Wiki

There are three packages which are meant to be in sync with each other:

• drupal/core-composer-scaffold
• drupal/core-recommended
• drupal/core-dev (This is development dependency)

1. [Outside Container] Checkout develop
2. [Outside Container] composer cgov-clean && composer install
3. [Inside Container] blt cgov:reinstall

1. [Outside Container] Checkout a new branch
2. [Inside Container] drush features:list:packages to view which installed modules do not match the features configuration.
3. [Inside Container] Foreach module marked as changed:
   1. drush features:export <module>
   2. review the changes that were exported to ensure they make sense. (e.g., you did not just get a bunch of configs for other modules)
   3. Commit the changes. (Yes, commit for each package for sanity)

1. Open the upgrade ticket and create a new comment with a running list of the steps you are typing. You will most likely need to redo the steps at some point, and you want to be able to copy and paste. Also you want this comment to end up being the actual steps followed for future reference. See example.
2. Review drupal.org issues for each of the patches for core, listed in the composer.json file. Do the following:
   1. If the issue is marked as fixed in the version you are upgrading to: remove the patch from composer.json
   2. If the comments mention that the patch did not work and someone made a new patch for the version you are upgrading to: update the patch url in the composer.json file to use the new patch.
3. Review each drupal.org issues for each module we have patched in composer.json file. Do the following:
   1. If the issue is marked as fixed in the same major version we have used you are upgrading to: remove the patch from composer.json
      • Say we are using 3.2.1 of a module and the issue is marked as fixed in 4.0.2, then an update of the module would not bump us to 4.0.2, and thus we still would need the patch.
   2. If the comments mention that the patch did not work, in a new feature/fix to the same major version we use, and someone made a new patch for the version you are upgrading to: update the patch url in the composer.json file to use the new patch.
4. [Outside Container] Upgrade Composer Packages
   1. Revert any pending git changes (git status should say there are no changes)
   2. composer cgov-clean && composer install
   3. composer require drupal/core-composer-scaffold:~x.y.z drupal/core-recommended:~x.y.z --update-with-dependencies --no-update
      • where x.y.z is the full version number (e.g. 9.2.9)
   4. composer require drupal/core-dev:~x.y.z --dev --update-with-dependencies --no-update
      • where x.y.z is the full version number (e.g. 9.2.9)
   5. composer update to actually download and update the packages.

      [!WARNING] This is now when things will break. This will most likely be do to a patch that can't be applied. If things break see Dealing with Bad Patches

5. In a new comment on the upgrade ticket, copy the information composer outputted for the upgrade, see https://github.com/NCIOCPL/cgov-digital-platform/issues/3769#issuecomment-1821561918 for an example of the output to look for. THIS IS IMPORTANT FOR TROUBLESHOOTING. If you miss this go back to the beginning.
6. [Outside Container] Run git status to find out what files were changed other than composer.json and composer.lock.
   • For example, the types of changes could be to .htaccess. If the file was going to not work correctly, then our patches should have failed , and you would have had to fix our patches. However, there could be files we had modified that Drupal is overwriting that would remove our changes, and that would be bad.

1. [Inside Container] drush updb
   • This all is assuming that drush updb works -- the update hooks can fail. If it fails you need to reinstall from the develop branch, and switch back to your upgrade branch. Go make a backup instead before starting.
2. Copy the list of update hooks that have been applied as a separate comment on the upgrade ticket. This is helpful for both debugging and also understanding which tickets had changes to the system and understanding what those changes are.
3. [Inside Container] drush features:list:packages to view which installed modules do not match the features configuration.
4. [Outside Container] Foreach module marked as changed:
   1. drush features:export <module>
   2. review the changes that were exported to ensure they make sense. (e.g., you did not just get a bunch of configs for other modules)
      • This is very important because many of those update hooks are adding important things to the configs that we do not want to lose.
   3. Commit the changes. (Yes, commit for each module for sanity)
5. [Inside Container] Run blt drupal:update --no-interaction to make sure that you can apply the features changes. This does not mean an upgrade can work on an existing instance, but it gets close.
6. Take a quick look at your local site to see if it works.
7. Review each ticket/code block of the update hooks that were executed. You want to be looking for code that is adding data to the site because of other data, BUT WOULD NOT ON A NEW SITE.
   • An example would be an update hook that adds permissions to a role based on existing role permissions. In this example you need to review our .install file that added the original permission and add the new permission to the update hook.
   • Another example would be REMOVING permission because of reasons. Same thing -- you need to modify the install hook to not do that. (However, if you miss this here you WILL encounter it doing installation testing... 😃 )

All Commands are run INSIDE the container

1. Validate the code:

   1. Edit phpstan.neon and remove all the excludePaths under the comment ## The following paths should be removed as each module is remediated
   2. blt validate - This will check coding standards.
      • This will throw errors for all of the code that had been excluded. So you will need to figure out what NEW items have come about.
      • This usually amounts to newly added code deprecations for the next major Drupal version. These are annoying because the code works today, but if we wait until the next major upgrade we will have lots of work. Addressing these will be on a case by case basis. Addressing is either:
        • Fix the code
        • Add to an ignore list.
   3. Restore the phpstan.neon file.

2. Run unit tests:

   1. blt test:phpunit - This runs unit tests (in sequential order so you can see errors).
      • Errors here means something broke. You must fix and re-test.
      • You may see deprecations here as well that you saw in the validate step.

3. Test installation

   1. blt cgov:reinstall --no-interaction
      • This will break in a myriad of ways. Configs, bad permissions, etc. You will need to address these.
   2. Look at the local site.

4. Test code regressions - by this point, you really should be able to push the code through the workflow. This is doing a blt setup and running cypress, the detailed instructions are TBD.

1. Remove <repo_root>/docroot/sites/default/files/translations if it exists.
2. Comment out the following two lines from <repo_root>/docroot/sites/settings/cgov_core.settings.php:

   $config['locale.settings']['translation']['use_source'] = 'local';
   $config['locale.settings']['translation']['path'] = DRUPAL_ROOT . '/translations';
   

3. Run blt cgov:reinstall --no-interaction
4. Delete all .po files from <repo_root>/docroot/translations
5. Copy all .po files from <repo_root>/docroot/sites/default/files/translations to <repo_root>/docroot/translations
6. Uncomment out the above two lines from <repo_root>/docroot/sites/settings/cgov_core.settings.php.
7. Remove <repo_root>/docroot/sites/default/files/translations if it exists.
8. Run blt cgov:reinstall --no-interaction
9. Verify translations work as well as they are going to work in Spanish. (e.g., no worse than before)

You can test on an ODE, but real testing must be on ACSF. It is SUPER important that you evaluate the admin interfaces of a new site against an upgraded site.

You will need to re-roll the patch to get it to work, which means, you need to make the patch.

There are roughly two types of patch types:

1. Patches under composer.json "patches" - These are patches for vendor php packages, e.g., modules, core. When these fail, you need to go find a new patch to apply. Since you probably reviewed all patches before attempting the update, this means there is currently no working patch. So you must re-roll the patch. This means:
   1. Getting a clean copy of Drupal code (this would be a the drupal core, or a module)
   2. Applying the code changes by hand
   3. Submit the patch onto the original ticket and make sure testing passes. (If it does not pass, then your patch is bad)
      • This all assumes that Drupal has good tests for the code you are changing, and well, that is not always the case.
2. Patches under post-drupal-scaffold-cmd - These are patches that get applied to Drupal scaffolding items. These are files that are always copied from Drupal core into the local project folder. (e.g., the root .gitattributes or docroot/.htaccess) These files must be patched after they are copied. So they are not a drupal/core patch, it is actually handled by drupal scaffold. Re-rolling is pretty similar to the "patches" section. The easiest way is:
   1. Remove the patch commands from post-drupal-scaffold-cmd related to this file.
   2. Modify the file in question
   3. Regenerate the patches.
      • TBD on the exact command.
      • For the .htaccess patches we made individual patches that must be applied on top of each other. We did this to keep track of WHY the specific files were being patched. This may or may not be needed. Based on experience though, diffing a diff in GitHub is near impossible.