Building the PC2 GitHub Website - pc2ccs/pc2v9 GitHub Wiki

The PC² github.io website is a set of static web pages which are dynamically created (updated) each time a new "build" is created. That is, each time an update is pushed to either the "develop" or "master" branch, a new (set of) web pages is built.

Building the web pages is done by a tool called Hugo. Hugo is invoked as one of the steps in the PC² Continuous Integration and Deployment (CI CD) process, which is controlled by the file .gitlab-ci.yml. (Specifically, this file gets executed (on GitLab) each time a GitHub change invokes the PC² GitLab build pipeline, and the file contains an invocation of Hugo as one of its stages.)

Hugo reads a set of files under the folder pc2v9/website to determine what the content of the web pages should be and how the pages should be formatted. Details of the complete Hugo directory structure under /website can be found here, but the following are the most relevant for the PC² website:

• /content : contains Markdown files describing the basic content of each of the main PC² website pages. These are the files that one would edit to change the basic content of a given web page.[1] Content pages (Markdown files) for the website currently include:

  • _index.md: the main "home page" for the website.
  • all-builds.md: a page containing links to PC² builds in chronological order.
  • current.md: a page with links to the most recent stable public release, next release candidate, and most recent nightly builds. This page also contains a link for downloading the current PC² Contest Administrator's Guide.
  • previous.md: a page with links to previous stable versions of PC².

• /layouts: contains Hugo HTML templates defining the static appearance of the web pages. For the PC² website, these templates are stored as Hugo shortcodes contained in subfolder /layouts/shortcodes.

• /static: holds files that are applied statically ("as-is", without modification) to the web page. This folder includes subfolders /doc, which holds documents such as the PC² Contest Adminstrator's Guide (linked from the Current Downloads page), and /img, which holds image files, such as the PC² logo, which are displayed in various web pages.

Hugo builds the PC² website by starting with the files found under /pc2v9/website/content. Each file under /content defines one website page and contains two basic components: Markdown text describing the content of that page, and Hugo action templates which direct Hugo to perform additional processing. These templates (which are actually Go templates) are "markup" in the form of Go actions and variables, and are denoted by commands contained within double curly braces ({{...}}).

As an example, the file /content/all-builds.md contains some introductory Markdown-formatted text defining the basic content of the All Builds page, followed by a single "action": {{< allbuilds >}}. This action instructs Hugo to insert at that point the contents of an action template named allbuilds. Hugo looks in the pc2v9/website/layouts/shortcodes folder to find action template files.

Each action template is a standard HTML file, augmented with "action markup" which instructs Hugo regarding data that is to be inserted into the basic HTML template. The "markup" is in the same form as above: actions and variables within {{...}}.

To continue the above example, the allbuilds template file, found in /layouts/shortcodes/allbuilds.html, looks like this:

{{ range $build := $.Site.Data.releases.all }}
<table class="table table-hover table-striped">
    <tbody>
        <tr>
            <td><b>Version {{ $build.version }}</b> &nbsp; &nbsp; (build date {{ $build.time }} on {{ $build.date }})</td>
        </tr>
    </tbody>
    
    <tr><td>
    
<table class="table table-hover table-striped">
    <tbody>
        {{ range $pc2 := $.Site.Data.pc2 }}
            {{ if isset $build.downloads $pc2.tool }}
            {{ $download := index $build.downloads $pc2.tool }}
            <tr>
                <td><b>{{ $pc2.name }}</b></td>
                <td align="right">{{ $download.size }} MB</td>
                <td>
                    <a href="{{ $download.urls.zip }}">{{ $pc2.tool }}-{{ $download.version }}.zip</a>
                    {{ if and (isset $download.urls "sha256") (isset $download.urls "sha512") }}
                    (<a href="{{ $download.urls.sha256 }}">sha256</a>, <a href="{{ $download.urls.sha512 }}">sha512</a>)
                    {{ end }}
                </td>
            </tr>
            {{ end }}
        {{ end }}
    </tbody>
</table>

</td></tr></table>

{{ end }}

Hugo processes this file by starting with the {{ range $build := $.Site.Data.releases.all }} statement (action), which basically says "find all the releases defined in the Site Data, assign each release to the variable $build, and process the following statements (down to the final {{end}}, which marks the end of the range) using the current value of $build each time (in other words, a "for loop" on the following HTML code using each value of $build within the loop).

The first block of HTML code is an HTML table which has two elements: a table row (<tr>) containing various values obtained from the current value of $build, and a second element which is itself another (inner) table. The inner table in turn has another "action" which loops through all the data found in Site.Data.pc2; it assigns the current data value to the variable $pc2 and then builds inner table rows containing that data. (Note that inclusion of some data in the table is based on conditional actions; see the Hugo and Go links above for further details.) The output when Hugo finishes processing this template file is the set of HTML commands describing the All Builds page, which can be seen at this location.

In addition to creating each of the pages defined under /content and laying them out as described under /layouts, Hugo arranges that pages (files) listed under /content automatically appear as selectable items in the web page's left-nav bar. Each page file under /content normally includes a Markdown "table" containing general information about the page; Hugo uses this table to obtain information for building the left-nav bar. A typical example of a web page Markdown file table is:

---
title: "Home"
description: "Programing Contest Control (PC^2) system implemented by the PC^2 Development Team for use at the ICPC World Finals and other programming contests"
date: 2019-11-12T21:10:52+01:00
draft: false
weight: 1
icon: fas fa-home
---

This table, which comes from the /content/_index.md file, causes Hugo to add a left-nav bar item named Home which links to the _index.md page. See the Hugo documentation for further information on exactly how this table is used by Hugo.

As described above, Hugo creates a set of static HTML pages. These pages are contained in files all named index.html - one for each website page. The main website page (the "home page") index.html file is stored at the root of the website structure, while additional web page index.html files are stored in subfolders corresponding to the page name. (For example, Hugo creates files current/index.html, all-builds/index.html, and previous/index.html corresponding respectively to the static web pages for the Current Downloads, All Builds, and Previous Stable Downloads pages.)

Following creation of the web page index.html files, and prior to pushing those files from GitLab to the pc2ccs.github.io website, the Continuous Integration and Deployment (CI CD) process controlled by the .gitlab-ci.yml file applies an additional post-processing step to the index.html files: it runs sed on them to replace occurences of the string & with a single & character. This is necessary to correct the rendering of certain character strings (for example, the string PC², which is expected to be rendered as "PC²"). Hugo expands the & character in PC² to &; this post-processing step restores it to a single & character.

This means that if a new web page is added under /content, that web page folder and file name should be added to the sed post-processing command in the .gitlab-ci.yml file. Otherwise, literals using HTML "ampersand character codes" will not be rendered properly on the page.

The process of creating new web pages and/or updating existing web pages is carried out in the same manner as making changes/updates to any other portion of PC²: by following the steps listed at Development Flow. However, the process of testing website changes (that is, observing the results in terms of "changed web pages") is complicated by the fact that website changes are reflected (only) in dynamically-created (Hugo-generated) web pages. Thus, in order to view the results of making website changes it is necessary to invoke Hugo.

One way to accomplish this would be to submit a Pull Request (PR) (see Guidelines for Submitting Pull Requests) and then have this PR be merged into the PC² code stream, causing the Continuous Integration and Deployment (CI CD) process to use Hugo to generate a new website (along with a new PC² distribution). However, this would be counter to the PC² Development Flow, which requires that changes be tested before they are merged into the PC² code stream.

The alternative is to directly invoke Hugo on your local machine to generate a (local copy of the) website, and then to view the changes in the local website. In order to do this it is first necessary to pull the changes into your local Git repository.

If the changes are part of a PC² GitHub Pull Request which you have been asked to review, pulling the changes onto your local machine can be done as follows:

• In your local Git repository, insure that you are on an up-to-date version of the develop branch. (Follow the instructions at this page to re-sync your develop branch to be sure it is up-to-date.)
• on the GitHub PR page, click the "Check out locally" drop-down box. This should show a list of the Git commands which need to be executed in your local Git repo to fetch the changes locally. Note: the commands can be copied to your clipboard by clicking the little "copy to clipboard" icon next to the drop-down list of commands.
• The commands will most likely refer to fetching things from the origin repository. In most cases the origin repository for the local repo for PC² developers will actually be their pc2v9 GitHub fork. However, the changes need to be pulled from the pc2v9 repo, not from a fork. If you have followed the instructions under Development Flow, the pc2v9 repo will be the repository referred to as upstream. Therefore, references to origin in the commands should be changed instead to refer to upstream.
• In your local Git repository, execute each of the specified commands, (again, be sure you are on the develop branch). This will create a local branch containing the updated code.

If you don't see a "Check out locally" button in the GitHub PR, you may be able to determine the necessary commands by going to the "Merge Pull Request" block in the GitHub PR and clicking on the text command line instructions. This should display two sets of commands: one set to pull the changes down to your machine, and a second set which would be used if you were going to push changes back up to GitHub. Use the first set of instructions, labeled Step 1, to pull the changes to your local machine (however, do NOT under any circumstances execute the second set (Step 2) instructions). Note that, as above, references to origin need to be changed to refer instead to upstream, and you should be sure you are on the develop branch before executing the commands.

Once the code for the changes has been loaded into your local Git repository, the next step is the following:

• From the pc2v9 directory, execute the ./website/scripts/populate-releases.py Python3 script (this requires having Python3 on your machine). This script updates the JSON release information which Hugo uses in building the website pages.

The next step is to use Hugo to generate the updated website pages. There are two approaches which can be used:

• generate the pages and then install them into an existing webserver (from where they can be viewed using any browser);
  or
• generate the pages and use a Hugo-provided webserver to serve the pages locally.

The first approach has the advantage that you will be able to view the website pages exactly as they will appear when served by any webserver. The disadvantage of this approach is that it requires access to an existing web server into which you are able to install the pages.

The second approach has the advantage that no "external" webserver is needed; Hugo will supply an embedded webserver which serves the pages to your browser. The disadvantage of this approach is that certain minor differences (related to display of HTML & literals; see below) may occur between the pages served by the Hugo embedded webserver and the way those pages will actually look when served by an external webserver.

The following steps are used to generate and view the website exactly as it will appear when served by a webserver.

• From the website directory, run Hugo (execute the command hugo in the website folder). (This requires having Hugo on your machine; see https://gohugo.io/getting-started/installing/)). This causes Hugo to generate the website pages in a new folder named public under the website folder.

• Change to the website/public folder.

• Execute the command sed -i -e 's/&/\&/' *.html */*.html.

  What this command does is update all references to HTML & literals in the index.html files so that they render as expected on the generated web pages. Omitting this command will simply mean that HTML strings such a PC² will be rendered literally as "PC²" instead of the expected "PC²".

• Copy the files under the website/public folder to an existing webserver (that is, to a location from which an existing webserver will serve them to a browser).

• Open a browser to the URL for the root (index.html) page for the website being served by the webserver.

• Use the browser to navigate around the website verifying the changes.

Instead of using the hugo command, which generates web pages that must then be installed into a webserver, it is possible to use Hugo to both generate the web pages and to also serve them to a browser. The steps to do this are:

• Change to the website folder.
• Execute the command hugo server.
• Open a browser to the URL http://localhost:1313. This opens the "Home page" of the website.
• Use the browser to navigate around the website verifying the changes.

The above commands cause Hugo to build the website pages in memory and then run an internal webserver to serve them. The Hugo webserver listens by default for connections on port 1313; opening a browser to that port will display the "Home page" of the website and allow navigation around the website. Note that the ./website/scripts/populate-releases.py Python3 script should still be run prior to running Hugo, regardless of whether you use hugo or hugo server.

There are certain advantages to using hugo server (rather than hugo) to examine website changes. In particular, the webserver created by hugo server listens for any changes to the website files and automatically rebuilds the site when changes occur. This means it is not necessary to reexecute the hugo command for every change.

One disadvantage of using hugo server is that the "post-processing" sed command described above will not be executed, meaning pages containing HTML & literals will be rendered "incorrectly" on your local machine (they should still render correctly when the website is built by the PC² Continuous Integration and Deployment (CI CD) pipeline).

hugo server has a wide variety of options which can be used to adjust its operation; for more information see https://gohugo.io/commands/hugo_server/.

[1] It is important to understand that, while the PC² github.io website actually exists as a separate repository which is visible in the pc2ccs organization (specifically, it is the repository https://github.com/pc2ccs/pc2ccs.github.io), one should NOT clone that repository nor attempt to edit any files within it. This is because that repository is dynamically regenerated, each time a PC² "build" occurs. The proper way to update the PC² github.io website is to use a branch to edit files under /website in the pc2v9 repository and then push those branch changes to the pc2v9 repo. Pushing the changes will initiate a Continuous Integration and Deployment (CI CD) build, which will regenerate the website.