Getting Started - BanklessDAO/content-gateway Wiki

📘 This document will help you get started with the Content Gateway codebase.

Development

📙 A note about operating systems: this guide will work with Linux and WSL environments. Other environments are not supported. This means that you can still work on the codebase, but you'll have to figure out how to do all the stuff that's outlined below on your own. 👇

Before you can start working on the codebase you'll need to install:

Environmental Variables

In order to be able to start working on the code you'll need to set the following environmental variables.

# Content Gateway API

export CGA_PORT="3333"                           # Will be used locally. On Heroku the PORT variable is used.
export MONGO_CGA_PORT=<figure_something_out>     # Used by Docker compose locally
export MONGO_CGA_USER=<figure_something_out>     # The MongoDB user name (will be used as a database name)
export MONGO_CGA_PASSWORD=<figure_something_out> # Used by Docker compose locally
export MONGO_CGA_URL=<your_mongodb_url>          
# 👆 The URL that points to your MongoDB instance. When you run CG locally you need to compose this from the port/user/password
# When running in the cloud you'll probably use Atlas or something similar
# 👇 This should be the default for local development
# export MONGO_CGA_URL="mongodb://${MONGO_CGA_USER}:${MONGO_CGA_PASSWORD}@localhost:${MONGO_CGA_PORT}/"


# Content Gateway Loader

export CGL_PORT=3334                          # Will be used locally. On Heroku the PORT variable is used
export CGA_URL=http://localhost:${CGA_PORT}   # This is necessary for the Loader even on Heroku.
export GHOST_API_KEY="<ghost_api_key>"        # For the Bankless website.
export YOUTUBE_API_KEY="<youtube_api_key>"    # For the Bankless podcast.
export PG_CGL_PORT=<figure_something_out>     # Used by Docker compose locally
export PG_CGL_USER=<figure_something_out>     # Used by Docker compose locally
export PG_CGL_PASSWORD=<figure_something_out> # Used by Docker compose locally
export PG_CGL_URL="<your_postgres_url>"       
# 👆 The URL that points to your PostgreSQL instance. When you run CG locally you need to compose this from the port/user/password
# When running in the cloud you'll probably use a hosted DB
# 👇 This should be the default for local development
# export PG_CGL_URL="postgresql://${PG_CGL_USER}:${PG_CGL_PASSWORD}@localhost:${PG_CGL_PORT}/${PG_CGL_USER}"

# Optional variables

export NODE_ENV="development" # or production
export RESET_DB="false"        # Set it to "true" if you want to reset the database when the application starts up

Heroku Setup

If you want to deploy an instance to Heroku, these are the necessary steps:

📙 Make sure that you call heroku login before trying to do this. You'll also need the Heroku CLI installed

First, we create the apps on Heroku:

heroku create content-gateway-api --remote cga # --team if you use teams
heroku create content-gateway-loader --remote cgl # --team if you use teams

Now you need to add the multi-procfile buildpack to them. This is because Heroku assumes that you have one app per repo by default, and this enables to have multiple Procfiles (deployments) in a repo

heroku buildpacks:add --app content-gateway-api heroku-community/multi-procfile
heroku buildpacks:add --app content-gateway-loader heroku-community/multi-procfile

Of course this won't work because Heroku doesn't know about node, so we need to add the node buildpack too:

heroku buildpacks:add --app content-gateway-api heroku/nodejs
heroku buildpacks:add --app content-gateway-loader heroku/nodejs

Then we'll have to tell Heroku where these Procfiles are:

heroku config:set --app content-gateway-api PROCFILE=apps/content-gateway-api/Procfile
heroku config:set --app content-gateway-loader PROCFILE=apps/content-gateway-loader/Procfile

Then we'll need to add a heroku-postbuild script to override the default build behavior of Heroku and let it build the project we need. This goes into the package.json in the root folder:

scripts: {
  "heroku-postbuild": "heroku-postbuild": "script/heroku-build $PROJECT_NAME"
}

📗 A note on the script folder: this project follows the Scripts to Rule them All guidelines. You'll find scripts for most tasks that you might want to execute there. If you call a script you'll see some documentation too.

Heroku needs to know the value of $PROJECT_NAME for each deployment so let's set them:

heroku config:set --app content-gateway-api PROJECT_NAME=content-gateway-api
heroku config:set --app content-gateway-loader PROJECT_NAME=content-gateway-loader

One last thing before we can push this: the CGA_URL needs to be set for the Heroku deployment too, so let's add it:

# at the time of writing this was the URL, it might change later 👇
heroku config:set CGA_URL=https://content-gateway-api.herokuapp.com/ --remote cgl

Finally, we push it to Heroku

📙 Don't forget to commit your changes before pushing 😅

git push cga master
git push cgl master

With this you'll be able to start the apps, but they won't work still 😅, because we haven't configured the databases yet!

Let's take a look at how to do this.

Database Setup

The Content Gateway Loader uses Postgresql. Adding Postgres to Heroku is relatively simple.

Fist you'll need to create a Heroku Postgres database for the loader.

📗 Note that this is already created for the production deployment, you'll only need this if you want to create your own Content Gateway instances.

heroku addons:create heroku-postgresql:standard-0 --remote cgl --name=pg-cgl --as=pg_cgl

The --as parameter will make sure that Heroku creates the proper environemntal variables (by default it will create DATABASE_URL which is not very nice).

📙 If you have different environments keep the --as parameter as-is, as it will affect the name of the environmental variable. So if you have a prod and a staging environment you'll call this command like this: heroku addons:create heroku-postgresql:standard-0 --remote prod_cgl --name=prod-pg-cgl --as=pg_cgl

As for the API we're going to need MongoDB. Heroku doesn't come with a MongoDB addon, so we'll need to use somethign else. What I'd suggest trying out is Atlas.

Once you set up the Atlas instance you can just set the MONGO_CGA_URL environment variable for the API instance and it'll work.

Local Development

If you don't want to deploy your own Content Gateway instances to Heroku, it is enough to set it up locally. We recommend using VS Code.

Extensions

There are a list of extensions that you will probably need:

Other not so important extensions:

If you choose to use Terminals Manager this is a good configuration for this project (belongs in .vscode/terminals.json):

{
    "autorun": true,
    "autokill": true,
    "terminals": [
        {
            "name": "api",
            "description": "Content Gateway API",
            "open": true,
            "focus": false,
            "recycle": false
        },
        {
            "name": "loader",
            "description": "Content Gateway Loader",
            "open": true,
            "focus": false,
            "recycle": false,
            "commands": ["script/prisma-generate cgl"]
        }
    ]
}

Once you've set your development environment up you can start working on the codebase. Make sure that you prepare the workspace with the script/setup script.

📗 A note on Nx: this project uses Nx which helps with handling the monorepo. Explaining how it works is out of the scope of this guide, and you probably won't need it anyway as most commands have a script. It is recommended however to learn how it works.

Docker

You don't need to use Docker for local development, but we have created a setup using Docker that's easy to use and takes care of all the infrastructure needs of the project.

📗 If you're not familiar with Docker you can read more about it here.

If you want to take advantage of this setup we have a script for creating and stopping a Docker enviroment.

📙 Make sure that you add the environmental variables (see above) before you run the Docker script. The Docker containers won't be created properly if Docker can't access the environmental variables it needs. See the docker-compose.yml file to determine what you need. Also, make sure that if you edit your .bashrc / .zshrc file then you either source it before you run this script or you restart your terminal.

📙 If your Docker containers don't seem to work no matter what you try you can remove and prune them. Docker won't remove volumes (disk drives) automatically and this can be problematic if you try to recover from a failed state. In this case the following command will help: docker container kill mongo_cga && docker container kill postgres_cgl && docker container prune -f && docker volume prune -f. Note that this can lead to data loss!

Use this script to start it:

script/docker-up

This will start a MongoDB and a Postgre SQL instance in Docker that's ready to be used from Content Gateway.

You can stop it with:

script/docker-down

📙 If you get permission errors make sure that these scripts have executable permissions. You can grant them by using chmod +x <file_path> in the terminal.

If you want to start the applications you can use the script/serve script with the appropriate parameters:

script/serve cga
script/serve cgl

📗 cga stands for Content Gateway API and cgl stands for for Content Gateway Loader.

If you thing you've messed something up you can also reset the databases by providing the RESET_DB="true" environmental variable.

For ease of use we recommend creating some aliases (you can put these in .bashrc or .zshrc):

alias nsa="script/serve cga"
alias nsl="script/serve cgl"
alias nsar="RESET_DB=true script/serve cga"
alias nslr="RESET_DB=true script/serve cgl"

script/serve will start a development environment that also supports hot code replace. If you need a production environment you can use the following commands to build the application and run it:

📗 For all scripts that accept a <project> parameter you can pass either cga or cgl.

script/build <project>
script/run <project>

Congratulations! You've completed the project setup and now you're ready to start working!

A Note on fp-ts

We're using fp-ts and io-ts in this project. These libraries implement strictly typed Functional Programming in TypeScript. If you're not familiar with the topic you can learn it quickly by following through these guides:

You can also take a look around the author's website

Directory Structure

We use the Nx guidelines for the directory structure. This means that we have the following folders: