< Home

Representation

The different stages version and move data towards running on a cluster.

• Code - whatever code is used to build your system. Code is stored in a Git repo and implements one or many microservices.

• Secrets, as JSON files, are stored and versioned in a separate Git repo. They configure runtime information for different contexts, cluster configurations, passwords, etc. They are used by command line scripts, populating K8s templates, cluster deployment, etc. WARNING: ensure your git repo is encrypted and secure as many free services, i.e. Github, are not.

• WovTools Files, in the code's git repo (wovtools directory), store WovTools specific files such as handlebars templated K8s Yaml files, project specific commands and services, Docker container scripts and the Database Registry. These templated files are packaged with the code, and compiled with data from Secrets to create versioned K8s files

• Databases and Data Sets - database schemas and snapshots are versioned and matched to deployments. Schema hashes quickly indicate changes. An SQL file (called a delta) is matched to a change to account for differences between versions.

Stages of Development

A microservice is built on a development machine locally using whatever language and development tools and without any WovTool considerations. Then, the project is inited and push and deployment stages begin.

• Build - The microservice(s) are built into their local archives.

  • Git Repos (~/.wovtools/config:.archives.coderepo) - a git server storing raw text files use to build and version the project

  • Local Archives - (~/.wovtools/[dba|dsa|sea] - The output of building a project, stored locally before being pushed.

• Push - The parts of the microservice are built, versioned and pushed into their respective archive.

  • Container Archive (wovtools/config.json:.archives.container) - stores the versions of the built Docker containers (Docker Server)

  • K8s Archive (wovtools/config.json:.archives.k8s) - stores versioned Kubernetes YAML files in a file server (AWS S3)

  • DB Archive (wovtools/db/archive) - stores database schemas, deltas, snapshot references and versions (Git repo)

• Deploy - A version of the project is sent to a K8s context to run.

  • Origin - A microservice defaults to running 'internal', i.e. in the cluster, but also as 'here' for local development.

Projects and Systems

Loosely, projects refer to the people/teams/repositories that serve a purpose and systems refer to the cluster(s) that the project runs on. Microservices by their nature make defining a "project" difficult and team/company naming conventions can conflict with this as well. Teams/subteams can "own" multiple projects but generally you should only have one team per Git repo.

• Master Project - the entire system/systems of microservices and projects (which may only be one project)

• (Team) Project -a Git repository containing one or many microservices that may be the master project or just a part of some master project. The 'Team' part emphasies ownership of the project by a team that manages/developes it.

• System - A top-level organizing name in which clusters live. So, there could be multiple clusters in a system, multiple providers, etc. This system name is the top level and could be something like a product of the company.

Naming Conventions

WovTools uses the existing structures in Git and Kubernetes to organize its work. Naming conventions and practices are a confusing and debatable topic. Nice thread talking about the ‘flavor’ concept I created here: https://groups.google.com/forum/#!topic/kubernetes-users/GPaGOGxCDD8".

• context - {cluster}-{namespace} - Fully defines where a microservice runs. ex. wov-aws-va-grape-api-dev

• cluster - {system}-{provider}-{regioncode}-{flavor} - The hardware. The Kubernetes cluster. ex. wov-aws-va-grape, for a system name of 'wov' on the AWS provider, in the Virginia region (us-east-1) with a name of grape, which is akin to a given name. A new cluster of wov-aws-va-apple could be created to redirect traffic to or test or experiment with cluster configuration.

• system - ex. wov. A system name in which clusters live. So, there could be multiple clusters in a system, multiple providers, etc. This system name is the top level and could be something like a product of the company.

• provider - ex. aws, gcp. A cluster hardware provider that Kubernetes is operating under. Only using aws at the moment.

• regioncode - ex. va. A short code given to a region, in this case 'va' refers to AWS's us-east-1, which is in Virginia (i.e. va).

• flavor - ex. live, blue, top. A name given to differentiate between clusters. You can have dev and prod stages inside a cluster, but sometimes you need different flavors of clusters to experiment.

• namespace - {aim}-{stage} ex. api-dev. A namespace separates parts of the system.

• project code - ex. api, app, plfb. An group of services running in a namespace to perform a task of the larger system. This might be considered a code project and has a single repository for this project. There can be one or multiple microservices in it.

• stage - ex. dev, text, acc, prod. Deployment stages in software.

• microservice {projectcode}{microservicecode} ex. apirest, apidb, wwwapp, plfbmonitor. Created as part of a project, it is a Kubernetes deployment of a Docker contianer, with pod(s), service and ingress, running on a cluster to do work for a project.

• container - {project/microservice}. A Docker container to fulfill a service. It is built using a recipe that prepares the code to deploy and runs Docker commands to build the container.

• user - ex. minikube, admin. A kubernetes user.

• node - An actual processing unit that does work. At least one per cluster.

Versioning

A pushed and deployed system is defined by the Primary, Secondary and Database version.

• The Primary version refers to a project's Git commit of the templates, code and recipes.
• The Secondary version refers to a Git commit of the JSON files that define a project's configuration.
• The Database version is a Git commit of a database schema and a delta file (of SQL commands) that transform from the previous schema version to the current.

Using the commits referred to by the Primary and Secondary, a Docker container is built and Kubernetes files are compiled, both which are pushed to their respective repositories, and tagged as '{Primary}_{Secondary}'. In this way, deployment is the retreival of the Kubernetes files of the necessary version and the sending of that to a cluster.

• TODO - force the pairing of the Database version to the Primary and Secondary version