• DevOps combines development (Dev) and operations (Ops) to increase the efficiency, speed, and security of software development and delivery compared to traditional processes. A more nimble software development lifecycle results in a competitive advantage for businesses and their customers.

One of the major areas of DevOps is building and maintaining Pipelines

CI/CD Pipelines are supposed to

• Build package
• Deploy Application in various environments
• Run the Automated Tests for each environment and Generate Reports

• Artifact: Build result is an artifact, the package is most important one to be preserved
• Repository: A storage location which can maintain history (i.e. versions or revisions )

• This is a source code repository which stores code and also the history which represents
  • the changes done
  • who has done it
  • when it was done
• Generally Version Control Systems are designed for multi users.
• Any VCS will have repositories which developers(users) will get a copy or equivalent to copy in their local workstation
• They get or make changes from/to VCS

Client Server:

• Subversion
• TFS
• ClearCase
• Perforce
• Distributed

• Mercurial
• Git

• In Git every one (client, server) have repositories
• IF you have to create a server you need to install same software as client (git) and for server features (USer management, Connectivity) - you can install additional components

• I have a huge room full of books, i want to send them to new home.

• Git has five stages, initially we will be focusing on three areas

• This configuration is done to set username and email id of the author
• git config --global user.name ""
• git config --global user.email ""
• Our current focus areas

• Create a new directory in your system and cd into it
• Now execute git init to initialize a git repository

• This is area of work for us
• In this we make changes, add/delete files etc

• This is area where the changes can be staged to be part of a commit

Other things to know

• log: This represents the history of the repo.
• status: This command represents what is your current status. For git the ideal status is working tree == local repo
• Tracked and Untracked files: Tracked files represent the files which were part of any commit i.e. they are part of git repo, whereas untracked files are not part of the repo yet.

• Lets create a new repo
• Git doesnot consider empty folders as change. Git considers only files as changes
• Every commit will have a unique commit id

• In Git we have a concept of reference object, We have two reference objects

  • branch
  • tag

• In Git by default we are working on a branch called as master

• According to our understanding so far, A branch points to the latest commit

Commands used

git log
git status
git log --oneline
git restore --staged <file path>

• Git internally uses SHA1 hash for everything

• Commit: This is hash of changes, parent commit, author, date time, message

For git

• file: blob
• folder: tree

To Staging Area

• Add all changes git add -A
• Add only modified files but not untracked

ignoring files

• To ignore certain files, in the root folder of repository create a file called as .gitignore
• To remove all untracked files from working tree git clean -fd .

Git branch

• Refer Here for docs
• Branch always points to latest commit on it
• In Git default branch is master and now a days majority of the organizations have moved to main as default branch.
• To rename branch git branch -m
• To create a new branch git branch

• To checkout to a branch (HEAD pointing to other branch)

• Now lets do a couple of commits

• Lets checkout to main

• Refer Here for docs
• Tag always points to a specific commit

• Bringing the changes from one branch to another
  • Merging
  • Rebase
  • Cherry-pick (specific commits not complete branch)
• Refer Here for merge documentation

• I have create a local repo to simulate the below graph

• Always check out to destination branch (branch which wants changes)

git checkout <destination-branch-name>

• Execute the command to merge from the source branch (branch from which changes are requrired)

git merge <source-branch>

• Overview

• Commands

• Overview

• use the same commands

• Overview

• Refer Here for docs.

• Commands: - checkout to destination or target branch git checkout - rebase from source branch git rebase

• Refer Here for docs

• We can get merge conflicts during rebase as well
• Watch classroom video for example

• Refer Here for docs

• Fixing commit message of latest commit

• Changing older commit messages
• Deleting commits
• combining commits (squash)
• rework on older commit

• Git has two types of logs
  • log
  • ref-log: This is a permanent log but available only in .git folder
• Watch classroom video for steps

• Detached Head refers to a state where HEAD is pointed towards a commit rather than a branch.
• We can use this for verifying changes but never start committing changes in this case, as changes might be lost.
• Best Practice: If you want to work on the older commit,
  • checkout to that commit
  • create a branch from there git switch -c ""
  • Now make changes in this branch
  • if the changes are required any where else use merge or cherry-pick.

• Git remote is another repo which is ideally containing the same code as yours.
• Git Remote can be hosted

  • locally on your system for local usage (This doesnot make sense)

  • Organizations hosting git on servers owned by them (Self Hosted Git)

    • Gitolite
    • Gerrit (Hosting + Code Reviews)
    • Gitlab Self Hosted Version
    • GitHub Enterprise (SelfHosted)

  • Cloud Hosted Git Remotes Cloud deployment consulting

    • GitHub
    • Azure Source Repos
    • AWS Code Commit
    • Atlassian Bit Bucket
    • GitLab

• Remote Repo

• Clone: When you donot have local copy of the repo and want it on your system you perform clone
• pull: to get the latest commits into your local repo (need to dive more)
• push: to send the commits from your local repo to remote repo
• Every remote repo has a url and a name. origin is the default remote repo name.

• In github we can create public repos or private repos.
• Let me create a public repo

• Let clone git clone repo-url

• lets cd into folder and execute git commands

• Lets create a commit and push the changes

• Every remote repo added to the git brings in additional branches
• One local repo can be connected to more than one remote repos.
• The command to push the changes is

• Git pull fetch + merge : DO this when you dont have local changes
• git pull –rebase fetch + rebase: DO this when you have local changes for cleaner history.
• Git push => git push
• Upstream branch: For the local branch in your repo what is the equivelent remote branch
• Syncing changes from one remote repo to other is a straight forward
  • Add two repos as remote to you local repo (clone from source repo and add remote of the destination)
  • now push branches by using git push

• Developer will be working on new features or defects
• Note: Watch the classroom recording

• To solve the problems with Big Bang Integration, Continuous Integration was proposed

• The Basic Idea of CI is - Perform Integration of all components on every change by any component, initially it might fail

• CI =

  • Build/package the code
  • Run tests:
    • unit tests, integration tests
    • smoke tests, sanity tests
  • Static Code Analysis

• Who should run CI?

  • Manual: Not feasible
  • Automated

• To Perform Automated CI, We need softwares. This is what CI/CD Engine is all about. Some Examples

  • Cruise Control
  • Jenkins/Hudson
  • Azure DevOps
  • Github Actions

• Know manual steps
• Improve Readabililty:
  • automated steps
  • executions

• I will be focusing of 4 tech stacks
  • java
  • .net
  • python
  • react/angular using node js
• We will Docker build steps

• Compiled: executable with no extra installation
  • C
  • C++
  • Golang
• Interpreted: directly run code with installed interpretor
  • Python
  • node js
• Hybrid: Get IL/Bytecode and run on IL with install runtime
  • Java
  • C#

• Basic Build steps
• Refer Here

• Compile a C language hello-world code
• Findout what GNU make tool.

• Create a free tier ubuntu vm on azure or aws

• We have javac which is basic java compiler
• For larger projects in very intitial phases, javac was used with make files
• Apache Ant was introduced which helps in building java projects. Ant projects have build.xml which will have build instructions. The build.xml is configuration of what has to be done

• Maven is a tool that is introduced to manage projects. Maven
  • manages dependencies
  • performs build, package, artifact management
  • create documentations
• Maven uses Convention over Configuration model

• Install openjdk 17

sudo apt install openjdk-17-jdk -y

• Install maven

mvn --version
Simple maven commands mvn <goal>

• Simple maven commands mvn

mvn compile
mvn test
mvn package

• Conventions:
  • pom.xml file is required in the root directory of the project
  • Folder structure
• We have created a pom.xml file with following content

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>io.learningthoughts.projects</groupId>
    <artifactId>helloworld</artifactId>
    <version>1.0-SNAPSHOT</version>

    <properties>
        <maven.compiler.source>17</maven.compiler.source>
        <maven.compiler.target>17</maven.compiler.target>
    </properties>
    <dependencies>
        <!-- https://mvnrepository.com/artifact/software.amazon.awssdk/s3 -->
        <dependency>
            <groupId>software.amazon.awssdk</groupId>
            <artifactId>s3</artifactId>
            <version>2.31.6</version>
            <scope>compile</scope>
        </dependency>
    </dependencies>

</project>

• We have created the folder structure src/main/java 

• Now execute mvn compile

• To package mvn package

• Projects skeleton or structure can be created using archetypes

mvn archetype:generate \
  -DgroupId=io.learningthoughts \
  -DartifactId=generated \
  -DarchetypeArtifactId=maven-archetype-quickstart \
  -DarchetypeVersion=1.5 \
  -DinteractiveMode=false

• Maven Goals:

  • Refer Here to this article

• POM Schema

• Maven Projects are generally organized in two styles

  • pom.xml with src folder (default structure): This represents a single project. Spring petclinic
  • pom.xml with modules which are directories which contain child pomsBroadleaf

• Commands

  • mvn :

    • mvn package

  • mvn :

    • mvn clean package

• Maven Remote Repositories can be hosted using Many tools
  • Jfrog Artifactory
  • Nexus
  • Github Pacakges
  • Azure Artifacts

• CI = build + test + package + archiveArtifact
• Sample Projects
  • Spring petclinic
  • Broadleaf Demosite

• This project requires jdk 17 and maven 3.9.9
• This requires downloading and setting up maven, refer classroom video for steps

• This project requires jdk 17 and maven 3.9.9

./mvnw package

• This was introduced by google for building andriod projects
• Gradle has features for both ant and maven
• Gradle has a feature of incremental build

• Here's how some common tasks compare between Maven and Gradle:

• Install .net sdk 9 Refer Here for steps
• Projects:
  • eShop
  • nopCommerce

• Lets try build,test, package of a react js aplication
• todoMvc is sample repo
• To install node and npm i prefer nvm
• Basic steps: Navigate to package.json to find out scripts
• Generally

npm install
npm run build
npm run test

• There are other alternatives like npx, yarn etc…

• This software helps in building CI/CD Pipelines

• A Pipeline is set of steps that needs to be executed in some order

• Popular Options:

  • Jenkins
  • Bamboo
  • Azure DevOps
  • Github Actions
  • Gitlab CI

• Popular styles of pipelines

  • UI Enabled Pipeline Creations:
    • Jenkins (Free Style Projects)
    • Azure DevOps (Classic Pipelines)
  • Pipeline as Code: Pipeline steps are part of version control.

• Azure DevOps offers complete ALM (Application Lifecycle Management) tools i.e. it offers

  • Boards (Plan, Organize projects)
  • Azure Source Repos (Version Control)
  • Azure Pipelines
  • Azure Test Plan
  • Azure Artifacts

• Azure DevOps Has two major options

  • Azure DevOps Server
  • Azure DevOps Service

• Signup For Azure Devops

• Azure DevOps Organization: Refer Here for creation, For creating Projects

• Findout the following terms in the context of Agile
  • Epic
  • User Story
  • Product Backlog
  • Sprint Backlog
  • Daily Standup Meetings

• Azure DevOps has a way to host version control systems. It supports git and TeamFoundation Version Control
• Azure DevOps Notifications can be linked to microsoft teams

• Agents are the systems where the pipelines are executed.
• Agent types
  • Microsoft Hosted Agents
  • Self Hosted Agents

• Import Spring petclinic into Azure DevOps Source Repos and ensure main is the default branch. Refer Here

• Set Git Credentials via ssh preferably

• Now create a develop branch
• Watch classroom video for basic pipeline
• Next Steps:
  • Configuring Self Hosted Agents
  • Learning to write basic pipelines in yaml
• Exercise: YAML and JSON

• Azure Pipelines are written in YAML Format.

• YAML is a data representation format with key value (name value) pairs as it basic structure.

• The tool will define th structure

  • Azure DevOps
  • K8s
  • Compose

• YAML represents data in name value

• name:

• Values can be of following types

  • text
  • number
  • boolean
  • list/array
  • map/object

• Text

title: Court
title: "Court"
title: 'Court'

• numbers

duration: 149
rating: 9.5

• boolean

imax: false
imax: no

• list or array

cast: ["Priyadarshi", "Harsh", "Shivaji"]
cast:
- Priyadarshi
- Harsh
- Shivaji

• map/dictionary/object:
• Crew:

  director: Ram Jagadeesh
  producer: Nani
  music: Vijay

• complete yaml

title: Court duration: 149 rating: 9.5 imaxEnabled: no Cast:

• Priyadarshi

• Harsh

• Shivaji Crew: director: Ram Jagadeesh producer: Nani music: Vijay

• Lets write about one more movie in the same yaml

title: Court duration: 149 rating: 9.5 imaxEnabled: no Cast:

• Priyadarshi
• Harsh
• Shivaji Crew: director: Ram Jagadeesh producer: Nani music: Vijay

title: Chhaava duration: 161 rating: 9.2 imaxEnabled: no Cast:

• Vicky

• Rashmika

• Akshay Crew: director: Laxman Utekar producer: Dinesh Vijan music: A.R Rehman

• If we want some one to fill the movie we define fields and types

title: <text>
duration: <number>
ratings: <number>
imaxEnabled: <boolean>
Cast: text array | list<text> | list<string>
Crew: <Crew>

Crew:
director: <text>
producer: <text>
music: <text>

• pipeline structure (First Version)

• An Agent i.e. a machine to build can be chosen at

  • Pipeline
  • Stage
  • Job

• To Simplify the pipeline

  • If your pipeline has one stage and muliple jobs, then represent pipeline has set of jobs
  • If your pipeline has one stage, one job and multiple steps, then represent pipeline as set of steps

• Lets create a pipeline which runs on microsoft hosted agent which will have Windows OS
• It will have 3 stages
  • Build
  • It will have 1 jobs
    • build the code
  • System Test
    • It will have 1 job which starts the selenium tests
  • Deploy
    • It will have 1 job which runs the terraform
• Pool defines the agents. Check for agent names
• This is dummy as it will only print the value but not execute

name: dummy pool: windows-2022 stages:

• stage: build jobs:
  • job: build steps:
    • script: echo Building
• stage: test jobs:
  • job: test steps:
    • script: echo testing
• stage: deploy jobs:
  • job: deploy steps:
    • script: echo deploying I have one stage, one job and 3 steps

name: dummy pool: ubuntu-latest steps:

• script: echo step1

• script: echo step2

• script: echo step3

• Pipeline generated in last session

• trigger:

  • main

• pool:

  • vmImage: ubuntu-latest

• steps:

• task: Maven@3

inputs:
    mavenPomFile: 'pom.xml'
    mavenOptions: '-Xmx3072m'
    javaHomeOption: 'JDKVersion'
    jdkVersionOption: '1.17'
    jdkArchitectureOption: 'x64'
    publishJUnitResults: true
    testResultsFiles: '**/surefire-reports/TEST-*.xml'
    goals: 'package'

• Refer Here for self hosted linux agents

• Softwares:

  • Openjdk
  • Maven

• Azure DevOps Agent Setup:

  • This requires a PAT (Personal Access Token)

• Lets create a pipeline to build the maven project
• Azure Devops YAML Schema
• Generic Pipeline:
  • When to run
  • What has to be executed
  • Where it has to run
  • metadata
• In Azure DevOps Pipelines
  • What is decided by steps
  • Where is decided by pool section
  • when is decided by triggers
• Manual step

git clone <url>
cd <folder>
mvn package

• When: whenver any change is pushed to main branch
• Where: ON a self hosted agent
• Refer Here for the first pipeline

name: spring-pet-clinic
name: spring-pet-clinic

trigger:
  - main

pool: Default

steps:
  - script: ./mvnw clean package
Lets try using Task

name: spring-pet-clinic

trigger:
  - main

pool: Default

steps:
  - task: Maven@4
    displayName: Maven Build
    inputs:
      mavenPOMFile: 'pom.xml'
      goals: 'package'
      testResultsFiles: '**/surefire-reports/TEST-*.xml'

• Refer Here for the changes with task

• A step is an activity in azure devops pipeline.
• Azure DevOps Steps types
• Azure DevOps Tasks

Directly allowing push to the branches might lead to failure propagations. To ensure that the each change is reviewed and the tests are passing, we use pull requests

• Azure DevOps PRs
• Watch classroom video for creating PRs
• Next Steps:
  • elaborate pipelines for
    • PR:
      • build
      • unit test
      • packaging
      • code analysis
    • Branch:
      • build
      • unit test
      • packaging
      • code analysis
      • Functional tests

• We have following usecases
  • Developers create PRs
  • PRs getting merged into development branch
  • Tester need to test the application
    • Unit tests, smoke tests, Sanity Test => every change (Day Builds)
    • System Tests => Happen once in a day which combines all of the development activtity (Night Builds)

• Microsoft hosted agents are hosted by ado
• For list of agents with softwares installed in it Refer Here

• Watch classroom video
• Run on every pull request to develop branch

---

pool:
  vmImage: ubuntu-latest

trigger: none

pr:
  - develop

steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Projects
    inputs:
      command: restore
      projects: "src/NopCommerce.sln"
  - task: DotNetCoreCLI@2
    displayName: Build the Web Project
    inputs:
      command: build
      projects: "**/Nop.Web.csproj"

• Run on every change pushed to develop branch i.e. this executes whenever a PR is merged.

pool:
  vmImage: ubuntu-latest

trigger:
  - develop


steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Projects
    inputs:
      command: restore
      projects: "src/NopCommerce.sln"
  - task: DotNetCoreCLI@2
    displayName: Build the Web Project
    inputs:
      command: build
      projects: "**/Nop.Web.csproj"
  - task: DotNetCoreCLI@2
    displayName: Run the tests
    inputs:
      command: test
      projects: "**/Nop.Tests.csproj"

• Run pipeline on schedule
  • day build every 2 hours on a weekday
  • nightly build on release branch at 10 PM
• Day build for every 2 hours on weekdays

pool:
  vmImage: ubuntu-latest

trigger: none

schedules:
  - cron: "30 */2 * * 1-5"
    displayName: "CI Schedule every 2 hours on Weekday"
    branches:
      include:
        - develop
    batch: false
    always: false


steps:
  - task: DotNetCoreCLI@2
    displayName: Restore Projects
    inputs:
      command: restore
      projects: "src/NopCommerce.sln"
  - task: DotNetCoreCLI@2
    displayName: Build the Web Project
    inputs:
      command: build
      projects: "**/Nop.Web.csproj"
  - task: DotNetCoreCLI@2
    displayName: Run the tests
    inputs:
      command: test
      projects: "**/Nop.Tests.csproj"

• Exercise: Try React Js Project

• Azure DevOps Pipeline Variables

• Azure DevOps has lots of predefined variables with information about

  • build
  • agent

• Allows us to use Environmental Variables

• Allows to create user defined variables

• Allows to organize variables in a variable group

• Add publish in the devops ci pipeline which runs once every 2 hours
• Also add configuration = Release for CI and configuration = Debug for PR pipeline

• Predefined variables
• user defined variables

• A Template is a reusable pipeline component
• Watch classroom Recording

• In Azure DevOps Marketplace we can find community task
• We can create our own custom tasks