CJE Notes - saviovettoor/DevOps-wiki GitHub Wiki

Certified Jenkins Engineer (CJE)
================================
1. Key CI/CD/Jenkins concepts: 
	Continuous Delivery/Continuous Integration Concepts
		Define continuous integration, continuous delivery, continuous deployment
			Countinues Delivery - means that code can be released/deployed at any time.
			Countinues Deployment - means code is released continuously as a  part of an automated pipeline to production.
			Continuous Integration – practice of integrating the code continuously/every checkin.
		Difference between CI and CD
		
		Stages of CI and CD
			Build(Checkout,Compile,Unit Test,Package), Deploy, and Test
		Continuous delivery versus continuous deployment
	Jobs
		What are jobs in Jenkins?
			A Job is any runnable task that is controlled by Jenkins.
			The Jenkins official documentation indicates that the term Job has been replaced by the term Project.
		Types of jobs - Freestyle, Maven, Pipeline, External Job, Multi Configuratio, Multi Job, Folder, MultiBranch Pipeline
			Freestyle Project- The build step for this type of project normally executes a shell (Linux) or batch (Windows) command. 
			Pipeline- This type of project used to be called a workflow. These projects are normally written in Jenkins Domain Specific Language (DSL).
			Multi-Configuration Project- This is for projects that will be tested on multiple environments, and require different configurations, depending on those environments.(like android project, ios, windows) by using diff build agents.
			Git-Hub Organization- This type of project can use the source control platform's organization and allow Jenkins to act on Jenkinsfiles stored within the organization’s repositories.
			Folder- This provides a method to group projects together. This is not technically a project. It acts as a type of directory structure for the projects, and the folder name becomes part of the path of the projects.
			Multi-Branch Pipeline- In this type of project, Jenkins uses a Jenkinsfile to mark repositories. If a branch is created in that repository, Jenkins will make a new project in Jenkins for that branch.
		Scope of jobs
			This includes all of the items that are part of that particular Job / Project. In some cases, there are Global libraries that are brought into the scope of a project simply by being included. Other items that are declared within a project only exist in that project’s scope, and are not available as a shared resource.
	Builds
		What are builds in Jenkins?
			Result of a Job/Project(Can be scoped to any typical IT task, defined in Jenkins).
		What are build steps, triggers, artifacts, and repositories?
			Build steps- a single operation within a project.
			triggers- something that starts a build (periodically{cron}, Poll SCM, Build after other projects are built,GitHub hook trigger , etc)
			artifacts- Output of a passed build(jar,zip,tar,...)
			repositories- SCM(Source Code Management)/VC(version Control): git,svn
		Build tools configuration
			Manage Jenkins -> Global Tool Configuration : here you can set build tool(mvn/ant/sbt/jdk) configuration.	
	Source
		What are source code management systems and how are they used?
			SCM(Source Code Management)/VC(version Control): git,svn
			which are used to track your code changes.
		Cloud-based SCMs
			Github
			GitLab
			Bitbucket
			AWS Code Commit
		Jenkins changelogs
			Contains data about a change in a SCM.
		Incremental updates v Clean Check Out
			Incremental updates: Instead of deleting and re-cloning a project repository into a workspace,it just pulling any changes alone.
			Clean Check Out: Clean the jobs workspace and re-cloning a project repository. 
		Checking in code
		Infrastructure-as-Code
			infrastructure from manual tasks to code,
		Branch and Merge Strategies
			master- latest Stable code.
			Development- enables isolation and concurrent development.
			Feature- Feature isolation is a special derivation of the development isolation
			Release
	Testing
		Benefits of testing with Jenkins
			Testing every build result findout the defects/issues ASAP.
		Define unit test, smoke test, acceptance test, automated verification/functional tests
			unit test- It tests a small piece of functionality, usually at the class, method level.
			smoke test-  aim at ensuring that the most important functions work/Main functionality.
			acceptance test- It tests against the product specifications.This is the handoff test of the software to the client. This is normally done by the client to ensure that the software meets their expectations.
			functional/verification tests- It tests the functionality of the project as a whole(does this feauture is working?).
	Notifications
		Types of notifications in Jenkins
			email
			slack
			hipchat
		Importance of notifications
			Result in fixing/identify the broken code faster.
	Distributed Builds
		What are distributed builds?
			Executing build on slave nodes/agents.
			When you want to preserve resources on the master node you use distributed builds.
			Master / Agent communication is via SSH ( preferred ) or JNLP ( TCP or HTTP ).
		Functions of masters and agents
			Master where you main jenkins server is running.
			Slave machine are used to execute jenkins jobs.
			Agents should be ”fungible” ( replaceable ). This means that local configuration on the agent should be kept to a minimum and global configuration on the master should be preferred.
	Plugins
		What are plugins?
			Plugins can really be used to add any functionality to Jenkins.
		What is the plugin manager?
			plugin manager- help to manage jenkins plugin{install, uninstall, upgrade, downgrad or search plugins}
			Dashboard Left Panel -> Manage Jenkins -> Manage Plugins
		How to install jenkins pluings?
			From the web UI - Manage Jenkins > Manage Plugins
			Using the Jenkins CLI - java -jar jenkins-cli.jar -s http://localhost:8080/ install-plugin SOURCE ... [-deploy] [-name VAL] [-restart]
									Installs a plugin either from a file, an URL, or from update center.
									 SOURCE    : If this points to a local file, that file will be installed. If this is an URL, Jenkins downloads the URL and installs that as a plugin.Otherwise the name is assumed to be the short name of the plugin in the existing update center (like "findbugs"),and the plugin will be installed from the update center.
									-deploy   : Deploy plugins right away without postponing them until the reboot.
									-name VAL : If specified, the plugin will be installed as this short name
									(whereas normally the name is inferred from the source name automatically).
			Console by uploading .hpi file
				Manage Jenkins > Manage Plugins page in the web UI.
				Click on the Advanced tab.
				Choose the .hpi file under the Upload Plugin section.
				Upload the plugin file.
		Jenkins Default pugins?
			
	Jenkins Rest API
		How to interact with it
			Using:XML , JSON with JSONP support, Python
		What can you do with it?
			Retrieve information from Jenkins for programmatic consumption.
			Trigger a new build
			Create/copy jobs
		Why use it?
			you can get info programmatic way.
	Security
		Authentication versus authorization
			Authentication identifies a user.
			Authorization dictates what a user is allowed to do.
		Matrix security
			Jenkins permissions are defined in a matrix-like structure, in which the administrator can set security based on a user or user group.
			Major categories: overall, Credentials, Agent, Job, Run, View and SCM
		Definition of auditing, credentials, and other key security concepts
			auditing: Tracking who did what on your Jenkins server.Or verifying that the access permissions are working as intended.
			credentials: to authenticate user.
		NOTE: Jenkins is an explicit allow model and that there is no deny; if something is not explicitly allowed then it is denied.
		Also, Jenkins permissions are additive and global -> parent -> job is how those allows are stacked. If something is allowed at a level above, and inheritance is enabled, then it is allowed in the levels below.
	Fingerprints
		What are fingerprints?
			Fingerprints are an md5sum that is used to track artifacts.
		How do fingerprints work?
			Jenkins maintains a database of md5sum, and for each md5sum, Jenkins records which builds of which projects used. This database is updated every time a build runs and files are fingerprinted.
	Artifacts
		How to use artifacts in Jenkins
			ArchiveArtifacts can be used to set an archiving strategy for a project in a Jenkinsfile.
		Storing artifacts
			you can archive it and store in nexus.
	Using 3rd party tools
		How to use 3rd party tools
			you can install the repective plugin and configure it by Manage Jenkins -> Global Tool Configuration
		Installation Wizard
			What is the Jenkins Installation Wizard?
				You can install via repos/docker or running jenkins jar using java
		How to use the Wizard?
			
		Which configurations are covered by the Installation Wizard?
			Create an Admin user with username and password
			Add common plugins
			Add specific plugins
2.Jenkins usage
	Jobs/Projects
		Organizing jobs in Jenkins
			Projects can be organized in either Folders or Views.
		Parameterized jobs
			To pass values while running a project.
			using Parameterized Trigger Plugin you can trigger another job aswell.
		Usage of Freestyle/Pipeline/Matrix jobs
			Freestyle- General-purpose build jobs, which provides a maximum of flexibility.
			Pipeline- Building Continuous Delivery pipelines through either a Web UI or a scripted(in Groovy) Jenkinsfile.
			Matrix/Multi Job
				When to use it
					If you'd like to stop the mess with downstream / upstream jobs chains definitions
					When you want to add full hierarchy of Jenkins jobs that will be executed in sequence or in parallel
					Add context to your buildflow implementing parameter inheritance from the MultiJob to all its Phases and Jobs. Phases are sequential, whilst jobs inside each Phase are parallel.
	Builds
		Setting up build steps and triggers
			Steps- Execute Shell, Invoke Ant, Build using SBT, Start/Stop Docker,Execute SonarScanner, etc.
			Triggers- periodically{cron}, Poll SCM, Build after other projects are built,GitHub hook trigger , etc
		Configuring build tools.
			Manage Jenkins -> Global Tool Configuration
		Running scripts as part of build steps
			Can run Shell/Batch scripts or Groovy script
			Groovy scripts can run as system or user level. System has access to Jenkins object model
	Source Code Management
		Polling source code management
			using poll scm
		Creating hooks
			- Enable "Poll SCM" option and  leave "Schedule" empty
			- In Github, go to your repo Settings -> Integrations & services->  Add a service -> add service "Jenkins (Git plugin)"
				Jenkins URL: https://(jenkins_master):8080
			-Now when you do a git push, it will automatically kick off the build in jenkins
			*Another way of doing it, using the trigger "GitHub hook trigger for GITScm polling" and in Github, add service called "Jenkins (GitHub plugin)"
			Add Jenkins hook url: http://(jenkins_master):8080/github-webhook/
			Test service button in Github services section is useful
		Including version control tags and version information
			Git allows you to create a tag for every build eg: git tag -a v1.4 -m "my version 1.4"
			Version Number plugin lets you include info in build name
	Testing
		Testing for code coverage
			code coverage test? It tests how well your code is tested.
			Cobertura and JaCoCo are most common coverage tools used for JAVA and SBT projects.
		Test reports in Jenkins
			Publish JUnit or TestNG reports
			In a JUnit report, if we set the “Health report amplification factor” to 3 and there’s a 10% failure, then health Score is 70%.
				{You would take 100% then subtract (10%*3), thus 70%.}
			How do you configure a junit report in a Pipeline? 
				junit “path to xml file”
		Displaying test results
		Integrating with test automation tools
		Breaking builds
			Based on unit test result we will be able to mark build as success or failure
	Notifications
		Setup and usage
			when can trigger notifications after post build action.
		Email notifications, instant messaging
			Email ext :lets you customize the message and tailor the recipients per trigger
			Sample pipeline:
				post {
					failure {
						emailext(
							subject: "${env.JOB_NAME [${env.BUILD_NUMBER}] Failed!",
							body: "<html> build Failed</html>",
							to: "[email protected]" )
						}
					}
				}
			Jabber and IRC for instant messaging
		Alarming on notifications
			Extreme notifications can have a video or audio cue in the real world
	Distributed Builds
		Setting up and running builds in parallel
			-Use Matrix/Multi Job Configuration 
			-Build Flow Plugin
			-Pipeline
				node {
					  git "[email protected]..."
					  parallel (
					   phase1: { sh "echo p1; sleep 20s; echo phase1" },
					   phase2: { sh "echo p2; sleep 40s; echo phase2" }
				      )
				     sh "run this after both phases complete"   
					}
		Setting up and using SSH agents, JNLP agents, cloud agents
			 Manage Jenkins->  Manage Nodes-> New Node ->Node Name 
				Then you can configure Launch Method SSH agents/ JNLP agents
		Monitoring nodes
			-Install Groovy plugin
			 Setup a freestyle project to Execute System Groovy Script
				int exitcode = 0
					for (slave in hudson.model.Hudson.instance.slaves) {
					 if (slave.getComputer().isOffline().toString() == "true"){
					 println('Slave ' + slave.name + " is offline!"); 
					 exitcode++;
					 }
					}
					if (exitcode > 0){
					 println("We have a Slave down, failing the build!");
					 return 1;
					}
			- You can use "Monitor and Restart Offline Slaves" plugin: to monitor and restart offline nodes if they are not disconnected manually.
	Plugins
		Setting up and using Plugin Manager
			Dashboard Left Panel -> Manage Jenkins -> Manage Plugins
			you can install plugin via CLI and the Plugin Manager
		Finding and configuring required plugins
			Updates tab – for upgrading plugin already installed
			Available tab – for downloading available plugin which are not installed.
			Advanced tab – for uploading plugin hpi/jpi file from local.
	CI/CD
		Using Pipeline (formerly known as “Workflow”)
		Integrating automated deployment
		Release management process
		Pipeline stage behavior
	Jenkins Rest API
		Using REST API to trigger jobs remotely, access job status, create/delete jobs
		Trigger a build: curl -X POST http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/build --user <USER_NAME>:<TOKEN> or
						 curl -I -X POST http://user_name:<TOKEN>@192.168.1.49:8080/job/MyJob/build -H "$CRUMB"
		Trigger a parameterized build:curl jenkinsurl:8080/jobs/jobname/buildnumber. –user <USER_NAME>:<TOKEN> –data-urlencode json=’{”paramater”: [{“name”:”id”},{ “somevalue”:”abcd”}]}’
		Retrieve a Job config.xml file: curl http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/config.xml --user <USER_NAME>:<TOKEN>
			 curl -X GET http://darealmc:[email protected]:8080/job/realmcjobs/job/pipeone/config.xml -H "$CRUMB" -o ./jobconfig.xml
		Console Output: curl http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/<build_number>/consoleText --user <USER_NAME>:<TOKEN> 
		Disable a Job: curl -X POST  http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/disable --user <USER_NAME>:<TOKEN> 
		Enable a Job: curl -X POST  http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/enable --user <USER_NAME>:<TOKEN> 
		Delete a Job : curl -X POST  http://10.150.18.229:8080/jenkins/view/DevOps/job/MyJob/doDelete --user <USER_NAME>:<TOKEN>  
		Create a new Job:curl -s -XPOST 'http://10.150.18.229:8080/createItem?name=yourJobName' --user <USER_NAME>:<TOKEN> --data-binary @config.xml -H "Content-Type:text/xml"
		Check if job exists:curl -X GET 'http://jenkins/checkJobName?value=yourJobFolderName' --user <USER_NAME>:<TOKEN>
		Safe Restart: curl -X POST  http://10.150.18.229:8080/SafeRestart --user <USER_NAME>:<TOKEN> 
		*TOKEN - Manage Jenkins ->Manage Users -> click on the users configure button->API Token
		* CRUMB=$(wget -q --auth-no-challenge --user username --password Password --output-document - 'http://192.168.1.49:8080/crumbIssuer/api/xml?xpath=concat(//crumbRequestField,":",//crumb)')
		if CSR protection is enabled you should use Crumb for API Calls
	Security
		Setting up and using security realms
			Manage Jenkins -> Configure Global Security
			- Jenkins own user database
			- LDAP
			- Active Directory
		User database, project security, Matrix security
			People link shows user list + committers
			Matrix based security – control privileges granularly using user ids/groups
			Project based matrix authorization security – Matrix based + set privileges on job configuration page as well
			Role based matrix authorization security – Manage Roles to control permissions by group. Adds groups/roles tabs to projects
		Setting up and using auditing
			Logs- Manage Jenkins -> System Log -> All Jenkins Logs
			Job Configuration History plugin – for job config history.
			Audit Trail plugin –This plugin adds an Audit Trail section in the main Jenkins configuration page.
		Setting up and using credentials
			Credentials plugin - provides a centralized way to define credentials that can be used by your Jenkins instance, plugins and build jobs.
			Credentials Binding plugin - allows you to configure your build jobs to inject credentials as environment variables.
			Global credentials-Credentials that should be available irrespective of domain specification to requirements matching.
			Credentials -> System ->Add domain
	Fingerprints
		Fingerprinting jobs shared or copied between jobs
		Fingerprint -plugin to Adds the ability to fingerprint files as build steps.
			$JENKINS_HOME/fingerprints -> where md5sum values are stored.
	Artifacts
		Copying artifacts
			Copy Artifact: plugin Adds a build step to copy artifacts from another project.
			job -> configure -> Build : Copy artifacts from another project
		Using artifacts in Jenkins
			job -> configure -> Post Build Action: archive artifacts
		Artifact retention policy
			By default, kept same length of time as build is there.
			job -> configure -> [x]Discard old builds -> Advanced: set a value for how many artifacts "Max # of builds to keep with artifacts"	
	Alerts
		Making basic updates to jobs and build scripts
		Troubleshooting specific problems from build and test failure alerts
			We can use plugin "email-ext" for alerts
3. Building Continuous Delivery (CD) Pipelines
	Pipeline Concepts
		Value stream mapping for CD pipelines
		Why create a pipeline?
			Code: Pipelines are implemented in code and typically checked into source control, giving teams the ability to edit, review, and iterate upon their delivery pipeline.
			Durable: Pipelines can survive both planned and unplanned restarts of the Jenkins master.
			Pausable: Pipelines can optionally stop and wait for human input or approval before continuing the Pipeline run.
			Versatile: Pipelines support complex real-world CD requirements, including the ability to fork/join, loop, and perform work in parallel.
			Extensible: The Pipeline plugin supports custom extensions to its DSL [1] and multiple options for integration with other plugins.
		Gates within a CD pipeline
			Provide a point for approval before continuing deployment within stages.
					
		How to protect centralized pipelines when multiple groups use same tools
			You can use approval in jenkins pipeline
		Definition of binary reuse, automated deployment, multiple environments
			Binary reuse- shared global library: {Configure System -> Global Pipeline Libraries}
			automated deployment- in postbuild action you can triiger another project or [x] Build after other projects are built
			multiple environments-
		Elements of your ideal CI/CD pipeline - tools
			Agent
			Source control repository
			Binary repository
			Automated testing
			Deployment
		Key concepts in building scripts (including security/password, environment information, etc.)
			Credentials plugin for password
			Keep environment information in source control
			Different script for each stage in the pipeline
	Upstream and downstream
		Triggering jobs from other jobs - in postbuild action you can triiger another project or [x] Build after other projects are built
		Setting up the Parameterized Trigger plugin - In General section of the job "[x]This project is parameterized"
		Upstream/downstreamjobs- The Upstream job triggers the downstream job, and vice versa.
	Triggering
		Triggering Jenkins on code changes:
			via hooks
			- Enable "Poll SCM" option and  leave "Schedule" empty
			- In Github, go to your repo Settings -> Integrations & services->  Add a service -> add service "Jenkins (Git plugin)"
				Jenkins URL: https://(jenkins_master):8080
			-Now when you do a git push, it will automatically kick off the build in jenkins
			*Another way of doing it, using the trigger "GitHub hook trigger for GITScm polling" and in Github, add service called "Jenkins (GitHub plugin)"
			Add Jenkins hook url: http://(jenkins_master):8080/github-webhook/
			Test service button in Github services section is useful
		Difference between push and pull
			When the source informs the build system of a code change, that’s pushing. -SCM polling trigger job.
			When the build system asks if there are changes to the source code, that’s pulling. - when hook configure for repo trigger a job.
		When to use push vs. pull
			When you don’t have control over SCM polling is used.
			You want trigger when there is change in SCM.
	Pipeline (formerly known as “Workflow”)
		Benefits of Pipeline vs linked jobs
			CI/CD as code.
			Changes are tracked.
			Pausable – can get manual approval
		Functionalities offered by Pipeline
			Build steps, pauses, parallelization, deploy, stash/unstash, etc
			Able to run diff stages in diff env in the same job. 
		How to use Pipeline
		Pipeline stage view
			The Pipeline Stage View plugin includes an extended visualization of Pipeline build history on the index page of a flow project, under Stage View.
	Folders
		- you organize Jenkins Projects/Jobs using folder and views.
		- Nested items determine folder health if you select "child item with worst health", job with worst health will determine folder health.	
		How to control access to items in Jenkins with folders
			Role Based Access Control can control folder
			Can control level as current/child/grandchild
		Referencing jobs in folders
			http://localhost:8080/job/myfolder/job/JobOne/
	Parameters
		Setting up test automation in Jenkins against an uploaded executable
			Using "File parameter" in Parameterized option you will be able to upload file.  
		Passing parameters between jobs
			"Trigger Parameterized  build on other project" in the post build action help you on this.
		Identifying parameters and how to use them: file parameter, string parameter
			file parameter: Parameterized option you will be able to upload file
			string parameter: Help you to pass string parameter to the build.eg: branch name
		Jenkins CLI parameters
			RUN: java -jar /var/lib/jenkins/jenkins-cli.jar -s http://localhost:8080/
			To view option: http://localhost:8080/cli/
			Run command with user name and password: java -jar jenkins-cli.jar -s http://localhost:8080/ safe-restart  --username admin --password admin
			jenkins-cli who-am-i -> Reports your credential and permissions.
			jenkins-cli build "Freestyles/My Freestyle Project"
			jenkins-cli version
			jenkins-cli shutdown
			jenkins-cli safe-shutdown
			jenkins-cli install-plugin thinBackup -restart
			jenkins-cli console "Freestyle/My Freestyle Project" 51
	Promotions
		Promotion of a job
			stage('Promote Development to Master') {
				when {
					branch 'development'
				}
		Why promote jobs?
				plugin allows you to distinguish good builds from bad builds by introducing the notion of 'promotion'.
		How to use the Promoted Builds plugin
			Using "when" condition in the stage
	Notifications
		How to radiate information on CD pipelines to teams
			via Email notifications or radiator plugin
	Pipeline Multibranch and Repository Scanning
		Usage of Multibranch jobs
			Suppose in dev branch your team is pushing code always and when the build is success you want to push/promote code to master we can use Multibranch pipeline
		Scanning GitHub and BitBucket Organization
		ScanningbasicSCMrepositories
			pipeline {
			 triggers {
				cron(* * * * *')
				pollSCM('* * * * *')
			}
	Pipeline Global Libraries
		How to share code across Pipelines
			-library that has methods accessible amongst different projects
			-extend the functionality of your pipeline
			-Groovy/pipeline syntax
			-Put a file "sayHello.groovy" inside jenkins-global-library/vars
				def call(String name = 'you') {
					echo "Hello, ${name}"
				  }
		Usages of the Shared Libraries
		Interaction with Folders and Repository scanning
		Security and Groovy sandbox
4. CD-as-code best practices
	Distributed builds architecture
		Running jobs on slave node
		Vertical growth – master is responsible for more jobs
		Horizontal growth – creation of more masters
	Fungible (replaceable) agents
		To include the tools' installation as part of the build process is the best practice.
		Best practice is to make slaves interchangeable, but can tie jobs to slaves
	Master-agent connectors and protocol
		SSH connector – preferred option. Slaves need SSHD server and public/private key
		JNLP/TCP connector – Java Network Launch Protocol start web agent on slave through JWS (Java Web Start). Can start via browser or OS service	JNLP/HTTP connector – like JNLP/TCP except headless and over HTTP
		Custom script – launch via command line.
	Tool installations on agents
		We can install manually on agent/ Jenkins can take care.
	Cloud agents
		CodeBuilder: AWS CodeBuild Cloud Agents Plugin
		Amazon EC2 Plugin: Allow Jenkins to start slaves on EC2 or Eucalyptus on demand, and kill them as they get unused.
		The JCloud plugin creates the possibility of executing the jobs on any cloud provider supported by JCloud libraries
	Traceability
		Docker Traceability plugin  allows the tracking of the creation of, and use of Docker containers in Jenkins.
		Artifacts and packages, Jenkins keep track of those using the fingerprint
	High availability
		High Availability plugin eliminates downtime due to master failures.Multiple Jenkins masters act as backups waiting for a primary master failure. Once a failure is detected, a backup master automatically boots up and acts as a failover.
		$JENKINS_HOME on a shared file system
		

-------------
Additional Notes
-------------
stage - the stage step remains but is now focused on grouping steps and providing boundaries for Pipeline segments.
lock - the lock step throttles the number of concurrent builds in a defined section of the Pipeline.
		lock can be also used to wrap multiple stages into a single concurrency unit:
milestone - the milestone step automatically discards builds that will finish out of order and become stale.

Security Realm is a dedicated database for user and passwords
4 kinds of Realms supported out of the box, covered below:
	Jenkins User Database
	Unix user/group Database
	Servlet Container
	External LDAP
DB is stored in XML files located in ${JENKINS_HOME}/users

some URIs must not require authentication
	Command line endpoint: /cli
	SCMs endpoint (hooks): /git, /subversion…​
	Jars download endpoints: /jnlpJars
	Current User Page: /whoAmI
PIPELINE BENEFITS
	The pipeline functionality is:
	Durable: The Jenkins master can restart and the Pipeline continues to run
	Pausable: can stop and wait for human input or approval
	Versatile: supports complex real-world CD requirements (fork, join, loop, parallelize)
	Extensible: supports custom extensions to its "DSL" (Domain-specific Language)
Executer can run on any Master or Agent

The JUnit step is coded in the Pipeline as:

steps {
   sh  './jenkins/test-all.sh'
   junit '**/surefire-reports/**/*.xml'
}

Scripted pipeline Uses node where Declarative uses agent

Run all Pipeline stages on a container based on the image centos:7:
	pipeline { agent { docker 'centos:7' } }
When you enable the master/agent mode, Jenkins automatically configures all your existing projects to stick to the master node.

Post-initialization script
	You can create a Groovy script file $JENKINS_HOME/init.groovy, or any .groovy file in the directory $JENKINS_HOME/init.groovy.d/, (See Configuring Jenkins upon start up for more info) to run some additional things right after Jenkins starts up.

Jenkins Features Controlled with System Properties
	Jenkins has several "hidden" features that can be enabled with system properties. System properties are defined by passing -Dproperty=value to the java command line to start Jenkins. Make sure to pass all of these arguments before the -jar argument, otherwise they will be ignored. Example:
		java -Dhudson.footerURL=http://example.org -jar jenkins.war
One question about how do you hide credentials ?
	Mask Passwords Plugin
Injecting Secrets into Jenkins Build Jobs
	Install the Credentials Plugin
	Credentials Binding plugin - allows you to configure your build jobs to inject credentials as environment variables.
finger print of an artifact?
	md5sum:	a093c93dfca2926e7b1483d3dfd8d1bf
	folder location: /var/lib/jenkins/fingerprints/a0/93/c93dfca2926e7b1483d3dfd8d1bf.xml
Health icons
	Sunny - more than 80% of Runs passing
	Partially Sunny - 61% to 80% of Runs passing
	Cloudy - 41% to 60% of Runs passing
	Raining - 21% to 40% of Runs passing
	Storm (Thunder)- less than 21% of Runs passing
Each build node connection will take 2-3 threads, which equals about 2 MB or more of memory. You will also need to factor in CPU overhead for Jenkins if there are a lot of users who will be accessing the Jenkins user interface.

Anatomy of a $JENKINS_HOME
---------------------------	
	JENKINS_HOME
	 +- config.xml     (Jenkins root configuration file)
	 +- *.xml          (other site-wide configuration files)
	 +- identity.key   (RSA key pair that identifies an instance)
	 +- secret.key     (deprecated key used for some plugins’ secure operations)
	 +- secret.key.not-so-secret  (used for validating _$JENKINS_HOME_ creation date)
	 +- userContent    (files served under your http://server/userContent/)
	 +- secrets        (root directory for the secret+key for credential decryption)
		 +- hudson.util.Secret   (used for encrypting some Jenkins data)
		 +- master.key           (used for encrypting the hudson.util.Secret key)
		 +- InstanceIdentity.KEY (used to identity this instance)
	 +- fingerprints   (stores fingerprint records, if any)
	 +- plugins        (root directory for all Jenkins plugins)
		 +- [PLUGINNAME]   (sub directory for each plugin)
			 +- META-INF       (subdirectory for plugin manifest + pom.xml)
			 +- WEB-INF        (subdirectory for plugin jar(s) and licenses.xml)
		 +- [PLUGINNAME].jpi   (.jpi or .hpi file for the plugin)
	 +- jobs           (root directory for all Jenkins jobs)
		 +- [JOBNAME]      (sub directory for each job)
			 +- config.xml     (job configuration file)
			 +- latest         (symbolic link to the last successful build)
			 +- builds         (stores past build records)
				 +- [BUILD_ID]     (subdirectory for each build)
					 +- build.xml      (build result summary)
					 +- log            (log file)
					 +- changelog.xml  (change log)
		 +- [FOLDERNAME]   (sub directory for each folder)
			 +- config.xml     (folder configuration file)
			 +- jobs           (sub directory for all nested jobs)
	 +- workspace      (working directory for the version control system)
		 +- [JOBNAME]      (sub directory for each job)
⚠️ **GitHub.com Fallback** ⚠️