Jenkins Publish Over SSH Plugin - chaitanyavangalapudi/devops-scripts GitHub Wiki

Publish Over SSH is one of the useful plugins to execute commands on Remote Linux Systems. Otherwise we have to execute series of ssh commands using sh like below.

stage (‘Deploy’) {
  script {
    sh ‘ssh [email protected] rm -rf /var/www/temp_deploy/dist/’
    sh ‘ssh [email protected] mkdir -p /var/www/temp_deploy’
    sh ‘scp -r dist [email protected]:/var/www/temp_deploy/dist/’
    sh ‘ssh [email protected] “rm -rf /var/www/example.com/dist/ && mv /var/www/temp_deploy/dist/ /var/www/example.com/”’
  }
}

In this article let us deploy Spring Boot application jar on remote CentOS Linux system using jenkins declarative pipeline.

Task: Assume that Remote System IP where we need to deploy our application is 192.34.22.9 and remote user is "deploymentuser" and we need to run sftp/scp/ssh from Jenkins box to Deployment box without the need to enter password.

Publish Over SSH plugin runs something similar to this. privatekey-rsa is supplied in Text Area in Jenkins UI or absolute path to this file on Jenkins box is supplied.

sftp -i /home/jenkins/.ssh/privatekey-rsa [email protected]

Step 1- Install Plugins - install Publish over SSH plugin & SSH Credentials plugin

To add ssh credentials on Jenkins server, we need to have ‘SSH Credentials’ plugin and **Credentials Binding Plugin ** installed on Jenkins server. Log in to Jenkins and navigate to System > Manage Jenkins > Manage Plugins

After this, Select available plug-in tab > type "Publish over SSH" in the search bar and select it for installation. Check to restart Jenkins when installation is complete, login to Jenkins after successful installation.

Step 2- Generate the Private and Public key pair

In order to set up communication between Jenkins server and Remote Linux server, we need to generate Private Key (Placed at Jenkins server) and Public key (Placed on Remote Deployment Linux server) pair.

Step 2.1 - Create Deployment user on Remote Host

SSH to your remote Linux Server 192.34.22.9 as root by ssh [email protected] Create user deploymentuser by running

useradd -m -s /bin/bash deploymentuser

Change password for deploymentuser by running

passwd deploymentuser

Step 2.2 - Login as deploymentuser and Generate keys

Once deploymentuser is created successfully. Generate keys using the following command Login as **deploymentuser ** by using su - deploymentuser

Let's generate key pair using ssh-keygen command.

ssh-keygen -f ~/.ssh/deploy-privatekey-rsa -o -t rsa -b 4096 -C "Deployment sshkey"

Type a strong passphrase and retype, Remember the passphrase for later use. Let us assume passphrase is "passphrase2019"

This will generate two files in ~/.ssh directory : deploy-privatekey-rsa & deploy-privatekey-rsa.pub

deploymentuser@remote:~$ ls -lah .ssh/ -rw------- 1 deploymentuser deploymentuser 1.7K Oct 19 11:56 deploy-privatekey_rsa -rw-r--r-- 1 deploymentuser deploymentuser 400 Oct 19 11:56 deploy-privatekey_rsa.pub

Copy the public key to authorized_keys by running

cat ~/.ssh/test-privatekey-rsa.pub >> ~/.ssh/authorized_keys

Run below commands

chmod go-w ~/
chmod 700 ~/.ssh
chmod 600 ~/.ssh/authorized_keys

By these commands we:

deny all, except the owner, to write in home directory;
reading, login and writes are available only for the owner with .ssh settings;
only the owner can read and save changes to file.ssh / authorized _ keys.

Step 2.3 - copy generated private key to Jenkins box by running

scp ~/.ssh/deploy-privatekey-rsa [email protected]:/home/jenkins/sshkeys/deploy-privatekey-rsa

For the first time we’re going to login by ssh directly from CI-server’s console. SSH always asks, whether we trust the key or not. If the answer is no, connection will be closed. If yes — key will be saved in file ~/.ssh/known_hosts.

After this setup, we will be able to run commands on remote deployment server without the need to type password. If you have entered passphrase, it will ask for passphrase.

To check it from the console, run the command:

ssh [email protected] ls /var/www/

Enter the passphrase: passphrase2019

Step 3 - Setup Private Key and other SSH settings on Jenkins Server UI

Now, Login to Jenkins server and follow the below steps to complete setup private key configuration.

Manage Jenkins > Configure System

Now go to option Publish Over SSH - and paste Private key here. You can choose to enter the absolute path to the private key on the Jenkins server. Select "Use password authentication, or use a different key" , Enter Server Name, Username, passphrase, Path to private key on Jenkins host.

You have successfully setup SSH/SFTP connection to remote Linux host from Jenkins Server. Now, you can execute commands using Publish over SSH plugin.

Step 4 - Execute SSH commands using Declarative pipeline

Use below snippet in your jenkins declarative pipeline to run commands remotely.

Note: You have to put the code from the snippet generator into script {} brackets because there is no declarative syntax for this plugin.

stage('SSH transfer') {
 script {
  sshPublisher(
   continueOnError: false, failOnError: true,
   publishers: [
    sshPublisherDesc(
     configName: "${env.SSH_CONFIG_NAME}",
     verbose: true,
     transfers: [
      sshTransfer(
       sourceFiles: "${path_to_file}/${file_name}, ${path_to_file}/${file_name}",
       removePrefix: "${path_to_file}",
       remoteDirectory: "${remote_dir_path}",
       execCommand: "ls -ltr"
      )
     ])
   ])
 }
}

By default, the plugin makes the build "UNSTABLE" if there are any errors. You can change this behavior by adding FailOnError: true, which tells it to fail the job if there are any issues during execution of commands (exit code non zero).

If you want to have more transfers or commands to run, repeat the sshTransfer block like:

transfers:[
  sshTransfer(
    execCommand: "ls -ltr"
   ),
  sshTransfer(
    sourceFiles:"${path_to_file}/${file_name}, ${path_to_file}/${file_name}",
    removePrefix: "${path_to_file}",
    remoteDirectory: "${remote_dir_path}",
    execCommand: "mkdir -p temp"
  )
])

Step 5 - Optional - Grant sudo access to deploymentuser user :

If there is a need to execute some commands as sudo user, the user can be granted sudo access by following procedure below.

# vi /etc/sudoers
and add this line: **deploymentuser - ALL=(ALL) NOPASSWD:ALL**

# User privilege specification
root    ALL=(ALL:ALL) ALL
deploymentuser     ALL=(ALL) NOPASSWD:ALL

Some Key Notes on Executing commands via Publish over SSH plugin:

The "Publish Over SSH plugin" opens SFTP connection to remote host using exec channel. In general, connection to remote host can be opened using either exec channel or shell channel.

With shell channel the shell (on Unix/Linux it's sh or bash or ksh) is started and console is created (the same you see on the screen if you run them manually). You have a prompt which you can parse or use for detection of completion of command.
With command/exec channel a shell instance is started for each command (actually the channel is opened for each command) and a command is passed as a parameter for the shell. It's easier to use command channel because you don't need to deal with command prompt. We can use compound commands if required in command channel. You can separate commands by ; or && or ||. So, to execute multiple commands, you could pass them to the exec channel by separating them with ; or ||
The command/exec channel does not allocate a pseudo terminal (PTY) for the session. As a consequence a different set of startup scripts is (might be) sourced (particularly for non-interactive sessions, .bash_profile is not sourced). And/or different branches in the scripts are taken, based on absence/presence of the TERM environment variable. So the environment might differ from the interactive session, you use with your SSH client. So, in your case, the PATH is probably set differently; and consequently the some commands like fuser executable cannot be found.

Ways to fix this, in preference order:

Fix the command not to rely on a specific environment. Use a full path to air in the command.
Fix your startup scripts to set the PATH the same for both interactive and non-interactive sessions.
You can use an ~/.ssh/environment file to set your HOME and PATH variables.

Since the plugin connects to remote host using SFTP, some of the commands can't be run directly other than those supported by sftp directly when you type help. In order to run the commands in shell mode via SFTP, you need to add "!" before the command. This will execute the commands in our local shell from where we are doing sftp, not on the remote shell.

Try below commands manually to check this like below:

!command                           Execute 'command' in local shell
!                                  Escape to local shell

[root@localhost ~]# sftp test@remotehost
test@remotehost's password:
Connected to remotehost.
sftp> fuser
Invalid command.
sftp> /usr/sbin/fuser
Invalid command.
sftp> !fuser  -k 9999/tcp
Shell exited with status 1
sftp> !fuser  -k 9999/tcp || true
sftp> quit

fuser command is used to kill a process running on a port. It returns error (exit code 1) if there is no process found. The shell returns exit code 127 if the command is not found because of some reasons explained above. To display the exit status of previously run command you can run:

$ echo $?

Output:

127

If you want the command to return success (exit code 0) irrespective of the situation (if you use failOnError: true setting at sshPublisher level), you can use compound statement like below:

 sshTransfer(
   execCommand: "!/usr/sbin/fuser -k 9999/tcp || true"
 )

Putting it all together

stage('Deploy') {
            steps {
               script {
                 sshPublisher(
                  continueOnError: false, failOnError: true,
                  publishers: [
                   sshPublisherDesc(
                    configName: "remotehost",
                    verbose: true,
                    transfers: [
                     sshTransfer(
                      sourceFiles: "target/spring-app.jar",
                      removePrefix: "target",
                      remoteDirectory: "/app/springbootsample",
                      execCommand: "ls -ltr /app/springbootsample"
                     ),
                     sshTransfer(
                      execCommand: "!/usr/sbin/fuser -k 7000/tcp || true"
                     ),
                     sshTransfer(
                      execCommand: "!nohup java -jar spring-app.jar --server.port=7000 &"
                     )
                    ])
                  ])
            }
        }
      }

All the commands executed here starting with "!" are executed on the localhost shell. Since Publish over SSH plugin runs the commands in Exec mode via SFTP, you can't run all commands. For more advanced configuration and execute other commands, use SSH Steps plugin, which we will discuss in another thread.