support new values for site.xcatdebugmode - xcat2/xcat-core GitHub Wiki
Currently, the site.xcatdebugmode support values are 0 1 or null. When site.xcatdebugmode=0 or site.xcatdebugmode value is null, then xcatdebugmode is off. When site.xcatdebugmode=1, the xcatdebugmode is on, the installer will halt for sles and ubuntu. After install xCAT on MN, the default action for xcatdebugmode is off. There is a need to add a new value to support logging the operation system installation without halt.
1. support values for xcatdebugmode
xCAT provides a batch of techniques to help user debug problems while using xCAT, especially on OS provision, such as collecting logs of the whole installation process and accessing the installing system via ssh, etc. These techniques will be enabled according to different xCAT debug levels specified by "xcatdebugmode".
planning support values:
xcatdebugmode=0: Only diagnose Log will be show in corresponding files.
xcatdebugmode=1: Diagnose Log will be show in corresponding files and debug port will be opened.
xcatdebugmode=2: SSH is supported while installing also with diagnose log show and debug port enable.List of Supported OS:
RHEL: 6.7 and above
SLES: 12 and above
UBT: 14.04.3 and aboveThe following behavior is observed during OS install:
+-----------------+--------------+--------------+--------------+
|**xcatdebugmode**| 0 | 1 | 2 |
+-----------------+----+----+----+----+----+----+----+----+----+
| |RHEL|SLES|UBT |RHEL|SLES|UBT |RHEL|SLES|UBT |
+=================+====+====+====+====+====+====+====+====+====+
| Diagnose Log | Y | Y | Y | Y | Y | Y | Y | Y | Y |
+-----------------+----+----+----+----+----+----+----+----+----+
|Enable Debug Port| N | N | N | Y | Y | Y | Y | Y | Y |
+-----------------+----+----+----+----+----+----+----+----+----+
| Enable SSH | N | N | N | N | N | N | Y | Y | Y |
+-----------------+----+----+----+----+----+----+----+----+----+Y means the behavior is supported by OS at current xcatdebugmode.
N means the opposite meaning.
2. collect logs during the diskfull installation process
The ability to collect logs during the diskfull installation process can be helpful when debugging installation problems.
- Pre-Install logs: the logs of pre-installation scripts, the pre-installation scripts include β%preβ section in anaconda, β<pre-scripts/>β section for SUSE and βpartman/early_commandβ and βpreseed/early_commandβ sections for ubuntu. The logs include the STDOUT and STDERR of the scripts as well as the debug trace output of bash scripts with βset -xβ
- Installer logs: the logs from the os installer itself, i.e, the logs of installation program(anaconda, autoyast and preseed,etc.)
- Post-Install logs: the logs of post-installation scripts, the post-installation scripts include β%postβ section in anaconda, β<chroot-scripts/>β and β<post-scripts/>β sections for SUSE and βpreseed/late_commandβ section for ubuntu. The logs include the STDOUT and STDERR of the scripts as well as the debug trace output of bash scripts with βset -xβ
- PostBootScript logs: the logs during the post boot scripts execution, which are specified in βpostbootscriptsβ attribute of node and osimage definition and run during the 1st reboot after installation.
The following behavior is observed during OS install:
+------------------+--------------+--------------+--------------+
|**xcatdebugmode** | 0 | 1 | 2 |
+------------------+----+----+----+----+----+----+----+----+----+
| |RHEL|SLES|UBT |RHEL|SLES|UBT |RHEL|SLES|UBT |
+=============+====+====+====+====+====+====+====+====+====+====+
| Pre-Install | MN | N | N | N |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | Y1 | Y2 | Y2 |
+-------------+----+----+----+----+----+----+----+----+----+----+
| Installer | MN | N | N | N | Y6 | Y6 | Y6 | Y6 | Y6 | Y6 |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | Y5 | Y5 | Y5 | Y5 | Y5 | Y5 | Y5 | Y5 | Y5 |
+-------------+----+----+----+----+----+----+----+----+----+----+
| Post-Install| MN | Y4 | Y3 | Y3 |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | Y1 | Y2 | Y2 |
+-------------+----+----+----+----+----+----+----+----+----+----+
| Post-Script | MN | Y4 | Y3 | Y3 |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | Y1 | Y2 | Y2 |
+-------------+----+----+----+----+----+----+----+----+----+----+Y1 means the installation logs can be saved to the CN in /var/log/xcat/xcat.log file without export to subsequent commands.
For RHEL
Running Kickstart Pre-Installation script...
...
installstatus installing
[get_install_disk]Information from /proc/partitions:
major minor #blocks name
...
Running Kickstart Post-Installation script...
...
running /xcatpost/mypostscript.post
...
/xcatpost/mypostscript.post return
For SLES
Running AutoYaST Pre-Installation script...
...
[get_install_disk]Information from /proc/partitions:
installstatus installing
major minor #blocks name
...
Running AutoYaST Chroot-Installation script...
ready
done
Running AutoYaST Post-Installation script...
...
running /xcatpost/mypostscript.post
...
/xcatpost/mypostscript.post return
For UBT
Running preseeding early_command Installation script...
...
[get_install_disk]Information from /proc/partitions:
major minor #blocks name
...
Running preseeding late_command Installation script...
...
Generating grub configuration file ...Y2 means the installation logs can be saved to the CN in /var/log/xcat/xcat.log file with export to subsequent commands.
For RHEL
Running Kickstart Pre-Installation script...
...
+ /tmp/baz.py 'installstatus installing'
installstatus installing
+ echo '[get_install_disk]Information from /proc/partitions:'
[get_install_disk]Information from /proc/partitions:
major minor #blocks name
...
Running Kickstart Post-Installation script...
...
+ echo '/opt/xcat/xcatinfo generated'
/opt/xcat/xcatinfo generated
...
running /xcatpost/mypostscript.post
...
+ echo '/opt/xcat/xcatinstallpost generated'
/opt/xcat/xcatinstallpost generated
...
/xcatpost/mypostscript.post return
service xcatpostinit1 disabled
For SLES
Running AutoYaST Pre-Installation script...
...
+ /tmp/bar.awk 'installstatus installing'
installstatus installing
+ echo '[get_install_disk]Information from /proc/partitions:'
[get_install_disk]Information from /proc/partitions:
major minor #blocks name
...
Running AutoYaST Chroot-Installation script...
...
+ /tmp/updateflag.awk
ready
done
...
Running AutoYaST Post-Installation script...
...
running /xcatpost/mypostscript.post
...
+ echo 'finished node installation, reporting status...'
finished node installation, reporting status...
...
/xcatpost/mypostscript.post return
service xcatpostinit1 disabled
For UBT
Running preseeding early_command Installation script...
...
[get_install_disk]Information from /proc/partitions:
major minor #blocks name
...
Running preseeding late_command Installation script...
...
+ echo 'postscripts downloaded successfully'
postscripts downloaded successfully
...
+ echo 'mypostscript returned'
mypostscript returned
...
+ echo 'finished node installation, reporting status...'
finished node installation, reporting status...Y3 means the installation logs can be forwarded to the MN in /var/log/xcat/computes.log file without export to subsequent commands.
/opt/xcat/xcatinfo generated
...
postscripts downloaded successfully
...
/xcatpost/mypostscript.post generated
...
/etc/init.d/xcatpostinit1 generated
...
/opt/xcat/xcatinstallpost generated
...
/opt/xcat/xcatdsklspost generated
...
running mypostscriptY4 means the error messages can be forwarded to the MN in /var/log/xcat/computes.log file only when critical error happens.
/usr/bin/openssl does not exist, halt ...
or
/usr/bin/wget does not exist, halt ...
or
failed to download postscripts from...
or
/xcatpost/getpostscript.awk does not exist, halt ...
or
generate mypostscript file failure, halt ...Y5 means the installer log can be saved to the CN in /var/log/anaconda /var/log/YaST2 /var/log/installer for RHEL SLES and UBT.
Y6 means the installer log can be forwarded to the MN in /var/log/xcat/computes.log file.
N means the logs can not be forwarded or saved.
3. collect logs during the diskless installation process
The ability to collect logs during the diskless installation process can be helpful when debugging installation problems.
- Provision logs: the logs during the diskless provision.
- PostBootScript logs: the logs during the post boot scripts execution, which are specified in βpostbootscriptsβ attribute of node and osimage definition and run during the 1st reboot after installation.
The following behavior is observed during OS install:
+---------------------+--------------+--------------+--------------+
| **xcatdebugmode** | 0 | 1 | 2 |
+---------------------+----+----+----+----+----+----+----+----+----+
| OS Distribution |RHEL|SLES|UBT |RHEL|SLES|UBT |RHEL|SLES|UBT |
+================+====+====+====+====+====+====+====+====+====+====+
| Provision | MN | N | Y3 | Y3 |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | N | N | N |
+----------------+----+----+----+----+----+----+----+----+----+----+
| PostBootScript | MN | Y3 | Y4 | Y4 |
+ logs +----+----+----+----+----+----+----+----+----+----+
| | CN | Y1 | Y2 | Y2 |
+----------------+----+----+----+----+----+----+----+----+----+----+Y1 means the installation logs can be saved to /var/log/xcat/xcat.log file on CN.
downloaded postscripts successfully
node booted successfully,reporting status.../var/log/xcat/xcat.log file on CN.
running /opt/xcat/xcatdsklspost
+
++
node booted successfully,reporting status...Y3(xcatdebugmode=1 or xcatdebugmode=2) means the installation logs can be forwarded to /var/log/xcat/computes.log file on MN for redhat and sles.
running xcatroot....
Extracting root filesystem:
Done....Y3(xcatdebugmode=1 or xcatdebugmode=2) means the installation logs can be forwarded to /var/log/xcat/computes.log file on MN for ubuntu.
running init script...
Extracting root filesystem:
Done...Y3(xcatdebugmode=0 or null) means the installation logs can be forwarded to /var/log/xcat/computes.log file on MN.
node booted successfully,reporting status...
ready
doneY4 means the installation logs and debug trace("set -x" or "-o xtrace") of bash scripts can be forwarded to /var/log/xcat/computes.log file on MN.
running /opt/xcat/xcatdsklspost
+
++
node booted successfully,reporting status...4. Use ssh when site.xcatdebugmode=2
The ssh access to the installer is enabled, the admin can login into the installer through:
- For RHEL, the installation wonβt halt, just login into the installer with ssh root@<node>.
- For SLES, the installation will halt after the ssh server is started, the console output looks like:
*** sshd has been started ***
*** login using 'ssh -X root@<node>' ***
*** run 'yast' to start the installation ***After the installation is finished, the system requires a reboot. The installation will halt again before the system configuration, the console output looks like:
*** Preparing SSH installation for reboot ***
*** NOTE: after reboot, you have to reconnect and call yast.ssh ***For sles11, the console output will looks different but similar:
*** Starting YaST2 ***
*** Preparing SSH installation for reboot ***
*** NOTE: after reboot, you have to reconnect and call ***
*** /usr/lib/YaST2/startup/YaST2.ssh ***Just as the message above suggests, the admin should run ssh -X root@<node> to access the installer and run yast.ssh or /usr/lib/YaST2/startup/YaST2.ssh to finish the installation.
Note: For sles12, during the second stage of an SSH installation YaST freezes. It is blocked by the SuSEFirewall service because the SYSTEMCTL_OPTIONS environment variable is not set properly. Workaround: When logged in for the second time to start the second stage of the SSH installation, call yast.ssh with the --ignore-dependencies as follows:
SYSTEMCTL_OPTIONS=--ignore-dependencies yast.ssh
- For UBT, the installation will halt on the following similar message in the console:
βββββββββββββ€ [!!] Continue installation remotely using SSH βββββββββββββ
β β
β Start SSH β
β To continue the installation, please use an SSH client to connect to β
β the IP address <node> and log in as the "installer" user. For β
β example: β
β β
β ssh installer@<node> β
β β
β The fingerprint of this SSH server's host key is: β
β <SSH_host_key> β
β β
β Please check this carefully against the fingerprint reported by your β
β SSH client. β
β β
β <Continue> β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββJust as the message show, the admin can run ssh installer@<node> with the password βclusterβ to login into the installer, the following message shows on login:
ββββββββββββββββββββββ€ [!!] Configuring d-i βββββββββββββββββββββββ
β β
β This is the network console for the Debian installer. From β
β here, you may start the Debian installer, or execute an β
β interactive shell. β
β β
β To return to this menu, you will need to log in again. β
β β
β Network console option: β
β β
β Start installer β
β Start installer (expert mode) β
β Start shell β
β β
βββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββββThe admin can open 2 sessions and then select βStart installerβ to continue installation in one session and select βStart shellβ in the other session to inspect the installation process in the installer.