Known Issues - calab-ntu/gpu-cluster GitHub Wiki

  • spock jobs using small amount of CPU might cause CPU high temperature. Solution: Use all CPUs(32) instead of small amount of number.

  • eureka05:

    Jul 27 20:10:05 eureka05 kernel: NVRM: Going over RM unhandled interrupt threshold for irq 400

  •  [system] Failed to activate service 'org.freedesktop.login1': timed out.

    This failure will slow down the system. After booting the system, check /var/log/messages first to see if this problem occurs. Solution: reboot

  •  kernel: NVRM: GPU at PCI:0000:41:00: GPU-d0ae3a6d-5280-244c-4b33-0c603a504553
 kernel: NVRM: GPU Board Serial Number:
 NVRM: Xid (PCI:0000:41:00): 31, pid=6286, Ch 00000008, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_T1_3 faulted @ 0x7fd7_654f5000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_WRITE

This issue usually happens when a GPU code is terminated manually. It will cause the GPU to hang. Solution: (1) see GPU Hangs , (2) reboot

  •  mcelog : AMD Processor family 23: mcelog does not support this processor. Please use the edac_mce_amd module instead.

  •  Kernel: perf: interrupt took too long (2502>2500), lowering kernel.perf_event_max_sample_rate to 79000

  •  Kernel: mlx5_core 0000:09:00.0: rsc_event_notifier:190:(pid 0):Async event for bogus resource 0x4d7

  •  kernel: gamer (62304): Using mlock ulimits for SHM_HUGETLB is deprecated
 kernel: pcieport 0000:00:03.1: AER: Corrected error received: id=0000
 kernel: pcieport 0000:00:03.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0019(Transmitter ID)
 kernel: pcieport 0000:00:03.1: device [1022:1453] error status/mask=00001000/00006000
 kernel: pcieport 0000:00:03.1: [12] Replay Timer Timeout

  • nvidia-smi shows

     Unable to determine the device handle for GPU 0000:41:00.0: GPU is lost. Reboot the system to recover this GPU

  •  kernel: MG_CPU[2818]: segfault at 0 ip 0000000000405e48 sp 00007fff55c688c0 error 4 in MG_CPU[400000+7000]

  •  kernel: python invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=0
 kernel: python cpuset=/ mems_allowed=0-1
 kernel: CPU: 22 PID: 72686 Comm: python Kdump: loaded Tainted: P           OE  ------------   3.10.0-1062.el7.x86_64 #1
 kernel: Hardware name: To Be Filled By O.E.M. To Be Filled By O.E.M./X399 Phantom Gaming 6, BIOS P1.30 08/27/2019

  •  pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.OU [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.o5636' failed with status=1, giving up after 4 attempts
 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.OU to [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.o5636                                                                Jul 20 15:27:39 eureka11 pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.ER [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.e5636' failed with status=1, giving up after 4 attempts
 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.ER to [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.e5636
 pbs_mom: LOG_ERROR::req_cpyfile, #012#012Unable to copy file /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.OU to [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.o5636#012*** error from copy#012Host key verification failed.#015#012lost connection#012*** end error output#012Output retained on that host in: 
 /var/spool/TORQUE/undelivered/5636.eureka00.gpucluster.calab.OU#012#012Unable to copy file 
 /var/spool/TORQUE/spool/5636.eureka00.gpucluster.calab.ER to [email protected]:/work1/jared/gamer-fork/bin/plummer/gamer.e5636#012*** error from copy#012Host key verification failed.#015#012lost connection#012*** end error output#012Output retained on that host in: 
 /var/spool/TORQUE/undelivered/5636.eureka00.gpucluster.calab.ER

  •  nvprof ./matrix_transpose
 ==15637== NVPROF is profiling process 15637, command: ./matrix_transpose
 ==15637== Warning: ERR_NVGPUCTRPERM - The user does not have permission to profile on the target device. See the following link for instructions to enable permissions and get more information: 
 https://developer.nvidia.com/ERR_NVGPUCTRPERM
 ==15637== Profiling application: ./matrix_transpose
 ==15637== Profiling result:
 No kernels were profiled.
 No API activities were profiled.
 ==15637== Warning: Some profiling data are not recorded. Make sure cudaProfilerStop() or cuProfilerStop() is called before application exit to flush profile data.

  •  kernel: pcieport 0000:00:03.1: AER: Corrected error received: id=0000
 kernel: pcieport 0000:00:03.1: PCIe Bus Error: severity=Corrected, type=Data Link Layer, id=0019(Receiver ID)
 kernel: pcieport 0000:00:03.1:   device [1022:1453] error status/mask=00000040/00006000

  •  kernel: traps: a.out[89599] general protection ip:4090b4 sp:7ffe6c084fa0 error:0 in a.out[400000+aa000]

  •  pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.OU [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.o7172' failed with status=1, giving up after 4 attempts
 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.OU to [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.o7172
 pbs_mom: LOG_ERROR::sys_copy, command '/usr/bin/scp -rpB /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.ER [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.e7172' failed with status=1, giving up after 4 attempts
 pbs_mom: LOG_ERROR::req_cpyfile, Unable to copy file /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.ER to [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.e7172                                                                                                                                                      Aug 17 16:52:09 eureka04 pbs_mom: LOG_ERROR::req_cpyfile, #012#012Unable to copy file /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.OU to [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.o7172#012*** error from copy#012Host key verification failed.#015#012lost connection#012*** end error output#012Output retained on that host in: /var/spool/TORQUE/undelivered/7172.eureka00.gpucluster.calab.OU#012#012Unable to copy file /var/spool/TORQUE/spool/7172.eureka00.gpucluster.calab.ER to [email protected]:/work1/jared/gamer-fork/bin/CDM/gamer.e7172#012*** error from copy#012Host key verification failed.#015#012lost connection#012*** end error output#012Output retained on that host in: /var/spool/TORQUE/undelivered/7172.eureka00.gpucluster.calab.ER

  • FTR. HDF5 on 33 is malfunctioning with the error Unable to create file (unable to lock file, errno = 37, error message = 'No locks available'). Setting export HDF5_USE_FILE_LOCKING=FALSE fixes it, as suggested here: https://github.com/h5py/h5py/issues/1082. But I don’t know why.

  • GPU freezes from time to time on all nodes. /var/log/messages shows

    kernel: NVRM: Xid (PCI:0000:41:00): 31, Ch 00000019, intr 00000000. MMU Fault: ENGINE GRAPHICS GPCCLIENT_GCC faulted @ 0x7fc0_e2d4b000. Fault is of type FAULT_PDE ACCESS_TYPE_VIRT_READ

    Reference: https://devtalk.nvidia.com/default/topic/1058707/linux/will-the-fault_pde-access_type_read-bug-in-the-nvidia-driver-ever-be-fixed-/ Useful info

    1. Uninstall driver: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-installation
    2. Uninstall driver: https://medium.com/@philliplakis/removing-nvidia-drivers-on-centos-7-for-upgrading-bf00a2a5f0df
    3. Package Manager Installation: https://docs.nvidia.com/cuda/cuda-installation-guide-linux/index.html#package-manager-installation

  • Node 192.168.0.153 is unstable. Seg fault occurs frequently after running gamer for a few hours. In addition, system freezes from time to time

    Error message example: kernel:watchdog: BUG: soft lockup - CPU#4 stuck for 23s!

  • Low I/O bandwidth in the new storage system (~ 20 MB/s reported by @tseng)

  • eater does not recognize user names

  • Debug for "No components were able to be opened in the pml framework" when running MPI (with UCX library)

    1. Check Infiniband status by ibstatus. For health node, message as below should show up:
    Infiniband device 'mlx5_0' port 1 status:
default gid:     fe80:0000:0000:0000:1c34:da03:0057:b0a8
base lid:        0x2
sm lid:          0x1
state:           4: ACTIVE
phys state:      5: LinkUp
rate:            100 Gb/sec (4X EDR)
link_layer:      InfiniBand
    1. If Infiniband status is fine. Try to dump more message by adding
    export OMPI_MCA_pml_ucx_verbose=100

    in the submit script, or using

    mpirun --mca pml_ucx_verbose 100

    A quick way for debug test is using mpirun --mca pml_ucx_verbose 100 -np 1 python -c "from mpi4py import MPI", which initiates the MPI by python so no need to compile anything.

    1. Check the error message. Use Infiniband band failure as example, one will see the following message:
    [eureka14:126041] pml_ucx.c:198 mca_pml_ucx_open: UCX version 1.12.0
[eureka14:126041] common_ucx.c:304 posix/memory: did not match transport list
[eureka14:126041] common_ucx.c:304 sysv/memory: did not match transport list
[eureka14:126041] common_ucx.c:304 self/memory0: did not match transport list
[eureka14:126041] common_ucx.c:304 tcp/lo: did not match transport list
[eureka14:126041] common_ucx.c:304 tcp/enp4s0: did not match transport list
[eureka14:126041] common_ucx.c:304 cma/memory: did not match transport list
[eureka14:126041] common_ucx.c:304 knem/memory: did not match transport list
[eureka14:126041] common_ucx.c:311 support level is none

    Compared to health Infiniband connection:

    [eureka16:10746] pml_ucx.c:198 mca_pml_ucx_open: UCX version 1.12.0
[eureka16:10746] common_ucx.c:304 posix/memory: did not match transport list
[eureka16:10746] common_ucx.c:304 sysv/memory: did not match transport list
[eureka16:10746] common_ucx.c:304 self/memory0: did not match transport list
[eureka16:10746] common_ucx.c:304 tcp/lo: did not match transport list
[eureka16:10746] common_ucx.c:304 tcp/enp4s0: did not match transport list
[eureka16:10746] common_ucx.c:199 driver '../../../../bus/pci/drivers/mlx5_core' matched by 'mlx*'
[eureka16:10746] common_ucx.c:295 rc_verbs/mlx5_0:1: matched both transport and device list
[eureka16:10746] common_ucx.c:311 support level is transports and devices

    this indicates IB device mlx5 is not detected by common_ucx.c. One can further check this by using PATH_TO_UCX/bin/ucx_info -d to see the device and transport setup. In the mal-functioning case, it will show:

    # Memory domain: mlx5_0
#     Component: ib
#             register: unlimited, cost: 180 nsec
#           remote key: 8 bytes
#           local memory handle is required for zcopy
#   < no supported devices found >

  • df command stuck or mounted directories stuck 

    kernel: INFO: task scp:48089 blocked for more than 120 seconds.                                                       
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.                                     
kernel: scp             D ffff88937b43d230     0 48089  48088 0x00000080

    Solution: Reboot the problematic node.