DirectPV Hardware issue and unable to discover data - cniackz/public GitHub Wiki
Links:
When you see **MountVolume.MountDevice failed for volume
- umount this drive -
umount /dev/sdc
- you have to repeat few times to umount the lazy mounts
- make sure the drive is completely umounted -
lsblk
- and restart directpv pod running on this node <------- restart pods
- post restart, the drive should have been mounted <---- Solution to mount, drive is mounted on pod restarts.
NOTE:
- if the drive is not mounted, then it should most likely be a bad disk and you will see an error in
kubectl directpv list drives --all
Additional:
$ kubectl directpv list drives --all
┌──────┬─────────┬───────────────────────────┬─────────┬─────────┬─────────┬────────┐
│ NODE │ NAME │ MAKE │ SIZE │ FREE │ VOLUMES │ STATUS │
├──────┼─────────┼───────────────────────────┼─────────┼─────────┼─────────┼────────┤
│ 25 │ sdb │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 25 │ sdc │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 25 │ sdd │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 25 │ sdf │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 25 │ nvme0n1 │ Micron_2200_MTFDHBA512TCK │ 477 GiB │ 477 GiB │ - │ Ready │
│ 25 │ nvme1n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
│ 25 │ nvme2n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
│ 26 │ sdb │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 26 │ sdc │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 26 │ sdd │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 26 │ sde │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 26 │ sdf │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 26 │ nvme1n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
│ 26 │ nvme2n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
│ 27 │ sdb │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 27 │ sdc │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 27 │ sdd │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 27 │ sde │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ - │ Ready │
│ 27 │ sdf │ SEAGATE XS1600LE70084 │ 1.5 TiB │ 1.5 TiB │ 1 │ Ready │
│ 27 │ nvme0n1 │ Micron_2200_MTFDHBA512TCK │ 477 GiB │ 477 GiB │ - │ Ready │
│ 27 │ nvme1n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
│ 27 │ nvme2n1 │ INTEL SSDPE2KX010T8 │ 932 GiB │ 931 GiB │ - │ Ready │
└──────┴─────────┴───────────────────────────┴─────────┴─────────┴─────────┴────────┘
$ kubectl directpv discover --all
Discovered node '25' ✔
Discovered node '26' ✔
Discovered node '27' ✔
┌─────────────────────┬──────┬─────────┬─────────┬─────────────┬──────────────────────────────────────────────┬───────────┬─────────────────────────────────┐
│ ID │ NODE │ DRIVE │ SIZE │ FILESYSTEM │ MAKE │ AVAILABLE │ DESCRIPTION │
├─────────────────────┼──────┼─────────┼─────────┼─────────────┼──────────────────────────────────────────────┼───────────┼─────────────────────────────────┤
│ 253:0$K+LEXs8Pjr... │ 25 │ dm-0 │ 672 MiB │ - │ vg1-thin--pool--1_tmeta │ NO │ Held by other device │
│ 253:1$XTA7h0zXBh... │ 25 │ dm-1 │ 1.3 TiB │ - │ vg1-thin--pool--1_tdata │ NO │ Held by other device │
│ 253:2$4+u+L8dxWn... │ 25 │ dm-2 │ 1.3 TiB │ - │ vg1-thin--pool--1-tpool │ NO │ Held by other device │
│ 253:3$kiwjkok/Wz... │ 25 │ dm-3 │ 1.3 TiB │ - │ vg1-thin--pool--1 │ NO │ Read only │
│ 253:4$/I8u1daV8L... │ 25 │ dm-4 │ 30 GiB │ - │ vg1-35515cc0--9a18--4139--9623--ae44706c3b84 │ YES │ - │
│ 253:5$oV9pKjQ0ML... │ 25 │ dm-5 │ 30 GiB │ - │ vg1-86e8b558--d625--476a--8086--f6b25118c66a │ YES │ - │
│ 253:6$DYFWb8ZGT1... │ 25 │ dm-6 │ 30 GiB │ - │ vg1-8436f57f--65b6--4b83--8053--a1e0fc2e1367 │ YES │ - │
│ 253:7$mWlarxxr0G... │ 25 │ dm-7 │ 30 GiB │ - │ vg1-3737e692--2bf8--4d5d--851c--2399f3c113c9 │ YES │ - │
│ 253:8$zWxYuwBwn6... │ 25 │ dm-8 │ 30 GiB │ - │ vg1-698b2df0--6c1f--47d9--8210--77263ac55420 │ YES │ - │
│ 253:9$M5/8sgyCsi... │ 25 │ dm-9 │ 30 GiB │ - │ vg1-9b0d258f--07e9--4277--b89d--d292b96a72d3 │ YES │ - │
│ 259:2$1cYA9UdyQ8... │ 25 │ nvme0n1 │ 477 GiB │ xfs │ Micron_2200_MTFDHBA512TCK │ NO │ Mounted; Used by DirectPV │
│ 259:0$iadvN6JMJY... │ 25 │ nvme1n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 259:1$pQ8GI8wGVm... │ 25 │ nvme2n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 8:0$100LgcjhLYzi... │ 25 │ sda │ 1.5 TiB │ - │ SEAGATE XS1600LE70084 │ NO │ Partitioned │
│ 8:1$kXaee14lqVPX... │ 25 │ sda1 │ 1.0 MiB │ - │ SEAGATE XS1600LE70084 (Part 1) │ NO │ Too small │
│ 8:2$ZClcBtc9Fltl... │ 25 │ sda2 │ 127 MiB │ vfat │ SEAGATE XS1600LE70084 (Part 2) │ NO │ Too small │
│ 8:3$gZ4PlDWPCXsi... │ 25 │ sda3 │ 384 MiB │ ext4 │ SEAGATE XS1600LE70084 (Part 3) │ NO │ Too small; Mounted │
│ 8:4$9dFn36EXsBJZ... │ 25 │ sda4 │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 (Part 4) │ NO │ Mounted │
│ 8:16$0coF0p96NdO... │ 25 │ sdb │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:32$2AmeEJ5TfbE... │ 25 │ sdc │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:48$vHoVo2O8SIM... │ 25 │ sdd │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:64$tXbZoGp9E/S... │ 25 │ sde │ 1.5 TiB │ LVM2_member │ SEAGATE XS1600LE70084 │ NO │ Held by other device │
│ 8:80$iHWzf58d6BX... │ 25 │ sdf │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 253:0$z2IPsDgqkf... │ 26 │ dm-0 │ 216 MiB │ - │ vg1-thin--pool--1_tmeta │ NO │ Too small; Held by other device │
│ 253:1$Z7Ij6/pNmw... │ 26 │ dm-1 │ 429 GiB │ - │ vg1-thin--pool--1_tdata │ NO │ Held by other device │
│ 253:2$IazGoqSX9s... │ 26 │ dm-2 │ 429 GiB │ - │ vg1-thin--pool--1-tpool │ NO │ Held by other device │
│ 253:3$0AHiKO4OBy... │ 26 │ dm-3 │ 429 GiB │ - │ vg1-thin--pool--1 │ NO │ Read only │
│ 253:4$dnQPu0BcWz... │ 26 │ dm-4 │ 30 GiB │ - │ vg1-bc91fcac--ff01--4e8a--a21d--7a6518a41683 │ YES │ - │
│ 253:5$aAOHd5WXNN... │ 26 │ dm-5 │ 30 GiB │ - │ vg1-7069d247--9f2e--4084--9286--e7f99f3541c4 │ YES │ - │
│ 253:6$Wrh2uiCAkr... │ 26 │ dm-6 │ 30 GiB │ - │ vg1-17c94729--95e7--477b--a8d1--b1ae4d22c32e │ YES │ - │
│ 253:7$p0fzgWlCFi... │ 26 │ dm-7 │ 30 GiB │ - │ vg1-2ca4d503--b7ba--42c2--8598--831af2695b45 │ YES │ - │
│ 253:8$72iyivDtHD... │ 26 │ dm-8 │ 30 GiB │ - │ vg1-e13c99e8--29ab--4169--94d1--333a2954aee1 │ YES │ - │
│ 253:9$AVV4b0SrSs... │ 26 │ dm-9 │ 30 GiB │ - │ vg1-e3f3efe3--496d--47e4--90ae--8985af17057d │ YES │ - │
│ 259:2$fbO4lYtUWw... │ 26 │ nvme0n1 │ 477 GiB │ LVM2_member │ Micron_2200_MTFDHBA512TCK │ NO │ Held by other device │
│ 259:1$os69Mc8uJ6... │ 26 │ nvme1n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 259:0$GLMtkQjRwq... │ 26 │ nvme2n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 8:0$sABSznucBOi1... │ 26 │ sda │ 1.5 TiB │ - │ SEAGATE XS1600LE70084 │ NO │ Partitioned │
│ 8:1$VM3APB52wevh... │ 26 │ sda1 │ 1.0 MiB │ - │ SEAGATE XS1600LE70084 (Part 1) │ NO │ Too small │
│ 8:2$HRM6Wju67Edn... │ 26 │ sda2 │ 127 MiB │ vfat │ SEAGATE XS1600LE70084 (Part 2) │ NO │ Too small │
│ 8:3$aYcmM44zO2WF... │ 26 │ sda3 │ 384 MiB │ ext4 │ SEAGATE XS1600LE70084 (Part 3) │ NO │ Too small; Mounted │
│ 8:4$3yasG7k9iOHv... │ 26 │ sda4 │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 (Part 4) │ NO │ Mounted │
│ 8:16$di1IdjokXoD... │ 26 │ sdb │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:32$IW/tgu3JPnO... │ 26 │ sdc │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:48$RseCKQPQz/C... │ 26 │ sdd │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:64$gtaE4yAHwMp... │ 26 │ sde │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:80$giKrqNu/VZI... │ 26 │ sdf │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 259:2$whnuln8r7m... │ 27 │ nvme0n1 │ 477 GiB │ xfs │ Micron_2200_MTFDHBA512TCK │ NO │ Mounted; Used by DirectPV │
│ 259:1$TWfAaGh4QH... │ 27 │ nvme1n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 259:0$N3W929YCqh... │ 27 │ nvme2n1 │ 932 GiB │ xfs │ INTEL SSDPE2KX010T8 │ NO │ Mounted; Used by DirectPV │
│ 8:0$llNuIE/q65lp... │ 27 │ sda │ 1.5 TiB │ - │ SEAGATE XS1600LE70084 │ NO │ Partitioned │
│ 8:1$YxtCSqnEjAx8... │ 27 │ sda1 │ 1.0 MiB │ - │ SEAGATE XS1600LE70084 (Part 1) │ NO │ Too small │
│ 8:2$VHe4QSv8tvSl... │ 27 │ sda2 │ 127 MiB │ vfat │ SEAGATE XS1600LE70084 (Part 2) │ NO │ Too small │
│ 8:3$U1hn3TY50fZA... │ 27 │ sda3 │ 384 MiB │ ext4 │ SEAGATE XS1600LE70084 (Part 3) │ NO │ Too small; Mounted │
│ 8:4$uIB2mkvlHGSl... │ 27 │ sda4 │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 (Part 4) │ NO │ Mounted │
│ 8:16$gWEYH2K2925... │ 27 │ sdb │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:32$6aEEgSbYIAr... │ 27 │ sdc │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:48$Xuxk57nirsX... │ 27 │ sdd │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:64$4lZq4gZrqFc... │ 27 │ sde │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
│ 8:80$pIDiaHyrdyM... │ 27 │ sdf │ 1.5 TiB │ xfs │ SEAGATE XS1600LE70084 │ NO │ Mounted; Used by DirectPV │
└─────────────────────┴──────┴─────────┴─────────┴─────────────┴──────────────────────────────────────────────┴───────────┴─────────────────────────────────┘
Generated 'drives.yaml' successfully.