Deploy AIStor on minikube for DirectPV - allanrogerr/public GitHub Wiki
Clone
cd ~
rm -rf ~/github/aistor && mkdir -p ~/github && cd ~/github
git clone https://github.com/miniohq/aistor.git
Drives
j=1
for i in vdb vdc vdd vde; do
sudo umount /dev/${i} # unmount single disk
sudo mkfs.xfs /dev/${i} -L DRIVE${j} -f # format xfs, labelling the drive similarly for easy identification
sudo mkdir -p /mnt/data${j} # At the same time create a matching directory
j=$((j+1))
done
sudo mount -a # mount all
lsblk -f # confirm
Output
vdb xfs DRIVE1 32218a29-35da-4803-b741-1902b16b9892
vdc xfs DRIVE2 d7fabbc9-e89a-482c-8752-8d06c179dfb0
vdd xfs DRIVE3 b76690b9-75a7-4d67-938b-7fcaa63bc499
vde xfs DRIVE4 76400069-5341-4fe2-bbed-31878f3523a9
Then run,
echo "
LABEL=DRIVE1 /mnt/data1 xfs defaults,noatime 0 2
LABEL=DRIVE2 /mnt/data2 xfs defaults,noatime 0 2
LABEL=DRIVE3 /mnt/data3 xfs defaults,noatime 0 2
LABEL=DRIVE4 /mnt/data4 xfs defaults,noatime 0 2
" | sudo tee -a /etc/fstab
Secondary applications
sudo systemctl stop docker.service && sudo systemctl stop docker.socket
sudo apt-get remove -y docker.io make npm
sudo apt-get install -y docker.io make npm
sudo systemctl enable docker.service && sudo systemctl enable docker.socket
sudo systemctl start docker.service && sudo systemctl start docker.socket
sudo groupadd docker
sudo usermod -aG docker ubuntu
newgrp docker
function get_latest_github_version() {
version=$(curl -s "https://github.com/$1/$2/tags" | grep -o 'href="/'$1'/'$2'/releases/tag/[^"]*' | head -n 1 | awk -F'/' '{print $NF}' )
echo ${version#v}
}
cd /tmp
wget -q https://storage.googleapis.com/minikube/releases/latest/minikube-linux-amd64
sudo chmod +x minikube-linux-amd64 && sudo mv minikube-linux-amd64 /usr/local/bin/minikube
sudo apt-get install -y conntrack socat
sudo ln -sf /usr/sbin/conntrack /usr/bin/conntrack
# CRICTL
VERSION_CRICTL=$(get_latest_github_version "kubernetes-sigs" "cri-tools")
echo $VERSION_CRICTL
wget -q https://github.com/kubernetes-sigs/cri-tools/releases/download/v$VERSION_CRICTL/crictl-v$VERSION_CRICTL-linux-amd64.tar.gz
sudo tar zxvf crictl-v$VERSION_CRICTL-linux-amd64.tar.gz -C /usr/local/bin
rm -f crictl-v$VERSION_CRICTL-linux-amd64.tar.gz
# CRI-DOCKERD
VERSION_CRI_DOCKERD=$(get_latest_github_version "Mirantis" "cri-dockerd")
echo $VERSION_CRI_DOCKERD
wget -q https://github.com/Mirantis/cri-dockerd/releases/download/v$VERSION_CRI_DOCKERD/cri-dockerd-$VERSION_CRI_DOCKERD.amd64.tgz
tar -xvf cri-dockerd-$VERSION_CRI_DOCKERD.amd64.tgz
sudo cp cri-dockerd/cri-dockerd /usr/local/bin/cri-dockerd
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/v$VERSION_CRI_DOCKERD/packaging/systemd/cri-docker.service
wget https://raw.githubusercontent.com/Mirantis/cri-dockerd/v$VERSION_CRI_DOCKERD/packaging/systemd/cri-docker.socket
sudo cp cri-docker.service cri-docker.socket /etc/systemd/system/
sudo sed -i -e 's,/usr/bin/cri-dockerd,/usr/local/bin/cri-dockerd,' /etc/systemd/system/cri-docker.service
sudo systemctl daemon-reload
sudo systemctl enable cri-docker.service
sudo systemctl enable --now cri-docker.socket
# CNI
VERSION_CNI=$(get_latest_github_version "containernetworking" "plugins")
echo $VERSION_CNI
CNI_PLUGIN_INSTALL_DIR="/opt/cni/bin"
wget -q https://github.com/containernetworking/plugins/releases/download/v$VERSION_CNI/cni-plugins-linux-amd64-v$VERSION_CNI.tgz
sudo mkdir -p "$CNI_PLUGIN_INSTALL_DIR"
sudo tar -xvf cni-plugins-linux-amd64-v$VERSION_CNI.tgz -C "$CNI_PLUGIN_INSTALL_DIR"
rm -f cni-plugins-linux-amd64-v$VERSION_CNI.tgz
Minikube
sudo sysctl -w fs.inotify.max_user_watches=1048576
sudo sysctl -w fs.inotify.max_user_instances=512
echo \"fs.inotify.max_user_watches = 1048576\" | sudo tee -a /etc/sysctl.conf
echo \"fs.inotify.max_user_instances = 512\" | sudo tee -a /etc/sysctl.conf
sudo chown -R ubuntu ~/.kube ~/.minikube
sudo resolvectl dns enp1s0 1.1.1.1 8.8.8.8 || true
sudo resolvectl dns eth0 1.1.1.1 8.8.8.8 || true
minikube stop
minikube start --driver=none
eval $(minikube docker-env)
alias k="minikube kubectl --"
Output
โ Stopping node "minikube" ...
๐ 1 node stopped.
๐ minikube v1.35.0 on Ubuntu 24.04 (kvm/amd64)
โจ Using the none driver based on existing profile
๐ Starting "minikube" primary control-plane node in "minikube" cluster
๐ Restarting existing none bare metal machine for "minikube" ...
โน๏ธ OS release is Ubuntu 24.04 LTS
๐ณ Preparing Kubernetes v1.32.0 on Docker 26.1.3 ...
โช kubelet.resolv-conf=/run/systemd/resolve/resolv.conf
๐คน Configuring local host environment ...
โ The 'none' driver is designed for experts who need to integrate with an existing VM
๐ก Most users should use the newer 'docker' driver instead, which does not require root!
๐ For more information, see: https://minikube.sigs.k8s.io/docs/reference/drivers/none/
โ kubectl and minikube configuration will be stored in /home/ubuntu
โ To use kubectl or minikube commands as your own user, you may need to relocate them. For example, to overwrite your own settings, run:
โช sudo mv /home/ubuntu/.kube /home/ubuntu/.minikube $HOME
โช sudo chown -R $USER $HOME/.kube $HOME/.minikube
๐ก This can also be done automatically by setting the env var CHANGE_MINIKUBE_NONE_USER=true
๐ Verifying Kubernetes components...
โช Using image gcr.io/k8s-minikube/storage-provisioner:v5
๐ Enabled addons: default-storageclass, storage-provisioner
๐ก kubectl not found. If you need it, try: 'minikube kubectl -- get pods -A'
๐ Done! kubectl is now configured to use "minikube" cluster and "default" namespace by default
DirectPV
cd ~
VERSION_DIRECTPV=$(get_latest_github_version "minio" "directpv")
echo $VERSION_DIRECTPV
wget -q https://github.com/minio/directpv/releases/download/v${VERSION_DIRECTPV}/kubectl-directpv_${VERSION_DIRECTPV}_linux_amd64 -O kubectl-directpv
chmod a+x kubectl-directpv
./kubectl-directpv install
./kubectl-directpv discover
./kubectl-directpv init drives.yaml --dangerous
./kubectl-directpv list drives
Output
Installing on unsupported Kubernetes v1.32
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100%
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโโโโโโโโโโ
โ NAME โ KIND โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโโโโโโโโโโค
โ directpv โ Namespace โ
โ directpv-min-io โ ServiceAccount โ
โ directpv-min-io โ ClusterRole โ
โ directpv-min-io โ ClusterRoleBinding โ
โ directpv-min-io โ Role โ
โ directpv-min-io โ RoleBinding โ
โ directpvdrives.directpv.min.io โ CustomResourceDefinition โ
โ directpvvolumes.directpv.min.io โ CustomResourceDefinition โ
โ directpvnodes.directpv.min.io โ CustomResourceDefinition โ
โ directpvinitrequests.directpv.min.io โ CustomResourceDefinition โ
โ directpv-min-io โ CSIDriver โ
โ directpv-min-io โ StorageClass โ
โ node-server โ Daemonset โ
โ controller โ Deployment โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโโโโโโโโโโ
DirectPV installed successfully
Discovered node 'miniohq-allan-5' โ
โโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโโโโฌโโโโโโโฌโโโโโโโโโโโโฌโโโโโโโโโโโโโโ
โ ID โ NODE โ DRIVE โ SIZE โ FILESYSTEM โ MAKE โ AVAILABLE โ DESCRIPTION โ
โโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโผโโโโโโโโโโโโโผโโโโโโโผโโโโโโโโโโโโผโโโโโโโโโโโโโโค
โ 253:16$YEZDNnw3G... โ miniohq-allan-5 โ vdb โ 50 GiB โ xfs โ - โ YES โ - โ
โ 253:32$2cJsOzfw0... โ miniohq-allan-5 โ vdc โ 50 GiB โ xfs โ - โ YES โ - โ
โ 253:48$Py/8+AVuw... โ miniohq-allan-5 โ vdd โ 50 GiB โ xfs โ - โ YES โ - โ
โ 253:64$eyptvpSfj... โ miniohq-allan-5 โ vde โ 50 GiB โ xfs โ - โ YES โ - โ
โโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโดโโโโโโโโโโโโโดโโโโโโโดโโโโโโโโโโโโดโโโโโโโโโโโโโโ
Generated 'drives.yaml' successfully.
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโ 100%
Processed initialization request '1b84e287-ca65-47aa-8c38-f70a5ea86ee7' for node 'miniohq-allan-5' โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโฌโโโโโโโโโโโโโโโโโโฌโโโโโโโโฌโโโโโโโโโโ
โ REQUEST_ID โ NODE โ DRIVE โ MESSAGE โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโผโโโโโโโโโโโโโโโโโโผโโโโโโโโผโโโโโโโโโโค
โ 1b84e287-ca65-47aa-8c38-f70a5ea86ee7 โ miniohq-allan-5 โ vdb โ Success โ
โ 1b84e287-ca65-47aa-8c38-f70a5ea86ee7 โ miniohq-allan-5 โ vdc โ Success โ
โ 1b84e287-ca65-47aa-8c38-f70a5ea86ee7 โ miniohq-allan-5 โ vdd โ Success โ
โ 1b84e287-ca65-47aa-8c38-f70a5ea86ee7 โ miniohq-allan-5 โ vde โ Success โ
โโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโโดโโโโโโโโโโโโโโโโโโดโโโโโโโโดโโโโโโโโโโ
โโโโโโโโโโโโโโโโโโโฌโโโโโโโฌโโโโโโโฌโโโโโโโโโฌโโโโโโโโโฌโโโโโโโโโโฌโโโโโโโโโ
โ NODE โ NAME โ MAKE โ SIZE โ FREE โ VOLUMES โ STATUS โ
โโโโโโโโโโโโโโโโโโโผโโโโโโโผโโโโโโโผโโโโโโโโโผโโโโโโโโโผโโโโโโโโโโผโโโโโโโโโค
โ miniohq-allan-5 โ vdb โ - โ 50 GiB โ 50 GiB โ - โ Ready โ
โ miniohq-allan-5 โ vdc โ - โ 50 GiB โ 50 GiB โ - โ Ready โ
โ miniohq-allan-5 โ vdd โ - โ 50 GiB โ 50 GiB โ - โ Ready โ
โ miniohq-allan-5 โ vde โ - โ 50 GiB โ 50 GiB โ - โ Ready โ
โโโโโโโโโโโโโโโโโโโดโโโโโโโดโโโโโโโดโโโโโโโโโดโโโโโโโโโดโโโโโโโโโโดโโโโโโโโโ
Wait
k -n direct-pv get pods -w
AIStor
k apply -f http://ns-3.k1.min.dev/dev/crds.yaml && k apply -f http://ns-3.k1.min.dev/dev/aistor.yaml
k patch service -n aistor aistor -p '{"spec":{"ports":[{"name": "http","port": 8444,"targetPort": 8444,"protocol": "TCP","nodePort":30080}],"type": "NodePort"}}'
Opt out of setting up DirectPV for now. miniohq/directpv should be used and this setup should be possible directly in AIStor but currently is not
Observe existing drives
Create an objectstore
Observe drive status