k8s-使用ceph分布式存储
1）k8s集群ceph环境准备
`下载key`
"k8s集群里所有节点都要执行，我这里只演示一个"
root@k8s-master01:~# wget -q -O- 'https://download.ceph.com/keys/release.asc' | apt-key add -
`导入ceph源`
"k8s集群里所有节点都要执行，我这里只演示一个"
root@k8s-master01:~# apt-add-repository 'deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-octopus/ buster main'
root@k8s-master03:~# cat /etc/apt/sources.list
deb https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-security main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-updates main restricted universe multiverse
deb https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb-src https://mirrors.aliyun.com/ubuntu/ focal-backports main restricted universe multiverse
deb http://mirrors.aliyun.com/ubuntu/ bionic main restricted universe multiverse
deb https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-mimic/ bionic main
root@k8s-master01:~# apt update
`搜索ceph版本信息`
root@k8s-master03:~# apt-cache madison ceph-common
ceph-common | 15.2.17-0ubuntu0.20.04.5 | https://mirrors.aliyun.com/ubuntu focal-updates/main amd64 Packages
ceph-common | 15.2.17-0ubuntu0.20.04.3 | https://mirrors.aliyun.com/ubuntu focal-security/main amd64 Packages
ceph-common | 15.2.1-0ubuntu1 | https://mirrors.aliyun.com/ubuntu focal/main amd64 Packages
ceph-common | 13.2.10-1bionic | https://mirrors.tuna.tsinghua.edu.cn/ceph/debian-mimic bionic/main amd64 Packages
ceph-common | 12.2.4-0ubuntu1 | http://mirrors.aliyun.com/ubuntu bionic/main amd64 Packages
ceph | 15.2.1-0ubuntu1 | https://mirrors.aliyun.com/ubuntu focal/main Sources
ceph | 15.2.17-0ubuntu0.20.04.3 | https://mirrors.aliyun.com/ubuntu focal-security/main Sources
ceph | 15.2.17-0ubuntu0.20.04.5 | https://mirrors.aliyun.com/ubuntu focal-updates/main Sources
`安装ceph-common`
root@k8s-master03:~# apt-get -y install ceph-common
2）准备ceph资源
2.1）准备存储池
[ceph@ceph-deploy ceph-cluster]$ ceph -s
cluster:
id: 93b70727-4253-4b8a-845e-28b766e34392
health: HEALTH_OK
services:
mon: 1 daemons, quorum ceph-mon-34
mgr: ceph-mgr-35(active)
osd: 3 osds: 3 up, 3 in
data:
pools: 0 pools, 0 pgs
objects: 0 objects, 0 B
usage: 3.0 GiB used, 147 GiB / 150 GiB avail
pgs:
`创建存储池`
[ceph@ceph-deploy ceph-cluster]$ ceph osd pool create rbd-data1 32 32
`在存储池启用rbd:`
[ceph@ceph-deploy ceph-cluster]$ ceph osd pool application enable rbd-data1 rbd
`初始化rbd:`
[ceph@ceph-deploy ceph-cluster]$ rbd pool init -p rbd-data1
`创建两个镜像img`
[ceph@ceph-deploy ceph-cluster]$ rbd create data-img01 --size 3G --pool rbd-data1 --image-format 2 --image-feature layering
[ceph@ceph-deploy ceph-cluster]$ rbd create data-img02 --size 5G --pool rbd-data1 --image-format 2 --image-feature layering
`列出镜像个多信息:`
[ceph@ceph-deploy ceph-cluster]$ rbd ls --pool rbd-data1 -l
NAME SIZE PARENT FMT PROT LOCK
data-img01 3 GiB 2
data-img02 5 GiB 2
2.2）准备ceph用户
`创建用户`
[ceph@ceph-deploy ceph-cluster]$ ceph auth get-or-create client.stj mon 'allow r' osd 'allow rwx pool=rbd-data1'
[client.stj]
key = AQCec7Rl+vvCHBAAiDR03Q47oMZKVniQnxh54Q==
`验证用户`
[ceph@ceph-deploy ceph-cluster]$ ceph auth get client.stj
exported keyring for client.stj
[client.stj]
key = AQCec7Rl+vvCHBAAiDR03Q47oMZKVniQnxh54Q==
caps mon = "allow r"
caps osd = "allow rwx pool=rbd-data1"
`导出用户信息到keyring文件`
[ceph@ceph-deploy ceph-cluster]$ ceph auth get client.stj > ceph.client.stj.keyring
[ceph@ceph-deploy ceph-cluster]$ cat client.stj.keyring
[client.stj]
key = AQCec7Rl+vvCHBAAiDR03Q47oMZKVniQnxh54Q==
caps mon = "allow r"
caps osd = "allow rwx pool=rbd-data1"
2.3）同步认证文件到k8s集群
[ceph@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.stj.keyring root@192.168.139.231:/etc/ceph/
[ceph@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.stj.keyring root@192.168.139.232:/etc/ceph/
[ceph@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.stj.keyring root@192.168.139.235:/etc/ceph/
[ceph@ceph-deploy ceph-cluster]$ scp ceph.conf ceph.client.stj.keyring root@192.168.139.236:/etc/ceph/
如果要动态创建卷，就要把ceph.conf拷贝到k8s的master节点，我这里我都拷贝了
2.4）k8s节点验证用户权限
root@k8s-master01:~# ceph --user stj -s
cluster:
id: 93b70727-4253-4b8a-845e-28b766e34392
health: HEALTH_OK
services:
mon: 1 daemons, quorum ceph-mon-34
mgr: ceph-mgr-35(active)
osd: 3 osds: 3 up, 3 in
data:
pools: 2 pools, 64 pgs
objects: 6 objects, 51 B
usage: 3.0 GiB used, 147 GiB / 150 GiB avail
pgs: 64 active+clean
3）k8s节点配主机名解析
-
k8s节点都要配置，这里我演示一个
root@k8s-master01:~# vim /etc/hosts
# 最后一行都要添加
192.168.139.33 ceph-osd-33
192.168.139.34 ceph-mon-34
192.168.139.35 ceph-mgr-35
192.168.139.36 ceph-deploy
手动配置rbd挂载
1）通过keyring文件挂载rbd
基于ceph提供的rbd实现存储卷的动态提供,由两种实现方式, - -是通过宿主机的keyring文件挂载Ibd,另外-一个是通过将keyring中key定义为k8s中的secret,然后pod通过secret挂载rbd。
1.1）通过keyring文件直接挂载-busybox
1.1.1）创建yaml并应用
root@k8s-master01:~# mkdir -p /stj-soft/yaml/ceph-rbd/busybox
root@k8s-master01:~# cd /stj-soft/yaml/ceph-rbd/busybox
`编写yaml文件`
root@k8s-master01:/stj-soft/yaml/ceph-rbd/busybox# cat busybox.yaml
apiVersion: v1
kind: Pod
metadata:
name: busybox
namespace: stj-ns
spec:
containers:
- name: busybox
image: busybox
command:
- sleep
- "3600"
imagePullPolicy: Always
volumeMounts:
- name: stj-rbd1
mountPath: /data
volumes:
- name: stj-rbd1
rbd:
monitors:
- '192.168.139.34:6789'
# - '192.168.139.38:6789'
# - '192.168.139.39:6789'
pool: rbd-data1 # 指定ceph创建的存储池
image: data-img01 # 指定该存储里创建的镜像
fsType: ext4 # 格式化rbd的文件系统类型
readOnly: false
user: stj # 使用哪个用户进行挂载
keyring: /etc/ceph/ceph.client.stj.keyring # 指定授权用户的文件路径
`应用yaml文件`
root@k8s-master01:/stj-soft/yaml/ceph-rbd/busybox# kubectl apply -f busybox.yaml
1.1.2）验证是否挂载
root@k8s-master01:/stj-soft/yaml/ceph-rbd/busybox# kubectl get pod -n stj-ns
NAME READY STATUS RESTARTS AGE
busybox 1/1 Running 0 106s
`进入该镜像`
root@k8s-master01:/stj-soft/yaml/ceph-rbd/busybox# kubectl exec -it busybox -n stj-ns sh
`验证是否挂载`
/ # df -h
Filesystem Size Used Available Use% Mounted on
overlay 35.0G 13.5G 21.4G 39% /
tmpfs 64.0M 0 64.0M 0% /dev
tmpfs 1.9G 0 1.9G 0% /sys/fs/cgroup
/dev/rbd0 2.9G 24.0K 2.9G 0% /data # 这个可以看到已经挂载
/dev/sda4 35.0G 13.5G 21.4G 39% /dev/termination-log
/dev/sda4 35.0G 13.5G 21.4G 39% /etc/resolv.conf
/dev/sda4 35.0G 13.5G 21.4G 39% /etc/hostname
/dev/sda4 35.0G 13.5G 21.4G 39% /etc/hosts
shm 64.0M 0 64.0M 0% /dev/shm
tmpfs 1.9G 12.0K 1.9G 0% /var/run/secrets/kubernetes.io/serviceaccount
tmpfs 1.9G 0 1.9G 0% /proc/acpi
tmpfs 64.0M 0 64.0M 0% /proc/kcore
tmpfs 64.0M 0 64.0M 0% /proc/keys
tmpfs 64.0M 0 64.0M 0% /proc/timer_list
tmpfs 64.0M 0 64.0M 0% /proc/sched_debug
tmpfs 1.9G 0 1.9G 0% /proc/scsi
tmpfs 1.9G 0 1.9G 0% /sys/firmware
1.1.3）验证是否可以写数据
`验证ceph的pool的空间`
[ceph@ceph-deploy ceph-cluster]$ ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
150 GiB 147 GiB 3.0 GiB 2.01
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd-data1 1 7.4 MiB 0.02 46 GiB 15
`进入该镜像生成100M数据`
root@k8s-master01:/stj-soft/yaml/ceph-rbd/busybox# kubectl exec -it busybox -n stj-ns sh
/data # dd if=/dev/zero of=data-file bs=1M count=100
/data # ls -lh
total 100M
-rw-r--r-- 1 root root 100.0M Jan 27 04:06 data-file
drwx------ 2 root root 16.0K Jan 27 03:55 lost+found
`验证ceph的pool的空间`
[ceph@ceph-deploy ceph-cluster]$ ceph df
GLOBAL:
SIZE AVAIL RAW USED %RAW USED
150 GiB 138 GiB 12 GiB 7.75
POOLS:
NAME ID USED %USED MAX AVAIL OBJECTS
rbd-data1 1 2.9 GiB 6.19 44 GiB 746
1.2）通过keyring文件直接挂载-nginx
1.2.1）创建yaml并应用
root@k8s-master01:~# mkdir -p /stj-soft/yaml/ceph-rbd/nginx
root@k8s-master01:~# cd /stj-soft/yaml/ceph-rbd/nginx
root@k8s-master01:/stj-soft/yaml/ceph-rbd/nginx# cat nginx.yaml
kind: Deployment
apiVersion: apps/v1
metadata:
labels:
app: nginx-stj-deployment-label
name: nginx-deployment
namespace: stj-ns
spec:
replicas: 1
selector:
matchLabels:
app: nginx-stj-selector
template:
metadata:
labels:
app: nginx-stj-selector
spec:
containers:
- name: nginx-stj-container
image: harbor.stangj.net/pub-images/nginx-1.24.0:v1
imagePullPolicy: Always
ports:
- containerPort: 80
protocol: TCP
volumeMounts:
- name: stj-rbd1
mountPath: /data
volumes:
- name: stj-rbd1
rbd:
monitors:
- '192.168.139.34:6789'
# - '192.168.139.38:6789'
# - '192.168.139.39:6789'
pool: rbd-data1 # 指定ceph创建的存储池
image: data-img01 # 指定该存储里创建的镜像
fsType: ext4 # 格式化rbd的文件系统类型
readOnly: false
user: stj # 使用哪个用户进行挂载
keyring: /etc/ceph/ceph.client.stj.keyring # 指定授权用户的文件路径
---
kind: Service
apiVersion: v1
metadata:
labels:
app: nginx-stj-label
name: nginx-stj-svc
namespace: stj-ns
spec:
type: NodePort
ports:
- name: http
protocol: TCP
port: 80
targetPort: 80
nodePort: 40001
- name: https
protocol: TCP
port: 443
targetPort: 443
nodePort: 40043
selector:
app: nginx-stj-selector
1.2.2）验证是否挂载
root@k8s-master01:/stj-soft/yaml/ceph-rbd/nginx# kubectl get pod -n stj-ns
NAME READY STATUS RESTARTS AGE
nginx-deployment-d6c55c674-2pgzx 1/1 Running 0 43s
root@k8s-master01:/stj-soft/yaml/ceph-rbd/nginx# kubectl exec -it nginx-deployment-d6c55c674-2pgzx -n stj-ns sh
kubectl exec [POD] [COMMAND] is DEPRECATED and will be removed in a future version. Use kubectl exec [POD] -- [COMMAND] instead.
sh-4.2# df
Filesystem 1K-blocks Used Available Use% Mounted on
overlay 36679172 14197984 22481188 39% /
tmpfs 65536 0 65536 0% /dev
tmpfs 1990824 0 1990824 0% /sys/fs/cgroup
/dev/rbd0 3021608 102428 2902796 4% /data # 以及挂载
/dev/sda4 36679172 14197984 22481188 39% /etc/hosts
shm 65536 0 65536 0% /dev/shm
tmpfs 1990824 12 1990812 1% /run/secrets/kubernetes.io/serviceaccount
tmpfs 1990824 0 1990824 0% /proc/acpi
tmpfs 1990824 0 1990824 0% /proc/scsi
tmpfs 1990824 0 1990824 0% /sys/firmware
1.2.3）验证是否可以写数据
`进入该镜像生成200M数据`
root@k8s-master01:/stj-soft/yaml/ceph-rbd/nginx# kubectl exec -it nginx-deployment-d6c55c674-2pgzx -n stj-ns sh
sh-4.2# dd if=/dev/zero of=data-file bs=2M count=100
sh-4.2# ls -lh
total 201M
-rw-r--r-- 1 root root 200M Jan 27 04:26 data-file
drwx------ 2 root root 16K Jan 27 03:55 lost+found
文章评论