部署ceph

https://github.com/ceph/ceph

部署方式

ceph-ansible: https://github.com/ceph/ceph-ansible # python
ceph-salt: https://github.com/ceph/ceph-salt #python
ceph-container: https://github.com/ceph/ceph-container #shell
ceph-chef: https://github.com/ceph/ceph-chef #Ruby

ceph-deploy: https://github.com/ceph/ceph-deploy #python

    ceph-deploy：是一个ceph官方维护的基于ceph-deploy命令行部署ceph集群的工具，基于ssh 执行可以sudo权限的shell命令以及一些python脚本实现ceph集群的部署和管理维护。
    Ceph-deploy 只用于部署和管理ceph集群，客户端需要访问ceph，需要部署客户端工具。

服务器准备:

三台服务器作为ceph集群OSD存储服务器，每台服务器支持两个网络,public网络针对客户端访问，cluster网络用于集群管理及数据同步，每台三块或以上的磁盘
- 主机IP规划
```
172.16.1.201/192.168.139.201
172.16.1.202/192.168.139.202
172.16.1.203/192.168.139.203
```
- 三台存储服务器磁盘划分:
```
/dev/sdb /dev/sdc /dev/sdd /dev/sde /dev/sdf  #100G
/dev/sdg /dev/sdh /dev/sdi                    #200G
```
三台服务器作为ceph集群Mon监视服务器，每台服务器可以和ceph集群的cluster网络通信。
```
172.16.1.204/192.168.139.204
172.16.1.205/192.168.139.205
172.16.1.206/192.168.139.206
```
两个ceph-mgr管理服务器，可以和ceph 集群的cluster网络通信。
```
172.16.1.207/192.168.139.207
172.16.1.208/192.168.139.208
```
一个服务器用于部署ceph 集群即安装 Ceph-deploy，也可以和ceph-mgr等复用。
```
172.16.1.209/192.168.139.209
```
创建一个普通用户，能够通过sudo执行特权命令，配置主机名解析，ceph 集群依赖于主机名,关闭各服务器防火墙、selinux、

系统环境准备:

时间同步
关闭selinux和防火墙
配置域名解析或通过DNS解析

部署RADOS集群

源的更新

每个主机都要执行

[root@ceph-osd-201 ~]# rpm -ivh https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch/ceph-release-1-1.el7.noarch.rpm
[root@ceph-osd-201 ~]# vim /etc/yum.repos.d/ceph.repo 
[Ceph]
name=Ceph packages for $basearch
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/$basearch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc

[Ceph-noarch]
name=Ceph noarch packages
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/noarch
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc

[ceph-source]
name=Ceph source packages
baseurl=https://mirrors.tuna.tsinghua.edu.cn/ceph/rpm-mimic/el7/SRPMS
enabled=1
gpgcheck=1
type=rpm-md
gpgkey=https://mirrors.tuna.tsinghua.edu.cn/ceph/keys/release.asc
[ceph@ceph-deploy ceph-cluster]$ sudo yum makecache fast
`发送源到其他主机，每一个主机都要发`
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.202:/etc/yum.repos.d/ceph.repo 
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.203:/etc/yum.repos.d/ceph.repo
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.204:/etc/yum.repos.d/ceph.repo
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.205:/etc/yum.repos.d/ceph.repo
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.206:/etc/yum.repos.d/ceph.repo
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.207:/etc/yum.repos.d/ceph.repo
[root@ceph-osd-201 ~]# scp /etc/yum.repos.d/ceph.repo 172.16.1.208:/etc/yum.repos.d/ceph.repo

创建ceph用户配置免密

每个主机都要执行

[root@ceph-osd-201 ~]# groupadd -g 2023 ceph &&useradd -u 2023 -g 2023 ceph && echo '123456' |passwd --stdin ceph

[root@ceph-osd-201 ~]# echo "ceph ALL = (root) NOPASSWD:ALL" | sudo tee /etc/sudoers.d/ceph
[root@ceph-osd-201 ~]# su - ceph 
[ceph@ceph-osd-201 ~]$ ssh-keygen 
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.201
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.202
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.203
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.204
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.205
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.206
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.207
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.208
[ceph@ceph-osd-201 ~]$ ssh-copy-id ceph@172.16.1.209

配置内部DNS

主机：ceph-deploy操作

[ceph@ceph-deploy ~]$ su - root 
[root@ceph-deploy ~]# yum -y install bind bind-chroot
[root@ceph-deploy ~]# vim/etc/named.conf 
options {
    listen-on port 53 { 192.168.139.209; };
    listen-on-v6 port 53 { ::1; };
    directory   "/var/named";
    dump-file   "/var/named/data/cache_dump.db";
    statistics-file "/var/named/data/named_stats.txt";
    memstatistics-file "/var/named/data/named_mem_stats.txt";
    recursing-file  "/var/named/data/named.recursing";
    secroots-file   "/var/named/data/named.secroots";
    allow-query     { any; };
     forwarders      { 192.168.139.2; };
    
    recursion yes;
    dnssec-enable no;
    dnssec-validation no;
[root@ceph-deploy ~]# cat /etc/named.rfc1912.zones 
zone "host.com" IN {
        type master;
        file "host.com.zone";
        allow-update { 192.168.139.209; };
};
[root@ceph-deploy ~]# cat /var/named/host.com.zone
$ORIGIN host.com.
$TTL 600        ; 10 minutes
@               IN SOA   dns.host.com. dnsadmin.host.com. (
                                2023090301 ; serial
                                10800      ; refresh (3 hours)
                                900        ; retry (15 minutes)
                                604800     ; expire (1 week)
                                86400      ; minimum (1 day)
                                )
                        NS      dns.host.com.
$TTL 60 ; 1 minute
dns                        A         192.168.139.209
CEPH-OSD-201               A         192.168.139.201
CEPH-OSD-202               A         192.168.139.202
CEPH-OSD-203               A         192.168.139.203
CEPH-MON-204               A         192.168.139.204
CEPH-MON-205               A         192.168.139.205
CEPH-MON-206               A         192.168.139.206
CEPH-MGR-207               A         192.168.139.207
CEPH-MGR-208               A         192.168.139.208
CEPH-DEPLOY                A         192.168.139.209
[root@ceph-deploy ~]# systemctl start named
[root@ceph-deploy ~]# systemctl enable named.service 
`修改DNS,每一个都要改`
[root@ceph-deploy ~]# vim /etc/sysconfig/network-scripts/ifcfg-eth0
DNS1=192.168.139.209
`每一个都要改`
[root@ceph-deploy ~]# echo 'search host.com' >> /etc/resolv.conf 
[root@ceph-deploy ~]# cat /etc/resolv.conf 
# Generated by NetworkManager
nameserver 192.168.139.209
nameserver 223.6.6.6
search host.com
[root@ceph-deploy ~]# ping -W1 -c1 ceph-osd-201
PING CEPH-OSD-201.host.com (192.168.139.201) 56(84) bytes of data.
64 bytes from 192.168.139.201 (192.168.139.201): icmp_seq=1 ttl=64 time=0.217 ms

安装ceph部署工具

主机：ceph-deploy-209操作

[ceph@ceph-deploy ~]$ sudo yum install ceph-deploy python-setuptools python2-subprocess32 -y

首先在管理节点上以cephadm用户创建集群相关的配置文件目录

[ceph@ceph-deploy ~]$ mkdir ceph-cluster
[ceph@ceph-deploy ~]$ cd ceph-cluster/

ceph-deploy参数解释

new:开始部署一个新的ceph存储集群, 并生成CLUSTER.conf集群配置文件和keyring认证文件。
install: 在远程主机上安装ceph相关的软件包，可以通过--release指定安装的版本。
rgw:管理RGW守护程序(RADOSGW,对象存储网关)。
mgr:管理MGR守护程序(ceph-mgr,Ceph Manager DaemonCeph管理器守护程序)。
mds:管理MDS守护程序(Ceph Metadata Server, ceph源数据服务器)。
mon:管理MON守护程序(ceph-mon,ceph监视器)。
gatherkeys:从指定获取提供新节点的验证keys ,这些keys会在添加新的MON/OSD/MD加入的时候使用。
disk: 管理远程主机磁盘。
osd:在远程主机准备数据磁盘，即将指定远程主机的指定磁盘添加到ceph集群作为osd使用。
repo:远程主机仓库管理。
admin:推送ceph集群配置文件和client.admin认证文件到远程主机。
config:将ceph.conf配置文件推送到远程主机或从远程主机拷贝。
uninstall:从远端主机删除安装包。
purgedata:从/var/lib/ceph 删除ceph数据，会删除/etc/ceph下的内容。
purge:删除远端主机的安装包和所有数据。
forgetkeys:从本地主机删除所有的验证keyring,包括client.admin, monitor,bootstrap等认证文件。
pkg:管理远端主机的安装包。
calamari:安装并配置-个calamari web节点，calamari 是一一个web监控平台。

初始化mon节点过程如下:

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy new --cluster-network 172.16.1.0/24 --public-network 192.168.139.0/24 ceph-mon-206
[ceph@ceph-deploy ceph-cluster]$ ll
total 12
-rw-rw-r-- 1 ceph ceph  271 Sep  3 15:14 ceph.conf
-rw-rw-r-- 1 ceph ceph 3967 Sep  3 15:14 ceph-deploy-ceph.log
-rw------- 1 ceph ceph   73 Sep  3 15:14 ceph.mon.keyring

初始化ceph存储节点:

初始化存储节点等于在存储节点安装了ceph 及ceph-rodsgw安装包，但是使用默认的官方仓库会因为网络原因导致初始化超时,因此推荐修改ceph仓库为阿里或者清华等国内的镜像源:

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-osd-201 
`为了让初始化更快`
[ceph@ceph-osd-202 ~]$ sudo yum -y install ceph ceph-radosgw
[ceph@ceph-osd-203 ~]$ sudo yum -y install ceph ceph-radosgw
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy install --no-adjust-repos --nogpgcheck ceph-osd-202 ceph-osd-203

mon节点并收集秘钥:

通过读取当前目录下的ceph.conf配置文件初始化mon节点，mon节点ha还可以后期横向扩容。

[ceph@ceph-mon-206 ~]$ sudo yum install ceph-mon -y
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy mon create-initial

验证mon服务

[ceph@ceph-mon-206 ~]$ ps -aux |grep mon |column -t |grep -v grep
dbus  784    0.0  0.0  58264   2600   ?      Ss   12:45  0:00  /usr/bin/dbus-daemon      --system      --address=systemd:  --nofork  --nopidfile  --systemd-activation
root  799    0.0  0.3  474332  8756   ?      Ssl  12:45  0:01  /usr/sbin/NetworkManager  --no-daemon
ceph  13355  0.0  0.9  465480  26928  ?      Ssl  15:50  0:00  /usr/bin/ceph-mon         -f            --cluster           ceph      --id         ceph-mon-01           --setuser  ceph  --setgroup  ceph

分发admin秘钥:

把配置文件和admin密钥拷贝至Ceph集群需要执行ceph管理命令的节点,从而不需要后期通过ceph命令访问与对ceph集群进行管理配置的时候每次都需要指定ceph-mon节点地址和ceph.client.admin.keyring文件。

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy admin ceph-osd-201 ceph-osd-202 ceph-osd-203

ceph节点验证密钥

[ceph@ceph-osd-201 ~]$ sudo tree /etc/ceph 
/etc/ceph
├── ceph.client.admin.keyring
├── ceph.conf
├── rbdmap
└── tmpJsIOI1

认证文件的属主和属组为了安全考虑,默认设置为了root用户和root组，如果需要ceph用户也能执行ceph命令，那么就需要对ceph用户进行授权,

[ceph@ceph-osd-201 ~]$ sudo setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring
[ceph@ceph-osd-202 ~]$ sudo setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring
[ceph@ceph-osd-203 ~]$ sudo setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring

配置 manager节点:

ceph的Luminious及以上版本有manager节点，早期的版本没有。

部署ceph-mgr节点:

mgr节点需要读取ceph的配置文件，即/etc/ceph 目录中的配置文件

[ceph@ceph-mgr-207 ~]$ sudo yum install ceph-mgr -y
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy mgr create ceph-mgr-207

验证ceph-mgr节点:

[ceph@ceph-mgr-207 ~]$ ps -ef | grep ceph
ceph       1909      1  0 16:42 ?        00:00:00 /usr/bin/ceph-mgr -f --cluster ceph --id ceph-mgr-207 --setuser ceph --setgroup ceph

ceph-deploy安装管理ceph集群

在ceph-deploy节点配置-下系统环境，以方便后期可以执行ceph管理命令。

[ceph@ceph-deploy ceph-cluster]$ sudo yum install ceph-common -y
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy admin ceph-deploy
[ceph@ceph-deploy ceph-cluster]$ sudo setfacl -m u:ceph:rw /etc/ceph/ceph.client.admin.keyring

测试ceph命令

[ceph@ceph-deploy ~]$ ceph -s
  cluster:
    id:     2a7d86ce-a8bb-4b31-a445-bd9af8920912
    health: HEALTH_WARN
            OSD count 0 < osd_pool_default_size 3
 
  services:
    mon: 1 daemons, quorum ceph-mon-206
    mgr: ceph-mgr-207(active)
    osd: 0 osds: 0 up, 0 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   0 B used, 0 B / 0 B avail
    pgs:

准备OSD节点:

列出ceph node节点磁盘:

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk list ceph-osd-201
[ceph_deploy.conf][DEBUG ] found configuration file at: /home/ceph/.cephdeploy.conf
[ceph_deploy.cli][INFO  ] Invoked (2.0.1): /bin/ceph-deploy disk list ceph-osd-201
[ceph_deploy.cli][INFO  ] ceph-deploy options:
[ceph_deploy.cli][INFO  ]  username                      : None
[ceph_deploy.cli][INFO  ]  verbose                       : False
[ceph_deploy.cli][INFO  ]  debug                         : False
[ceph_deploy.cli][INFO  ]  overwrite_conf                : False
[ceph_deploy.cli][INFO  ]  subcommand                    : list
[ceph_deploy.cli][INFO  ]  quiet                         : False
[ceph_deploy.cli][INFO  ]  cd_conf                       : <ceph_deploy.conf.cephdeploy.Conf instance at 0x7f7ea2216320>
[ceph_deploy.cli][INFO  ]  cluster                       : ceph
[ceph_deploy.cli][INFO  ]  host                          : ['ceph-osd-201']
[ceph_deploy.cli][INFO  ]  func                          : <function disk at 0x7f7ea2261938>
[ceph_deploy.cli][INFO  ]  ceph_conf                     : None
[ceph_deploy.cli][INFO  ]  default_release               : False
[ceph-osd-201][DEBUG ] connection detected need for sudo
[ceph-osd-201][DEBUG ] connected to host: ceph-osd-201 
[ceph-osd-201][DEBUG ] detect platform information from remote host
[ceph-osd-201][DEBUG ] detect machine type
[ceph-osd-201][DEBUG ] find the location of an executable
[ceph-osd-201][INFO  ] Running command: sudo fdisk -l
[ceph-osd-201][INFO  ] Disk /dev/sda: 214.7 GB, 214748364800 bytes, 419430400 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdd: 107.4 GB, 107374182400 bytes, 209715200 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdh: 214.7 GB, 214748364800 bytes, 419430400 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdi: 214.7 GB, 214748364800 bytes, 419430400 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdc: 107.4 GB, 107374182400 bytes, 209715200 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdf: 107.4 GB, 107374182400 bytes, 209715200 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdg: 214.7 GB, 214748364800 bytes, 419430400 sectors
[ceph-osd-201][INFO  ] Disk /dev/sdb: 107.4 GB, 107374182400 bytes, 209715200 sectors
[ceph-osd-201][INFO  ] Disk /dev/sde: 107.4 GB, 107374182400 bytes, 209715200 sectors
[ceph-osd-201][INFO  ] Disk /dev/mapper/centos-root: 53.7 GB, 53687091200 bytes, 104857600 sectors
[ceph-osd-201][INFO  ] Disk /dev/mapper/centos-swap: 4160 MB, 4160749568 bytes, 8126464 sectors
[ceph-osd-201][INFO  ] Disk /dev/mapper/centos-home: 155.8 GB, 155818393600 bytes, 304332800 sectors

使用ceph-deploy disk zap擦除各ceph node的ceph数据磁盘:

# ceph-osd-201 ceph-osd-202 ceph-osd-203 的存储节点磁盘擦除过程如下:
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-201 /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-201 /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-201 /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-201 /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-201 /dev/sdf

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-202 /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-202 /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-202 /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-202 /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-202 /dev/sdf

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-203 /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-203 /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-203 /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-203 /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy disk zap ceph-osd-203 /dev/sdf

ceph-deploy disk zap ceph-osd-201 /dev/sdb
ceph-deploy disk zap ceph-osd-201 /dev/sdc
ceph-deploy disk zap ceph-osd-201 /dev/sdd
ceph-deploy disk zap ceph-osd-201 /dev/sde
ceph-deploy disk zap ceph-osd-201 /dev/sdf
ceph-deploy disk zap ceph-osd-202 /dev/sdb
ceph-deploy disk zap ceph-osd-202 /dev/sdc
ceph-deploy disk zap ceph-osd-202 /dev/sdd
ceph-deploy disk zap ceph-osd-202 /dev/sde
ceph-deploy disk zap ceph-osd-202 /dev/sdf
ceph-deploy disk zap ceph-osd-203 /dev/sdb
ceph-deploy disk zap ceph-osd-203 /dev/sdc
ceph-deploy disk zap ceph-osd-203 /dev/sdd
ceph-deploy disk zap ceph-osd-203 /dev/sde
ceph-deploy disk zap ceph-osd-203 /dev/sdf

添加OSD

数据分类保存方式:
1. Data:即ceph保存的对象数据
2. Block: rocks DB数据即元数据
3. block-wal:数据库的wal日志

添加OSD:

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-201 --data /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-201 --data /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-201 --data /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-201 --data /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-201 --data /dev/sdf

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-202 --data /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-202 --data /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-202 --data /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-202 --data /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-202 --data /dev/sdf

[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-203 --data /dev/sdb
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-203 --data /dev/sdc
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-203 --data /dev/sdd
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-203 --data /dev/sde
[ceph@ceph-deploy ceph-cluster]$ ceph-deploy osd create ceph-osd-203 --data /dev/sdf

ceph-deploy osd create ceph-osd-201 --data /dev/sdb
ceph-deploy osd create ceph-osd-201 --data /dev/sdc
ceph-deploy osd create ceph-osd-201 --data /dev/sdd
ceph-deploy osd create ceph-osd-201 --data /dev/sde
ceph-deploy osd create ceph-osd-201 --data /dev/sdf
ceph-deploy osd create ceph-osd-202 --data /dev/sdb
ceph-deploy osd create ceph-osd-202 --data /dev/sdc
ceph-deploy osd create ceph-osd-202 --data /dev/sdd
ceph-deploy osd create ceph-osd-202 --data /dev/sde
ceph-deploy osd create ceph-osd-202 --data /dev/sdf
ceph-deploy osd create ceph-osd-203 --data /dev/sdb
ceph-deploy osd create ceph-osd-203 --data /dev/sdc
ceph-deploy osd create ceph-osd-203 --data /dev/sdd
ceph-deploy osd create ceph-osd-203 --data /dev/sde
ceph-deploy osd create ceph-osd-203 --data /dev/sdf

设置 OSD服务自启动:

`查看osd`
[ceph@ceph-osd-201 ~]$ ps -ef | grep osd |column -t
2023  3197  1     0  09:30  ?      00:00:05  /usr/bin/ceph-osd  -f            --cluster  ceph  --id  0  --setuser  ceph  --setgroup  ceph
2023  3733  1     0  09:31  ?      00:00:06  /usr/bin/ceph-osd  -f            --cluster  ceph  --id  1  --setuser  ceph  --setgroup  ceph
2023  4261  1     0  09:31  ?      00:00:05  /usr/bin/ceph-osd  -f            --cluster  ceph  --id  2  --setuser  ceph  --setgroup  ceph
2023  4779  1     0  09:31  ?      00:00:05  /usr/bin/ceph-osd  -f            --cluster  ceph  --id  3  --setuser  ceph  --setgroup  ceph
2023  5293  1     0  09:31  ?      00:00:05  /usr/bin/ceph-osd  -f            --cluster  ceph  --id  4  --setuser  ceph  --setgroup  ceph
`设置osd自动启动`
[ceph@ceph-osd-201 ~]$ sudo systemctl enable ceph-osd@0 ceph-osd@1 ceph-osd@2 ceph-osd@3 ceph-osd@4
[ceph@ceph-osd-202 ~]$ sudo systemctl enable ceph-osd@5 ceph-osd@6 ceph-osd@7 ceph-osd@8 ceph-osd@9
[ceph@ceph-osd-203 ~]$ sudo systemctl enable ceph-osd@10 ceph-osd@11 ceph-osd@12 ceph-osd@13 ceph-osd@14

验证ceph集群

[ceph@ceph-deploy ceph-cluster]$ ceph -s
  cluster:
    id:     2a7d86ce-a8bb-4b31-a445-bd9af8920912
    health: HEALTH_OK
 
  services:
    mon: 1 daemons, quorum ceph-mon-206
    mgr: ceph-mgr-207(active)
    osd: 15 osds: 15 up, 15 in
 
  data:
    pools:   0 pools, 0 pgs
    objects: 0  objects, 0 B
    usage:   15 GiB used, 1.5 TiB / 1.5 TiB avail
    pgs: