Scope/Description
- This article walks through the process of adding an external Ceph Cluster to a Proxmox Virtual Enviroment for Virtual Machine (VM) Image storage.
Prerequisites
- A running Ceph Cluster
- A storage pool created for rbd storage
- A running Proxmox 6.2 or greater environment
Steps for EC Pools
- The process differs depending on storage pool type, replicated or erasure code.
- Note that you can add as many replicated pools as you want, only one erasure code pool is supported.
- On the Proxmox server ensure the following directories exist. Create them if not
- /etc/ceph
- /etc/pve/priv/ceph
- Copy the ceph.conf and ceph.client.admin.keyring into /etc/ceph from a ceph monitor to the proxmox node
[root@monitor~]# scp ceph.conf root@<proxmoxhost>:/etc/ceph [root@monitor~]# scp ceph.client.admin.keyring root@<proxmoxhost>:/etc/ceph
- First, we’ll cover adding an Erasure Code pool for storage into Ceph.
- Since Erasure Code RBDs store there metadata in a replicated rbd pool, two pools are needed to complete the following steps.
- pve_rbd. A replicated rbd pool to store the rbd metadata.
- pve_rbd_ec. The dedicated EC data pool to store the actual rbd data.
- SSH into Proxmox host, or use Proxmox GUI shell
- Copy the ceph.conf into /etc/pve/priv/ceph, changing the name of the file to the name of the rbd pool. In the below example the rbd pool name is “pve_rbd_ec”
[root@proxmox~]# cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/pve_rbd_ec.conf
- Copy the admin key to /etc/pve/priv/ceph, changing the name of the file to the name of the rbd pool. In the below example the rbd pool name is “pve_rbd_ec”
[root@proxmox~]# cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/pve_rbd_ec.keyring
- Open this file in a text editor and add the bolded lines at the bottom of the file. In the below example the rbd Erasure Code pool is called “pve_rbd_ec”.
[global] cluster network = 192.168.0.0/16 fsid = cc1be5c3-7b91-4226-9732-22f43b69d34e mon host = [v2:192.168.20.11:3300,v1:192.168.20.11:6789],[v2:192.168.20.12:3300,v1:192.168.20.12:6789],[v2:192.168.20.13:3300,v1:192.168.20.13:6789] mon initial members = zmon1,zmon2,zmon3 osd pool default crush rule = -1 public network = 192.168.0.0/16 [client] rbd default data pool = pve_rbd_ec
Note: For Proxmox 7.0, this field should be included under [global] without [client]
Example below.
[global] cluster network = 192.168.0.0/16 fsid = cc1be5c3-7b91-4226-9732-22f43b69d34e mon host = [v2:192.168.20.11:3300,v1:192.168.20.11:6789],[v2:192.168.20.12:3300,v1:192.168.20.12:6789],[v2:192.168.20.13:3300,v1:192.168.20.13:6789] mon initial members = zmon1,zmon2,zmon3 osd pool default crush rule = -1 public network = 192.168.0.0/16 rbd default data pool = pve_rbd_ec
- Access the webUI to now create the Ceph RBD storage there.
- From the webUI, click on Datacenter, then Storage, and finally Add and RBD:
- In the dialog box that pops up add the ID. This must match the name of the keyring file you created in /etc/pve/priv/ceph.
- Add the name of the pool. This is the “pve_rbd” replicated pool, it will use “pve_rbd_ec” as the default data pool.
- Add the monitor hosts. Use IP addresses unless you have a DNS that can translate the FQDN.
- Add the user name. In this case it is admin.
- Check off KRBD.
- The Proxmox host can now use Ceph RBD’s to create disks for VM’s
ProxMox 8
Run the following from one of the proxmox host
pvesm add rbd <storage-name> –pool <replicated-pool> –data-pool <ec-pool>
Add the mon ips in the proxmox UI under the newly added RBD under storage
Copy the keyring to “/etc/pve/priv/ceph.client.admin.keyring” and to “/etc/pve/priv/ceph” and name the keyring your ec pool name (ID)
Steps for Replicated Pools
- Next, we’ll cover Replicated pools. At least one replicated pool is needed to complete the following steps.
- pve_rbd. A replicated rbd pool to store the rbd metadata and rbd data.
- SSH into Proxmox host, or use Proxmox GUI shell
- Copy the ceph.conf into /etc/pve/priv/ceph, changing the name of the file to the name of the rbd pool. In the below example the rbd pool name is pve_rbd.
[root@proxmox~]# cp /etc/ceph/ceph.conf /etc/pve/priv/ceph/pve_rbd.conf
- Copy the admin key to /etc/pve/priv/ceph, changing the name of the file to the name of the rbd pool. In the below example the rbd pool name is pve_rbd.
[root@proxmox~]# cp /etc/ceph/ceph.client.admin.keyring /etc/pve/priv/ceph/pve_rbd.keyring
- Access the webUI to now create the Ceph RBD storage there.
- From the webUI, click on Datacenter, then Storage, and finally Add and RBD:
- In the dialog box that pops up add the ID. This must match the name of the keyring file you created in /etc/pve/priv/ceph.
- Add the name of the pool.
- Add the monitor hosts. Use IP addresses unless you have a DNS that can translate the FQDN.
- Add the user name. In this case it is admin.
- Check off KRBD.
- The Proxmox host can now use Ceph RBD’s to create disks for VM’s
Verification
- After creating a disk, verify the ec pool is set as the rbd data pool. The naming convention of proxmox rbds is “VM-ID-DISK#”. In the below example the VM ID is 133 and its the second disk attached to the VM. Bolded below, you can see the data_pool is correctly set to the ec pool
[root@proxmox~]# rbd info vm-133-disk-1
rbd image 'vm-133-disk-1':
size 32 GiB in 8192 objects
order 22 (4 MiB objects)
snapshot_count: 0
id: 65cfae27458d39
data_pool: rbd_ec
block_name_prefix: rbd_data.1.65cfae27458d39
format: 2
features: layering, exclusive-lock, object-map, fast-diff, deep-flatten, data-pool
op_features:
flags:
create_timestamp: Wed Oct 14 16:11:36 2020
access_timestamp: Wed Oct 14 16:11:36 2020
modify_timestamp: Wed Oct 14 16:11:36 2020
Troubleshooting
Views: 7587