KB045233 - Adding HBA and Port Buckets to CRUSH Map
Posted on September 7, 2020 by Brett Kelly
Adding HBA and Port Buckets to CRUSH Map
This article will walk through how to add "hba" and "port" buckets to an existing CRUSH map.
These CRUSH buckets then can be used as failure domains in a CRUSH rule.
e.g You want to create a ceph storage pool that will distribute objects or objects chunks across OSDs grouped by the HBA or card they attached to rather than at the host level.
- An existing ceph cluster built with 45Drives Storinators. (15,30,45,60)
- When using ceph-ansible-45d to build the cluster, this process is to be done after running the "core.yml" playbook BEFORE any pools are created.
- Non 45Drives brand servers are not supported in the process outlined below.
- Storinator Hybrid Servers are not currently supported in the process outlined below.
- Storinator "Base" AV15s are not currently supported in the process outlined below.
- OSD nodes must have the client.admin keyring present. They can be removed after the process is complete if necessary.
- If "ceph -s" errors on each OSD node, you need the client.admin keyring copied over from a mon node.
- The below steps are to be done on each storinator OSD node in the cluster.
- Remote into one of the OSD nodes
# curl -LO https://raw.githubusercontent.com/45Drives/tools/master/opt/tools/modify-crush-map.sh
- Run "modify-crush-map.sh"
- You must specify how many HBA cards are in the system. For example an Q30:
# sh modify-crush-map.sh -H 2
- Repeat these steps for each OSD node in the cluster
- Verify that the hba, port and OSD are created and moved by observing "ceph osd tree"
- If the script is terminated early or by accident, it is safe to just run again. It will not create or move and buckets or OSDs that are already in their intended location.
- Script Errors: