KB450110 – Updating Ceph

See here for detailed explanation of the Ceph release cycle.

To summarize there are (2) types of a Ceph cluster update.

Both can be completed without cluster downtime, but release notes should be reviewed in both cases.

1. Minor Updates
2. Major Updates

Major Updates are released every 9 months, they are the most invasive of the update procedures. Editing of ansible group_vars required. It is recommended to read the release notes and highlight any potential problem areas in your environment.

Ansible is required to complete an automated install. It updates each service one at a time, one node at a time

Services are updated in the following order:

1. Monitor services
2. OSD services
3. MDS services
4. RGW services

Minor Updates are minor bug fixes released every 4-6 months. These updates are quick and can be done safely with “yum update.”

To apply minor Ceph cluster updates run:

yum update

If a new kernel is installed, a reboot will be required to take effect. If there is no kernel update you can stop here.

Set osd flag noout and norebalance to prevent the rest of the cluster from trying to heal itself while the node reboots

ceph osd set flag noout
ceph osd set flag norebalance

Then reboot each node one at a time. Do not reboot the next node until the one before is back up and in the cluster.

reboot

After each node is rebooted unset the flags set earlier and you are all done.

ceph osd unset flag noout
ceph osd unset flag norebalance

Major updates are applied with Ansible.

To upgrade to the next major release edit the group_vars/all.yml in your ceph-ansible-45d directory.

In the INSTALL heading, find the line ceph_stable_release, replace the existing release with the next stable release.

For example - Updating from Luminous (12.2.X) to Mimic (13.2.X)
vim ~/ceph-ansible-45d/group_vars/all.yml
> Change "ceph_stable_release: luminous"
> To     "ceph_stable_release: mimic"

Now run the “rolling-updates.yml” playbook

cd ~/ceph-ansible-45d/
ansible-playbook infrastructure-playbooks/rolling_update.yml

It will take some time but all the data is up and accessible during the update. For detailed information on what Ansible is doing see this blog.

NOTE: Updating from Luminous to Mimic will require manual intervention to get the dashboard back up and running.

Verify you are running the new version with the following command:

ceph versions
----------
{
  "mon": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
  },
  "mgr": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
  },
  "osd": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 44
  },
  "mds": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 3
  },
  "rgw": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 1
  },
  "overall": {
      "ceph version 13.2.0 (79a10589f1f80dfe21e8f9794365ed98143071c4) mimic (stable)": 54
  }
}