KB450157 – Repairing Damaged Objects

Last modified: July 3, 2019
You are here:
Estimated reading time: 1 min

Repairing Damaged Objects

This page will walk through how to fix a damaged object when the cluster health message below occurs.

  cluster:
    id:     cec9ca98-b59f-4d91-8ddd-43802195c735
    health: HEALTH_ERR
            1 scrub errors
            Possible data damage: 1 pg inconsistent
   ....
  data:
    pools:   10 pools, 1120 pgs
    objects: 29.66 k objects, 99 GiB
    usage:   320 GiB used, 7.7 TiB / 8.0 TiB avail
    pgs:     1119 active+clean
             1    active+clean+inconsistent
  • To find the damaged PG run “ceph health detail”
    • OSD_SCRUB_ERRORS 1 scrub errors
      PG_DAMAGED Possible data damage: 1 pg inconsistent
          pg 13.6 is active+clean+inconsistent, acting [4,18,14]
  • The problematic PG is pg 13.6
  • Attempt to repair the PG
    • ceph pg repair 13.6
    • Watch that the a repair has begun in either the UI or terminal
      •   data:
            pools:   10 pools, 1120 pgs
            objects: 29.66 k objects, 99 GiB
            usage:   320 GiB used, 7.7 TiB / 8.0 TiB avail
            pgs:     1119 active+clean
                     1    active+clean+scrubbing+deep+inconsistent+repair
    • If successful the cluster will return to a healthy state.
Was this article helpful?
Dislike 1
Views: 105
Unboxing Racking Storage Drives Cable Setup Power UPS Sizing Remote Access