45Drives Knowledge Base
KB450472 - Intelligent Tiering with Lifecycle Management on Ceph (S3)
https://knowledgebase.45drives.com/kb/kb450472-intelligent-tiering-with-lifecycle-management-on-ceph-s3/

KB450472 - Intelligent Tiering with Lifecycle Management on Ceph (S3)

Posted on September 7, 2021 by Mitchell Hall


Scope/Description

Prerequisites

Steps

Creation of new storage class and pool

Note: If you only wish to create a rule that expires and deletes objects, rather than moving them to a new storage tier, this section can be skipped, and you can move directly to “Creating S3 lifecycle management policies”

root@labosd1:~# radosgw-admin zonegroup list

root@labosd1:~# radosgw-admin zone list

root@labosd1:~# radosgw-admin zonegroup placement list

Add a new storage class

root@labosd1:~# radosgw-admin zonegroup placement add \

--rgw-zonegroup default \

--placement-id default-placement \

--storage-class COLD

root@labosd1:~# radosgw-admin zone placement add \

--rgw-zone default \

--placement-id default-placement \

--storage-class COLD \

--data-pool default.rgw.cold.data \

--compression lz4

Creating S3 Lifecycle management policies

root@octrgw1:~# apt install s3cmd

root@octrgw1:~# s3cmd –configure

Lifecycle configuration

<LifecycleConfiguration>

  <Rule>

    <ID>Transition then Expire Rule</ID>

    <Filter>

       <Prefix></Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Transition>

      <Days>30</Days>

      <StorageClass>COLD</StorageClass>

    </Transition>

    <Expiration>

      <Days>60</Days>

    </Expiration>

  </Rule>

</LifecycleConfiguration>
<LifecycleConfiguration>

  <Rule>

    <ID>Expire after 1 year</ID>

    <Filter>

       <Prefix></Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Expiration>

      <Days>365</Days>

    </Expiration>

  </Rule>

</LifecycleConfiguration>
<LifecycleConfiguration>

  <Rule>

    <ID>Transition objects to COLD storage class</ID>

    <Filter>

       <Prefix></Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Transition>

      <Days>30</Days>

      <StorageClass>COLD</StorageClass>

    </Transition>

  </Rule>

</LifecycleConfiguration>
<LifecycleConfiguration>

  <Rule>

    <ID>prefix expiration</ID>

    <Filter>

       <Prefix>dir1</Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Expiration>

      <Days>120</Days>

    </Expiration>

    <Filter>

       <Prefix>dir2</Prefix>

    </Filter>

    <Status>Enabled</Status>

    <Expiration>

      <Days>60</Days>

    </Expiration>

  </Rule>

</LifecycleConfiguration>

 

root@octrgw1:~# s3cmd setlifecycle lifecycle-expire.xml s3://lifecycletest

root@octrgw1:~# s3cmd info s3://lifecycletest

When using transition rules instead of expire rules, s3cmd info will not properly list it. You will need a tool like S3 browser to view the lifecycle rule. S3 browser can also be used to create these rules. The only downside of using S3 Browser is that it does not allow the use of custom storage classes such as the use of “COLD”. It uses only standard Amazon S3 naming, so when using transition rather than expiration it is still better to use s3cmd and the XML files.

Verification

rgw lc debug interval = 60
root@octrgw1:~# systemctl restart ceph-radosgw@rgw.octrgw1

Troubleshooting

root@octrgw1:~# radosgw-admin lc list
root@octrgw1:~# radosgw-admin lc process
Ceph RGW Archives - 45Drives Knowledge Base
Did you know  45Drives offers free  public and private  webinars ? Click here to learn more  & register! Build & Price

KB450472 – Intelligent Tiering with Lifecycle Management on Ceph (S3)

Scope/Description This guide runs through in detail what lifecycle management is, how to deploy it, and how to create custom storage classes with RGW’s. Prerequisites This guide assumes you have a Ceph cluster deployed with 2 RGW’s in place and the required RGW pools created such as buckets.data, buckets.index, buckets.non-ec, rgw.log rgw.meta. If this is […]

Read more »

KB450295 – Creating & Using Ceph S3 Buckets

Scope/Description This article will provide a step-by-step on how to create a S3 user, creating a S3 bucket and then how to link that bucket to either push or pull data. Prerequisites Ceph Cluster An S3 service to make use of Ceph’s S3 offering, such as Amazon, Azure, etc. Rados Gateway services must be installed […]

Read more »

KB450270 – Configuring Ceph RGW to use SSL with HAProxy

Scope/Description This article will walk through the process of configuring Ceph RGW to use SSL with HAProxy Loadbalancer. Prerequisites Ceph Cluster running either Nautilus(v14) or Octopus(v15) RGW gateways configured and operating ceph-ansible-45d version 1.4.2 or greater SSL Certificate Steps Obtain SSL Certificate Run these commands on the ansible master node. When generating the Certificate Signing […]

Read more »

KB450422 – Configuring Ceph Object Storage to use Multiple Data Pools

Scope/Description This guide will run through the process of updating the default data pool that Ceph RGW uses to store object data for buckets. If your cluster is only going to use a single EC pool to store your object data on, you can simply update the default data placement policy, however, if the end-user […]

Read more »

KB450421 – Ceph RGW Garbage Collection

Scope/Description This article discusses the key garbage collection tunables to play with with large delete workloads on a RGW Ceph cluster Prerequisites Ceph cluster with RGW(s) deployed Steps The tunables for garbage collection below. Here is description of each For heavy delete RGW workloads it is recommended to start with the following. rgw_gc_max_concurrent_io = 20 […]

Read more »
© 2024 - 45Drives Knowledge Base
Unboxing Racking Storage Drives Cable Setup Power UPS Sizing Remote Access