LAB 17 ‐ Object Storage Configuration - gautam-amitesh/OCI GitHub Wiki
There are two types of data Structured Data and Unstructured Data. Structured data uses a predefined and expected format which means the fields are fixed. Some common usecases include CRM (Customer Relationship Management) forms, online transactions, and stock data. Unstructured data lacks definition which come in many forms to be stored as objects: images, audio, video, document files, and other file formats. Unlike Structured data we can't search for a specific file based on its actual content in the database. It's not surprising that this type typically makes up about 80% of an organization's data. Storing these two distinct data types in block storage will be expensive so there is a popular method of cloud storage which is often used for storing unstructured data called Object Storage also called object-based storage, this method is scalable (elastic storage) and efficient (store infinite amount of data at comparitively lower cost without performance degradation). It is used particularly for backup, archiving, or any instances where data does not have to be dynamically updated (eg. data of ex-patients which might be used in future for study purpose which is currently of no use). Unlike Block Storage where data is stored in fixed-size blocks in Object Storage data is stored as objects in buckets. Containers holding data are known as buckets.
-
Cost efficient (Block volume vs Object storage)
-
Block Volume (200GB + Ultra high performance)
-
Object Storage (Archive + 200GB)
-
Object Storage (Infrequent access + 200GB)
-
Object Storage (Standard + 200GB)
-
Bucket Creation
-
Navigation
Navigation menu > Storage > Object Storage & Archive Storage > Buckets -
Click
Create bucket -
On Create bucket page fill Bucket name
-
Select Bucket scope (Scope in which the bucket is unique)
Namespace: Default value is NAMESPACE. Bucket scope as NAMESPACE means that the bucket is unique only in the owning namespace/tenancy. Other tenancies can have a bucket with same name in their namespace. To find your namespace navigate to tenancy details and find Object storage namespace.Region: Bucket scope as REGION means that the bucket is regionally unique. No other tenancy can have a bucket with same name and scope REGION.
-
Select Default storage tier
Standard: The Standard storage tier is "hot" storage used for data that you need to access quickly, immediately, and frequently.Archive: The Archive Storage tier is "cold" storage used for rarely access, but that must be retained and preserved for long periods of time.
-
If
Standardis selected in Default storage tier then Enable auto-tiering option appears enable that for automatically moving infrequently accessed objects from the Standard tier to less expensive storage. -
Enable object versioning for creating an object version when a new object is uploaded, an existing object is overwritten, or when an object is deleted.
-
Enable Emit object events for creating automation based on object state changes using the Events Service.
-
Enable Uncommitted multipart uploads cleanup for creating a lifecycle rule to automatically delete uncommitted multipart uploads (uploading large files in chunks) older than 7 days
-
Select Encryption
-
Click
Create bucketat the bottom right -
You will see that bucket is successfully created
Uploading Objects
-
Navigation
Bucket details (bucket-20260724-0945) > Objects -
Click
Upload objects -
On Upload objects page provide Object name prefix for more accessibility but this field is not required
-
Select Storage tier
Standard: "hot" storage used for data that you need to access quickly, immediately, and frequently.Infrequent access: "cool" storage used for data that you access infrequently, but that must be available immediately when needed.Archive: "cold" storage used for data seldom or rarely access, but that must be retained and preserved for long periods of time.
-
Choose Files from your Computer (you can choose multiple files) and click
Next -
Review your inputs and click
Upload objects -
Once you see your files are done uploading click
Close -
Now you will be able to see your uploads inside the bucket also you can notice the
prefixis added before the object name
Sharing Objects
There are two ways to share objects:
- Change Visibility from
PrivatetoPublicbut enabling public visibility will let anonymous and unauthenticated users access data stored in the bucket. - Create Pre-authenticated requests (PAR)
To enable public visibility head over to bucket details page from there click on Actions dropdown then click on Edit visibility and select Public and click Update.
Creating pre-authenticated request
-
Navigation
From bucket details page switch to Management section
-
Click
Create pre-authenticated request -
On Create pre-authenticated request page fill the Name
-
Select Pre-authenticated request target if you select
Objectit will ask you to provide the particular object name -
Select Access type and Expiration
-
Click
Create pre-authenticated request -
Pre-authenticated request details page will now show up from there copy Pre-authenticated request URL for your records. It will not be shown again and click
Close -
Now you can paste that URL in browser and it will open that object
Creating Lifecycle Policy Rules
-
Navigation
From bucket details page switch to Policies section
-
Under Lifecycle policy rules click
Create rule -
On the Create lifecycle rule page scroll down if you get an alert saying "Are you missing IAM policy statements in your tenancy?" then click
Try to add statementsorCopy statements to clipboardand manually add it. For me it was throwing ERROR so I manually added it by modifying the policy statement and selectedroot compartmentAllow service objectstorage-us-ashburn-1 to manage object-family in tenancy -
After you add IAM policy you will not get that alert to verify go back and try creating lifecycle rule again
-
Type the Name for rule
-
Select Target
-
Select Lifecycle action
-
Type Number of days and click
Create rule -
Later if you want to edit/disable/delete rule you can do it by clicking on
Actions/three dot and thenEdit rule
Creating Retention Rules
If you have active retention rules, you can’t update, overwrite, or delete objects and their metadata, or delete buckets, until the retention duration expires, or the retention rule is deleted.
Note: If bucket has versioning enabled (BucketVersioningEnabled) you cannot create retention rule. To suspend Object versioning head over to bucket details page and click on Suspend. If you need you can re-enable it from the same place.
-
Navigation
From bucket details page switch to Policies section
-
Scroll down and under Retention rules click
Create rule -
On the Create retention rule page type Name
-
Select Retention rule type
- Time-bound: Object modification is prevented for the retention duration you specify.
- Indefinite: Object modification is prevented until you delete the retention rule.
-
If selected Time-bound then specify Retention duration (Days/Years)
-
While enabling retention rule lock be very carefull because when a rule is locked, only an increase in the retention duration is allowed and the rule can only be deleted by deleting the bucket. A bucket must be empty to be deleted. And to empty a bucket you will need to wait untill the retention period expires.
-
Click
Create rule