12 Object Storage ‐ basic - KleberVales/OCI-Architect-Associate GitHub Wiki

1. Object Storage

1.1 Introduction

1.2 OCI Storage Services at a Glance

1.3 Key Characteristics of Object Storage

1.4 Use Cases of Object Storage

2. Object Storage Resources

2.1 Introduction

2.2 Namespace

2.3 Objects and Metadata

2.4 Buckets

2.5 Object Storage Endpoint

2.6 Prefixes and Simulated Directory Structure

2.7 Benefits of Using Prefixes

3. Object Storage Characteristics

3.1 Introduction

3.2 Strong Consistency

3.3 High Durability

3.4 Data Integrity

3.5 Security

3.6 Scalability and Object Size

4. Multipart Uploads

4.1 Introduction

4.2 What Are Multipart Uploads?

4.3 Recommendations and Limits

4.4 Object Lifecycle Policy Rules

4.5 Multipart Uploads Using the API

4.6 Multipart Uploads Using the CLI

5. Managing Buckets and Objects

5.1 Introduction

5.2 Naming and Structural Rules

5.3 Object Immutability and Versioning

5.4 Pre-Authenticated Requests

5.5 Retention Rules

5.6 Replication Policy

5.7 Security Zones

5.8 Object Copy

6. Object Storage Tiers

6.1 Introduction

6.2 Purpose of Storage Tiers

6.3 Bucket Creation and Storage Tiers

6.4 Object-Level Storage Tier Assignment

6.5 How Storage Tiers Affect Cost

6.6 Standard Tier

6.7 Infrequent Access Tier (Cool Tier)

6.8 Archive Storage Tier

7. Auto-Tiering

7.1 Introduction

7.2 Recap: Standard vs. Infrequent Access Tiers

7.3 What Is Auto-Tiering?

7.4 How Auto-Tiering Works

7.5 Use Cases for Auto-Tiering

8. Object Lifecycle Management

8.1 Introduction

8.2 What Is Object Lifecycle Management?

8.3 Object Name Prefix and Pattern Matching

8.4 Rule Priority

8.5 Auto-Tiering and Lifecycle Rules

8.6 Limits and Archive Bucket Rules

8.7 Data Uploaded Before Policy Creation

8.8 Object Name Filters

8.9 Precedence of Rules During Evaluation

8.10 Supported Options in Lifecycle Policy Rules

.
.
.
.
.

1. Object Storage

In this lesson, we explored the overview of object storage within OCI. We learned that object storage and archive storage provide unlimited capacity with high durability, supporting objects up to 10 TiB. Object storage is a highly durable, scalable, secure, cost-efficient, internet-scale, and high-performance platform designed for 99.9% availability. It is ideal for unstructured data like images, videos, and analytic data, and supports private access through a Service Gateway. With multiple storage tiers—Standard, Infrequent Access, and Archive—it offers cost and performance flexibility. Finally, we reviewed key use cases, including storing application logs, serving as a primary data repository for big data, and preserving data for extended periods.

1.1 Introduction

Welcome to this lesson on the Overview of Object Storage. In this lesson, we will explore Oracle Cloud Infrastructure (OCI) storage services, with a particular focus on object storage and archive storage. Let's get started.

1.2 OCI Storage Services at a Glance

This lesson provides a summary of OCI storage services. The main focus will be on object storage and archive storage. These services offer:

Unlimited capacity with high durability
Ability to store objects up to 10 TiB (1 TiB equals nearly 1.1 TB)
Object storage is used for unstructured data such as images and videos
Archive storage is used for long-term archival needs

1.3 Key Characteristics of Object Storage

Object storage is a highly durable, scalable, secure, cost-efficient, internet-scale, and high-performance storage platform. It is also highly reliable, designed for 99.9% availability.

Key features include:

Unlimited data storage – You can store an unlimited amount of data
Thousands of buckets per account – Each bucket can host an unlimited number of objects
Object size flexibility – Objects can range from 0 bytes to 10 TiB
Designed for unstructured data – Ideal for images, videos, and analytic data
Regional service – Data can be accessed from anywhere using object storage endpoints
Private access support – Supports private access from OCI resources in a VCN through a Service Gateway, allowing connection to public endpoints using private IP addresses in a private subnet
Cost and performance flexibility – Offers multiple storage tiers:
- Standard
- Infrequent Access
- Archive

1.4 Use Cases of Object Storage

Object storage provides a highly available and durable content repository for data, images, logs, and videos. Common use cases include:

Storing application log data for analysis and troubleshooting
Optimized for unstructured and semi-structured data
Primary data repository for big data
Preserving data for longer periods of time

2. Object Storage Resources

2.1 Introduction

Object Storage is a cloud storage service designed to store large amounts of unstructured data (such as text files, images, videos, backups, system logs, and audio files).

Unlike a traditional hard drive (which organizes files into folders and subfolders hierarchically), Object Storage treats each file as an autonomous "object".

How does it work internally?

Each stored object consists of three main elements:

The Data: The file itself (e.g., a photo, a PDF document, or a compressed backup).
Metadata: A set of information that describes the object (such as creation date, size, file type, and custom tags).
A Unique Identifier (Key): A unique address or name that allows you to find and retrieve that file instantly.

Main Features

Flat Structure: Files are not saved in actual physical folders, but rather loosely stored within a large container called a Bucket. (As we saw in the previous lesson, the idea of "folders" that we see in the panel is just a visual simulation created using prefixes in the / slashes).
Virtually Infinite Scalability: You can start by storing a few megabytes and grow to petabytes of data without needing to resize disks or worry about running out of physical space.
High Durability: Data is automatically replicated across multiple locations to ensure you never lose information.
API/HTTP Access: Files are accessed and managed primarily through web requests, APIs, command line interface (CLI), or SDKs, rather than being mapped as a traditional network drive (like a hard drive connected to the computer).

2.2 Namespace

The first construct is the namespace.

Think of it as a top-level container for all buckets and objects
At account creation time, each OCI tenant is assigned one unique object storage namespace

2.3 Objects and Metadata

The data you store in object storage is stored as an object along with its metadata.

Objects are the actual data items you upload and retrieve
Each object carries metadata that describes it

2.4 Buckets

Next, we have the bucket, which is a logical container for storing objects.

Key rules about bucket naming:

The name for a bucket must be unique in a region within each tenancy
Example: You can create a bucket named demo-bucket in the India West (Mumbai) region
- You cannot create another bucket with the same name demo-bucket in the Mumbai region
- But you can create a bucket with the same name demo-bucket in the India South (Hyderabad) region

Important note: Within an object storage namespace, buckets and objects exist in a flat structure (flat hierarchy).

2.5 Object Storage Endpoint

Now, let's see how the object storage endpoint looks.

In the example:

Object name: oci.jpg
Bucket name: training
Namespace name: ociarchassociate

The endpoint is constructed as follows:

The object name comes after /o
The bucket name comes after /b
The namespace name comes after /n
You can also see the service name (object storage) and the region name in the endpoint

2.6 Prefixes and Simulated Directory Structure

We previously discussed that buckets and objects exist in a flat hierarchy. However, we can simulate a directory structure by adding a prefix string that includes one or more forward slashes to an object name.

Example:

Object name: oci.jpg
Prefix created: training/oci

So the full path becomes training/oci/oci.jpg, where training/oci is the prefix and oci.jpg is the object name.

2.7 Benefits of Using Prefixes

Can you think of any benefit of using prefixes? Here are some key advantages:

Bulk deletes – Perform bulk deletes of all objects using CLI or API
Bulk downloads – Perform bulk downloads of all objects using CLI or API
Hierarchical view – Use the console to display a hierarchical view of objects

3. Object Storage Characteristics

In this lesson, we examined the key characteristics of object storage in OCI. We learned that object storage provides strong consistency by always serving the most recent copy of data on read requests. It offers eleven 9s durability by storing objects redundantly across three availability domains (or three fault domains in single-AD regions). Data integrity is actively monitored using checksums, with corrupt data automatically detected and repaired without customer intervention. From a security perspective, all data is encrypted by default, tightly integrated with OCI IAM, and transferred over HTTPS via SSL endpoints. Finally, object storage is highly scalable, supporting unlimited data, thousands of buckets per account, unlimited objects per bucket, and object sizes ranging from 0 bytes to 10 TiB.

3.1 Introduction

Welcome to this lesson on Object Storage Characteristics. In this lesson, we will explore the key characteristics that make object storage in Oracle Cloud Infrastructure (OCI) a reliable, durable, and secure storage solution. Let's get started.

3.2 Strong Consistency

When a read request is made, object storage always serves the most recent copy of the data that was written to the system. This means object storage provides strong consistency.

Read requests always return the latest written data
No eventual consistency delays

3.3 High Durability

Object storage is designed to be highly durable, providing eleven 9s durability (99.999999999%).

How is this achieved?

Regions with multiple availability domains: Each object is stored redundantly across three different availability domains
Regions with a single availability domain: Each object is stored redundantly across three different fault domains

3.4 Data Integrity

Data integrity is actively monitored using checksums.

Corrupt data is detected and automatically repaired
Any loss in data redundancy is detected
Remediation is performed without customer intervention or impact

3.5 Security

From a security standpoint, object storage provides robust protection:

All stored data is encrypted by default
The service is tightly integrated with OCI Identity and Access Management (IAM)
Data is uploaded and downloaded over SSL endpoints using the HTTPS protocol

3.6 Scalability and Object Size

Object storage offers virtually limitless scalability:

You can store an unlimited amount of data
You can create thousands of buckets per account
Each bucket can host an unlimited number of objects
Stored objects can be as small as 0 bytes and as large as 10 tebibytes (TiB)

4. Multipart Uploads

4.1 Introduction

Instead of sending a large file all at once (which can easily fail if the connection drops), Multipart Upload splits the file into smaller pieces and sends them in parallel.

Main advantage: If sending a piece fails, you only need to resend the part that failed, not the entire file.
Flexibility: It's possible to pause the sending of specific parts and resume it later.

4.2 What Are Multipart Uploads?

Multipart uploads upload data in parallel, making them faster and more efficient than uploading a large object in a single upload.

Key benefit:

If a multipart upload fails for any reason, instead of restarting the entire object, you only need to retry uploading the part that failed

4.3 Recommendations and Limits

The recommendation is that if the object size is greater than 100 mebibytes (MiB), you should consider using multipart uploads.

Key limits and specifications:

An object can be divided into a maximum of 10,000 parts
Each part must be at least 10 MiB in size
The upper limit of an object part is 50 gibibytes (GiB)
You have the flexibility to pause uploads of individual parts and resume when schedule and resources allow

4.4 Object Lifecycle Policy Rules

You can use object lifecycle policy rules to automatically delete any uncommitted or failed multipart upload after a specific number of days.

This operation can be performed using the OCI Console

4.5 Multipart Uploads Using the API

Let's look at how multipart uploads can be performed using the Multipart Upload API. There are four main steps:

Step 1: Create the Parts

Before you initiate an upload, you need to create the parts.

Example: Object size is around 200 MB, split into 10 parts of 20 MB each

Step 2: Initiate an Upload

Make a CreateMultipartUpload REST API call.

Object storage provides an Upload ID
The upload is marked as active and remains active until you commit or abort it

Step 3: Upload Object Parts

Make an UploadPart request.

Object storage returns an entity tag (ETag) value for each part uploaded
When committing the upload, you require both the part number and the corresponding ETag value for each part

Step 4: Commit the Upload

Make a CommitMultipartUpload request.

Object storage reconstructs the object from its constituent parts

These are the steps required to perform a multipart upload using the Multipart Upload API.

4.6 Multipart Uploads Using the CLI

Now let's look at how to perform multipart uploads using the CLI. There are some key differences compared to using the API:

No need to split the object into parts – Just specify the part size and object storage will split the object into parts
Set maximum parallel parts – You can choose to set the maximum number of parts to be uploaded in parallel
No commit required – You don't have to perform any commit after the upload is complete

CLI Command Notes

You must specify the mandatory part size flag, and the value must be an integer
- This denotes the size of each part in mebibytes (MiB)
Optionally, use parallel upload count to set the maximum number of parallel uploads allowed

5. Managing Buckets and Objects

5.1 Introduction

Rules, resources, and advanced features for managing Buckets and Objects in OCI Object Storage.

5.2 Naming and Structural Rules

While specifying the bucket name or object name, please note the following:

Names are case sensitive
You cannot nest buckets – a bucket cannot contain other buckets
You can store objects up to 10 tebibytes (TiB)

5.3 Object Immutability and Versioning

Important points about working with objects:

You cannot edit or append data to an object
However, you can replace the entire object
You can enable object versioning to retain previous versions of objects

5.4 Pre-Authenticated Requests

There is a concept of pre-authenticated requests (PARs) that provide a way to let you access a bucket or an object without having your own credentials.

Useful for granting temporary access to specific resources

5.5 Retention Rules

You can apply retention rules at the bucket level to provide immutable object storage options for data that is written to object storage.

Use cases for retention rules:

Governance
Regulatory compliance
Legal requirements

5.6 Replication Policy

Using a replication policy for a bucket, you can automatically replicate the objects in one object storage bucket to another bucket in the same or different region.

5.7 Security Zones

Security zones ensure that your cloud resources comply with Oracle security principles.

Example: Buckets in a security zone must be private

5.8 Object Copy

You can copy objects to other buckets in the same region and to buckets in other regions using the object copy feature.

5.9 Object Lifecycle Policies

Using object lifecycle policies, you can automatically manage the archiving and deletion of objects according to a predefined schedule.

6. Object Storage Tiers

6.1 Introduction

This lesson covers the Storage Tiers of OCI Object Storage, which serve to balance access performance and storage cost according to the frequency of data usage.

6.2 Purpose of Storage Tiers

Storage tiers help you:

Maximize access performance where appropriate
Minimize storage costs

6.3 Bucket Creation and Storage Tiers

There are two options at bucket creation level with respect to storage tiers:

Standard storage tier bucket – Can contain a mix of objects with different storage tier assignments
Archive storage tier bucket – Can only contain objects with an archive storage tier assignment

Important notes:

When creating a bucket, you cannot change the default storage tier for the bucket
You specify the default storage tier during bucket creation (either Standard or Archive)

6.4 Object-Level Storage Tier Assignment

If the object is being uploaded to a standard default storage tier bucket, you can explicitly assign any storage tier to the object:

Standard
Infrequent Access
Archive

Note: You can also change the storage tier for an object, but this is only possible in the case of a standard storage tier bucket.

6.5 How Storage Tiers Affect Cost

Every object uploaded to object storage is assigned to a storage tier, which determines:

Its storage cost
Associated retrieval fees

There are three storage tiers: Standard, Infrequent Access, and Archive.

6.6 Standard Tier

Default storage tier
Used for data you need to access quickly, immediately, and frequently
Has higher storage cost compared to other tiers
When you choose a standard default storage tier during bucket creation, you can also enable auto-tiering

Use cases:

Content repository for scalable data (images, logs, videos)
Data repository for Hadoop or big data workloads

6.7 Infrequent Access Tier (Cool Tier)

Used for data that you access infrequently, but that must be available immediately when needed
Storage costs are lower than standard
Has a minimum storage retention period of 31 days
You will be charged the prorated cost of storing the data for the full 31 days if you delete or overwrite objects in this tier
When you need to access objects, you are charged a per GB data retrieval fee

Use cases:

Backup for on-premises data

6.8 Archive Storage Tier

Meant for data that is rarely accessed, but must be retained for long periods of time
Archive storage tier buckets can only contain objects with archive storage tier assignment
Provides the lowest storage cost
Objects must be restored before they are available for access
Minimum storage retention period is 90 days
When you restore objects, you are returning those objects to the standard tier for access
Restore time is close to an hour

Use cases:

Compliance and audit mandates
Storing historical data

7. Auto-Tiering

7.1 Introduction

This lesson covers the Auto-Tiering feature in OCI Object Storage, which is used to dynamically manage the movement of files between tiers without manual effort.

1. What is Auto-Tiering?

It's a bucket-level enabled feature that monitors data access patterns and automatically moves files between the Standard (frequent) and Infrequent Access tiers, saving you time you would spend writing complex lifecycle policies.

7.2 Recap: Standard vs. Infrequent Access Tiers

Just to reinforce what we learned in the previous lesson:

Standard Storage Tier:

Optimized for data you need to access quickly, immediately, and frequently
No retrieval charges
No retention period requirements

Infrequent Access Storage Tier:

Ideal for data that is accessed infrequently
Lower storage costs compared to standard
Comes with a minimum storage retention period of 31 days
Data retrieval fees apply

7.3 What Is Auto-Tiering?

Auto-tiering determines which storage tier your data should be in based on your usage patterns.

Key benefit:

Saves time from writing complex lifecycle policy rules
Eliminates deliberating over which storage tier is best for your data

7.4 How Auto-Tiering Works

As shown in the diagram:

Data is initially in the frequent access (standard) tier
Based on the access pattern, the service moves it to the infrequent access tier if it is infrequently accessed
Later, if the objects are accessed more frequently, it is moved back to the standard tier

Important notes:

This feature is enabled at a bucket level
If you enable auto-tiering, data access patterns are monitored
The feature does not cost anything
Instead, it helps you reduce costs by automatically moving objects larger than 1 MiB from standard to the more cost-effective infrequent access tier
It performs the transition of objects based on access patterns
No retrieval charges apply
Storage costs are pro-rated

7.5 Use Cases for Auto-Tiering

Auto-tiering is particularly useful in the following scenarios:

New application data storage – When there are no established access patterns
Data storage with changing access patterns – When usage patterns shift over time

8. Object Lifecycle Management

8.1 Introduction

8.2 What Is Object Lifecycle Management?

Object lifecycle management lets you manage the lifecycle of your object storage data through automated archiving and deletion, reducing storage costs and saving time.

Works by taking automated actions based on the rules you define

Supported lifecycle actions:

Move to Infrequent Access
Move to Archive
Delete

Supported resources:

Objects
Object versions
Uncommitted or failed multipart uploads

8.3 Object Name Prefix and Pattern Matching

Object-related rules can have an object name prefix and pattern matching condition to narrow the scope of individual lifecycle policy rules.

8.4 Rule Priority

A rule that deletes an object always takes priority over a rule that would move that same object to another storage tier

8.5 Auto-Tiering and Lifecycle Rules

If auto-tiering is enabled, you cannot create a rule that moves objects to the Infrequent Access tier

8.6 Limits and Archive Bucket Rules

You can create up to 1,000 lifecycle rules per bucket
You can also create lifecycle policies on an archive storage bucket
- However, only delete rules are supported

8.7 Data Uploaded Before Policy Creation

Question: What happens to the data uploaded to the bucket before policy creation?

Answer: Lifecycle policies apply to data uploaded before the policy was created.

Example:

A lifecycle policy rule archives all objects over 30 days old
The bucket contains objects that are 40 days old
Those objects are identified immediately by the service as candidates for archiving, and the archiving process begins

8.8 Object Name Filters

There are two types of object name filters:

1. Prefix Matching

Exact matching of the left-most characters of an object name
Does not support wildcard characters
Example: training/oci can serve as a prefix for matching purposes in training/oci/os.jpg

2. Pattern Matching

Matching happens on the entire object name
Supports wildcard characters
Example: *.jpg can match oci.jpg and also training/oci/test.jpg

8.9 Precedence of Rules During Evaluation

Let's discuss the precedence of rules during evaluation of object name filters. There are three types of filters:

Include by prefix
Include by pattern
Exclude by pattern

Precedence order (first to last):

Pattern exclusions
Pattern inclusions
Prefix inclusions

8.10 Supported Options in Lifecycle Policy Rules

The supported options in lifecycle policy rules include:

Move or delete all objects in the bucket
Move or delete objects in the bucket that match the object name filters you specify
- Can use both object name prefixes and pattern matching
Delete uncommitted or failed multipart uploads
If object versioning is enabled or suspended on a bucket:
- Create lifecycle rules that move or delete previous versions of all objects in the bucket
- Also applies to the latest version of objects
- Can also specify object name filters

Example Lifecycle Policy

Consider the following example:

Move standard tier objects to the archive tier 30 days after creation or last update
After 180 days, automatically delete those archived objects
Additionally, create another lifecycle policy rule that deletes uncommitted or failed multipart uploads after 5 days