CB‐Spider Object Storage API Guide - cloud-barista/cb-spider GitHub Wiki

CB-Spider Object Storage API Guide

CB-Spider provides AWS S3-compatible Object Storage API that enables unified management of Object Storage across various Cloud Service Providers (CSPs).

Key Features

  • Multi-Cloud Support: Unified management of Object Storage across AWS, IBM Cloud, GCP, and other CSPs
  • AWS S3 Compatibility: Provides compatibility with existing AWS S3 tools and SDKs
  • PreSigned URL REST API: Provides PreSigned URL generation REST API
  • Simple Authentication: Easy authentication with ConnectionName parameter

API List

1. Bucket Management

Function HTTP Method Endpoint Example
List Buckets GET /spider/s3 Example
Create Bucket PUT /spider/s3/{bucket-name} Example
Get Bucket Info GET /spider/s3/{bucket-name} Example
Check Bucket Exists HEAD /spider/s3/{bucket-name} Example
Get Bucket Location GET /spider/s3/{bucket-name}?location Example
Delete Bucket DELETE /spider/s3/{bucket-name} Example

2. Object Management

Function HTTP Method Endpoint Example
Upload Object (File) PUT /spider/s3/{bucket-name}/{object-key} Example
Upload Object (Form) POST /spider/s3/{bucket-name} Example
Download Object GET /spider/s3/{bucket-name}/{object-key} Example
Get Object Info HEAD /spider/s3/{bucket-name}/{object-key} Example
Delete Object DELETE /spider/s3/{bucket-name}/{object-key} Example
Delete Multiple Objects POST /spider/s3/{bucket-name}?delete Example

3. Multipart Upload

Function HTTP Method Endpoint Example
Initiate Multipart Upload POST /spider/s3/{bucket-name}/{object-key}?uploads Example
Upload Part PUT /spider/s3/{bucket-name}/{object-key}?uploadId={id}&partNumber={num} Example
Complete Multipart Upload POST /spider/s3/{bucket-name}/{object-key}?uploadId={id} Example
Abort Multipart Upload DELETE /spider/s3/{bucket-name}/{object-key}?uploadId={id} Example
List Parts GET /spider/s3/{bucket-name}/{object-key}?uploadId={id}&list-type=parts Example
List Multipart Uploads GET /spider/s3/{bucket-name}?uploads Example

4. Versioning Management

Function HTTP Method Endpoint Example
Get Bucket Versioning GET /spider/s3/{bucket-name}?versioning Example
Set Bucket Versioning PUT /spider/s3/{bucket-name}?versioning Example
List Object Versions GET /spider/s3/{bucket-name}?versions Example
Delete Versioned Object DELETE /spider/s3/{bucket-name}/{object-key}[?versionId=version-id] Example

5. CORS Management

Function HTTP Method Endpoint Example
Get Bucket CORS GET /spider/s3/{bucket-name}?cors Example
Set Bucket CORS PUT /spider/s3/{bucket-name}?cors Example
Delete Bucket CORS DELETE /spider/s3/{bucket-name}?cors Example

6. CB-Spider Special Features

Function HTTP Method Endpoint Example
Generate PreSigned URL (Download) GET /spider/s3/presigned/download/{bucket-name}/{object-key} Example
Generate PreSigned URL (Upload) GET /spider/s3/presigned/upload/{bucket-name}/{object-key} Example
Force Empty Bucket DELETE /spider/s3/{bucket-name}?empty Example
Force Delete Bucket DELETE /spider/s3/{bucket-name}?force Example

Note: CB-Spider special features are unique to CB-Spider and not part of AWS S3 standard.

API Usage Examples

1. Bucket Management

List Buckets

CB-Spider Standard Method

curl -X GET "http://localhost:1024/spider/s3?ConnectionName=aws-config01"

AWS S3 Standard Method

curl -X GET "http://localhost:1024/spider/s3" \
  -H "Authorization: AWS4-HMAC-SHA256 Credential=aws-config01"

Note: AWS S3 standard method will be omitted in subsequent examples for brevity.

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<ListAllMyBucketsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Owner>
    <ID>aws-config01</ID>
    <DisplayName>aws-config01</DisplayName>
  </Owner>
  <Buckets>
    <Bucket>
      <Name>spider-test-bucket</Name>
      <CreationDate>2025-09-04T04:18:06Z</CreationDate>
    </Bucket>
  </Buckets>
</ListAllMyBucketsResult>

Create Bucket

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response:

HTTP/1.1 200 OK

Get Bucket Info

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>spider-test-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
</ListBucketResult>

Check Bucket Exists (HEAD)

curl -I "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response for existing bucket:

HTTP/1.1 200 OK
Vary: Origin
X-Amz-Id-2: 1756959525
X-Amz-Request-Id: 1756959525
Date: Thu, 04 Sep 2025 04:18:45 GMT

Response for non-existing bucket:

HTTP/1.1 404 Not Found
Content-Type: application/xml
Date: Thu, 04 Sep 2025 14:11:05 GMT
Server: CB-Spider/1.0

Get Bucket Location

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?location&ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<LocationConstraint xmlns="http://s3.amazonaws.com/doc/2006-03-01/">ap-southeast-2</LocationConstraint>

Delete Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

2. Object Management

Upload Object (From File)

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  --data-binary @local-file.txt \
  -H "Content-Type: text/plain"

Response:

HTTP/1.1 200 OK
ETag: "5d41402abc4b2a76b9719d911017c592"

Upload Object (Form Data)

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -F "key=test-file.txt" \
  -F "[email protected]"

Response:

HTTP/1.1 200 OK
Content-Type: application/xml
ETag: "5d41402abc4b2a76b9719d911017c592"
X-Amz-Request-Id: 1756965697

Notes:

  • The key form field specifies the object name/key
  • The file form field contains the file to upload
  • Optional Content-Type form field can be specified
  • Response includes ETag header for uploaded object
  • Supports browser-based HTML form uploads

Download Object

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  -o downloaded-file.txt

Response:

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 1024
ETag: "5d41402abc4b2a76b9719d911017c592"
Last-Modified: Thu, 04 Sep 2025 14:15:30 GMT

[File content here]

Get Object Info (HEAD)

curl -I "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01"

Response:

HTTP/1.1 200 OK
Content-Length: 41
Content-Type: application/octet-stream
Etag: 1e575261e49dd31e40c50f283bbe1187
Last-Modified: Thu, 04 Sep 2025 04:19:23 GMT
Vary: Origin
X-Amz-Id-2: 1756959581
X-Amz-Request-Id: 1756959581
Date: Thu, 04 Sep 2025 04:19:41 GMT

Delete Object

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

Delete Multiple Objects

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket?delete&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<Delete xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Object>
    <Key>file1.txt</Key>
  </Object>
  <Object>
    <Key>file2.txt</Key>
  </Object>
  <Object>
    <Key>file3.txt</Key>
  </Object>
</Delete>'

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<DeleteResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Deleted>
    <Key>file1.txt</Key>
  </Deleted>
  <Deleted>
    <Key>file2.txt</Key>
  </Deleted>
  <Deleted>
    <Key>file3.txt</Key>
  </Deleted>
</DeleteResult>

Notes:

  • The delete query parameter is required
  • Request body must be XML with proper namespace declaration
  • Supports deleting multiple objects in a single request
  • Returns detailed results for each deleted object

3. Multipart Upload

Important Notes:

  • Part Size Limit: Each part (except the last one) must be at least 5MB in size for AWS S3 compatibility
  • ETag Format: Multipart files have ETag format as {hash}-{part_count} (e.g., 50f9c71a2e1d2cd7706f6dfd0b12c9fd-3)
  • Upload ID Lifecycle: CB-Spider automatically invalidates Upload IDs after a period of inactivity (unlike AWS S3 standard where Upload IDs never expire). It's recommended to upload parts immediately after initiation

Initiate Multipart Upload

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploads&ConnectionName=aws-config01" \
  -H "Content-Type: application/octet-stream"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<InitiateMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Bucket>spider-test-bucket</Bucket>
  <Key>large-multipart-file.bin</Key>
  <UploadId>bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--</UploadId>
</InitiateMultipartUploadResult>

Upload Part

# Upload Part 1 (6MB)
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploadId=bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--&partNumber=1&ConnectionName=aws-config01" \
  --data-binary @large-part1.bin \
  -H "Content-Type: application/octet-stream"

Response:

HTTP/1.1 200 OK
Etag: da6a0d097e307ac52ed9b4ad551801fc
Vary: Origin
X-Amz-Id-2: 1756960451
X-Amz-Request-Id: 1756960451
Date: Thu, 04 Sep 2025 04:34:11 GMT
Content-Length: 0

Complete Multipart Upload

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploadId=bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUpload>
  <Part>
    <PartNumber>1</PartNumber>
    <ETag>"da6a0d097e307ac52ed9b4ad551801fc"</ETag>
  </Part>
  <Part>
    <PartNumber>2</PartNumber>
    <ETag>"0ac5c730711202deffd434a3aa9ed578"</ETag>
  </Part>
  <Part>
    <PartNumber>3</PartNumber>
    <ETag>"49d186241a929edcf287fa2bd2b05e0f"</ETag>
  </Part>
</CompleteMultipartUpload>'

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<CompleteMultipartUploadResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Location>/spider-test-bucket/large-multipart-file.bin</Location>
  <Bucket>spider-test-bucket</Bucket>
  <Key>large-multipart-file.bin</Key>
  <ETag>"50f9c71a2e1d2cd7706f6dfd0b12c9fd-3"</ETag>
</CompleteMultipartUploadResult>

Note: The ETag ending with -3 indicates this file was assembled from 3 parts.

Abort Multipart Upload

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/abort-test-file.bin?uploadId=bXbASMlldIaDaT5MMtIe.PgQqmoDh3zpvaKDs5jueADDxI9IbEAK497EuAgHX8n_nWslGtvwVvuj1hAcZ4NJnQ2BPpEocEQgji5tnR7iMk5GCHCONbohmGyMP453R0K82hcIBtvUSx0ojZd5aYhdjA--&ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content
Vary: Origin
X-Amz-Id-2: 1756960740
X-Amz-Request-Id: 1756960740
Date: Thu, 04 Sep 2025 04:39:00 GMT

List Parts

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket/multipart-test-file.txt?uploadId=5_4rVl9cerQt63BTCq21WBdNSA_j8v3f3D_D6DW0U8YlTwfSrxg0X1d7i9iB32hKSRd8WrBhGTIopaTgzxmB.Ramr3kODk.D_l74WjjYjM8DWNEaZIVVmoHO60HW6bPo_MqOavo3FXiR5uAHMHwYNw--&list-type=parts&ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<ListPartsResult>
  <Bucket>spider-test-bucket</Bucket>
  <Key>multipart-test-file.txt</Key>
  <UploadId>5_4rVl9cerQt63BTCq21WBdNSA_j8v3f3D_D6DW0U8YlTwfSrxg0X1d7i9iB32hKSRd8WrBhGTIopaTgzxmB.Ramr3kODk.D_l74WjjYjM8DWNEaZIVVmoHO60HW6bPo_MqOavo3FXiR5uAHMHwYNw--</UploadId>
  <PartNumberMarker>0</PartNumberMarker>
  <NextPartNumberMarker>0</NextPartNumberMarker>
  <MaxParts>1000</MaxParts>
  <IsTruncated>false</IsTruncated>
  <Initiator>
    <ID>arn:aws:iam::123456789012:user/example-user</ID>
    <DisplayName>example-user</DisplayName>
  </Initiator>
  <Owner>
    <ID>7d9f77b078e963b2aca023ffa7dc39a3cac3b138397f2214d2d66e33bee3384d</ID>
    <DisplayName>example-owner</DisplayName>
  </Owner>
  <StorageClass>STANDARD</StorageClass>
</ListPartsResult>

Optional Parameters:

  • part-number-marker: Start listing after this part number
  • max-parts: Maximum number of parts to return (default: 1000)

List Multipart Uploads

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?uploads&ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<ListMultipartUploadsResult>
  <Bucket>spider-test-bucket</Bucket>
  <KeyMarker></KeyMarker>
  <UploadIdMarker></UploadIdMarker>
  <NextKeyMarker>multipart-test-file.txt</NextKeyMarker>
  <NextUploadIdMarker>n.O260Ytu6cYAqeOo1I5NR_OWLXCG1KQ8XUSrZ_4.OQXyodWwD_0fxaTN0K5U.sXuvyvLNKkgVXSsECsZwNdVdKBPXqs2pV70bbzBekYAQVcU8R1rlSoSnwHRlPxVDyQENP2PJa3BelO7AanbGtu0A--</NextUploadIdMarker>
  <MaxUploads>1000</MaxUploads>
  <IsTruncated>false</IsTruncated>
  <Upload>
    <Key>lifecycle-test.bin</Key>
    <UploadId>jAGGob7vC5OEYMi.d6RZx6.4MfnVa88ZH6sKbdIXMb3Uxd1hiGdH0NwjtP9KFojcpykb6fy65j3XhwGB.S6.c8LVBsbBAsVuarc9sWlmaB7IOEkkxSPukafDjhY_.i85J_uOmuWFEfA5HIU_2t24dw--</UploadId>
    <Initiated>2025-09-04T04:55:04Z</Initiated>
    <StorageClass>STANDARD</StorageClass>
    <Initiator>
      <ID>arn:aws:iam::123456789012:user/example-user</ID>
      <DisplayName>example-user</DisplayName>
    </Initiator>
    <Owner>
      <ID>7d9f77b078e963b2aca023ffa7dc39a3cac3b138397f2214d2d66e33bee3384d</ID>
      <DisplayName>example-owner</DisplayName>
    </Owner>
  </Upload>
  <Upload>
    <Key>multipart-test-file.txt</Key>
    <UploadId>ukiFkcnk68wgORdZCOKjSdnDNXCniobwwmHAPsPpDfH.kWfqePTSL_pIDWKVb88eQ3Kg4AFMDusX3qKEuDRdmKu4v8F4F048LyEgEDgH8UAkyK6qQhzCJ52ITloFOTeoSXcAZJhbWHWVeQUn8NfRDg--</UploadId>
    <Initiated>2025-09-04T04:30:32Z</Initiated>
    <StorageClass>STANDARD</StorageClass>
    <Initiator>
      <ID>arn:aws:iam::123456789012:user/example-user</ID>
      <DisplayName>example-user</DisplayName>
    </Initiator>
    <Owner>
      <ID>7d9f77b078e963b2aca023ffa7dc39a3cac3b138397f2214d2d66e33bee3384d</ID>
      <DisplayName></DisplayName>
    </Owner>
  </Upload>
  <Prefix></Prefix>
  <Delimiter></Delimiter>
</ListMultipartUploadsResult>

Optional Parameters:

  • prefix: Only list uploads with keys beginning with this prefix
  • key-marker: Start listing after this key name
  • upload-id-marker: Start listing after this upload ID
  • delimiter: Character used to group keys
  • max-uploads: Maximum number of uploads to return (default: 1000)

Multipart Upload Technical Notes

Upload ID Lifecycle:

  • Upload IDs must be used immediately after initiation
  • CB-Spider automatically invalidates unused Upload IDs (differs from AWS S3 standard)
  • Always initiate a new multipart upload for each file upload session

Part Size Requirements:

  • Each part (except the last) must be at least 5MB for AWS S3 compatibility
  • Maximum of 10,000 parts per multipart upload
  • Part numbers must be between 1 and 10,000

ETag Format:

  • Single uploads: Standard MD5 hash (e.g., "da6a0d097e307ac52ed9b4ad551801fc")
  • Multipart uploads: Hash with part count suffix (e.g., "50f9c71a2e1d2cd7706f6dfd0b12c9fd-3")

4. Versioning Management

Get Bucket Versioning

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Status>Enabled</Status>
</VersioningConfiguration>

Set Bucket Versioning

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Status>Enabled</Status>
</VersioningConfiguration>'

Response:

HTTP/1.1 200 OK

To suspend versioning:

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<VersioningConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Status>Suspended</Status>
</VersioningConfiguration>'

List Object Versions

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?versions&ConnectionName=aws-config01"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<ListVersionsResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>spider-test-bucket</Name>
  <Prefix></Prefix>
  <KeyMarker></KeyMarker>
  <VersionIdMarker></VersionIdMarker>
  <NextKeyMarker></NextKeyMarker>
  <NextVersionIdMarker></NextVersionIdMarker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
  <Version>
    <Key>test-file.txt</Key>
    <VersionId>yb4PgjnFVD2LfRZHXBjjsHBkQRHlu.TZ</VersionId>
    <IsLatest>true</IsLatest>
    <LastModified>2025-09-04T04:24:12Z</LastModified>
    <ETag>23228a38faecd0591107818c7281cece</ETag>
    <Size>23</Size>
    <StorageClass>STANDARD</StorageClass>
    <Owner>
      <ID>aws-config01</ID>
      <DisplayName>aws-config01</DisplayName>
    </Owner>
  </Version>
</ListVersionsResult>

Delete Versioned Object

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?versionId=yb4PgjnFVD2LfRZHXBjjsHBkQRHlu.TZ&ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

Versioning Important Notes

Creating Multiple Versions: When versioning is enabled, uploading an object with the same key creates a new version:

# Upload version 1
echo "Version 1 content" > file-v1.txt
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/same-file.txt?ConnectionName=aws-config01" \
  --data-binary @file-v1.txt -H "Content-Type: text/plain"

# Upload version 2 (same key)
echo "Version 2 content" > file-v2.txt
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/same-file.txt?ConnectionName=aws-config01" \
  --data-binary @file-v2.txt -H "Content-Type: text/plain"

Versioning States:

  • Enabled: New versions are created for each upload
  • Suspended: New uploads don't create versions (but existing versions remain)
  • Unversioned: Default state (no versioning)

5. CORS Management

Get CORS Configuration

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01"

Response Example (when CORS is configured):

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <ExposeHeader>x-amz-server-side-encryption</ExposeHeader>
    <ExposeHeader>x-amz-request-id</ExposeHeader>
    <ExposeHeader>x-amz-id-2</ExposeHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>

Error Response (when CORS is not configured):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchCORSConfiguration</Code>
  <Message>The CORS configuration does not exist</Message>
  <Resource>/spider-test-bucket</Resource>
  <RequestId>1756963456</RequestId>
</Error>

Set CORS Configuration

Basic CORS Configuration:

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>DELETE</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
    <MaxAgeSeconds>3000</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>'

Advanced CORS Configuration (Multiple Rules):

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01" \
  -H "Content-Type: application/xml" \
  -d '<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration>
  <CORSRule>
    <AllowedOrigin>https://example.com</AllowedOrigin>
    <AllowedOrigin>https://app.example.com</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedHeader>Content-Type</AllowedHeader>
    <AllowedHeader>Authorization</AllowedHeader>
    <ExposeHeader>ETag</ExposeHeader>
    <MaxAgeSeconds>1800</MaxAgeSeconds>
  </CORSRule>
  <CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <MaxAgeSeconds>300</MaxAgeSeconds>
  </CORSRule>
</CORSConfiguration>'

Response:

HTTP/1.1 200 OK

Test CORS with OPTIONS Request

curl -X OPTIONS "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  -H "Origin: https://example.com" \
  -H "Access-Control-Request-Method: PUT" \
  -H "Access-Control-Request-Headers: Content-Type" \
  -v

Response:

HTTP/1.1 204 No Content
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Origin: *
Allow: OPTIONS, DELETE, GET, HEAD, POST, PUT
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
Date: Thu, 04 Sep 2025 05:26:35 GMT

Delete CORS Configuration

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01"

Response:

HTTP/1.1 200 OK

CORS Configuration Elements

Element Description Required Example
AllowedOrigin Specifies the origins allowed to access the bucket Yes * or https://example.com
AllowedMethod Specifies the HTTP methods allowed Yes GET, PUT, POST, DELETE
AllowedHeader Specifies the headers allowed in the request No *, Content-Type, Authorization
ExposeHeader Specifies which headers are exposed to the browser No ETag, x-amz-request-id
MaxAgeSeconds Specifies how long the browser can cache the preflight response No 3600 (1 hour)

CORS Configuration Examples

Allow All Origins (Development):

<CORSRule>
  <AllowedOrigin>*</AllowedOrigin>
  <AllowedMethod>GET</AllowedMethod>
  <AllowedMethod>PUT</AllowedMethod>
  <AllowedMethod>POST</AllowedMethod>
  <AllowedMethod>DELETE</AllowedMethod>
  <AllowedHeader>*</AllowedHeader>
  <MaxAgeSeconds>3000</MaxAgeSeconds>
</CORSRule>

Production Configuration (Specific Domains):

<CORSRule>
  <AllowedOrigin>https://example.com</AllowedOrigin>
  <AllowedOrigin>https://app.example.com</AllowedOrigin>
  <AllowedMethod>GET</AllowedMethod>
  <AllowedMethod>PUT</AllowedMethod>
  <AllowedHeader>Content-Type</AllowedHeader>
  <AllowedHeader>Authorization</AllowedHeader>
  <ExposeHeader>ETag</ExposeHeader>
  <MaxAgeSeconds>1800</MaxAgeSeconds>
</CORSRule>

Read-Only Access:

<CORSRule>
  <AllowedOrigin>*</AllowedOrigin>
  <AllowedMethod>GET</AllowedMethod>
  <AllowedHeader>Range</AllowedHeader>
  <ExposeHeader>Content-Length</ExposeHeader>
  <ExposeHeader>Content-Range</ExposeHeader>
  <MaxAgeSeconds>300</MaxAgeSeconds>
</CORSRule>

CORS Best Practices

  1. Security: Use specific origins instead of * in production
  2. Performance: Set appropriate MaxAgeSeconds to reduce preflight requests
  3. Headers: Only allow necessary headers to minimize security risks
  4. Methods: Only specify required HTTP methods
  5. Testing: Use browser developer tools to verify CORS behavior https://www.example.com GET PUT Content-Type Authorization 3000
```

6. CB-Spider Special Features

Generate PreSigned URL (Download)

curl -X GET "http://localhost:1024/spider/s3/presigned/download/spider-test-bucket/test-file.txt?ConnectionName=aws-config01&expires=3600"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<PresignedURLResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <PresignedURL>https://spider-test-bucket.s3.dualstack.ap-southeast-2.amazonaws.com/test-file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA***EXAMPLE%2F20250904%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20250904T061448Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=***-signature</PresignedURL>
  <Expires>3600</Expires>
  <Method>GET</Method>
</PresignedURLResult>

Optional Parameters:

  • expires: URL expiration time in seconds (default: 3600)
  • method: HTTP method for the URL (default: GET, also supports PUT)
  • response-content-disposition: Content-Disposition header for downloads

Usage Example:

# Download a file using the generated PreSigned URL
curl -X GET "{PresignedURL}" -o downloaded-file.txt

Note: This feature is unique to CB-Spider and not part of AWS S3 standard.

Generate PreSigned URL (Upload)

curl -X GET "http://localhost:1024/spider/s3/presigned/upload/spider-test-bucket/test-file.txt?ConnectionName=aws-config01&expires=3600"

Response Example:

<?xml version="1.0" encoding="UTF-8"?>
<PresignedURLResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <PresignedURL>https://spider-test-bucket.s3.dualstack.ap-southeast-2.amazonaws.com/test-file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA***EXAMPLE%2F20250904%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20250904T061457Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=***-signature</PresignedURL>
  <Expires>3600</Expires>
  <Method>PUT</Method>
</PresignedURLResult>

Optional Parameters:

  • expires: URL expiration time in seconds (default: 3600)

Usage Example:

# Upload a file using the generated PreSigned URL
curl -X PUT "{PresignedURL}" --data-binary "@local-file.txt" -H "Content-Type: text/plain"

Note: This feature is unique to CB-Spider and not part of AWS S3 standard.

Force Empty Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?empty&ConnectionName=aws-config01" \
  -H "X-Force-Empty: true"

Response:

HTTP/1.1 204 No Content

Error Response (without header):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidRequest</Code>
  <Message>Force empty requires 'empty' query parameter or X-Force-Empty header</Message>
  <Resource>/spider-test-bucket</Resource>
  <RequestId>1756963732</RequestId>
</Error>

Verification:

# Check that bucket is now empty
curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Empty Bucket Response:

<?xml version="1.0" encoding="UTF-8"?>
<ListBucketResult xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
  <Name>spider-test-bucket</Name>
  <Prefix></Prefix>
  <Marker></Marker>
  <MaxKeys>1000</MaxKeys>
  <IsTruncated>false</IsTruncated>
</ListBucketResult>

Note: This feature is unique to CB-Spider and not part of AWS S3 standard. The X-Force-Empty header is required as a safety mechanism.

Force Delete Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?force&ConnectionName=aws-config01" \
  -H "X-Force-Delete: true"

Response:

HTTP/1.1 204 No Content

Error Response (bucket not empty, without force):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>BucketNotEmpty</Code>
  <Message>The bucket you tried to delete is not empty. It contains 2 objects. Use force=true parameter to force delete.</Message>
  <Resource>/spider-test-bucket</Resource>
  <RequestId>1756963800</RequestId>
</Error>

Error Response (without header):

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>InvalidRequest</Code>
  <Message>Force delete requires 'force' query parameter or X-Force-Delete header</Message>
  <Resource>/spider-test-bucket</Resource>
  <RequestId>1756963817</RequestId>
</Error>

Verification:

# Check that bucket no longer exists
curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Deleted Bucket Response:

<?xml version="1.0" encoding="UTF-8"?>
<Error>
  <Code>NoSuchBucket</Code>
  <Message>aws-config01, spider-test-bucket: does not exist!</Message>
  <Resource>/spider-test-bucket</Resource>
  <RequestId>1756963825</RequestId>
</Error>

Note: This feature forcefully deletes the bucket and all its contents. This feature is unique to CB-Spider and not part of AWS S3 standard. The X-Force-Delete header is required as a safety mechanism.

Related Links

Add a custom footer
⚠️ **GitHub.com Fallback** ⚠️