CB‐Spider Object Storage API Guide (JSON Format) - cloud-barista/cb-spider GitHub Wiki

CB-Spider Object Storage API Guide (JSON Format)

CB-Spider provides AWS S3-compatible Object Storage API that enables unified management of Object Storage across various Cloud Service Providers (CSPs). This guide demonstrates JSON format input/output examples.

Key Features

  • Multi-Cloud Support: Unified management of Object Storage across AWS, IBM Cloud, GCP, and other CSPs
  • AWS S3 Compatibility: Provides compatibility with existing AWS S3 tools and SDKs
  • JSON Format Support: Supports both XML and JSON response formats based on Accept header
  • PreSigned URL REST API: Provides PreSigned URL generation REST API
  • Simple Authentication: Easy authentication with ConnectionName parameter

JSON Format Usage

To receive JSON responses, include the following header in your requests:

Accept: application/json

Without this header, the API returns XML format responses (default S3 behavior).

API List

1. Bucket Management

Function HTTP Method Endpoint Example
List Buckets GET /spider/s3 Example
Create Bucket PUT /spider/s3/{bucket-name} Example
Get Bucket Info GET /spider/s3/{bucket-name} Example
Check Bucket Exists HEAD /spider/s3/{bucket-name} Example
Get Bucket Location GET /spider/s3/{bucket-name}?location Example
Delete Bucket DELETE /spider/s3/{bucket-name} Example

2. Object Management

Function HTTP Method Endpoint Example
Upload Object (File) PUT /spider/s3/{bucket-name}/{object-key} Example
Upload Object (Form) POST /spider/s3/{bucket-name} Example
Download Object GET /spider/s3/{bucket-name}/{object-key} Example
Get Object Info HEAD /spider/s3/{bucket-name}/{object-key} Example
Delete Object DELETE /spider/s3/{bucket-name}/{object-key} Example
Delete Multiple Objects POST /spider/s3/{bucket-name}?delete Example

3. Multipart Upload

Function HTTP Method Endpoint Example
Initiate Multipart Upload POST /spider/s3/{bucket-name}/{object-key}?uploads Example
Upload Part PUT /spider/s3/{bucket-name}/{object-key}?uploadId={id}&partNumber={num} Example
Complete Multipart Upload POST /spider/s3/{bucket-name}/{object-key}?uploadId={id} Example
Abort Multipart Upload DELETE /spider/s3/{bucket-name}/{object-key}?uploadId={id} Example
List Parts GET /spider/s3/{bucket-name}/{object-key}?uploadId={id}&list-type=parts Example
List Multipart Uploads GET /spider/s3/{bucket-name}?uploads Example

4. Versioning Management

Function HTTP Method Endpoint Example
Get Bucket Versioning GET /spider/s3/{bucket-name}?versioning Example
Set Bucket Versioning PUT /spider/s3/{bucket-name}?versioning Example
List Object Versions GET /spider/s3/{bucket-name}?versions Example
Delete Versioned Object DELETE /spider/s3/{bucket-name}/{object-key}[?versionId=version-id] Example

5. CORS Management

Function HTTP Method Endpoint Example
Get Bucket CORS GET /spider/s3/{bucket-name}?cors Example
Set Bucket CORS PUT /spider/s3/{bucket-name}?cors Example
Delete Bucket CORS DELETE /spider/s3/{bucket-name}?cors Example

6. CB-Spider Special Features

Function HTTP Method Endpoint Example
Generate PreSigned URL (Download) GET /spider/s3/presigned/download/{bucket-name}/{object-key} Example
Generate PreSigned URL (Upload) GET /spider/s3/presigned/upload/{bucket-name}/{object-key} Example
Force Empty Bucket DELETE /spider/s3/{bucket-name}?empty Example
Force Delete Bucket DELETE /spider/s3/{bucket-name}?force Example

Note: CB-Spider special features are unique to CB-Spider and not part of AWS S3 standard.

API Usage Examples

1. Bucket Management

List Buckets

curl -X GET "http://localhost:1024/spider/s3?ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Owner": {
    "ID": "aws-config01",
    "DisplayName": "aws-config01"
  },
  "Buckets": {
    "Bucket": [
      {
        "Name": "test-json-guide-bucket",
        "CreationDate": "2025-09-04T16:15:25Z"
      }
    ]
}

Create Bucket

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -H "Accept: application/json"

Response:

HTTP/1.1 200 OK

Get Bucket Info

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Name": "spider-test-bucket",
  "Prefix": "",
  "Marker": "",
  "MaxKeys": 1000,
  "IsTruncated": false,
  "Contents": null
}

Check Bucket Exists (HEAD)

curl -I "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response for existing bucket:

HTTP/1.1 200 OK
Vary: Origin
X-Amz-Id-2: 1756959525
X-Amz-Request-Id: 1756959525
Date: Thu, 04 Sep 2025 04:18:45 GMT

Response for non-existing bucket:

HTTP/1.1 403 Forbidden
Vary: Origin
Date: Thu, 04 Sep 2025 16:45:31 GMT

Get Bucket Location

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?location&ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "LocationConstraint": "ap-southeast-2"
}

Delete Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

2. Object Management

Upload Object (From File)

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  --data-binary @local-file.txt \
  -H "Content-Type: text/plain" \
  -H "Accept: application/json"

Response:

HTTP/1.1 200 OK
ETag: "5d41402abc4b2a76b9719d911017c592"

Upload Object (Form Data)

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -F "key=test-file.txt" \
  -F "[email protected]" \
  -H "Accept: application/json"

Response:

HTTP/1.1 200 OK
Content-Type: application/json
ETag: "5d41402abc4b2a76b9719d911017c592"
X-Amz-Request-Id: 1756965697

Notes:

  • The key form field specifies the object name/key
  • The file form field contains the file to upload
  • Optional Content-Type form field can be specified
  • Response includes ETag header for uploaded object
  • Supports browser-based HTML form uploads

Download Object

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  -o downloaded-file.txt

Response:

HTTP/1.1 200 OK
Content-Type: text/plain
Content-Length: 1024
ETag: "5d41402abc4b2a76b9719d911017c592"
Last-Modified: Thu, 04 Sep 2025 14:15:30 GMT

[File content here]

Get Object Info (HEAD)

curl -I "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01"

Response:

HTTP/1.1 200 OK
Content-Length: 41
Content-Type: application/octet-stream
Etag: 1e575261e49dd31e40c50f283bbe1187
Last-Modified: Thu, 04 Sep 2025 04:19:23 GMT
Vary: Origin
X-Amz-Id-2: 1756959581
X-Amz-Request-Id: 1756959581
Date: Thu, 04 Sep 2025 04:19:41 GMT

Delete Object

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

Delete Multiple Objects

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket?delete&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "Delete": {
      "Objects": [
        {"Key": "file1.txt"},
        {"Key": "file2.txt"},
        {"Key": "file3.txt"}
      ],
      "Quiet": false
    }
  }'

JSON Response Example:

{
  "Deleted": [
    {
      "Key": "file1.txt"
    },
    {
      "Key": "file2.txt"
    },
    {
      "Key": "file3.txt"
    }
  ],
  "Error": null
}

Notes:

  • The delete query parameter is required
  • Uses JSON format with Content-Type: application/json and Delete.Objects array
  • Supports deleting multiple objects in a single request
  • Returns detailed results for each deleted object
  • Quiet parameter: Set to true to suppress successful deletion responses

3. Multipart Upload

Important Notes:

  • Part Size Limit: Each part (except the last one) must be at least 5MB in size for AWS S3 compatibility
  • ETag Format: Multipart files have ETag format as {hash}-{part_count} (e.g., 50f9c71a2e1d2cd7706f6dfd0b12c9fd-3)
  • Upload ID Lifecycle: CB-Spider automatically invalidates Upload IDs after a period of inactivity (unlike AWS S3 standard where Upload IDs never expire). It's recommended to upload parts immediately after initiation

Initiate Multipart Upload

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploads&ConnectionName=aws-config01" \
  -H "Content-Type: application/octet-stream" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Bucket": "spider-test-bucket",
  "Key": "large-multipart-file.bin",
  "UploadId": "bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--"
}

Upload Part

# Upload Part 1 (6MB)
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploadId=bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--&partNumber=1&ConnectionName=aws-config01" \
  --data-binary @large-part1.bin \
  -H "Content-Type: application/octet-stream"

Response:

HTTP/1.1 200 OK
Etag: da6a0d097e307ac52ed9b4ad551801fc
Vary: Origin
X-Amz-Id-2: 1756960451
X-Amz-Request-Id: 1756960451
Date: Thu, 04 Sep 2025 04:34:11 GMT
Content-Length: 0

Complete Multipart Upload

curl -X POST "http://localhost:1024/spider/s3/spider-test-bucket/large-multipart-file.bin?uploadId=bpV9AokiRqAUEYA2z9mhMfPZ8rUh2OHx1f22lGPmw8CdfeqLmvqSSIw9rLoAH19DoGR5zk8i0p_mc_d0kpjuu5ZuwdPE09oWQcb62irqFjtzjWg3ifjY0TeiXATCy9yCsHTRQucIqAQYSCaFFhHfPg--&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "Parts": [
      {"PartNumber": 1, "ETag": "\"da6a0d097e307ac52ed9b4ad551801fc\""},
      {"PartNumber": 2, "ETag": "\"0ac5c730711202deffd434a3aa9ed578\""},
      {"PartNumber": 3, "ETag": "\"49d186241a929edcf287fa2bd2b05e0f\""}
    ]
  }'

JSON Response Example:

{
  "Location": "/spider-test-bucket/large-multipart-file.bin",
  "Bucket": "spider-test-bucket",
  "Key": "large-multipart-file.bin",
  "ETag": "\"50f9c71a2e1d2cd7706f6dfd0b12c9fd-3\""
}

Note: The ETag ending with -3 indicates this file was assembled from 3 parts.

Abort Multipart Upload

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/abort-test-file.bin?uploadId=bXbASMlldIaDaT5MMtIe.PgQqmoDh3zpvaKDs5jueADDxI9IbEAK497EuAgHX8n_nWslGtvwVvuj1hAcZ4NJnQ2BPpEocEQgji5tnR7iMk5GCHCONbohmGyMP453R0K82hcIBtvUSx0ojZd5aYhdjA--&ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content
Vary: Origin
X-Amz-Id-2: 1756960740
X-Amz-Request-Id: 1756960740
Date: Thu, 04 Sep 2025 04:39:00 GMT

List Parts

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket/multipart-test-file.txt?uploadId=5_4rVl9cerQt63BTCq21WBdNSA_j8v3f3D_D6DW0U8YlTwfSrxg0X1d7i9iB32hKSRd8WrBhGTIopaTgzxmB.Ramr3kODk.D_l74WjjYjM8DWNEaZIVVmoHO60HW6bPo_MqOavo3FXiR5uAHMHwYNw--&list-type=parts&ConnectionName=aws-config01"

JSON Response Example:

{
  "Bucket": "spider-test-bucket",
  "Key": "multipart-test-file.txt",
  "UploadId": "5_4rVl9cerQt63BTCq21WBdNSA_j8v3f3D_D6DW0U8YlTwfSrxg0X1d7i9iB32hKSRd8WrBhGTIopaTgzxmB.Ramr3kODk.D_l74WjjYjM8DWNEaZIVVmoHO60HW6bPo_MqOavo3FXiR5uAHMHwYNw--",
  "PartNumberMarker": 0,
  "NextPartNumberMarker": 0,
  "MaxParts": 1000,
  "IsTruncated": false,
  "Initiator": {
    "ID": "arn:aws:iam::123456789012:user/example-user",
    "DisplayName": "example-user"
  },
  "Owner": {
    "ID": "7d9f77b078e963b2aca023ffa7dc39a3cac3b138397f2214d2d66e33bee3384d",
    "DisplayName": "example-owner"
  },
  "StorageClass": "STANDARD",
  "Parts": null
}

Optional Parameters:

  • part-number-marker: Start listing after this part number
  • max-parts: Maximum number of parts to return (default: 1000)

List Multipart Uploads

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?uploads&ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Bucket": "spider-test-bucket",
  "KeyMarker": "",
  "UploadIDMarker": "",
  "NextKeyMarker": "multipart-test.txt",
  "NextUploadIDMarker": "UYlkOeACYiuiiSJ4OU7qvJ9hfeIge69fSqD0EPeZI4udhk4OpRU4jo7l_IP4i2USl3ozO2zhbbioTRpn_2fd61hCR2_Y11MTLOUcjxc97uvnQX3py2MjvTrLy7pyHkytNqqC4qyRmehHOx49fyu9rA--",
  "MaxUploads": 1000,
  "IsTruncated": false,
  "Uploads": [
    {
      "Key": "multipart-test.txt",
      "UploadID": "UYlkOeACYiuiiSJ4OU7qvJ9hfeIge69fSqD0EPeZI4udhk4OpRU4jo7l_IP4i2USl3ozO2zhbbioTRpn_2fd61hCR2_Y11MTLOUcjxc97uvnQX3py2MjvTrLy7pyHkytNqqC4qyRmehHOx49fyu9rA--",
      "Initiated": "2025-09-04T16:16:13Z",
      "StorageClass": "STANDARD",
      "Initiator": {
        "ID": "arn:aws:iam::123456789012:user/example-user",
        "DisplayName": "example-user"
      },
      "Owner": {
        "ID": "7d9f77b078e963b2aca023ffa7dc39a3cac3b138397f2214d2d66e33bee3384d",
        "DisplayName": "example-owner"
      }
    }
  ],
  "Prefix": "",
  "Delimiter": ""
}

Optional Parameters:

  • prefix: Only list uploads with keys beginning with this prefix
  • key-marker: Start listing after this key name
  • upload-id-marker: Start listing after this upload ID
  • delimiter: Character used to group keys
  • max-uploads: Maximum number of uploads to return (default: 1000)

Multipart Upload Technical Notes

Upload ID Lifecycle:

  • Upload IDs must be used immediately after initiation
  • CB-Spider automatically invalidates unused Upload IDs (differs from AWS S3 standard)
  • Always initiate a new multipart upload for each file upload session

Part Size Requirements:

  • Each part (except the last) must be at least 5MB for AWS S3 compatibility
  • Maximum of 10,000 parts per multipart upload
  • Part numbers must be between 1 and 10,000

ETag Format:

  • Single uploads: Standard MD5 hash (e.g., "da6a0d097e307ac52ed9b4ad551801fc")
  • Multipart uploads: Hash with part count suffix (e.g., "50f9c71a2e1d2cd7706f6dfd0b12c9fd-3")

4. Versioning Management

Get Bucket Versioning

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Status": "Enabled"
}

Set Bucket Versioning

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "Status": "Enabled"
  }'

Response:

HTTP/1.1 200 OK

To suspend versioning:

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?versioning&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "Status": "Suspended"
  }'

List Object Versions

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?versions&ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example:

{
  "Name": "spider-test-bucket",
  "Prefix": "",
  "KeyMarker": "",
  "VersionIdMarker": "",
  "NextKeyMarker": "",
  "NextVersionIdMarker": "",
  "MaxKeys": 1000,
  "IsTruncated": false,
  "Versions": [
    {
      "Key": "test-file.txt",
      "VersionId": "yb4PgjnFVD2LfRZHXBjjsHBkQRHlu.TZ",
      "IsLatest": true,
      "LastModified": "2025-09-04T04:24:12Z",
      "ETag": "23228a38faecd0591107818c7281cece",
      "Size": 23,
      "StorageClass": "STANDARD",
      "Owner": {
        "ID": "aws-config01",
        "DisplayName": "aws-config01"
      }
    }
  ]
}

Delete Versioned Object

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?versionId=yb4PgjnFVD2LfRZHXBjjsHBkQRHlu.TZ&ConnectionName=aws-config01"

Response:

HTTP/1.1 204 No Content

Versioning Important Notes

Creating Multiple Versions: When versioning is enabled, uploading an object with the same key creates a new version:

# Upload version 1
echo "Version 1 content" > file-v1.txt
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/same-file.txt?ConnectionName=aws-config01" \
  --data-binary @file-v1.txt -H "Content-Type: text/plain"

# Upload version 2 (same key)
echo "Version 2 content" > file-v2.txt
curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket/same-file.txt?ConnectionName=aws-config01" \
  --data-binary @file-v2.txt -H "Content-Type: text/plain"

Versioning States:

  • Enabled: New versions are created for each upload
  • Suspended: New uploads don't create versions (but existing versions remain)
  • Unversioned: Default state (no versioning)

5. CORS Management

Get CORS Configuration

curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Response Example (when CORS is configured):

{
  "CORSRules": [
    {
      "AllowedOrigins": ["*"],
      "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
      "AllowedHeaders": ["*"],
      "ExposeHeaders": ["ETag", "x-amz-server-side-encryption", "x-amz-request-id", "x-amz-id-2"],
      "MaxAgeSeconds": 3000
    }
  ]
}

JSON Error Response (when CORS is not configured):

{
  "Error": {
    "Code": "NoSuchCORSConfiguration",
    "Message": "The CORS configuration does not exist",
    "Resource": "/spider-test-bucket",
    "RequestId": "1756963456"
  }
}

Set CORS Configuration

Basic CORS Configuration:

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "CORSRule": [
      {
        "AllowedOrigin": ["*"],
        "AllowedMethod": ["GET", "PUT", "POST", "DELETE"],
        "AllowedHeader": ["*"],
        "MaxAgeSeconds": 3000
      }
    ]
  }'

Advanced CORS Configuration (Multiple Rules):

curl -X PUT "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01" \
  -H "Content-Type: application/json" \
  -H "Accept: application/json" \
  -d '{
    "CORSRule": [
      {
        "AllowedOrigin": ["https://example.com", "https://app.example.com"],
        "AllowedMethod": ["GET", "PUT"],
        "AllowedHeader": ["Content-Type", "Authorization"],
        "ExposeHeader": ["ETag"],
        "MaxAgeSeconds": 1800
      },
      {
        "AllowedOrigin": ["*"],
        "AllowedMethod": ["GET"],
        "MaxAgeSeconds": 300
      }
    ]
  }'

Response:

HTTP/1.1 200 OK

Test CORS with OPTIONS Request

curl -X OPTIONS "http://localhost:1024/spider/s3/spider-test-bucket/test-file.txt?ConnectionName=aws-config01" \
  -H "Origin: https://example.com" \
  -H "Access-Control-Request-Method: PUT" \
  -H "Access-Control-Request-Headers: Content-Type" \
  -v

Response:

HTTP/1.1 204 No Content
Access-Control-Allow-Headers: Content-Type
Access-Control-Allow-Methods: GET,HEAD,PUT,PATCH,POST,DELETE
Access-Control-Allow-Origin: *
Allow: OPTIONS, DELETE, GET, HEAD, POST, PUT
Vary: Origin
Vary: Access-Control-Request-Method
Vary: Access-Control-Request-Headers
Date: Thu, 04 Sep 2025 05:26:35 GMT

Delete CORS Configuration

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?cors&ConnectionName=aws-config01"

Response:

HTTP/1.1 200 OK

CORS Configuration Elements

Element Description Required Example
AllowedOrigin Specifies the origins allowed to access the bucket Yes * or https://example.com
AllowedMethod Specifies the HTTP methods allowed Yes GET, PUT, POST, DELETE
AllowedHeader Specifies the headers allowed in the request No *, Content-Type, Authorization
ExposeHeader Specifies which headers are exposed to the browser No ETag, x-amz-request-id
MaxAgeSeconds Specifies how long the browser can cache the preflight response No 3600 (1 hour)

CORS Configuration Examples

Allow All Origins (Development):

{
  "CORSRules": [
    {
      "AllowedOrigins": ["*"],
      "AllowedMethods": ["GET", "PUT", "POST", "DELETE"],
      "AllowedHeaders": ["*"],
      "MaxAgeSeconds": 3000
    }
  ]
}

Production Configuration (Specific Domains):

{
  "CORSRules": [
    {
      "AllowedOrigins": ["https://example.com", "https://app.example.com"],
      "AllowedMethods": ["GET", "PUT"],
      "AllowedHeaders": ["Content-Type", "Authorization"],
      "ExposeHeaders": ["ETag"],
      "MaxAgeSeconds": 1800
    }
  ]
}

Read-Only Access:

{
  "CORSRules": [
    {
      "AllowedOrigins": ["*"],
      "AllowedMethods": ["GET"],
      "AllowedHeaders": ["Range"],
      "ExposeHeaders": ["Content-Length", "Content-Range"],
      "MaxAgeSeconds": 300
    }
  ]
}

CORS Best Practices

  1. Security: Use specific origins instead of * in production
  2. Performance: Set appropriate MaxAgeSeconds to reduce preflight requests
  3. Headers: Only allow necessary headers to minimize security risks
  4. Methods: Only specify required HTTP methods
  5. Testing: Use browser developer tools to verify CORS behavior

6. CB-Spider Special Features

Generate PreSigned URL (Download)

curl -X GET "http://localhost:1024/spider/s3/presigned/download/spider-test-bucket/test-file.txt?ConnectionName=aws-config01&expires=3600" \
  -H "Accept: application/json"

JSON Response Example:

{
  "PresignedURL": "https://spider-test-bucket.s3.dualstack.ap-southeast-2.amazonaws.com/test-file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA***EXAMPLE%2F20250904%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20250904T061448Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=***-signature",
  "Expires": 3600,
  "Method": "GET"
}

Optional Parameters:

  • expires: URL expiration time in seconds (default: 3600)
  • method: HTTP method for the URL (default: GET, also supports PUT)
  • response-content-disposition: Content-Disposition header for downloads

Usage Example:

# Download a file using the generated PreSigned URL
curl -X GET "{PresignedURL}" -o downloaded-file.txt

Note: This feature is unique to CB-Spider and not part of AWS S3 standard.

Generate PreSigned URL (Upload)

curl -X GET "http://localhost:1024/spider/s3/presigned/upload/spider-test-bucket/test-file.txt?ConnectionName=aws-config01&expires=3600" \
  -H "Accept: application/json"

JSON Response Example:

{
  "PresignedURL": "https://spider-test-bucket.s3.dualstack.ap-southeast-2.amazonaws.com/test-file.txt?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=AKIA***EXAMPLE%2F20250904%2Fap-southeast-2%2Fs3%2Faws4_request&X-Amz-Date=20250904T061457Z&X-Amz-Expires=3600&X-Amz-SignedHeaders=host&X-Amz-Signature=***-signature",
  "Expires": 3600,
  "Method": "PUT"
}

Optional Parameters:

  • expires: URL expiration time in seconds (default: 3600)

Usage Example:

# Upload a file using the generated PreSigned URL
curl -X PUT "{PresignedURL}" --data-binary "@local-file.txt" -H "Content-Type: text/plain"

Note: This feature is unique to CB-Spider and not part of AWS S3 standard.

Force Empty Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?empty&ConnectionName=aws-config01" \
  -H "X-Force-Empty: true"

Response:

HTTP/1.1 204 No Content

JSON Error Response (without header):

{
  "Error": {
    "Code": "InvalidRequest",
    "Message": "Force empty requires 'empty' query parameter or X-Force-Empty header",
    "Resource": "/spider-test-bucket",
    "RequestId": "1756963732"
  }
}

Verification:

# Check that bucket is now empty
curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Empty Bucket Response:

{
  "Name": "spider-test-bucket",
  "Prefix": "",
  "Marker": "",
  "MaxKeys": 1000,
  "IsTruncated": false,
  "Contents": null
}

Note: This feature is unique to CB-Spider and not part of AWS S3 standard. The X-Force-Empty header is required as a safety mechanism.

Force Delete Bucket

curl -X DELETE "http://localhost:1024/spider/s3/spider-test-bucket?force&ConnectionName=aws-config01" \
  -H "X-Force-Delete: true"

Response:

HTTP/1.1 204 No Content

JSON Error Response (bucket not empty, without force):

{
  "Error": {
    "Code": "BucketNotEmpty",
    "Message": "The bucket you tried to delete is not empty. It contains 2 objects. Use force=true parameter to force delete.",
    "Resource": "/spider-test-bucket",
    "RequestId": "1756963800"
  }
}

JSON Error Response (without header):

{
  "Error": {
    "Code": "InvalidRequest",
    "Message": "Force delete requires 'force' query parameter or X-Force-Delete header",
    "Resource": "/spider-test-bucket",
    "RequestId": "1756963817"
  }
}

Verification:

# Check that bucket no longer exists
curl -X GET "http://localhost:1024/spider/s3/spider-test-bucket?ConnectionName=aws-config01" \
  -H "Accept: application/json"

JSON Deleted Bucket Response:

{
  "Error": {
    "Code": "NoSuchBucket",
    "Message": "aws-config01, spider-test-bucket: does not exist!",
    "Resource": "/spider-test-bucket",
    "RequestId": "1756963825"
  }
}

Note: This feature forcefully deletes the bucket and all its contents. This feature is unique to CB-Spider and not part of AWS S3 standard. The X-Force-Delete header is required as a safety mechanism.

Related Links