Batch expiry demo - allanrogerr/public GitHub Wiki

Create bucket and set versioning

mc alias set play https://play.min.io:9000 Q3AM3UQ867SPQQA43P2F zuf+tfteSlswRu7BJ86wekitnifILbZam1KYY3TG
mc admin info play
mc rm --recursive --versions --force play/batch-expiry
mc rb --force play/batch-expiry
mc mb play/batch-expiry
mc version enable play/batch-expiry

Copy some sample data

mc cp --recursive /etc/hosts play/batch-expiry/24/Jan

Create batch expiry template

mc batch generate play/batch-expiry expire

Edit template (see below example)

cat << EOF > batch-expire.yaml
expire:
  apiVersion: v1
  bucket: batch-expiry # Bucket where this job will expire matching objects from
  prefix: 24/Jan # (Optional) Prefix under which this job will expire objects matching the rules below.
  rules:
    - type: deleted # objects with delete marker as their latest version
      purge:
          retainVersions: 0 # (default) delete all versions of the object. This option is the fastest.

  notify:
    endpoint: https://webhook.site/8c209e85-26eb-4a3d-946a-ff9f793b1fbe # notification endpoint to receive job completion status

  retry:
    attempts: 10 # number of retries for the job before giving up
    delay: 500ms # least amount of delay between each retry
EOF

Create a delete marker. Show result

mc rm play/batch-expiry/24/Jan/hosts.allow
mc ls --versions play/batch-expiry/24/Jan

[2024-03-11 11:56:21 UTC]   221B STANDARD b693b82b-8b2d-437b-80c6-c5b826dc917b v1 PUT hosts
[2024-03-11 12:06:25 UTC]     0B STANDARD 44eeaa16-f9ed-4a24-8e87-329e2ee909e7 v2 DEL hosts.allow
[2024-03-11 11:56:21 UTC]   411B STANDARD 086dbeec-e50c-41da-a0f9-e18ceddf7466 v1 PUT hosts.allow
[2024-03-11 11:56:21 UTC]   711B STANDARD b681a08d-7013-4f13-80fa-0e0bcc838de8 v1 PUT hosts.deny

Start batch expire job. Observe status

mc batch start play/ batch-expire.yaml

Successfully started 'expire' job `WuFEXXS7BLULZgQgFoWrTr:-1` on '2024-03-11 12:07:59.441743305 +0000 UTC'

mc batch status play/ PTnY89WeJ48MY4SV2sXF3T:-1

mc: Unable to find an active job, attempting to list from previously run jobs
✔ ✔ ✔ 
JobType:       	expire	
Objects:       	0     	
FailedObjects: 	0     	
CurrObjName:

Observe all versions, of files with delete markers, are purged

mc ls --versions play/batch-expiry/24/Jan

[2024-03-11 11:56:21 UTC]   221B STANDARD hosts
[2024-03-11 11:56:21 UTC]   711B STANDARD hosts.deny
⚠️ **GitHub.com Fallback** ⚠️