Minio - bcgov/common-service-showcase GitHub Wiki

MinIO

MinIO is a cloud-native open source application tool which offers high-performance, S3 compatible object storage. It is offered under the GNU AGPL v3 license.

We are potentially considering to leverage MinIO to power a part of our upcoming Common Document Management solution.

Licensing

  • Open Source (GNU AGPL v3 license)
  • May have limitations with more permissive licensing models
  • Commerical license used for paid options

Cost

  • Community Tier is free; Enterprise options available (details here)
  • Enterprise options are good to know but not worth pursuing for our needs

Ecosystem

  • Minio server application designed as a containerized microservice
    • Serves a configurable hosted S3 endpoint
  • Minio gateway application designed as a containerized microservice
    • Acts as a web-frontend viewer for existing S3 bucket endpoints
  • Minio client application designed as a command line tool
    • Allows users to interface with existing S3 buckets via terminal

Features

  • Offers multi-bucketing redundancy support with error-correction erasure code support
  • Support for 3rd-party federated OIDC authentication
  • Dropbox-like management interface for server and gateway
  • Supports temporary share link generation
  • Can interface with multiple databases for persistence
  • Has SDKs available in Java, Golang, Python, JavaScript, .NET and Haskell
    • Generally follows/extends S3 bucket API

Resources

Proof of Concepts

Gateway

The gateway is relatively easy to spin up - just requires a few environment variables and it can run immediately (in-memory database mode).

export MINIO_ROOT_USER=<BUCKETNAME>
export MINIO_ROOT_PASSWORD=<SECRETKEY>
./minio gateway s3 https://nrs.objectstore.gov.bc.ca

It is able to provide a nice web interface for managing your S3 bucket - there appears to be some degree of access control concepts available, but this UX appears to be mainly designed more for administrators than end users.

MinIO Gateway Listing Figure 1 - MinIO Gateway listing S3 Directory contents

MinIO also has the ability to generate direct access links for sharing objects. These links are time-boxed to expire after 7 days for example:

MinIO Temporary Share Figure 2 - MinIO Gateway generating temporary share link

Client

The mc command line tool is quite flexible. After it is configured, you can use this tool as a way to directly manipulate your bucket contents at will.

./mc alias set objStore https://nrs.objectstore.gov.bc.ca <BUCKETNAME> <SECRETKEY>
./mc ls objStore/egejyy/chefs/dev/submissions
[2020-11-16 02:38:59 PST]     0B _$folder$
[2021-12-16 16:34:06 PST]     0B 009b6eb4-3bc6-4b83-9e2a-0e97f552b50d/
[2021-12-16 16:34:06 PST]     0B 0905e4d9-1edd-46b8-b7b0-4e06e1130494/
[2021-12-16 16:34:06 PST]     0B 155771d6-3bff-4abd-935e-1f0a475dfd4c/
[2021-12-16 16:34:06 PST]     0B 1b011a20-b0ee-4471-80b7-b0952956c0e4/
[2021-12-16 16:34:06 PST]     0B 1cf3026f-9047-46bc-9943-8571bda51244/
[2021-12-16 16:34:06 PST]     0B 1d47b4c0-5166-4c88-b2ef-1d5c5eb2b61c/
[2021-12-16 16:34:06 PST]     0B 293fec62-8a09-49bd-925e-6f3a6c125d87/
[2021-12-16 16:34:06 PST]     0B 2c7cb67d-115a-4ec9-804c-4a2cfcddf887/
[2021-12-16 16:34:06 PST]     0B 322cc48f-1270-4fb3-91ed-c0fa97260f2c/
[2021-12-16 16:34:06 PST]     0B 3cff3be4-b549-4cbe-be29-f458ae867c68/
[2021-12-16 16:34:06 PST]     0B 435ebdb1-7917-4212-9c41-9f5fe20450e4/
[2021-12-16 16:34:06 PST]     0B 561717eb-b156-4bb6-9535-7d1dd82c7e07/
[2021-12-16 16:34:06 PST]     0B 5a6c88b9-044c-4fe8-aa56-9d8acac6b7a7/

Evaluation

  • MinIO and its application ecosystem appears quite functional, with out of the box support for many general file management tasks.
  • Especially considering where we will likely land in infrastructure (Dell ECS Object Storage), only Minio Gateway and Client may prove useful.
  • MinIO provides nice SDK libraries for interfacing with MinIO directly, but may not offer a ton of value overhead versus directly connecting with the underlying S3 bucket.
  • Unfortunately not sure if there is a MinIO RESTful API that is published and can be directly interacted with - this will likely limit our options for decoupled integration.
  • Configuration options appear relatively limited with MinIO - while it can do many things already, it is relatively opinionated and may be hard for us to adapt future business requirements around MinIO.
  • Can use MinIO as a role model for some of our DOMO features, but using MinIO as a core engine/wrapper may not be a wise design choice as it can restrict our options for future business design development.
⚠️ **GitHub.com Fallback** ⚠️