Error code 500 Internal server Error on the AWS S3 - isgaur/AWS-BigData-Solutions GitHub Wiki

This error indicates that Amazon S3 is unable to handle the request at that time.This issue can usually be solved by the application taking an exponential backoff period after it gets the 500 error the first time. This is usually related to the application making too many calls in a short period of time which generates the error. If you are able to execute the same command after a retry, that would ideally solves the problem.

Because Amazon S3 is a distributed service, a very small percentage of 5xx errors are expected during normal use of the service. All requests that return 5xx errors from Amazon S3 can and should be retried , so I always recommend that applications making requests to Amazon S3 have a fault-tolerance mechanism to recover from these errors.

Enable a retry mechanism in the application making requests - Because of the distributed nature of Amazon S3, requests that return 500 errors can be retried. It's a best practice to build retry logic into applications that make requests to Amazon S3. We recommend that you set the retry count for your applications to 10.

All AWS SDKs have a built-in retry mechanism with an algorithm that uses exponential backoff. This algorithm implements increasingly longer wait times between retries for consecutive error responses. Most exponential backoff algorithms use jitter (randomized delay) to prevent successive collisions. For more information, see Error Retries and Exponential Backoff in AWS.