Download and Decompression Rate (c4.large) - Sentimentron/Utah GitHub Wiki

url -s https://aws-publicdatasets.s3.amazonaws.com/common-crawl/crawl-data/CC-MAIN-2015-11/segments/1424936462472.19/warc/CC-MAIN-20150226074102-00293-ip-10-28-5-156.ec2.internal.warc.gz | pv | gzip -d >/dev/null

Reached about 36.4MB/s, more than enough.

pigz (gzip decompress in parallel) was no faster.

⚠️ **GitHub.com Fallback** ⚠️