mc mirror tests without watch and stalled network with 70% lost packets - allanrogerr/public GitHub Wiki
test scenarios:
test if mc mirror hangs even if we don't use --watch
use --limit-upload + upload large file + lxc script to nuke network for X seconds (30 is probably enough)
Test if object or xl.meta ends up in an invalid state
that means: .. IF mc think it's fully copied and does not copy it again .. but the object is not in a proper state on the target: (xl.meta is wrong, size is too small, not readable, etc.. )
.. it's ok if the FIRST run of the mc mirror breaks objects when the network is bad.. but the SECOND run should realize and fix/overwrite.
Connect to source/client
ssh -p 20431 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Connect to target
ssh -p 20673 [email protected] -o "ServerAliveInterval=5" -o "ServerAliveCountMax=100000" -o "StrictHostKeyChecking=off"
Versioned
rm -rf ~/images
mkdir ~/images && cd ~/images && wget https://sample-videos.com/img/Sample-jpg-image-30mb.jpg -O image.jpg
for i in {1..10}; do cp image.jpg image$i.jpg; done
mc rb --force source/versioned
mc mb source/versioned
mc rb --force target/versioned
mc mb target/versioned
mc version enable source/versioned
mc version enable target/versioned
mc cp --recursive ~/images source/versioned
mc cp --recursive ~/images source/versioned
mc cp --recursive ~/images source/versioned
1.-
Add mirrors
ubuntu@mc-mirror-source:~/images$ mc mirror --json source/versioned target/versioned
{
"status": "success",
"source": "source/versioned/images/image1.jpg",
"target": "target/versioned/images/image1.jpg",
"size": 30789588,
"totalCount": 2,
"totalSize": 61579176
}
{
"status": "success",
"source": "source/versioned/images/image.jpg",
"target": "target/versioned/images/image.jpg",
"size": 30789588,
"totalCount": 1,
"totalSize": 30789588
}
{
"status": "success",
"source": "source/versioned/images/image10.jpg",
"target": "target/versioned/images/image10.jpg",
"size": 30789588,
"totalCount": 3,
"totalSize": 92368764
}
{
"status": "success",
"source": "source/versioned/images/image2.jpg",
"target": "target/versioned/images/image2.jpg",
"size": 30789588,
"totalCount": 4,
"totalSize": 123158352
}
{
"status": "success",
"source": "source/versioned/images/image3.jpg",
"target": "target/versioned/images/image3.jpg",
"size": 30789588,
"totalCount": 5,
"totalSize": 153947940
}
{
"status": "success",
"source": "source/versioned/images/image4.jpg",
"target": "target/versioned/images/image4.jpg",
"size": 30789588,
"totalCount": 6,
"totalSize": 184737528
}
{
"status": "success",
"source": "source/versioned/images/image5.jpg",
"target": "target/versioned/images/image5.jpg",
"size": 30789588,
"totalCount": 7,
"totalSize": 215527116
}
{
"status": "success",
"source": "source/versioned/images/image6.jpg",
"target": "target/versioned/images/image6.jpg",
"size": 30789588,
"totalCount": 8,
"totalSize": 246316704
}
{
"status": "success",
"source": "source/versioned/images/image7.jpg",
"target": "target/versioned/images/image7.jpg",
"size": 30789588,
"totalCount": 9,
"totalSize": 277106292
}
{
"status": "error",
"error": {
"message": "Failed to copy `https://mc-mirror-source.lab.min.dev:9000/versioned/images/image3.jpg`.",
"cause": {
"message": "Head \"https://mc-mirror-source.lab.min.dev:9000/versioned/images/image3.jpg\": dial tcp 1.2.3.4:9000: i/o timeout",
"error": {
"Op": "Head",
"URL": "https://mc-mirror-source.lab.min.dev:9000/versioned/images/image3.jpg",
"Err": {
"Op": "dial",
"Net": "tcp",
"Source": null,
"Addr": {
"IP": "1.2.3.4",
"Port": 9000,
"Zone": ""
},
"Err": {}
}
}
},
"type": "error"
}
}
{
"status": "success",
"source": "source/versioned/images/image8.jpg",
"target": "target/versioned/images/image8.jpg",
"size": 30789588,
"totalCount": 10,
"totalSize": 307895880
}
{
"status": "success",
"total": 307895880,
"transferred": 307895880,
"speed": 17099394.94355873
}
Immediately lose packets
sudo lxc exec mc-mirror-source -- sudo tc qdisc add dev eth0 root netem loss 75% && sleep 60 && sudo lxc exec mc-mirror-source -- sudo tc qdisc del dev eth0 root
Get basic stats
ubuntu@mc-mirror-source:~/images$ mc ls --recursive --summarize --versions source/versioned
[2024-02-15 23:36:05 UTC] 29MiB STANDARD 9d295b22-0b31-4c53-9550-59c27c3e1258 v3 PUT images/image.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD 233deab9-3505-497f-81e0-cd06ebb10c93 v2 PUT images/image.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD 01997afb-afba-4541-b9a2-99163bd14a0a v1 PUT images/image.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD 8b9b44f9-cd2e-40bb-a1d0-5461cc030d24 v3 PUT images/image1.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD 329a69a3-73ca-491a-bc58-d89fcdd454d5 v2 PUT images/image1.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD 55e5b1e8-0738-4e38-8d1f-a23c95e6c146 v1 PUT images/image1.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD 9e387e5c-4635-41ef-9bc3-4f4e640ad88d v3 PUT images/image10.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD e4cdbf89-aaf8-4dc8-866f-531f66f65d09 v2 PUT images/image10.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD 0a3bd645-b2b2-4199-9e32-bad594ba9ae0 v1 PUT images/image10.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD e83712fe-8968-49ca-ab20-587963461283 v3 PUT images/image2.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD 1183b3d8-d80f-4132-a10e-aadbc53ba67c v2 PUT images/image2.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD 2434d1f9-4941-40b8-a09c-eb71bdd34f54 v1 PUT images/image2.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD 6daeeeeb-de19-419e-a47e-314201b4c403 v3 PUT images/image3.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD 4f071b33-af3c-4a3a-b946-51d0926c3108 v2 PUT images/image3.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD 01a665e7-dff7-4247-b8b9-7a51ac13121e v1 PUT images/image3.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD aba3c6f5-b838-4a3b-86b6-60c04ab04afe v3 PUT images/image4.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD 6cee802b-2d31-4bdd-a1a4-43cb027b6edd v2 PUT images/image4.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD e939db9a-131a-4be7-9e2c-17e66bbc73f8 v1 PUT images/image4.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD 72741b2a-4964-4c3d-873d-a9574e6148eb v3 PUT images/image5.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD d5f531af-f143-4fa1-9f37-10cf6bd7c9e3 v2 PUT images/image5.jpg
[2024-02-15 23:36:02 UTC] 29MiB STANDARD be594d10-3b9b-4ca1-944b-3bb2c1f806db v1 PUT images/image5.jpg
[2024-02-15 23:36:06 UTC] 29MiB STANDARD 776cc044-9e8f-4078-b766-59f32f04aa04 v3 PUT images/image6.jpg
[2024-02-15 23:36:04 UTC] 29MiB STANDARD e7f562bc-71c3-4b54-bb0d-0fdcc1e7df8b v2 PUT images/image6.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD efa039ed-65cc-4334-929b-f1da5a354298 v1 PUT images/image6.jpg
[2024-02-15 23:36:07 UTC] 29MiB STANDARD 11cb58ce-531b-47fd-aeb0-65967cb57a4c v3 PUT images/image7.jpg
[2024-02-15 23:36:05 UTC] 29MiB STANDARD c6f6311d-08c5-4a0c-a743-8a7f28f198dd v2 PUT images/image7.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD c0efcf0d-0179-41a6-a526-329f1ee7a61c v1 PUT images/image7.jpg
[2024-02-15 23:36:07 UTC] 29MiB STANDARD 06c7bf3f-2149-403f-9205-6901aa3a34e3 v3 PUT images/image8.jpg
[2024-02-15 23:36:05 UTC] 29MiB STANDARD 6a51cb03-8904-4405-8c3a-e3531057c269 v2 PUT images/image8.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD 85b73317-c5f9-4b62-b904-bb11cab6144d v1 PUT images/image8.jpg
[2024-02-15 23:36:07 UTC] 29MiB STANDARD 96f36c15-aa64-4d51-9150-e1107492fc76 v3 PUT images/image9.jpg
[2024-02-15 23:36:05 UTC] 29MiB STANDARD 108e6976-45da-4a94-ad8f-10453f619fb6 v2 PUT images/image9.jpg
[2024-02-15 23:36:03 UTC] 29MiB STANDARD bc8ee7df-d041-4d55-b160-bf199b23c238 v1 PUT images/image9.jpg
Total Size: 969 MiB
Total Objects: 33
ubuntu@mc-mirror-source:~/images$ mc ls --recursive --summarize --versions target/versioned
Total Size: 0 B
Total Objects: 0
ubuntu@mc-mirror-source:~/images$
Test validity of object. Not applicable since no objects were actually copied.
for i in {1..10}; do mc cat target/versioned/images/image$1.jpg > /dev/null; done
2.-
Redo tests with --limit-upload [tested with 1k and 1M]
Add mirrors
mc mirror --json --limit-upload 1k --limit-download 1k source/versioned target/versioned
Immediately lose packets
sudo lxc exec mc-mirror-source -- sudo tc qdisc add dev eth0 root netem loss 75% && sleep 60 && sudo lxc exec mc-mirror-source -- sudo tc qdisc del dev eth0 root
Get basic stats
ubuntu@mc-mirror-source:~/images$ mc ls --recursive --summarize --versions target/versioned
Total Size: 0 B
Total Objects: 0
Test validity of object. Not applicable since no objects were actually copied.
for i in {1..10}; do mc cat target/versioned/images/image$1.jpg > /dev/null; done
3.-
Add mirrors again after network is restored
mc mirror --json source/versioned target/versioned
Get basic stats
ubuntu@mc-mirror-source:~/images$ mc ls --recursive --summarize --versions target/versioned
[2024-02-15 23:52:45 UTC] 29MiB STANDARD 4d3d568b-9bc8-44f0-a803-ed40437387df v1 PUT images/image.jpg
[2024-02-15 23:52:45 UTC] 29MiB STANDARD 52c4845c-8663-4800-83ed-33b230d55ee0 v1 PUT images/image1.jpg
[2024-02-15 23:52:46 UTC] 29MiB STANDARD d17d8c03-788d-4e30-ab3b-30986d367c82 v1 PUT images/image10.jpg
[2024-02-15 23:52:46 UTC] 29MiB STANDARD 9f56a4fc-cca6-4bbd-8adb-cd3617e60299 v1 PUT images/image2.jpg
[2024-02-15 23:52:47 UTC] 29MiB STANDARD 173197cd-49eb-4376-92b2-2e269e140a03 v1 PUT images/image3.jpg
[2024-02-15 23:52:47 UTC] 29MiB STANDARD 086b9f48-4301-4063-afe2-e4486683973b v1 PUT images/image4.jpg
[2024-02-15 23:52:48 UTC] 29MiB STANDARD 81e137ff-3d69-4eb9-8edf-d4dd28f7c67c v1 PUT images/image5.jpg
[2024-02-15 23:52:48 UTC] 29MiB STANDARD 18490217-789c-4449-8c4c-a68d92255cbf v1 PUT images/image6.jpg
[2024-02-15 23:52:49 UTC] 29MiB STANDARD 74f512a5-757c-4c28-ade6-02e86024a198 v1 PUT images/image7.jpg
[2024-02-15 23:52:49 UTC] 29MiB STANDARD 62d9d64a-064b-4ff1-ae31-bbee9642a984 v1 PUT images/image8.jpg
[2024-02-15 23:52:49 UTC] 29MiB STANDARD 8d0113f4-acd9-40ed-9ad8-2de40982cd60 v1 PUT images/image9.jpg
Total Size: 323 MiB
Total Objects: 11
Test validity of object. No errors thrown
for i in {1..10}; do mc cat target/versioned/images/image$1.jpg > /dev/null; done