Crawldaddy Benchmark 23.10.2018 - linovaldivia/crawldaddy GitHub Wiki

Tests were run on a dual-core machine with 8GB RAM, on a ~25 Mbps connection (downstream) Sites were crawled 5 times and the number of internal links (links to pages hosted on the same host) crawled was limited to 3000.

Site	Alexa Global Ranking	Total links found	Int links crawled	Pass 1	Pass 2	Pass 3	Pass 4	Pass 5	Avg time
jsoup.org	155,846	649	542	28.314s	22.911s	19.16s	18.124s	18.407s	21.354s
kafka.apache.org	1,450	654	131	27.787s	26.89s	25.437s	25.407s	24.185s	25.781s
www.fgc.cat	205,838	692	431	5m 14.108s	7m 5.740s	6m 16.857s	6m 2.936s	5m 49.711s	6m 5.870s
www.docker.com	2,556	1,559	253	54.160s	26.936s	26.921s	26.896s	26.817s	32.346s
thethriftytravellers.com	16M+	2,201	1692	5m 2.879s	4m 54.876s	5m 10.367s	5m 7.825s	5m 3.648s	5m 3.919s
www.eclipse.org	3,927	6,186	3,000	15m 44.692s	15m 24.673s	20m 49.264s	20m 40.880s	15m 33.415s	17m 38.584s