EXH Refresh Throttling - Serendipitously/pfive GitHub Wiki

Because there is no 'search', or 'browsable' API available for exhentai.org, the only way to detect new galleries being added to the system (and receive their gallery_id + gallery_token combinations) we must hit the main exhentai home page and scan the HTML for the links to the 25 newest galleries.

However, exhentai has throttling and bot detection systems on this endpoint which will IP ban you after a certain number of automated refreshes. Below are some of the current test cases to see what these throttling limits are to determine a scheme to low-profile them.

##The following tests are hitting the g.e-hentai.org homepage (simply because I don't have authentication set up quite yet)##

• Refresh 5 times in 5 seconds: 1 Hour ban on 6th retry.
• Refresh 4 times in 5 seconds, then 2 times 1 min later: 1 Hour ban on third retry.
• Refresh 4 times in 5 seconds, then 3 times 15 min later: 24 Hour ban on 4th retry. (likely because I had been banned on this IP before)
• Requesting increasing page numbers each second, refreshed 6 times in 10 seconds: 1 Hour ban on 7th retry.
• Requesting increasing page numbers once a min, refreshed 7 times before 1 Hour ban on 8th retry