Problematic Sites

In recent years, we've seen more and more sites become more and more difficult for FanFicFare to access.

Often, this has been because of the site using Cloudflare or other tools to protect themselves from hostile attacks and overly aggressive web crawlers.

Unfortunately, FanFicFare has many commonalities with these bad actors, itself being an automated page scraper and downloader. With the result that, intentionally or not, FanFicFare is frequently blocked on some sites. (Personally, I suspect the explosion of people and organizations harvesting anything and everything they possibly can for AI training is a significant factor.)

FanFicFare has two major features to help work around these problems. Neither is a perfect solution. These are Browser Cache and Proxy Features.

FanFicFare has also implemented several site specific features to address specific issues.

Browser Cache Feature

FanFicFare has code to allow it to try reading pages from your regular web browser's cache on disk (Chrome, Firefox, Edge, etc). Originally, this was only designed to work with FanFiction.net, but it was later found to be useful with other sites as well.

The feature was originally designed to work with the stories pages being loaded (and cached) manually or using other download tools, and to try direct download if a cached page wasn't found.

In practice, it's become common to only use cached pages and to automatically open those pages in your default web browser with these settings:

use_browser_cache:true
use_browser_cache_only:true
open_pages_in_browser:true

This works better with sites like Archive of Our Own where the is_adult flag causes FanFicFare to append additional URL parameters to page requests. It's difficult at best to manually pre-cache those pages.

While Browser Cache can't log in for you (there's no way to do an HTTP POST request via open_pages_in_browser), it can work on stories requiring login with sites like Archive of Our Own when you are already logged in in your web browser.

See the BrowserCacheFeature wiki page for more details.

Proxy Features

FanFicFare has code to allow it to call two different proxy servers:

• fanfictionnet_ff_proxy was created specifically for use with FanFicFare and fanfiction.net, but hasn't been updated in some time.
• FlareSolverr is a more generic proxy that sees more regular updates, but doesn't allow for image downloads.

In practice, I believe more people use Browser Cache now than Proxies. See the ProxyFeatures wiki page for more details.

Site Specific Problems

The most popular fanfiction sites, FanFiction.net (aka ffnet) and Archive of Our Own (aka AO3), as of this writing in April 2025, have been having site reliability issues for months. ffnet has had problems on and off for years at this point.

FanFiction.net

For most FanFicFare users, ffnet has been blocking direct download by FanFicFare since 2021. At this point, I don't really recommend FanFicFare for new users for downloading from ffnet. Most people who still use FanFicFare with ffnet use Browser Cache to do so.

There are a few additional (non-Cloudflare) problems specific to ffnet. FanFicFare has features to help address these that off by default and need to be set true under [www.fanfiction.net] in your personal.ini setting to take affect. See the personal.ini wiki page for more details.

Chapter List Not Updating

ffnet has a tendency to not update the chapter list right away. It's become common for new chapter notification emails to arrive and the new chapter URL to work, but the first chapter of the story to not show it for hours, days, or sometimes ever. This has happened on and off since at least 2015.

FanFicFare implements the check_next_chapter option to help:

## If set true, FFF will look at the highest numbered chapter listed
## on the first chapter and attempt to load the *next* chapter number
## after that.  Note that if the last chapter *is* listed, this will
## result in several attempts to open a chapter that isn't there.  If
## you are using open_pages_in_browser, this can be extra annoying.
## All metadata (word count, characters, etc) will be taken from the
## 'next' chapter, if found.

check_next_chapter:true

Other Metadata Not Updating

Somewhat more recently (since 2022), ffnet has had tendency, even after the chapter list has updated, to not update the other metadata (word count, characters, etc) for a story right away on the first chapter.

FanFicFare implements the meta_from_last_chapter option to help:

## ffnet sometimes doesn't update the metadata on the first chapter
## immediately.  meta_from_last_chapter:true will instruct the adapter
## to take the metadata from the last chapter instead of chapter 1.
## Unfortunately, we've seen cases where that chapter isn't up to date
## either, even in cases where the last chapter was found by
## check_next_chapter.  (No, it is not feasible for FFF to look at
## every chapter for the most update to date metadata.)

meta_from_last_chapter:true

Random Story Chapters Fail

Very recently (2025), ffnet has had tendency for random story chapters to fail to load.

FanFicFare implements the somewhat kludgey try_shortened_title_urls workaround feature to address it.

Users noticed that while ffnet apparently 'remembers' the failure (ie, retrying with the same URL continues to fail), if you try a slightly different URL, like removing part of the story name, it might work. Again, this is only for fanfiction.net.

## When try_shortened_title_urls:true, FFF will, on chapter failure,
## try again after removing a character from the end of the story name
## in the chapter URL.  And it will try up to twice, ie, it will try:

## https://www.fanfiction.net/s/999999/26/This-is-the-Story-Title
## https://www.fanfiction.net/s/999999/26/This-is-the-Story-Titl
## https://www.fanfiction.net/s/999999/26/This-is-the-Story-Tit

try_shortened_title_urls:true

This feature isn't currently documented in defaults.ini because I'm really hoping ffnet get their servers fixed.

NOTE: If you turn on check_next_chapter:true and try_shortened_title_urls:true, you will get a lot of attempts if there isn't a next chapter to find. If you have open_pages_in_browser:true, you'll see a bunch of attempts pop up in new browser tabs.

Archive of Our Own

In early 2025, AO3 started having issues, first with more Cloudflare blocking, and then with high rates of server errors. Given that the reported(#1158) server errors occur far more commonly with FanFicFare than to users in web browsers, it's probable that these are also, at least partially, an automated downloader blocking mechanism.

AO3 publicly reported they were having server issues in early 2025 with plans to get new servers. So hopefully, things will improve in future.

Mirror Sites

AO3 has a number of official alternate domains. Most just redirect to archiveofourown.org, which doesn't help us.

Two, however, appear to be full mirrors (copies) of archiveofourown.org: archive.transformativeworks.org and archiveofourown.gay. And don't seem to throw the same server errors as we get on the main site.

That gives us a couple options that don't require using Browser Cache.

I learned about archive.transformativeworks.org first and implemented a workaround feature:

## When set true, this setting causes the AO3 code in FFF to replace
## archiveofourown.org with archive.transformativeworks.org before
## every network request.  This does *not* change the story or chapter
## URLs saved by FFF, they will still be archiveofourown.org.

use_archive_transformativeworks_org:true

This, however, still has a problem. The server for archive.transformativeworks.org appears to be misconfigured because you cannot login on archive.transformativeworks.org, even in your web browser. So user-restricted stories cannot be downloaded.

Sometime later I learned about archiveofourown.gay, which is also a mirror, not a redirect.

## When set true, this setting causes the AO3 code in FFF to replace
## archiveofourown.org with archiveofourown.gay before every network
## request.  This does *not* change the story or chapter URLs saved by
## FFF, they will still be archiveofourown.org.

use_archiveofourown_gay:true

archiveofourown.gay does allow login, and therefore downloading of user-restricted stories.

In versions prior to 2025-05-04, any stories with images stored on media.archiveofourown.org (mostly on works archived by Open Doors) will also have the URLs changed to an invalid DNS and fail to load. This was fixed.

NOTE: If you try to set both use_archive_transformativeworks_org:true and use_archiveofourown_gay:true, you will get archive.transformativeworks.org. Set one or the other, not both.

Browser Cache Feature

FanFicFare's Browser Cache Feature will, with some limitation, work always_loginwith AO3. As mentioned above, if you also have is_adult:true set, or the work requires 'adult' permission, it will work better with open_pages_in_browser:true to get the cached pages' URLs exactly correct.

The biggest limitation is that AO3 allows authors to restrict their works' visibility to registered, logged in AO3 users. So you also need to be logged in to the site in your default web for best results.

NOTE: While using Browser Cache Feature with AO3, you should comment out always_login and username under [archiveofourown.org] in personal.ini. If username is set, FanFicFare will attempt to login for Get Story URLs from Web Page. This was a compromise between needing always_login and adding another setting specifically for Get URLs because user-restricted stories don't even appear in series when you're not logged in.

NOTE: Use either Browser Cache or a Mirror Site as described above. It is not recommended to try and use both at the same time.