base_xenforoforum adapter sites

Three sites are supported using base_xenforoforum adapters:

SpaceBattles Creative Writing(SB)

SufficientVelocity User Fiction(SV)

Questionable Questing(QQ)

Forums are not really designed to host stories, but people use them that way anyway. When I started reading stories from the Worm fandom (an original web serial by Wildbow), the largest collection of stories I found for it was on a couple of near identical forum sites: SB & SV.

(QQ was later added by request because it is also nearly identical. QQ is the only one that supports user/password login to access its NSFW sections.)

The biggest problem with downloading stories posted to a forum is picking out the 'chapter' posts from all the other posts, since most stories on these sites combine chapters and reader comments. Plus, there's more than one way such stories are organized.

Supported story forms

I've identified and supported three different story forms:

• Threadmarks First, a story thread can have 'Threadmarks' which link to specific posts. If a thread URL is given, and the thread has 2 or more Threadmarks, each Threadmarked post is considered a chapter--a number of stories I found had a single Threadmark for an 'Index' post. It's not uncommon to also have an Index post (see below), but Threadmarks take priority. You can give FanFicFare the full thread URL, but it will remember and use the 'normalized' URL without the thread title. Examples:

  • https://forums.spacebattles.com/threads/323254/
  • https://forums.sufficientvelocity.com/threads/advice-and-trust-nge.8768/ (normalizes to https://forums.sufficientvelocity.com/threads/8768/)

• First Post Index Second, a story thread can have an 'Index' in the first post containing links to chapter posts. If a thread URL is given (and there are fewer than minimum_threadmarks Threadmarks), it's assumed that the first post of the thread is the Index. As an added complication, sometimes the Index post also contains the first chapter. The first post/index is included as the first chapter titled 'First Post', unless it's also linked to explicitly in the post itself. Examples:

  • https://forums.spacebattles.com/threads/297826/
  • https://forums.spacebattles.com/threads/bug-on-a-wire-worm.272435/

• Post Index URL Third, a story can have an Index post that isn't the first post of the thread. If a specific post URL is given, that post is used as the Index and included as 'First Post' if not explicitly linked. Or you can give an Index post URL to explicitly not use the Threadmarks, for example if Omakes are listed in the index post but not in threadmarks. As with thread URLs, you can give a full post URL, but it will be normalized by FanFicFare. Examples:

  • https://forums.spacebattles.com/posts/12990114/
  • https://forums.spacebattles.com/threads/hey-missy-worm-peggy-sue-on-hiatus.318440/#post-15732928 (normalizes to https://forums.spacebattles.com/posts/15732928/)

• One-shot Post A post URL can also be used to download a one-shot or single chapter story, as are common in the Discussion/Idea threads. But because they take the thread title by default, you'll have to set the title manually afterwards or setup an Individual Story Section. Examples:

  • https://forums.spacebattles.com/threads/wormverse-ideas-recs-and-fic-discussion-thread-35.305746/page-75#post-14833156 (normalizes to https://forums.spacebattles.com/posts/14833156/)
  • https://forums.sufficientvelocity.com/posts/1647460/

In all cases, the title of the story is taken from the thread title and the author from the first post.

In all Index post cases, an Index post can contain links to chapter posts in other threads. Only links to posts on the same site will be included. Example: https://forums.spacebattles.com/threads/309577/

As of this writing (Oct 2016), SB has reinstated thread length limits, forcing arbitrary thread splits. I don't believe threadmarks can be used to point to posts in previous threads. So stories may be forced to abruptly split. Example: https://forums.spacebattles.com/threads/426988/

Unsupported story forms

In addition to the supported story forms, there are some common, but unsupported story forms:

• No Index I've seen story threads that contain multiple chapter posts but lack either Threadmarks or an Index post. These stories are not supported because of the difficulty of distinguishing chapter posts from comment posts. Example:

  • https://forums.spacebattles.com/threads/313243/

• Story Only (without Index) 'Story Only' threads aren't supported unless they have an index because FanFicFare has no way to know that it really is a Story Only thread. The potential for abuse (downloading lengthy discussion threads) is just too high to provide a version that can download all posts.

Other details:

• Normalized URLs FanFicFare 'normalizes' both the story URLs and chapter URLs for base_xenforoforum adapter sites. This is because the URLs as shown in your browser, which are meant to be more human-readable, can change if the thread title is changed or if a post is moved from one thread to another.

• Update Notices I have yet to find an effective 'update notice' email mechanism for these sites. 'Watching' a thread works for story threads that have fallen dormant, but it's common for discussion (and derailments) to continue for weeks after the last chapter update. Email notice on OP update may work for you, depending on the activity of the thread.

• Dates are a bit uncertain.

• datePublished is set to oldest Threadmark date, or posted date of the Index post.

• dateUpdated is set to newest Threadmark date, or updated date of the Index post.

• Unknown Metadata status (Completed/In-Progress) and numWords are not set from site data. In the Calibre Plugin version, there are two options for counting words after downloading: FanFicFare Config->Basic->Calculate Word Count (preferred) and FanFicFare Config->Count Pages->Word Count (legacy). The common metadata genre and category are also empty. Instead, forumtags and tagsfromtitle are included in subject/tags.

• In-line Post Index URL I don't have the example off hand, but I've seen at least one story that included the most up-to-date index in the same post as a later chapter without linking to it. FanFicFare has no way to detect that, which will place that chapter out of order in First Post.

Useful INI Setting for [base_xenforoforum]

There are a number of [base_xenforoforum] specific INI settings and changes to common default settings.

• strip_chapter_numbers:false is explicitly set for [base_xenforoforum] because of how many chapters labeled simply '1.1', '1.2', etc I saw. So even if you set strip_chapter_numbers:true in your [defaults], it will be false for [base_xenforoforum] unless you also set it in [base_xenforoforum] in personal.ini.

• Some fairly complex replace_metadata magic is included by default for [base_xenforoforum] to find 'tags' in the thread title and include them as subject/tags and on the Title Page, but not in the title. It also removes 'Thread 5' etc. This is instead of the usual genre and category metadata we collect from most sites.

  • Example: Way of the Open Hand (Worm/D&D) becomes Title: Way of the Open Hand with Tags: Worm, D&D

• SB & SV now include thread Tags, but not all threads use them. By default, those tags will be populated in the metadata field forumtags and included as subject/tags when the index was found in Threadmarks. You can set always_use_forumtags:true to use them without Threadmarks. By default, forumtags are automatically capitalized. Set capitalize_forumtags:false if you prefer the original lowercase.

• description is set from the content of the first post or Index post. description_limit:500 is set by default for [base_xenforoforum] adapters because the first/Index post is often also the first chapter.

• Even when using the first post as the chapter list, the first post/chapter will not be updated in the normal operation of the Update EPUB feature because Update assumes past chapters don't change. You can force it to be updated by setting always_reload_first_chapter:true.

• always_include_first_post:true can be set to force inclusion of the first post. This is useful if there are Threadmarks, but the first post isn't threadmarked or if index post is not the first post of the thread but is linked in the post itself. There's a chance this can cause the post to be included more than once.

• The title 'First Post' can be changed with INI parameter first_post_title.

• The number of Threadmarks required to use threadmarks for chapters can be adjusted with INI parameter minimum_threadmarks.

• Because base_xenforoforum adapters can pull chapter URLs from human written posts, the odds of errors in the chapter URLs are vastly higher than for most sites. Usually, a bad chapter URL will abort the entire download. You can set continue_on_chapter_error:true to continue after failing to download a chapter and instead record an error message in the ebook for that chapter.

• normalize_text_links if set true will normalize all links in chapter texts, if they match chapter URLs. This will both fix partial URLs that wouldn't otherwise work and allow for the next setting:

• internalize_text_links if set true will search all links in chapter texts and, if they match any included chapter URLs, replace them with links to the chapter in the download. Only works with epub and html output formats. base_xenforoforum adapters should also use normalize_text_links with this. Now all those links in the index post correctly link to the chapters in your epub instead of going back to the site.

Settings per Individual Story

Given the diversity of forum stories, you may want to use different settings for different stories. This is possible using Individual Story URL Sections. Note that you should use Normalized URLs and match the thread or post URL exactly.

Here are some examples:

## Change story title, add categories.
[https://forums.spacebattles.com/threads/426988/]
replace_metadata:
 title=>.*=>The Battleshippening
extracategories:Kancolle, Alternate Universe, Cute Battleships Doing Cute Battlethings

## Change story title and author, add categories.
[https://forums.spacebattles.com/posts/20867703/]
replace_metadata:
 title=>.*=>Belated Battleships: An Old Cat
 author=>.*=>Skywalker_T-65
 authorUrl=>.*=>https://forums.spacebattles.com/members/skywalker_t-65.318921/
extracategories:Kancolle, Alternate Universe, Cute Battleships Doing Cute Battlethings

## Change First Post title, add tags.
[https://forums.spacebattles.com/threads/346640/]
always_include_first_post:true
first_post_title:Prologue and Chapter 1
extratags: Original, Sci-Fi, AI

## replace_br_with_p for this story only.
[https://forums.spacebattles.com/threads/384594/]
replace_br_with_p:true