In the end, what we all truly care about when using NAI is getting coherent, well-written, creative stories. This is the reason we tinker with generator settings, author's note, biases, etc… But without a good measure of story quality, it is hard (if not impossible) to judge if and how these things really affect the output. Does increasing top k sampling by .05 impact coherence or creativity? Does it depend on the randomness setting? Or do all these parameters actually do nothing at all, and we are just debating placebo effects? If we had a good, well-constructed measure for human ratings, we could run actual studies on these things and put together how all these settings influence the story on a grand scale, piece by piece.

The tl;dr is AI researcher are extremely intelligent people, and papers on language models are usually way over my head. However, they are not psychometricians, and so their way of rating story quality is… often not great. Getting usable data from human raters require you to understand how they approach stories, and building your scale based on real data.

Here is the thing – measuring a complex concept like “fluency” or "interestingness" with a single question is terrible. If you are really interested in a longer discussion of this topic, see this article, for example.

But in a nutshell: The measures commonly used by AI researchers for rating stories frequently have serious weakness and would not probably not pass review in fields focussed on measuring human attitudes and opinions. Even if you happen to ask well-written questions (some are not, though), results will still be unreliable and have low predictability. That is just the fate of measuring complex, multi-faceted concepts with a single question. Single-Item measures are also terrible for many types of statistical analyses (even just reporting things like the mean or correlation is dubious). The fact that none of the commonly used questions did go through any preliminary testing to see if they work well, probably makes the whole issue even worse.

Furthermore, just because your theory says that the deciding aspects of AI-generated stories are fluency, relevancy, coherence or whatever, does not mean that the average human actually considers these categories when rating stories. The constructs used by human to assess stories might have a very different structure than your theory. Which is why you need to check your scale before you wreck your scale! Otherwise, you might happen to think you are measuring one thing, but you are really just measuring a bunch of nonsense (called low validity). Or you are measuring the thing you want to measure, but you picked questions that are so imprecise to be basically worthless (called low reliability).

© Nevit Dilmen
Hey, look, you already learned the basics of psychometrics!

I regularly consult people on scale construction in my job, and I have never seen a purely theoretical scale survive real-world data. Humans are complicated…go figure.

I drafted a set of 73 potential questions. These are way more questions than we need, but out of experience, I expected to throw away 50% of the questions or more because they turn out to not be useful for various reasons. That is part of the point of the study – to figure out which questions actually work by looking how people actually approach these questions.

Questions were taken from existing research, drafted by me or… Sigurd (pro-tip for behavioral researchers: NAI is great for drafting scales!). I categorized them into 7 theoretical aspects that made sense to me at the time: Cohesion, consistent characterizations, creativity, general story quality, repetitiveness, style, pacing.

I recruited people for rating the stories in two ways (roughly 50/50):

• Asking people on the NAI and AIM discord as well as the NAI Reddit to participate
• Recruiting participants from panels for academic research (SurveySwap.io and SurveyCircle.com)

Using nrt, stories were generated by giving Euterpe a short prompt to establish the genre (High Fantasy, Hard Sci-Fi, Historical Romance or Horror). Euterpe ran through 30 generations with one of 8 default presets. This included every default preset except for ProWriter (was not a default preset when I sampled the stories) and Moonlit Chronicler (nrt did not support top-A sampling yet). The results were 320 stories (40 per preset) that would each take about 5 minutes to read through.

The full dataset of all stories can be downloaded here.

Oh, you really clicked that thing? Well, hello there, curious traveler!

The following table should hopefully give a rough idea of the steps I went through in the analysis. This is an attempt to explain the general idea for people without statistical expertise. It is not a full explanation of the statistical procedures! This is the very rough, “overly simplified and probably wrong” version. Really, I just hope no statistician is reading this…

Nonetheless, “statistical procedure” in the table is the technical description of what I actually did. It is there for the people with stats education.

(Although, if you are interested in a full report of the analysis, maybe better wait until I did a technical write-up of the whole thing. I'll link it here when it is done.)

This way I ended up with 8 questions for coherence, 7 for creativity, 3 for consistent characterization, 4 for pace and 3 for avoiding repetition. I used all of those for the analyses below.

However, the difference in precision you get from 8 versus 6 questions should be minimal. And I would like to avoid unnecessarily long questionnaires. Guidelines for constructing questionnaires like this generally recommend 4-6 questions per domain for these reasons.

So, I capped the number of questions per aspect to 6 for future studies. So, the final AISS v1 ends up with 6 questions for coherence, 6 for creativity, 3 for consistent characterization, 4 for pace and 3 for avoiding repetition – 22 questions total. This should take about 3 minutes or less to finish, so that should be fine for shorter studies.

The following graph shows the performance of the presets for continuous story generation on the 5 story aspects.

A value of 0 represents the expected average over all presets. So bars that point downwards indicate performance below the average, bars that point upwards indicate performance above the average.

(These results are corrected for differences between prompts. So if a genre got more “difficult” prompts by bad luck, this is corrected for in this graph).

Note that small deviations from average are probably just due to randomness. The following differences were strong enough that we have statistical proof to assume that these differences represent something real (testing with alpha = .1):

• Genesis: Less coherent than the average
  (stats with sum (deviation) contrast coding are b = -.31, adjusted p = .04))
• Ace of Spades: Less repetition than the average
  (b = .37, adjusted p = .01))
• Morpho: More repetitions than the average
  (b = .72, adjusted p < .001))

At least for continuous story generation, I would therefore that you do not use Genesis. It seems to underperform on coherence, with no benefit. It is, of course possible, that Genesis does well on some other aspect that was not measured, but at this point there is no proof for this.

Based on these results, my recommendation for continuous story generation would be to use Ace of Spades as the default preset for continuous story generation. It does better at avoiding repetition than the rest. A possible caveat would be that it is unclear if Ace of Spades hurts memory or lorebook usage. However, Ace of Spades repetition penalty is not especially high (Genesis and Basic Coherence have higher rep pen). Instead, it seems to benefit from its extremely high repetition penalty slope. So early parts of the context should not be affected much from rep pen, while later parts will be strongly affected. Since memory and lorebook is usually at the beginning of the context, Ace of Spades should actually penalize these less than many other presets.

Morpho's lack of rep-pen really hurts its performance without user intervention. The story visible degrades the longer it continues with this preset. (Example outputs here. Arguably, Morpho was never intended to be used without user intervention. So, I do not recommend Morpho for continuous story generation. It might have its use to spice up single outputs or if you do not mind heavy editing and steering, but there currently is no data on this.

Basic Coherence, Ouroboros, and Fandango more or less perform the same on all story aspects. They are solid presets for continuous story generation, but at the same time they do not really stand out in any aspect.

Low Rider and All-Nighter also are more or less average on all story aspects. However, they turned out to differ in their consistency in some aspect. This might make them useful for special purposes, or at the very least make them interesting case studies. Read on below for more about this.

Okay, if you are not familiar with statistics and the concept of variance, the following analysis might be a bit confusing. However, I think the findings are quite interesting, and I'll do my best to walk you through it.

So let's take a few steps and think about what presets might do:

1. Cut out bad tokens – this should increase the average scores on story aspects. These average scores over several outputs are what we have analyzed so far.
2. Change the distribution of token probabilities. The clearest case is the randomness setting. This can either make likely tokens even more likely, leading to fairly consistent outputs. Or it can make likely tokens less prominent and strengthen somewhat unlikely tokens. This would make outputs more varied. In other words, presets might differ in their variance of story aspect scores. Two presets might differ in how consistent their outputs are, even if they have the same average.

To visualize this point, imagine the following example:

• Preset 1 only produces 50% outputs that are complete incoherent garbage (score 1) and 50% outputs with perfect stories that are totally plausible with a clear theme (score 5)
• Preset 2 only produces outputs that are okay – somewhat incoherent in places, but in general fine (all score 3)

Both presets have the same average, but clearly different variance. Preset 1 is maximally inconsistent, preset 2 is maximally consistent. These are extreme examples, but it might be interesting to test if we have differences of consistency within presets. This is what this analysis does.

The result of this analysis was:

• Presets differed in their consistency regarding coherence:
  • All-Nighter was less consistent on coherence than the average (adjusted p = .01).
• Presets differed in their consistency regarding Pace:
  • Morpho was more consistent on Pace than the average (adjusted p = .08). Remember that Morpho also had lower pace than average in general. So, in other words, Morpho will consistently produce stories that have a low pace.
  • Low Rider, on the other hand, was less consistent on Pace than the average (adjusted p = .08).
• Presets showed more or less the same variance when it came to Creativity, Repetition and Consistent Characterizations.
• Prompts don't differ in variance for any of the story aspects, in case you were wondering.

So visualizing variance in a way that is easy to grasp is a bit tricky, but we will try box plots. (Strictly speaking, box plots don't visualize variance, but rather distance between percentiles. However, they are the easier to process than other visualizations and should get the point across best.)

Below are two graphs that compare the consistency of the relevant presets to the rest. For our purposes, the important thing to understand about the two graphs below is that the “box” in these graph represents the middle 50% of the data. So, a narrow box means that most data is restricted to a narrow range. A long box means that values have a wider spread.

As you can see, All-Nighter has about the same average as the rest, but outputs are much more varied. All-Nighter will give you more outputs that are incoherent nonsense, but you should also have a higher chance for the occasional stroke of genius and get something that is more coherent than usual.

Low Rider has about average pace over all its stories, but the stories are much more varied in pace than usual. Low Rider will generate more stories with low pace, but also more fast-paced stories.

Morpho has a lower pace than average (see above), and seems very consistent about generating stories with low pace.

Remember that none of the presets produce more coherent or more fast-paced stories on average than the rest. However, All-Nighter shows less consistency when it comes to coherence, while Low Rider is less consistent about pace. As long as you do not mind spamming retry, these properties of the presets could be utilized if you want more coherent or fast-paced output:

• If you want strongly coherent outputs and are willing to retry a bunch, try All-Nighter. You'll have to deal with more nonsense outputs, but you'll also get more good ones in exchange.
• If you want fast-paced stories and don't mind retrying slow-paced outputs, try Low Rider. You will get more slow-paced output than usual, but in exchange, you will also get more fast-paced ones.

We will start with stating some of the more obvious observations.

First, a large part of how well a preset does, seems to be determined by how well it avoids repetition. Morpho unsurprisingly doesn't do well here, since it applies no rep pen at all. Arguably, keeping repetition at bay isn't even the goal of Morpho. But for continuous story generation, the amount of repetition just tanks Morpho's performance. Other presets apply a rep pen between 2.165 (Ouroboros) and 3.05 (Basic Coherence). In general, within this range there does not seem to be large differences when it comes to repetitions – these presets perform about the same on avoiding repetition, except for Ace of Spades. This one positive exception is noteworthy, however. Ace of Spades applies a relative extreme repetition penalty slope of 7.02, while other presets stay well below 1.0. The fact that Ace of Spades seems more successful than other presets at avoiding repetitions with seemingly no downside suggests that it might be worth experimenting more with high values of rep pen slope.

Another preset that should be looked at more closely is Genesis. The default preset for Euterpe underperformed on coherence, and it is not immediately obvious why it should. Genesis could be characterized as very light sampling with Nucleus and TFS (0.975 for both) combined with low randomness (0.63). Something in that combo does not seem to work as intended, but it is unclear what exactly. Morpho comes somewhat close as a preset with no sampling and low randomness (0.6889), but surprisingly, it does not show the same issues as Genesis. Morpho, of course, does not have any rep pen, while Genesis is relatively heavy on rep pen (2.975). So, maybe low randomness combined with strong rep pen is somehow problematic? I have no clue why that should be the case, though…

Speaking of randomness, All-Nighter was an interesting case in that it performed around average coherence, but did so much less consistently than other presets. It had much more pronounced variance on coherence than other presets. We do not need to search long for the reason for this increased randomness. It is… randomness!

All-Nighter is essentially strong TFS sampling (0.836) combined with high randomness (1.33). Dialing up randomness does not seem to hurt average coherence much, but it does increase the spread around this average. There are two more results from low randomness presets that seem to fit well with this: Morpho (randomness = 0.6889) and Basic Coherence (randomness = 0.585), both had a tiny bit lower variance when it comes to variance. I did not list these results above because the evidence for these effects is fairly weak and might still just be due to chance (both adjusted ps = 0.149). However, it does add weight to the suspicion that randomness does increase the variance of coherence.

There is one last finding on presets that I am not quite sure what to make of: Low Rider and Morpho were less consistent than other presets on pace. Honestly, no idea why. The evidence on these effects are somewhat weak (both adjusted ps = 0.08) so maybe these are just coincidences. Future studies will have to show…

The prompt for Horror was this:

I woke up to hear knocking on glass. At first, I thought it was the window until I heard it come from the mirror again. I got out of bed and walked over to the mirror. When I looked into it, there was a face looking back at me.

Two typical outputs are here

In both cases, it feels like Euterpe understood what things sound scary in theory. But she just throws them in with no sense of connection to the story.

Just take the plot points from excerpt 2:

• Scary demon appears in mirror in bedroom
• Police is called (police is super common with this prompt somehow)
• They go up to the bedroom to find the Sheriff lying there dead (?)
• Protagonist floats up for some reason (?!)
• Sheriff gets up as a monster and attacks the mom who is now suddenly his wife (?!?)
• Protagonist floats downwards towards a light and finds himself reborn (?????)
• Protagonist is now reborn, lives a good life and is fighting the devil… █▄▄ ███ █▄▄ █▄█▄█ █▄█ ▀█▀

Yeah, okay, that will feel like Griffin lead to low high pace and low coherence ratings, alright. We have lots of stuff happening, and taken for themselves, they are not necessarily out of place in a horror story. But they have no real connection to each other. On the one hand, this could just be a fine-tune thing. There might not be enough horror stories in the fine-tune dataset for Euterpe to be any good at this. On the other hand, this type of prompt might just be confusing for language models in general: The horror comes in this prompt stems from something being “terribly wrong” - mirrors should not work like that. Language models should in most cases be trained to avoid selecting tokens that are “wrong” like that. In most stories, we would want mirrors to show a reflection, not the face of a stranger. So in a way, this prompt might be an uphill battle for most language models.