Other general considerations - cogstat/cogstat GitHub Wiki

Fast, preliminary results vs. slow, precise results.

Some analysis might be slow to run (bootstrapping, some of the diffusion analyses, etc.). Some descriptions note that the faster solutions might be preferred. From the viewpoint of published results, the most precise available solution should be preferred. (A research project takes weeks or more frequently months or even years to be completed, then published. In this time frame, even the relatively slow data processing solutions are rather fast (usually taking minutes or occasionally hours to run). So after spending some months with a work, a researcher probably wouldn't want to lose quality to save a few minutes.) Still, in some exploratory phase of the research, faster but less precise solutions might be reasonable options to choose. But these faster and more imprecise results should be used only for a first fast check and not for publications.

• For this reason, CogStat may provide two solutions. A "fast preliminary" version and a "slow precise" version
• The user might choose between them on the appropriate dialog with a checkbox (or with other simple solution).
• Because CogStat offers a fast way to check many aspects of the data, the "fast preliminary" option could be the default option.
• When running the fast preliminary version, the output always should warn the user that "This is a fast preliminary calculation. Use the Precise option for calculating the to be published results."

Present the results of suboptimal procedures

Usually, best practice solutions are preferred for the CogStat pipeline. However, in some cases it might make sense to use less ideal solutions, when those solutions are widespread and can help readers to compare the actual results with former findings. In those cases, the output should warn the user that these suboptimal solutions should be used only for comparing the present result with former results.

What not to include

There are several indices that can be calculated, in fact, calculated in other packages but are less useful in the sense that alternative indices are easier to interpret or use. For example, skewness and kurtosis are mostly not interesting for themselves but are used to estimate whether a variable is normally distributed, but for the latter aim, there are better tools (e.g., hypothesis tests that may also consider the variation of the relevant statistics depending on the sample size). As another example, the standard error is usually not interesting in itself but is, for example, a tool to find an interval estimate; for the latter aim, it is better to check the interval estimate directly. In those cases, these indices shouldn't be displayed because there are better alternatives to reach the relevant information.