Comment Coding - tufts-oir/QualtricsTools GitHub Wiki

OIR adds value to survey reports through comment coding. Research Analysts and Research Assistants use NVivo to code comments to text entry questions. These coded comments are then tabled in survey reports to provide a summary of text responses.

The current comment coding appendix generation functions work with standard comment coding only. Integration with hierarchical comment coding is planned as a future development.

QualtricsTools makes it easy to create survey summary report appendices of full text comments using make_text_appendices and make_split_text_appendices. In addition, users can easily make text appendices that include both full text comments and frequency reports of categorized "coded comments" using the functions make_coded_comments and make_split_coded_comments.

Coded comments must follow one of two standardized formats (see below for details and examples). These coded comment data formats correspond to how Tufts Office of Institutional Research codes and exports coded comment data. The parameter code_type specifies whether the coded comment data are exported from FileMaker Pro (or have been modified to appear as such) or exported from NVivo using the matrix export.

Coded Comment Appendices with the Command Line in R

There are two functions to create coded comment appendices:

  1. make_coded_comments creates coded comment appendix for an overall survey summary report
  2. make_split_coded_comments creates coded comment appendices for split supgroup reports

Both of these functions require the following arguments (input):

  • qsf_path: The path location of the .qsf file to be processed. This must be passed as text string (surrounded by "") and, on a Windows machine, all '' must be replaced by '/'. If no qsf_path is provided, the user will be asked to choose a qsf.

  • csv_path: The string path location of the .csv file to be processed. This requires path specifications as above. If no csv_path is provided, the user will be prompted to choose from the dialogue box.

  • headerrows: Specifies the number of header rows in the CSV data. Use 3 for data exported under current OIR specifications (legacy exporter, unchecked legacy view results)

  • sheets_dir: The string path location of the directory which contains Excel documents for each coded comment using one of the two formats specified. All coded comments must use the same data output format that follows the NVivo or FileMaker Pro export format.

  • output_dir: The string path for the directory to which R should write the coded comment report. If not specified, your coded comment appendix will be written to a temporaryy file.

  • n_threshold: The the number of verbatim comments which must appear before an appendix of coded comments will be included. This defaults to 15, but it can be set to any level. Split appendices may require a lower n_threshold since some subgroups are smaller.

  • code_type: The type of coded comment data export. These two types of data exports are described above. If no code_type is specified, the default is to use NVivo crosstab export with the ResponseID in the first row and second column labeled with the varnmae and containing response presence 1/0 indicator. For old filemaker pro format, use "fmp" specification.

  • filename: The desired name of the output file including extension (e.g. .docx). If this is not specified, it will default to "Text Appendices with Coded Comments.docx".

    • filename is relevant for make_coded_comments ONLY. Split coded comment appendices generated using make_split_coded_comments on the command line will be named based on the split group.

  • split_by: A list specifying which columns of the response date should be used to split the respondents. This can be one column or a combination of columns. The split_by specifications will be used to create filenames for split coded comment appendices.

    • split_by ia relevant for make_split_coded_comments ONLY, since make_coded_comments is used for overall reports.
    • Example for splitting reports by undergraduate school: split_by = c("ENorLA")

Coded Comment Appendices with the QualtricsTools App

This functionality is under development as of Spring 2020 but is not yet available. Please use the command line to generate coded comment appendices.

Formatting Coded Comment Data for QualtricsTools

Examples of coded comment data exports for each code_type format are included in the QualtricsTools package. In the prescribed workflow, each individual text entry component of a survey which needs to be categorized must have a corresponding coded comments ,xlsx file in the specified sheets_dir. The example below has only one question with coded comments, but the coded comment appendix functions will add coded comment tables for any text entry survey question or question component which has a corresponding excel sheet.

NVivo Matrix Export

NVivo is useful for coding comments and the tool that Tufts OIR uses as of April 2020. When coding comments with NVivo, use the matrix export for each node to produce a coded comment table compatible with QualtricsTools (code type = nvivo). Coded comments using the code_type specification "nvivo" contain an unlabeled first column of ResponseIDs. The second column is labeled with the variable name and includes 1's and 0's indicating response presence for each respondent (i.e. did the person respond to the question). The variable name in the second column header will be used to link coded comment responses to the appropriate survey question. NVivo matrix export can include data for all respondents, The response presence column will be used to filter out non-respondents and provide an accurate count of total comments coded. Columns to the right of the named response presence column contain coded category data. Each column has a coded comment category as its headers with 1's and 0's indicating whether the row's response was assigned to that category. Matrix exports from NVivo may also contain a "Total" column. This "Total" column is not necessary and will be ignored when creating the coded comment table.

Sample data are shown below to illustrate the code_type = "nvivo" data export format.

NVivo Matrix Export Example

FileMaker Pro Export

Prior to adopting NVivo for comments coding, Tufts OIR used FileMaker Pro (FMP) to code comments. Coded Comment data exported form FileMaker can be turned into coded comment appendices with the code_type specification "fmp". FMP data exports must contain ResponseID in the first column. There must be a column named varname that includes the variable name values in all rows. Any columns between the initial ResponseID and varname The column immediately right of varname is labeled with the question text and contains full response text for each respondent. Columns to the right of this included coded comment data. Each column title represents a coded comment category, and "1" entries in cells indicate that a respondent's response has been assigned to that category.

Sample data are shown below to illustrate the code_type = "fmp" data export format.

FMP Export Example

A Note on Coded Comment Data Export Types

Tufts OIR transitioned their comment coding process from FMP to NVivo in Summer 2018. Between Summer 2018 and Spring 20202, data exported from NVivo was manually modified to mimic FMP data export format so the data would be compatible for QualtricsTools. As of April 2020, QualtricsTools functions to create coded comment appendices have been modified to accept the code_type parameter specification and will work with data exported under either format. Please look at the coded comment data files when generating reports from older surveys that may have coded comments prepared in FMP or using the NVivo to FMP manual manipulations. Use code_type="fmp" specification to generate appendices from these data.

Sample Output from make_coded_comments

Below is a sample of the coded comment appendix file created using make_coded comments.

Note: This example includes specification n_threshold = 5 since there are only six responses to the text question q5_position.

Coded Comment Appendix Example

If adjustments to the survey are necessary before exporting the coded comments, the survey data can be loaded through get_setup, edited and adjusted, and then processed with coded comments through make_coded_comments.

# Load and process the survey
get_setup(qsf_path = "C:/Example/Path/to/SurveyFile.qsf",
          csv_path = "C:/Example/Path/to/ResponseFile.csv",
          headerrows = 3)

# Exclude a particular question and reinsert it into the blocks
questions[1](/tufts-oir/QualtricsTools/wiki/1)['qtSkip'](/tufts-oir/QualtricsTools/wiki/'qtSkip') = TRUE
blocks <- insert_questions_into_blocks(questions, blocks)

# Insert coded comments and render text appendices
make_coded_comments(
    sheets_dir = "C:/Example/Directory/For/Coded/Comments",
    output_dir = "C:/Example/Path/Output Testing/",
    filename = "Testing Comment Coding.docx",
    n_threshold = 15)