Ordering open answers (categorizing free text)

In brief

Short free texts from a questionnaire (for example "What was missing in the seminar?") cannot be counted directly. Here they are sorted into categories: the person analysing the data defines the categories and assigns each answer; Statable counts, shows the frequencies and, on request, writes columns into the data that can be used for further analysis (for example the category by degree program). Statable does not assign anything itself: no AI, and no text is sent anywhere.

What the table looks like

One row per person and one text column with the answer (one answer per row). An ID column (Nr, ID) helps: Statable recognises the rows by it; without an ID the row number counts. The result is new columns: with one category per answer a group column with the codes 1 to k and the categories as value labels, with several categories per answer (under "More options") one 0/1 column per category, created as a multiple response set.

Steps

1. Read a sample of the answers and create categories. Inductive means the categories emerge from the material; deductive means they come from theory or the research question. Often it is a mixture (Kuckartz & Rädiker, 2022; Mayring, 2022).
2. Record a short description and an anchor example for each category (coding guide). It says what belongs in the category and helps a second person to assign in the same way.
3. Assign answers: one at a time or as a list, by click or with the keys 1 to 9 (0 removes the assignment). The keys only work while the focus is in the assignment, never in a text field.
4. Check the counter: how many people have no answer and how many answers are still open.
5. "Write to the data" writes the columns (one undo step). Afterwards "Evaluate now" leads to the question "How is something distributed?".

Three states per row

Every row is in exactly one of three states: no answer (empty text), answer still open or assigned. Empty and open count as missing: they are counted and never silently filled with "no category" or 0. If "fits nowhere" should be recorded, create a category "Other".

Reading the result

• The table gives n for each category and the percentage of people with an assigned answer. The base is in the column heading ("base 31").
• With several categories per answer the percentage of mentions is added. The percentages of people then add up to more than 100% because one answer counts several times; Statable says so in a sentence.
• Answers that are still open are missing from the table; their number is shown above ("12 answers still open").
• The table describes these answers. It tests nothing and cannot be generalized to other people: no test, no p value.

Second coding and agreement

A single person codes subjectively. So a second person assigns the same answers with the same categories without seeing the first assignment. "Create a second coding" makes a separate assignment for this; for the second coding the first one is hidden. Once both codings are written to the data, "Check agreement" opens the question "Do ratings agree?" with both columns (Cohen's kappa, Krippendorff's alpha). With several categories per answer agreement is checked per category. For a truly independent coding the second person works in an own copy of the project file.

Requirements and limits

• This is meant for short free texts in a questionnaire (one to a few sentences), not for interviews or long texts. Programs for qualitative content analysis such as MAXQDA, ATLAS.ti, QCAmap or QualCoder suit those. Statable is not a coding program: no memos, no nested category system.
• There is no automatic categorization and no AI (privacy, traceability). Every assignment comes from a person.
• How dependable the result is depends on the coding guide and on the agreement of two people, not on the number of answers.
• Rows are recognised by the ID column or the row number. Without an ID, deleting or re-sorting rows shifts the assignment.
• Paragraph breaks in long answers become spaces when the file is read.
• When written again, the own columns are overwritten; another column of the same name is left untouched and the new column gets a number in its name.

Of 31 answers with an assignment, 12 people (38.7%) mentioned practice; 2 answers were still open.

For specialists

Own code, no test statistic. Percent of people = answers with the category / answers with at least one category; percent of mentions = n / sum of all assignments. One category per answer: codes 1 to k in the order of the category list, missing for empty and open. Several: 0/1 per category (1 assigned, 0 answered with another category, missing empty or open), multiple response set with count value 1. The reference values come from R (script scripts/welle4-erwartet.R); the columns created in PSPP are checked against them. Writing to the data is one PSPP command (COMPUTE and IF with $CASENUM, VALUE LABELS, VARIABLE LEVEL).

Typical examples

• "What was missing in the seminar?": categories such as practice, feedback, time; afterwards a crosstab shows whether the degree programs differ.
• The field "Other, namely …" of a question with fixed answers.

Sources

Kuckartz, U. & Rädiker, S. (2022). Qualitative Inhaltsanalyse. Methoden, Praxis, Umsetzung mit Software und künstlicher Intelligenz (ch. 3 and 4). Beltz Juventa. Mayring, P. (2022). Qualitative Inhaltsanalyse. Grundlagen und Techniken. Beltz.