Case Studies

Annotation in Practice: Two Mini Case Studies

Two concise, real-world annotation examples with token level decisions and clear guideline rationale.

Sep 2025

Annotation quality depends on more than abstract understanding. It lives in the decisions we make token by token. These short examples show how I apply guidelines, document edge cases, and balance consistency with context.

Case Study 1, Pain Descriptor Labelling

Raw text:

“It feels like burning needles.”

Annotation

  • Metaphor based scheme
    • burning → pain: metaphor/heat
    • needles → pain: metaphor/sharp-object
  • Clinical scheme
    • whole phrase → pain: neuropathic

Commentary, I use a dual layer approach to reconcile priorities from metaphor research, medical literature, and annotator usability. Decisions are logged in a shared sheet with definitions and examples so others can apply categories consistently.

Case Study 2, Speech Data Verification

Transcript (audio)

I’m gonna get it.

Model output

I am going to get it.

Annotation

  • I’m → I am → correct (acceptable normalisation)
  • gonna → going to → correct (acceptable normalisation)

Commentary, I mark normalisations as correct when they preserve meaning and follow the guideline. If a model inserts words not spoken or erases dialect features, I mark error: hallucination or error: bias and log a note for error analysis.

Want the quantitative side, see my companion piece, Measuring Annotation Quality, A Mini Evaluation Project.