Week 3 – Tutorial: Annotation Disagreement
Recognizing Causes of Annotation Disagreement
Q1. Two annotators from different cultural backgrounds disagree on whether children’s shouting during play is hostile or dominant.
Q2. Two annotators show perfect disagreement due to handwritten label sheets where values appear shifted by one.
Q3. Annotators disagree whether manipulating an object without looking counts as engagement or disengaged object play.
Q4. Two annotators sit in different stadium locations and miss different events due to obstructions and glare.
Q5. A neutral face with “yeah sure” leads to disagreement in compliance labeling.
Q6. Annotators interpret “intentional gesture” differently (system-recognized vs user-deliberate).
Q7. Flow is used to validate an away-game experience, but annotators struggle to label it consistently.
Q8. Two annotators disagree on expressiveness using Laban Movement Analysis categories.
Q9. Upright vs slouched posture annotations agree in acted data but not in real-world walking data.