Detecting temporal cognition in text: Comparison of Judgements by self, expert and machine

Erin Walsh, Janie BUSBY GRANT

Research output: Contribution to journalArticle

2 Downloads (Pure)

Abstract

Background: There is a growing research focus on temporal cognition, due to its importance in memory and planning, and links with psychological wellbeing. Researchers are increasingly using diary studies, experience sampling and social media data to study temporal thought. However, it remains unclear whether such reports can be accurately interpreted for temporal orientation. In this study, temporal orientation judgements about text reports of thoughts were compared across human coding, automatic text mining, and participant self-report.

Methods: 214 participants responded to randomly timed text message prompts, categorically reporting the temporal direction of their thoughts and describing the content of their thoughts, producing a corpus of 2505 brief (1–358, M = 43 characters) descriptions. Two researchers independently, blindly coded temporal orientation of the descriptions. Four approaches to automated coding used tense to establish temporal category for each description. Concordance between temporal orientation assessments by self-report, human coding, and automatic text mining was evaluated. Results: Human coding more closely matched self-reported coding than automated methods. Accuracy for human (79.93% correct) and automated (57.44% correct) coding was diminished when multiple guesses at ambiguous temporal categories (ties) were allowed in coding (reduction to 74.95% correct for human, 49.05% automated). Conclusion: Ambiguous tense poses a challenge for both human and automated coding protocols that attempt to infer temporal orientation from text describing momentary thought. While methods can be applied to minimize bias, this study demonstrates that researchers need to be wary about attributing temporal orientation to text-reported thought processes, and emphasize the importance of eliciting self-reported judgements.
Original languageEnglish
Article number2037
Pages (from-to)1-6
Number of pages6
JournalFrontiers in Psychology
Volume9
Issue numberOCT
DOIs
Publication statusPublished - 26 Oct 2018

Fingerprint

Cognition
Data Mining
Research Personnel
Self Report
Text Messaging
Social Media
Psychology

Cite this

@article{ecea09bec0554920b8dc73bf838f96f9,
title = "Detecting temporal cognition in text: Comparison of Judgements by self, expert and machine",
abstract = "Background: There is a growing research focus on temporal cognition, due to its importance in memory and planning, and links with psychological wellbeing. Researchers are increasingly using diary studies, experience sampling and social media data to study temporal thought. However, it remains unclear whether such reports can be accurately interpreted for temporal orientation. In this study, temporal orientation judgements about text reports of thoughts were compared across human coding, automatic text mining, and participant self-report.Methods: 214 participants responded to randomly timed text message prompts, categorically reporting the temporal direction of their thoughts and describing the content of their thoughts, producing a corpus of 2505 brief (1–358, M = 43 characters) descriptions. Two researchers independently, blindly coded temporal orientation of the descriptions. Four approaches to automated coding used tense to establish temporal category for each description. Concordance between temporal orientation assessments by self-report, human coding, and automatic text mining was evaluated. Results: Human coding more closely matched self-reported coding than automated methods. Accuracy for human (79.93{\%} correct) and automated (57.44{\%} correct) coding was diminished when multiple guesses at ambiguous temporal categories (ties) were allowed in coding (reduction to 74.95{\%} correct for human, 49.05{\%} automated). Conclusion: Ambiguous tense poses a challenge for both human and automated coding protocols that attempt to infer temporal orientation from text describing momentary thought. While methods can be applied to minimize bias, this study demonstrates that researchers need to be wary about attributing temporal orientation to text-reported thought processes, and emphasize the importance of eliciting self-reported judgements.",
keywords = "Self-report, Stanford Natural Language Parser, Temporal cognition, Temporal orientation, Tense extraction",
author = "Erin Walsh and {BUSBY GRANT}, Janie",
year = "2018",
month = "10",
day = "26",
doi = "10.3389/fpsyg.2018.02037",
language = "English",
volume = "9",
pages = "1--6",
journal = "Frontiers in Psychology",
issn = "1664-1078",
publisher = "Frontiers Media S.A.",
number = "OCT",

}

Detecting temporal cognition in text: Comparison of Judgements by self, expert and machine. / Walsh, Erin; BUSBY GRANT, Janie.

In: Frontiers in Psychology, Vol. 9, No. OCT, 2037, 26.10.2018, p. 1-6.

Research output: Contribution to journalArticle

TY - JOUR

T1 - Detecting temporal cognition in text: Comparison of Judgements by self, expert and machine

AU - Walsh, Erin

AU - BUSBY GRANT, Janie

PY - 2018/10/26

Y1 - 2018/10/26

N2 - Background: There is a growing research focus on temporal cognition, due to its importance in memory and planning, and links with psychological wellbeing. Researchers are increasingly using diary studies, experience sampling and social media data to study temporal thought. However, it remains unclear whether such reports can be accurately interpreted for temporal orientation. In this study, temporal orientation judgements about text reports of thoughts were compared across human coding, automatic text mining, and participant self-report.Methods: 214 participants responded to randomly timed text message prompts, categorically reporting the temporal direction of their thoughts and describing the content of their thoughts, producing a corpus of 2505 brief (1–358, M = 43 characters) descriptions. Two researchers independently, blindly coded temporal orientation of the descriptions. Four approaches to automated coding used tense to establish temporal category for each description. Concordance between temporal orientation assessments by self-report, human coding, and automatic text mining was evaluated. Results: Human coding more closely matched self-reported coding than automated methods. Accuracy for human (79.93% correct) and automated (57.44% correct) coding was diminished when multiple guesses at ambiguous temporal categories (ties) were allowed in coding (reduction to 74.95% correct for human, 49.05% automated). Conclusion: Ambiguous tense poses a challenge for both human and automated coding protocols that attempt to infer temporal orientation from text describing momentary thought. While methods can be applied to minimize bias, this study demonstrates that researchers need to be wary about attributing temporal orientation to text-reported thought processes, and emphasize the importance of eliciting self-reported judgements.

AB - Background: There is a growing research focus on temporal cognition, due to its importance in memory and planning, and links with psychological wellbeing. Researchers are increasingly using diary studies, experience sampling and social media data to study temporal thought. However, it remains unclear whether such reports can be accurately interpreted for temporal orientation. In this study, temporal orientation judgements about text reports of thoughts were compared across human coding, automatic text mining, and participant self-report.Methods: 214 participants responded to randomly timed text message prompts, categorically reporting the temporal direction of their thoughts and describing the content of their thoughts, producing a corpus of 2505 brief (1–358, M = 43 characters) descriptions. Two researchers independently, blindly coded temporal orientation of the descriptions. Four approaches to automated coding used tense to establish temporal category for each description. Concordance between temporal orientation assessments by self-report, human coding, and automatic text mining was evaluated. Results: Human coding more closely matched self-reported coding than automated methods. Accuracy for human (79.93% correct) and automated (57.44% correct) coding was diminished when multiple guesses at ambiguous temporal categories (ties) were allowed in coding (reduction to 74.95% correct for human, 49.05% automated). Conclusion: Ambiguous tense poses a challenge for both human and automated coding protocols that attempt to infer temporal orientation from text describing momentary thought. While methods can be applied to minimize bias, this study demonstrates that researchers need to be wary about attributing temporal orientation to text-reported thought processes, and emphasize the importance of eliciting self-reported judgements.

KW - Self-report

KW - Stanford Natural Language Parser

KW - Temporal cognition

KW - Temporal orientation

KW - Tense extraction

UR - http://www.scopus.com/inward/record.url?scp=85055527869&partnerID=8YFLogxK

U2 - 10.3389/fpsyg.2018.02037

DO - 10.3389/fpsyg.2018.02037

M3 - Article

VL - 9

SP - 1

EP - 6

JO - Frontiers in Psychology

JF - Frontiers in Psychology

SN - 1664-1078

IS - OCT

M1 - 2037

ER -