Sensitivity to subtle cues regarding the emotions that others are experiencing is vital for effective interactions in social situations. The "Eyes Test" has been proposed, and utilized, as a measure of mild social impairment amongst adults. In this paper we consider the nature of this impairment and identify some limitations in the use of the Eyes Test as a measure of perceiver sensitivity to the psychological state of others. In addition, 25 participants completed the "Eyes Test" in the absence of the stimulus photograph. For over half of the items, participants selected the correct item. Implications for the Eyes Test and measurement of social impairment are considered. In summary, we advocate that future tests of social impairment use targets for which the affective state is known, use objective accuracy criterion, bear in mind the differentiation between genuine and posed expressions of emotion, and consider of the impact of contextual factors.