Humans are sensitive to different categories of actions due to their importance in social interactions. However, biological motion research has been heavily tilted toward the use of walking figures. Employing point-light animations (PLAs) derived from motion capture data, we investigated how different activities (boxing, dancing, running, and walking) related to each other during action perception, using a visual search task. We found that differentiating between actions requires attention in general. However, a search asymmetry was revealed between boxers and walkers, i.e., searching for a boxer among walkers is more efficient than searching for a walker among boxers, suggesting the existence of a critical feature for categorizing these two actions. The similarities among the various actions were derived from hierarchical clustering of search slopes. Walking and running proved to be most related, followed by dancing and then boxing. Signal detection theory was used to conduct a non-parametric ROC analysis, revealing that human performance in visual search is not fully explained by low-level motion information.