When the two eyes are presented with conflicting stimuli, perception starts to fluctuate over time (i.e., binocular rivalry). A similar fluctuation occurs when two patterns are presented to a single eye (i.e., monocular rivalry), or when they are swapped rapidly and repeatedly between the eyes (i.e., stimulus rivalry). Although all these cases lead to rivalry, in quantitative terms these modes of rivalry are generally found to differ significantly. We studied these different modes of rivalry with identical intermittently shown stimuli while varying the temporal layout of stimulation. We show that the quantitative differences between the modes of rivalry are caused by the presence of monocular interactions between the rivaling patterns; the introduction of a blank period just before a stimulus swap changed the number of rivalry reports to the extent that monocular and stimulus rivalries were inducible over ranges of spatial frequency content and contrast values that were nearly identical to binocular rivalry. Moreover when monocular interactions did not occur the perceptual dynamics of monocular, binocular, and stimulus rivalries were statistically indistinguishable. This range of identical behavior exhibited a monocular (∼50 ms) and a binocular (∼350 ms) limit. We argue that a common binocular, or pattern-based, mechanism determines the temporal constraints for these modes of rivalry.