We evaluate the economic significance of linguistic barriers to communication in 226 US cities from 1980 to 2010. We address the question: to what extent do linguistic barriers across social groups inhibit the benefits of knowledge exchange? The empirical results show that linguistic, racial and composite diversity increase the average income of working age population in American cities. This positive effect of diversity, however, diminishes the higher is the proportion of foreign-born population who lack English fluency. We call this the 'paradox of diversity'. Overall, our findings provide important policy insights about how social diversity may enhance economic performance within cities.