The Australian Weed Risk Assessment protocol (WRA) is often considered the standard approach for pre-border screening of new plant introductions. Here we assess its robustness against three key criteria: ability to discriminate success or failure of species at three stages of the invasion process (introduction, naturalisation and spread); sensitivity to taxonomic range and target region; and dependence on knowledge of invasive behaviour elsewhere. We address these issues by retrospectively testing the WRA using pine (Pinus) introductions to New Zealand and Great Britain. For both regions we calculated WRA scores for 115 species, and classified all species according to whether they had been introduced, which of these had naturalised, and the extent of their naturalised distribution (spread). Using regression models, we assessed whether WRA scores could predict success at each stage. We repeated this procedure using WRA scores calculated without information on species naturalisation behaviour elsewhere. In both regions, the WRA could discriminate among species in the same genus at the introduction and naturalisation stages, but not at the spread stage. The outcome at the naturalisation stage depended on prior knowledge of naturalisation behaviour elsewhere. Without this information the WRA may be unable to distinguish among closely related species, and should be used cautiously where data on invasive behaviour elsewhere is lacking. Human selection played a strong role in the invasion process both through introducing pine species likely to naturalise in New Zealand and Great Britain in the first instance, and subsequent use of many of these species for forestry in the target regions.