Earlier this year, the Council of Canada Academies released a report that examines successful international practices for supporting natural sciences and engineering research and how some of these could be applied in Canada. According to the report, greater experimentation with novel alternatives, combined with rigorous data collection on these experiments, can help NSE funders better assess the costs and benefits of new approaches. One such approach is the use of partial lotteries, and If NSERC were to pursue this direction, Canada will follow funding bodies in Switzerland, New Zealand, Germany, and the U.K. to use mechanisms involving random chance to decide which scientific projects get funded. Given the novelty of funding-by-lottery, methodological questions concerning how and when lotteries are introduced into funding decisions remain poorly understood. In this paper, I will attempt shed light on this issue by utilizing a philosophical framework of scientific inquiry inspired by the American pragmatists and, more especially, the pluralism of Paul Feyerabend. I will go onto show how this framework suggests a reinterpretation of existing evidence on the reliability of peer review and is broadly consistent with available data on the predictability of future science.