The detection of black-hole binaries at high redshifts is a cornerstone of the science case of third-generation gravitational-wave interferometers. The star-formation rate peaks at z~2 and decreases by orders of magnitude by z~10. Any confident detection of gravitational waves from such high redshifts would imply either the presence of stars formed from pristine material originating from cosmological nucleosynthesis (the so-called population III stars), or black holes that are the direct relics of quantum fluctuations in the early Universe (the so-called primordial black holes). Crucially, detecting sources at cosmological distances does not imply inferring that sources are located there, with the latter posing more stringent requirements. To this end, we present two figures of merit, which we refer to as "z-z plot" and "inference horizon", that quantify the largest redshift one can possibly claim a source to be beyond. We argue that such inference requirements, in addition to detection requirements, should be investigated when quantifying the scientific payoff of future gravitational-wave facilities.