Multi-moded horn antennas can be used as high efficiency feeds for bolometers when diffraction limited resolution is not required. For example, such horns are proposed for the PLANCK Surveyor, a satellite telescope due to be launched in 2007 to make definite measurements of the Cosmic Microwave Background. In a previous paper we described an accurate approach involving electromagnetic modelling using a rigorous mode matching technique to obtain both the horn aperture fields and the corresponding far field radiation patterns. In this paper we extend this description to determine the "phase center" of such horns when used on large reflecting telescopes. The "phase center" is ill defined as the individual spatially coherent fields making up the far field pattern all appear to come from different phase centers. The best average phase center location is therefore redefined in terms of the virtual beam waist position behind the horn aperture at which the focus of the telescope should be located in order to optimise angular resolution and on-axis gain for the beam on the sky. A number of alternative techniques to locating the phase center are discussed in detail in this paper.