Catalysis of the transfer of phosphate from one side of the membrane to the other. In high affinity transport the transporter is able to bind the solute even if it is only present at very low concentrations.