Interacting selectively and non-covalently with an RNA polymerase II transcription factor, which may be a single protein or a complex, in order to stop, prevent, or reduce the frequency, rate or extent of transcription from an RNA polymerase II promoter. A protein binding transcription factor may or may not also interact with the template nucleic acid (either DNA or RNA) as well.