Interacting selectively and non-covalently with a specific DNA sequence that is part of the core promoter for mitochondrial RNA polymerase in order to modulate transcription by mitochondrial RNA polymerase. The transcription factor may or may not also interact selectively with a protein or macromolecular complex.