Any process that stops, prevents or reduces the frequency, rate or extent of dipeptide transport by stopping, preventing or reducing the frequency, rate or extent of transcription from an RNA polymerase II promoter.