Any process that stops, prevents, or reduces the frequency, rate or extent of invasive growth in response to glucose limitation by stopping, preventing, or reducing the frequency, rate or extent of transcription from an RNA polymerase II promoter.