Any process in which a virus stops, prevents, or reduces the frequency, rate or extent of host poly(A)-binding protein (PABP) activity. PABP binds to the poly(A) tail of mRNA to facilitate translation.