Any protein complex that interacts with RNA polymerase II to increase (positive transcription elongation factor) or reduce (negative transcription elongation factor) the rate of transcription elongation.