Any process that decreases the rate, frequency or extent of the red or far-red signaling pathway, the series of molecular signals initiated upon sensing by photoreceptor molecules of red light or far red light.