Any process that modulates the rate, frequency or extent of a response to interferon-gamma. Response to interferon gamma is a change in state or activity of a cell or an organism (in terms of movement, secretion, enzyme production, gene expression, etc.) as a result of an interferon-gamma stimulus.