Any process in which an organism stops, prevents, or reduces the frequency, rate or extent of the accumulation of callose that occurs as part of the defense response of a second organism, where the two organisms are in a symbiotic interaction.