Any process in which an organism stops, prevents, or reduces the frequency, rate or extent of the accumulation of callose in cell walls that occurs as part of the defense response of a second organism, where the two organisms are in a symbiotic interaction.