The chemical reactions and pathways resulting in the breakdown of cyanogenic glycosides, any glycoside containing a cyano group that is released as hydrocyanic acid on acid hydrolysis; such compounds occur in the kernels of various fruits.