The chemical reactions and pathways resulting in the breakdown of a deoxyribonucleoside diphosphate, a compound consisting of a nucleobase linked to a deoxyribose sugar esterified with diphosphate on the sugar.