The chemical reactions and pathways resulting in the breakdown of a prosthetic group, the non-amino acid portion of certain protein molecules. Prosthetic groups may be inorganic or organic and are usually required for the biological activity of the protein.