Interacting selectively and non-covalently with a SET domain of a protein. SET domains are named after three Drosophila proteins that contain this domain: Su(var), E(z) and trithorax. SET domains are associated with histone lysine methylation.