The aggregation, arrangement and bonding together of a set of components to form a sexual spore wall, the specialized envelope lying outside the cell membrane of a spore derived from a product of meiosis.