The aggregation, arrangement and bonding together of a set of components to form an ascospore wall. During sporulation in Ascomycota, each ascospore nucleus becomes surrounded by a specialized spore wall, formed by deposition of spore wall components in the lumenal space between the outer and inner leaflets of the prospore membrane. An example of this process is found in Saccharomyces cerevisiae.