The cell cycle process in which aggregation, arrangement and bonding together of a set of components to form the spindle midzone. The spindle midzone is the area in the center of the spindle where the spindle microtubules from opposite poles overlap.