PVT (Physical Vapor Transport) method has advantages in producing high quality, large scale wafers where many researches are being carried out to commercialize nitride semiconductors. However, complex process variables cause various defects when it had non-equilibrium growth conditions. Annealing process after crystal growth has been widely used to enhance the crystallinity. It is important to set appropriate temperature, pressure, and annealing time to improve crystallinity effectively. In this study, the effect of the annealing conditions on the crystalline structure variation of the AlN single crystal grown by PVT method was investigated with synchrotron whitebeam X-ray topography, electron backscattered diffraction (EBSD), and Rietveld refinement. X-ray topography analysis showed secondary phases, sub-grains, impurities including carbon inclusion in the single crystal before annealing. EBSD analyses identified that sub-grains with slightly tilted basal plane appeared and the overall number of grains increased after the annealing process. Rietveld refinement showed that the stress caused by the temperature gradient during the annealing process between top and bottom in the hot zone not only causes distortion of grains but also changes the lattice constant.