Single-crystal X-ray diffraction structural data of four Fe-Mg tourmalines with different Fe contents from Xinjiang, Sichuan, and Yunnan Provinces, China, were collected at room temperature and −100℃. The intrinsic dipole moments of polyhedra and the total intrinsic dipole moment of the unit cell were calculated. By comparing the intrinsic electric dipole moments of the X, Y, Z, T, and B site polyhedra, it is found that the T site polyhedron makes the greatest contribution to the total intrinsic dipole moment. The pyroelectric coefficients of four Fe-Mg tourmalines were experimentally determined, and the influence of intrinsic dipole moments on their pyroelectric properties was investigated. The experimental results show that, compared with the case at room temperature, the intrinsic dipole moments change with the total Fe content at −100℃ in a completely different way. With the decrease of temperature, the total intrinsic dipole moments of tourmaline decrease. Over the same temperature interval, the pyroelectric coefficients increase with the increase in intrinsic dipole moment.