An artificial neural network (ANN) based transfer learning model is built for quality of transmission (QoT) prediction in optical systems feasible with different modulation formats. Knowledge learned from one optical system can be transferred to a similar optical system by adjusting weights in ANN hidden layers with a few additional training samples, where highly related information from both systems is integrated and redundant information is discarded. Homogeneous and heterogeneous ANN structures are implemented to achieve accurate Q-factor-based QoT prediction with low root-mean-square error. The transfer learning accuracy under different modulation formats, transmission distances, and fiber types is evaluated. Using transfer learning, the number of retraining samples is reduced from 1000 to as low as 20, and the training time is reduced by up to four times.