With exponential traffic growth projected from mobile and cloud applications, optical networks powering the Internet backbone will require orders-of-magnitude transmission capacity gains. At the core of our innovation in this area are agile digital signal processing (DSP)-based approaches, including digital Nyquist spectral shaping, DSP-based nonlinearity compensation and advanced soft-decision forward error correction (FEC) that optimize spectrum utilization, maximize transmission reach and provide data rate flexibility.