This paper presents a deep learning-based framework to detect and localize the pedestrians’ anomaly behaviors in videos captured at the grade crossing. A skeleton detection and tracking algorithm are employed to capture the key point trajectories of body movements of the pedestrians. A deep recurrent neural network is applied to learn the normal patterns of pedestrians’ movements using dynamics skeleton trajectories features. An anomaly behaviors detection and localization algorithm are developed by analyzing each pedestrian’s reconstructed trajectories. In the experiments, a video dataset involving normal pedestrian behaviors is established by collecting data at multiple grade crossing spots with different camera angles. Then the proposed framework is trained on the dataset to learn the regularity patterns of normal pedestrians and localize the anomaly behaviors during the testing phase. To the best of our knowledge, it is the first attempt to analyze pedestrians’ behavior at a grade crossing. The experimental results show that the proposed framework can detect and localize the anomaly behaviors, such as squatting down, lingering, and other behaviors that may cause safety issues at the grade crossing. Our study also points out the direction for further improvement of the present development to meet the need for real-world applications.