Abstract: The converter steelmaking process represents a pivotal aspect of steel metallurgical production, with the characteristics of the flame at the furnace mouth serving as an indirect indicator of the internal smelting stage. Effectively identifying and predicting the smelting stage poses a significant challenge within industrial production. Traditional image-based methodologies, which rely on a single static flame image as input, demonstrate low recognition accuracy and inadequately extract the dynamic changes in smelting stage. To address this issue, the present study introduces an innovative recognition model that preprocesses flame video sequences from the furnace mouth. Subsequently, it employs a convolutional recurrent neural network (CRNN) to extract spatiotemporal features and derive recognition outputs. Additionally, we adopt feature layer visualization techniques to verify the model’s effectiveness and further enhance model performance by integrating the Bayesian optimization algorithm. The results indicate that the ResNet18 with Channel Attention Block Attention Module (CBAM) in the convolutional layer demonstrates superior image feature extraction capabilities, achieving an accuracy of 90.70% and an area under the curve of 98.05%. The constructed Bayesian Optimization-Convolutional Recurrent Neural Network (BO-CRNN) model exhibits a significant improvement in comprehensive performance, with an accuracy of 97.01% and an area under the curve of 99.85%. Furthermore, statistics on the model’s average recognition time, computational complexity, and parameter quantity (Average recognition time: 5.49 ms, floating-point operations per second: 18260.21 M, parameters: 11.58 M) demonstrate superior performance. Through extensive repeated experiments on real-world datasets, the proposed Convolutional Recurrent Neural Network model is capable of rapidly and accurately identifying smelting stages, offering a novel approach for converter smelting endpoint control.