Abstract: Accurate modeling of material constitutive relationships under compositional fluctuations poses significant challenges. Traditional mechanism-driven methods struggle to capture the complex nonlinear behavior of material properties as composition varies, while data-driven deep learning approaches, despite their high accuracy and robustness, lack strict constraints from physical metallurgical mechanisms, often leading to substantial prediction deviations. To address this critical issue, this study proposes a multimodal deep learning model based on an encoder-decoder framework, integrating physical metallurgical theory with machine learning to achieve high-precision prediction of material constitutive relationships under complex loading and compositional fluctuations. Firstly, the performance of three encoder architectures, long short-term memory (LSTM), gated recurrent unit (GRU), and temporal convolutional network (TCN), was systematically compared, with the TCN encoder-based model demonstrating the best performance, achieving mean absolute error (MAE), root mean square error (RMSE), mean absolute percentage error (MAPE), and correlation coefficient (R) values of 1.84 MPa, 2.75 MPa, 4.48%, and 0.9918, respectively. By comparing with purely data-driven models, the critical guiding role of physical metallurgy in the deep learning process was validated. Furthermore, the accuracy and generalizability of the proposed model in material processing scenarios were verified by embedding it into a numerical simulation framework and applying it to out-of-domain data. Finally, the influence of elemental content on material mechanical properties was analyzed using the developed model. This study provides an efficient and reliable alternative method for obtaining material constitutive relationships, avoiding the high costs associated with traditional experimental approaches, and offering potential for computer-aided material design and process optimization for material processing.