In this work, we address the challenge of accurately predicting latency in packet-switched Xhaul networks, enabling the convergent transport of fronthaul (FH) and midhaul (MH) traffic within radio access networks (RANs). Although deterministic worst-case (WC) models provide strict latency bounds, they tend to significantly overestimate actual flow latencies, leading to inefficient resource allocation. To address this limitation, we propose a machine learning-based (ML) latency prediction framework that leverages quantile regression (QR) to provide more accurate estimates of maximum one-way transmission latency for FH and MH flows—an essential requirement for reliable RAN operation. Our approach enhances WC estimations by incorporating additional latency-related features, and is validated using an extensive data set generated through the simulation of diverse ring and mesh topologies. We integrate the QR-based latency predictions into a mixed-integer linear programming (MILP) model for optimal flow routing and distributed unit (DU) placement. A comparative analysis reveals that QR-based latency prediction outperforms WC latency estimations, significantly improving network performance by reducing the number of active DU processing nodes by up to 20% without compromising latency constraints. The results highlight the potential of ML techniques to enhance the accuracy of latency modeling in dynamic, latency-sensitive Xhaul scenarios, thereby contributing to the realization of RAN Digital Twin systems envisioned for future 6G networks.