The increase in mental health disorders in college populations necessitates novel assessment strategies that circumvent the limitations of existing self-report instruments. To address this issue, this paper presents a new deep learning framework for mental health monitoring in academia by integrating multimodal passive sensing data collected from smartphones. A cross-modal attention network, searched by a new metaheuristic algorithm the Addax Optimization Algorithm for neural architecture search, was trained and first evaluated on the StudentLife dataset. To further validate the results due to the extremely limited number of samples in the test set (N = 7), we subsequently performed a zero-shot transfer and fine-tuning evaluation on the College Experience Study (CES) dataset. CES is a large longitudinal dataset of over 200 students collected over 5 years including passively-collected sensor, survey and brain-imaging data. The validation of the model in 140 local participants resulted in classification accuracy for depression of 87.4% on StudentLife (with adjusted 95% CI 62–98%), suggesting considerable uncertainty. On CES the performance of the zero-shot transferred model reached 72.3% of classification accuracy and 83.9% when fine-tuned. This means the initial 87.4% result can be interpreted as overoptimistic. Modality weight analysis showed the importance of survey in predicting depression and the effect of activity on stress predictions. This paper serves as a proof-of-concept of a novel system for screening mental health disorders passively using mobile phones within academia, and suggests it may have a role to play in the early detection of risk.