Vehicular Ad Hoc Networks (VANETs) and Flying Ad Hoc Networks (FANETs) are essential for safety-critical and data-intensive applications in intelligent transportation and aerial systems. However, high node mobility, dynamic topologies, heterogeneous communication technologies, and limited energy resources in FANETs cause frequent handovers, resulting in increased latency, packet loss, and reduced network performance. Conventional threshold-based and heuristic mechanisms struggle to adapt to these highly dynamic environments. This paper proposes a unified Q-learning based intelligent handover framework for both VANETs and FANETs operating in heterogeneous communication settings. The framework models the handover decision as a Markov Decision Process (MDP) and employs a lightweight, model-free reinforcement learning approach. It dynamically learns optimal policies by jointly considering multiple real-time parameters: received signal strength, node mobility, traffic load, network occupancy, and residual energy. In VANETs, the framework enables intelligent vertical handovers between high-bandwidth Li-Fi and reliable RF (IEEE 802.11p) links. In FANETs, it incorporates energy-aware decision-making to extend network lifetime under 3D mobility. Extensive simulations using OMNeT++ integrated with simulation of urban mobility (SUMO) demonstrate significant performance gains over RSS-based, heuristic, SDN-assisted, and existing RL methods. The proposed framework achieves a handover success rate of up to 91.6% under high mobility, improves throughput to 15.8 Mbps in hybrid Li-Fi/RF VANET scenarios, and maintains a packet delivery ratio of 92.8% under heavy traffic. It further reduces end-to-end delay by up to 38%, lowers handover latency to 41 ms (VANET) and 54 ms (FANET), decreases energy consumption per packet to 0.48 J, and extends FANET network lifetime to 168 min. The proposed Q-learning-based approach provides a scalable, adaptive, and energy-efficient solution for seamless handover management in next-generation heterogeneous vehicular and aerial ad hoc networks.