Autism Spectrum Disorder (ASD) is now one of the most prevalent neurodevelopmental disorders, characterized by socio-communicative and cognitive challenges. Amongst them, memory challenges in ASD have been reported early, but only recently have studies suggested a prevalent memory impairment in autistic individuals. More importantly, little is known about the brain correlates associated with the memory challenges in ASD. In the current study, we used both quantitative and qualitative approaches to review a total of 42 studies (22 were included in the ALE analysis) to reveal the consistent findings in the literature on the memory functions in ASD, and found that the memory challenges in ASD are probably accompanied by an imbalance between brain functions in the left and right hemispheres, with deficient functions in fronto-parietal regions on the left hemisphere and an excessive engagement of the counterparts on the right hemisphere. In addition, the analysis suggested a top-down deficit from the frontal lobe during the memory tasks, and the atypical functions probably depended on the stimulus format and task types.