Flexible decision making depends on retrieving and recombining memories. Yet because this process unfolds covertly, its governing principles remain unknown. Here we use gaze reinstatement to uncover the hidden dynamics and computational logic of memory retrieval during flexible behavior. As people deliberated on a blank screen, they directed their gaze toward the encoding locations of decision-relevant experiences, and these fixations shaped their evolving choice. A task-optimized recurrent neur