Vision is context dependent, with neuronal responses shaped not only by local features but also by surrounding visual input. While classical studies, using grating stimuli, show that iso-oriented surrounds suppress responses more than orthogonal surrounds, the role of contextual modulation under natural stimulus conditions remains less clear. Using recordings from mouse primary visual cortex (V1), we trained convolutional neural network models to predict neuronal responses to natural images and synthesized surround stimuli that selectively suppressed or facilitated responses to optimal center inputs. In vivo experiments confirmed these predictions. Facilitatory surrounds resembled naturalistic continuations of the optimal center stimulus, consistent with natural image statistics, whereas suppressive surrounds deviated from these predictions. Applying the same approach to macaque V1 revealed similar principles across species. Both models accurately predicted responses to classical grating stimuli. We formalize these results in a normative Bayesian model, showing that neuronal activity for preferred center features reflects posterior beliefs about likely center-surround configurations.