Within theories of concepts and categorization, a concept is considered coherent to the extent that it is perceived as a structured whole. Although this notion plays an important role in the study of conceptual processes such as categorization and reasoning, it has proven difficult to define and operationalize. In this paper, we propose a formal framework in which the coherence of a concept depends on the patterns of covariation among its properties. Combining the theory of conceptual spaces with ideas from Principal Component Analysis, we argue that the coherence of a concept can be estimated by how efficiently its features can be compressed into a simpler, lower-dimensional structure. We discuss how this view explains why natural kind concepts are generally seen as more coherent than artifact concepts, and why coherent concepts support stronger inductive inferences. Furthermore, we apply these ideas to the conjunction fallacy and propose that the drive for coherent representations can override probabilistic intuitions when reasoning about social categories.