Represented structure versus representational structure: A challenge for interpreting LLMs

An adequate theory of representation should distinguish between the structure of a representation and the structure of what it represents. I argue that the simplest sorts of transformers (the architecture that underlies most familiar Large Language Models) have only a very lightweight structure for their representations: insofar as they work with the structure of language, they represent it but do not use it. In addition to being interesting in its own right, this also shows how we may use high-