Large Language Models (LLMs) are increasingly integrated into software systems as automated decision-making components. These systems rely on instruction prompts written in natural language to encode complex workflows. However, debugging these prompts when LLMs produce undesired outputs remains challenging due to their black-box nature and the impracticality of manually inspecting large, complex inputs. Unlike traditional software, LLMs provide no access to execution paths or intermediate states, making it difficult to identify which input fragments are responsible for unexpected behavior. This paper investigates whether delta debugging can be effectively applied to identify and isolate problematic parts of LLM inputs that lead to undesired outputs. We introduce semantic markers as an instrumentation technique that embeds unique identifiers in LLM inputs and extracts traceability information from chain-of-thought reasoning. We systematically evaluate whether these markers accurately identify causal input fragments and enable delta debugging to isolate minimal subsets responsible for incorrect outputs. Through experiments on two benchmarks representing development and production scenarios, we demonstrate that delta debugging with semantic markers can systematically pinpoint problematic input fragments in both development and production settings. Our investigation shows that this approach transforms prompt debugging from an ad-hoc manual process into a systematic methodology, enabling engineers to efficiently identify and address the root causes of unexpected LLM behavior in real-world applications.
