Reconstruction-based anomaly detection is appealing for industrial inspection because it reconstructs anomaly-free references and produces interpretable pixel-level residual maps. However, in multiclass settings, it often suffers from, first, identity mapping, where abnormal regions are overreconstructed and residuals vanish, and second, cross-category feature entanglement, which introduces artifacts and weakens localization. Residual scoring is further biased because it mixes true defect discrepancies with model-induced reconstruction errors, causing false alarms and confusing explanations. We propose a semantically guided counterfactual model (SGCAD) for multiclass anomaly detection and localization. SGCAD applies a masking intervention to a candidate region, reconstructs the intervened input, and contrasts factual and intervened residuals to estimate reconstruction bias, yielding a debiased anomaly map by treating consistent differences as anomalies. To stabilize reconstruction, we introduce a dual-branch bottleneck with latent constraints to mitigate identity mapping while preserving normal details, and a hierarchical adaptive prompt to reduce cross-category confusion during decoding without class priors. Extensive experiments on four public benchmarks demonstrate that while SGCAD maintains competitive image-level detection performance, it achieves superior state-of-the-art results in pixel-level localization. By effectively mitigating bias-induced false positives, SGCAD offers a more precise and reliable solution for practical industrial inspection.