Theoretical Map of Thought Chains: Evaluating Explanatory Power Based on Task-Theory Matching

School of Mathematics and Physics, University of Science and Technology Beijing, Beijing, China

^* Corresponding author’s email: This email address is being protected from spambots. You need JavaScript enabled to view it.

Abstract

The Chain-of-Thought (CoT) method brings significant enhancements to the complex reasoning ability of large language models. However, it is still disputable on what exactly CoT mechanism improves and lacks a proper quantitative metric to characterize CoT.Therefore, this paper defines a new metric relative improvement rate, and proposes a meta-analysis to evaluate the explanatory ability of six mainstream theories for CoT phenomenon over 10 tasks. Fisher’s exact test is adopted in the statistical test. The main contributions of this paper include: On highly parallel-difficulty tasks (e.g., Circuit Value, Last Letter), “expressiveness/complexity theory” view shows the strongest explanatory power, which is consistent with recent discoveries in symbolic reasoning. On real-world math reasoning tasks (e.g., GSM8K, AQuA), “statistical learning theory” view shows the strongest explanatory power, and its “intermediate steps serve as supervisory signals” view gets further supports. Besides, this paper also builds task-theory matching lookup table, which could give direct guidance for practical CoT design. This paper aims to provide a robust quantitative foundation for understanding the CoT mechanism and offers valuable guidance for the advancement of prompt engineering. For detailed code, please refer to: https://github.com/Htian-66/CoT.git