Can Large Language Models Truly Reason? From Stochastic Parrots to In-Context Learners

PhD Thesis Proposal Defence


Title: "Can Large Language Models Truly Reason? From Stochastic Parrots to
In-Context Learners"

by

Miss Tsz Ting CHUNG


Abstract:

The rapid advancement of Large Language Models (LLMs) has sparked a 
fundamental debate: are these models genuinely reasoning, or are they merely 
"stochastic parrots"? This thesis addresses this question through a 
comprehensive evaluation framework organized around four criteria that a 
truly reasoning-capable LLM must satisfy: (1) going beyond memorization, (2) 
closing the human—machine gap, (3) exhibiting reasoning traceability, and (4) 
handling global dependencies.

Guided by these criteria, we systematically investigate LLM reasoning across 
three complementary domains, each exposing deficiencies along different 
dimensions. In the physical domain, we design paired tasks at two cognitive 
levels to isolate memorization from genuine understanding, revealing a ~40% 
accuracy gap between LLMs and humans and thereby exposing the stochastic 
parrot phenomenon. In the logical domain, we construct counterintuitive 
premises grounded in propositional logic to neutralize commonsense shortcuts, 
showing that standard LLMs achieve near-random performance (<36%) while 
humans reach 86.7%, demonstrating that LLMs lack genuine logical deduction 
when semantic heuristics are stripped away. Extending our investigation to 
the long-context domain, we design a benchmark satisfying also the global 
dependency, where correct answers cannot be obtained through local retrieval 
pattern-matching alone but require reasoning over globally scattered 
evidence. Results reveal that reasoning-oriented LLMs lag behind humans by 
>15% in accuracy and >30% in reasoning quality.

Each evaluation study highlights where LLMs must improve along different 
criteria, collectively painting a comprehensive picture of current reasoning 
limitations. Finally, we shift from diagnosing failures to enabling genuine 
learning by reframing in-context learning as a pedagogical curriculum, 
demonstrating that structured context facilitates in-context test-time 
learning. Collectively, this thesis charts a path from stochastic parrots to 
in-context learners.


Date:                   Monday, 11 May 2026

Time:                   3:00pm - 5:00pm

Venue:                  Room 5506
                        Lift 25/26

Committee Members:      Prof. Dit-Yan Yeung (Supervisor)
                        Prof. Nevin Zhang (Chairperson)
                        Dr. Yangqiu Song