The rise of large language models (LLMs) like ChatGPT has sparked speculation about whether AI is on the cusp of human-like reasoning abilities. But while these chatbots can hold fluent conversations and generate coherent text, some AI researchers argue their reasoning capabilities are an illusion. 

In a new paper, researchers at Arizona State University systematically probe whether iterative prompting - asking an LLM to refine its answers repeatedly - can improve performance on logical reasoning tasks. Their results reveal limitations in LLMs' ability to self-critique their solutions.

Down the Rabbit Hole  

The researchers chose graph coloring, a classic AI reasoning challenge to test iterative prompting. The goal is to assign colors to each node in a graph so no two adjacent nodes share the same color. It's the kind of constrained optimization problem humans solve using logical deduction.

First, the researchers directly prompted the LLM GPT-4 with 100 random graph coloring problems. Without any feedback or prompting, GPT-4 solved only 16 out of 100 instances correctly. No big surprise there.

Things got more interesting when the researchers tried having GPT-4 critique its solutions. The idea was that GPT-4 could propose a coloring, check it for errors, and then refine its solution based on those errors. But instead of improving, performance plummeted - GPT-4 solved just 1 out of 100 problems this way. 

Why the failure? The researchers found GPT-4 needed help to verify its solutions accurately. When it generated a correct coloring, it would falsely claim errors and undo the solution. Like Alice falling down the rabbit hole, iterative self-critiquing led the LLM astray.

External Feedback 

Next, the researchers tried having an external program, which was guaranteed to be correct, to double-check GPT-4's solutions. This time, iterative prompting did improve performance to around 40% accuracy. 

But here's the twist - the researchers found the feedback content didn't matter. Just telling GPT-4 "try again" worked nearly as well as providing specific error details. This suggests the performance gains come simply from making more attempts, not from GPT-4 internalizing logical critiques.

To confirm this, the researchers tried having GPT-4 generate multiple candidate solutions in one shot without any prompting or verification. With 15 tries, GPT-4 could solve 40% of problems as effectively as the iterative approach.

Off With Their Heads!

Based on these experiments, the researchers conclude today's LLMs have limited ability to analyze and refine their reasoning critically:

"Our results thus call into question claims about the self-critiquing capabilities of state of the art LLMs."

Their findings contribute to a growing debate about overselling the current capabilities of large language models. Unbridled enthusiasm risks misleading the public and policymakers about how far we are from accurate artificial general intelligence.

We are still in the early days of a technology that could ultimately transform society. But separating hype from reality is critical as we chart a responsible path forward. Like Alice's journey through Wonderland, discovering the limits of today's AI systems can orient us to what's actual versus illusion.

The Looking Glass 

So, where does this leave us? Here are the key facts and implications of this research:

• LLMs need help to evaluate their solutions to reasoning problems accurately. Self-critiquing does not improve performance.
• External feedback from a correct program enables iterative prompting gains, but the content of the feedback may not matter much.
• Multiple tries without prompts or verification work nearly as well as iterative prompting. This suggests performance gains may come from having more attempts. 

The takeaway? We should be cautious about claims that LLMs can refine solutions through logical self-reflection. Their reasoning capabilities remain quite brittle and limited.

Rather than obsessing over whether models are "truly" reasoning, we should focus evaluation on how they perform on social priorities that matter - education, healthcare, science, and governance—moving beyond narrow AI benchmarks toward real-world usefulness.

There's no magic shortcut to artificial general intelligence. But models like ChatGPT hint at the transformative potential of AI while revealing challenges ahead. With eyes open to possibilities and limitations, we can navigate judiciously to an emerging technology that serves humanity.

Try building your own AI assistant on CPROMPT.AI. CPROMPT democratizes AI by letting anyone generate prompt applications through a simple web interface - no coding required. Join the community reshaping the future of AI.

ChatGPT is an AI chatbot that can hold remarkably human-like conversations and generate coherent text on various topics. It has exploded in popularity since its launch late last year, with users leveraging it for everything from essay writing to programming assistance. However, as an AI system continuously updated behind the scenes, how exactly is ChatGPT changing over time? 

A comprehensive new study from Stanford and UC Berkeley researchers tested different versions of the key AI models behind ChatGPT - GPT-3.5 and GPT-4 - from March to June 2023. They evaluated the models across eight diverse tasks, including solving math problems, answering sensitive questions, taking opinion surveys, reasoning over multiple information sources, generating code, answering medical exam questions, and visual reasoning.

The findings from the study were both intriguing and revealing. Here's what they uncovered:

• On a math test identifying prime vs. composite numbers, GPT-4's accuracy plummeted from 84% to 51% between March and June. Researchers found this was partly due to GPT-4 becoming less amenable to "chain of thought" prompting in June, a technique where the AI is guided through step-by-step reasoning. In contrast, GPT-3.5 substantially improved on this task.
• When taking OpinionQA surveys, GPT-4's response rate plunged from 97.6% in March to just 22.1% in June, a nearly 75% drop. The June version often refused to answer questions it had previously responded to, stating it has no opinions as an AI system. 
• In code generation, the percentage of Python code samples that could be directly executed without errors sank substantially from March to June for GPT-3.5 and GPT-4. The June versions frequently failed to follow prompts requesting the code snippet, often tacking on non-executable text.

While there were areas of regression, it's worth noting that GPT-4 performed better at multi-hop reasoning in June compared to March.

The study emphasizes that while continuous improvement of AI systems like ChatGPT is desired, its crucial changes do not result in the loss of essential functions or increase unpredictability. Although ChatGPT may enhance some capabilities over time, the findings reveal it risks regressing in other areas or becoming less adept at specific practical prompting techniques. Greater transparency from AI developers about the nature of system updates is essential.

As ChatGPT continues to evolve, how do you envision its role in your daily tasks? Have you interacted with ChatGPT recently? We'd love to hear if you've observed any changes in its behavior. Drop your experiences in the comments below, and remember to subscribe for the latest insights into ChatGPT's evolution!