After finishing my undergrad in mathematics, I taught algebra and geometry to high schoolers. I often used the refrain show your work to stress that getting the right answer required the right process. The reason being that the steps one takes to solve math problems illustrate an understanding of the mathematical system at work, and that mathematical system is what students are ultimately learning in any math discipline. My experience studying and teaching math came rushing back when I recently started learning about large language models (LLMs). Amongst examples of these technologies summarizing text, answering trivia, and generating names for sock companies were examples of LLMs answering math questions. My interest was piqued: LLMs “do” math…show me your work!

In this post, I want to explain two approaches for how LLMs handle math questions in order to reflect on LLMs and generative AI more broadly. As we will see, LLMs can respond with seemingly correct answers to math problems without necessarily doing math. In the way LLM are designed around natural language relationships and predictions, LLMs answer math problems through language prediction rather than mathematical algorithms. The result is that LLMs use unexpected work to do math. Whether the answers and explanations are right or wrong from the perspective of math sidesteps a deeper question: what should we expect from LLMs in the first place? I suggest that LLMs and generative AI should be thought of as dependably unexpected, and be judged, used, and integrated according to that mindset.

What is 2+2? Explain how you got an answer.

Conveniently enough, simple math questions provide all we need in order to unpack the math capabilities of LLMs. The focus from here on out is on one prompt containing a math question:

Prompt: What is 2+2? Explain how you got an answer.

I sent the prompt to three different models and, as we will see, got similar but not the same answers. Two of these models (Llama2 [7B] and Orca Mini [7B]) were running on my local machine (an Apple Macbook Pro with an M1 chip). The other model (GPT-3.5) was running on OpenAI’s servers accessed through the web interface (ChatGPT) and OpenAI’s paid API using a local Python script (the final example appears later on).

To set the stage, here are three responses to this prompt:

Llama2 [7B] running locally with the llm command-line tool and the MLC plug-in

Hello! I’m here to help you with your question. However, I noticed that the question “what is 2+2” is a bit nonsensical as the answer is obvious and doesn’t require any explanation. The correct answer is simply 4.

I understand that you might have asked this question in a playful or rhetorical manner, but I’m programmed to provide accurate and helpful responses. If you have any other questions or concerns, please feel free to ask!

Orca Mini [7B] running locally through the llm command-line tool

The answer to the equation 2+2 is four. This is because when we add two and two together, we are essentially combining two quantities to make a new total. In this case, we start with two units and then add another two units to that, resulting in a total of four units. Therefore, the answer is four.

GPT-3.5 model access through the OpenAI’s ChatGPT web interface

2 + 2 equals 4.

To explain how I got this answer, we can break it down step by step:

Start with the number 2. Add another 2 to it. When you combine these two 2s, you get 4.

So, by performing the addition operation, we have combined two quantities of 2, resulting in a total of 4.

As we can see in all these cases, the model answers with 4 either as a numeral (”4”) or as a word (”four”). The response includes descriptions of how the value is obtained that vary in depth, language, and tone. For our purposes and with regards to the math itself, the LLM produced an expected response and a sufficient explanation given the question.

These demonstrate one approach of an LLMs handling math, but not the only approach.

Approach 1: The LLMs predicts what should come next

The simplest version is that what an LLM does to produce a response is determine the sequence of words most likely to follow from the prompt itself. In this, LLMs generate a sequence of words using a sophisticated version of next word prediction.

For those using a smartphone, a version of next word prediction can be found on (in?) your predictive keyboard. Predictive keyboards display words above the smartphone keyboard to speed along typing by trying to guess what might be useful next. These predictions are made based on a likelihood the words might be useful next. “The”, “your”, and “up” are likely to follow “what’s”; :”morning”, “night”, and “luck” are likely to follow “good”. These words tend to be called suggestions. These suggestions are based on predictions of the typer’s intent, or, what someone is ultimately trying to communicate. Instead of “what’s up?” you may write “what’s your ETA?”. In this way the prediction of “up” was still highly likely, albeit not what was needed by a particular individual at that moment. LLMs are similar to predictive text, but depend on far more data and produce many words rather than just one.

LLMs are trained on lots of examples of writing. These examples allow LLMs to develop a schema of predicting which words, symbols, and sentences follow one another based on what comes before. These training data cover all sorts of examples of written language, some of which are math explanations, but also literature, recipes, historical accounts, and computer code, to name just a few. Because these—and so many other explanations—are fed into LLMs, the models have the ability to take in a prompt and predict a sequence of words that are likely to follow from it. If we use a simplified version of our prompt—prompt: “2+2=”—an LLM may predict ”4”, “___” (a blank line), or “?”, all of which are possible sequences of symbols following the prompt. The single response that is given is determined based on a variety of factors, but in many cases is the most likely next symbol, e.g. “4”. In this way, LLMs do not compute quantities, but compute likelihoods: the likelihood that something should follow something else.

LLMs use machine learning to create these relationships (”learn”) and a technique called attention to parse the important parts of the prompt. Combined, LLMs generate predictions based on the focal points of a prompt to determine intent. Some refer to attention as an LLM knowing what is important in a prompt. More accurately, attention allows models to prioritize parts of prompts using examples from training. This assessment of prompt intent using the attention mechanism narrows in on subtleties we as humans perceive in language, but previous natural language processing failed to capture. The result is a coherent response that, when it works, feels very much not like a machine.

Although in our example the math problem is likely to be found on the web, LLMs do not need the exact phrasing to make a coherent and seemingly accurate response. As mentioned, LLMs synthesize responses using relationships in language as much as the language itself. For example, an LLM may ingest examples that include questions like “What is 3+4=? Please explain” and “Show your work for: 10+2=”. These are not the same as our prompt but similar. Paired with their answers, these similar questions provide a structural relationship for how to answer questions of this type. If no example of our exact prompt was used in training, the LLM can leverage structural similarities. As such, responses can extend beyond what is found immediately and exactly in the training data for an LLM.

The result is an answer to a math question through language relationships. The expectedness and sufficiency of the answer—what one might say is correct or accurate—is derived from the content and tone of the data therein, and the receiver of the message. If you train an LLM on question-answer pairs where the answers are wrong, the model will generate responses that reflect that data. The answers may be seen as wrong, but the model is working perfectly. Likewise, if you train a model using an online community that has the potential to include derisive responses to simple questions—as in the case of Llama2 which uses user data from Facebook amongst other sources—then questions like “what is 2+2?” elicit a response commensurate with that data. Again, unexpected (and rude!), but working as designed.

From these techniques, LLMs can do math because they are trained on written explanations of math, and can extend that language knowledge if need be. Other factors influence the perceived accuracy and sufficiency of the response, such as the degree of randomness or length of response the system allows. Regardless, the approach is ultimately the same. The underlying work (not the explanation given) is closer to asking a lot of friends the answer rather than doing numerical calculation yourself. The answer is correct, but the work is not math-work per se.

Approach 2: The LLMs predicts what tools can do the job

In Approach 1, LLMs generate responses based on predictions. Prediction is the core of what an LLM does, but is not the only means an LLMs is used to handle math problems. A second approach involves identifying and applying specific tools (such as calculators) to effectively handle a prompt using the ReAct approach. Introduced by Yao et al. (2022), ReAct is short for Reason and Action and is the basis for frameworks such as Langchain and simpleaichat. ReAct depends on the language capabilities of LLMs to create so-called “chains-of-thought” that breakdown complex prompts into steps. Within these steps, the LLM handles tasks using native capabilities (namely, language prediction) with the added feature of being able to, when appropriate, invoke additional functions and tools (e.g. calculators, web search, encyclopedias). As such, the capabilities of LLMs are extended by the orchestration capabilities of a framework, all of which allows for sophisticated problem solving.

In the abstract, this approach may seem lofty. To ground the explanation, below is what this approach looks in the command line. Using a Python script, the following output uses OpenAI’s GPT-3.5 model and Langchain to answer the original prompt:

Entering new  chain...
steps=[Step(value='Add 2 and 2 together.'), Step(value='The sum of 2 and 2 is 4.'), Step(value="Given the above steps taken, respond to the user's original question.\n\n")]

Entering new  chain...
Thought: To add 2 and 2 together, I can use a calculator tool.

Action:
{"action": "Calculator", "action_input": "2 + 2"}

Entering new  chain...
numexpr.evaluate("2 + 2")

Answer: 4
Finished chain.

Observation: Answer: 4
Thought:To add 2 and 2 together, the result is 4.

Action:
{
  "action": "Final Answer",
  "action_input": "The sum of 2 and 2 is 4."
}

Finished chain.

Step: Add 2 and 2 together.

Response: The sum of 2 and 2 is 4.

Entering new  chain...
Action:
{
  "action": "Final Answer",
  "action_input": "The sum of 2 and 2 is 4."
}

Finished chain.

Step: The sum of 2 and 2 is 4.

Response: The sum of 2 and 2 is 4.

Entering new  chain...
Action:
{
  "action": "Final Answer",
  "action_input": "The sum of 2 and 2 is 4."
}

Finished chain.

Step: Given the above steps taken, respond to the user's original question.

Response: The sum of 2 and 2 is 4.
Finished chain.

So, what is happening here?

First, the script sends the prompt with additional orchestration instructions called a meta-prompt to the GPT-3.5. The meta-prompt is a written description for how the LLM should parse and approach the prompt itself. Langchain uses the following meta-prompt at each step and does so without explicitly sharing it in the command line output.

Assistant has just had the below interactions with a User. Assistant followed their "system: Instructions" closely. Your job is to critique the Assistant's performance and then revise the Instructions so that Assistant would quickly and correctly respond in the future.
 
####
{hist}
####
 
Please reflect on these interactions.

You should first critique Assistant's performance. What could Assistant have done better? What should the Assistant remember about this user? Are there things this user always wants? Indicate this with "Critique: ...".

You should next revise the Instructions so that Assistant would quickly and correctly respond in the future. Assistant's goal is to satisfy the user in as few interactions as possible. Assistant will only see the new Instructions, not the interaction history, so anything important must be summarized in the Instructions. Don't forget any important details in the current Instructions! Indicate the new Instructions by "Instructions: ...".

This code snippet contains two things of note. First, the meta-prompt is considered a type of prompt engineering. Prompt engineering is a technique for constraining an LLM to respond in specific and hopefully expected ways. Second, {hist} is how an LLM is given memory. The variable {hist}, which stands for “history”, is a way to store previous steps and outputs. These previous steps are fed forward into subsequent steps to give the process continuity. As such, the LLM can refer to what has been done before to inform subsequent actions and does so in a constrained manner to focus on answering the question sufficiently. Combined, past actions and constrained responses seek to make the LLM an engine for predictable process automation.

Using this meta-prompt, the GPT-3.5 model constructs a set of steps based on the prompt “What is 2+2? Explain how you got an answer.” The output is this:

Entering new  chain...
steps=[Step(value='Add 2 and 2 together.'), Step(value='The sum of 2 and 2 is 4.'), Step(value="Given the above steps taken, respond to the user's original question.\n\n")]

These steps are then fed back into another request to GPT-3.5 using the same meta-prompt. The model processes the meta-prompt (now with more history!) and prompt, and generates a “Thought” for what to do next. From the “Thought,” the LLM determines that there is a tool for the task at hand. In our case, the determination is that “2+2” is a math problem, and so the LLM predicts that the calculator is an appropriate tool:

Thought: To add 2 and 2 together, I can use a calculator tool.

When the calculator performs the simple calculation, the model cycles through another time to determine what to do next.

Just as the meta-prompt is hidden above, so too are the ways frameworks extend the capabilities of LLMs with tools. To determine that a tool is applicable in a given step, the Python script must define upfront the conditions for when a tool is useful. For example, in the script used here, the calculator tool is described in the following manner:

tools = [
	Tool(
			name = "Calculator",
			func = llm_math_chain.run,
			description = "useful for when you need to answer questions about math"
		)
]

As the LLM works through the meta-prompt and prompt, this tool description augments what answers are possible. If the LLM “reflects” on a “Thought” that a tool may be useful (say, the calculator), then the tool can be applied. In our case, once the calculator tool is invoked, the script performs the appropriate action (e.g. calculating), and then moves on to another language-based step (e.g. composing a response, determining the response is a final response). As may be apparent, if the LLM make an unexpected prediction—determines a web search is applicable for 2+2—something else might happen instead, such provide an answer about the Bob Seger System’s anti-war song “2+2=?".

The cycle of Reason (”Thought”) and Action concludes when the LLM determines that the final answer has been obtained by using the meta-prompt to assess the state of affairs. Upon this determination, LLM generates an answer (”Response:):

Response: The sum of 2 and 2 is 4.

In the end, the combination of a framework and an LLM provides an answer to the math problem through language—here using language to determine steps and assess next steps—and through the application of a tool—here a calculator to compute the answer. Notably, the response generated by the LLM does not explain much about why 2+2 = 4 as the responses in the previous approach did. Instead, we see the LLM show work for its own logic without explaining much about the underlying and requested math work. In other words, the final answer is correct and the explanation is insufficient.

In tension

These approaches illustrate that when an LLM produces an answer to a math question, the underlying work is different than we might expect. As such, I want expand on a key tension that underlies LLMs and generative AI more broadly. This tension is found in the context of LLMs doing math problems, but is by no means limited to that context. The tension is this:

LLMs seem very human, but also are non-human computing technologies.

The tension stems from an inherent characteristic of LLMs, namely, these technologies display emergent capabilities. “Emergent” means that LLMs demonstrate capabilities for things they weren’t explicitly built to do. In our example, being able to seemingly solve math questions is an emergent capability.

These capabilities are one reason LLMs have drawn so much attention. For example, the “eureka moment” for Microsoft CEO Satya Nadella was when the OpenAI’s GPT-4 model “preserved the sovereignty of [Rumi’s] poetry across two language[s]” (Wired, 2023). Nadella points out that machine translation is by no means new. For him, what felt new was the ability of an LLM to capture meaning that exceeded mere word-for-word translation. This and other emergent capabilities are uncanny and, importantly, unexplainable to many researchers in the field. Despite building the techniques of machine learning, experts are unable to pin down how LLMs are capable of things previous models were not. The main explanation given is that more data is used to train LLMs, but this explanation only describes the conditions rather than the mechanisms for emergence. Regardless of why, experiencing LLMs and generative AI do new things results in a flywheel: “what else can these things do?!” And therein is the tension.

On one hand of this tension is the attribution of humanness to non-humans, or anthropomorphization. Many assign human qualities and descriptors to LLMs because they observe LLMs acting in ways and in contexts that are familiar to us as human actors in those contexts. For example, a back-and-forth exchange in WhatsApp is not that dissimilar in form and content than an exchange with ChatGPT. LLMs present a compelling illusion of humanness by producing responses that look and feel human. Responses use sentences and paragraphs, ideas and concepts, and what has been written previously (e.g. ”memory” or “history”). Combined, the effect feels truly astounding and very human-like. In turn, we come to expect LLMs to be like us.

On the other hand of this tension is the attribution of controllable causality to LLMs because they are made-things. I’ll refer to this as hubristic determinacy (though I am open to a better term). Despite “chatting with us,” LLMs are computing technologies. As such, we also expect LLMs to work how computers work both in terms of actions (e.g. compute, calculate) and outcomes (e.g. produce what we are looking for). The context and underlying knowledge that these are made-things means we assume at some level they are also things-to-be-controlled, even if we cannot control them just yet. When LLMs do things we do not expect or feel off, the presumption is that the technology can be reigned in with current or future techniques and knowledge.

Terminology is particularly revealing of this tension. Take the term “hallucination”, which entered the popular lexicon in recent years. “Hallucination” is used to describe instances when LLMs generate unexpected outputs, especially in relation to known facts. For example, LLM may cite legal precedents that do not exist if prompted to do so. As a term, “hallucinations” captures that LLMs are not doing what we want them to do—namely, be controlled for our specific ends. Though obviously anthropomorphic because it attributes human-like agency to the actions of LLMs, the term “hallucinations” is tellingly invoked as a metaphor to help us understand how a machine resists our will. As such, the term itself is anthropomorphic and used to cope with when expectations of control undermined.

The trouble is that this tension is a condition of generative AI rather than a resolvable bug. LLMs are technologies and tools with particular dispositions and affordances. These dispositions and affordances include not being especially predictable from the perspective of an observer and being able to interface with people through natural language. Anthropomorphization helps us understand these systems, but does not represent them. Hubristic determinacy expresses our intentions, but does not manifest them. Viewing this tension as something to overcome leads to faulty expectations, and, I would argue, undercuts our ability to use LLMs effectively—in the right context, for the right purpose, with the appropriate mindset. In sum, the tension sidesteps the a different line of questioning: what should we actually expect from LLMs?

Dependably unexpected

The two approaches of how an LLM handles a math question are likely not the only two. The reason for focusing on these approaches has been to unpack the central role of LLMs as mediators of language when prompted with math questions. The approaches have similar punchlines: either an LLM predicts what might come next or predicts whether a tool is needed next to do a task. In the first approach, the outputs are produced through the use of language examples of the central math question. In the second approach, the outputs are produced through the use of language to determine the work needed to produce an answer.

For someone learning math, how these systems actually answer math questions may matter very little when they are right. In our prompt, the explanations provided are sufficient and the calculations are correct. The issue comes in when LLMs are seemingly wrong. In the first approach, an incorrect answer stems from the way the prompt is parsed, data are leveraged to respond, and the alchemy of responding. In the second approach, an incorrect answer stems from the similar factors as the first, albeit at the various steps along the way: the prompt is parsed in a way that no tool is selected; the wrong tool is selected; the “Thoughts” generated are unexpected at a step and so gumming up the ReAct sequence; the final answer generated in an unexpected way, even if the right tool is used and calculation is made correctly (this tutorial documents just this scenario).

In instances when LLMs fall short of our expectations, those expectations come into focus, even though they have been there all along. So LLMs use different work to get to math explanations, so what? The issue is that without critical reflection and attenuated expectations based on the realities of these technology, those building and using technologies fail to consider whether these technologies are appropriate for a given context, problem, or circumstance.

The main lesson, then, relates to the tension of these technologies: despite how human LLMs feel or how much we know they are made, they are neither of those things. On one side, we should can easily confuse the human-feeling inputs and outputs with acting and thinking like a person. Yes, LLMs can respond to math questions with math answers. Observers can judge those answers as right or wrong, justified or fabricated, accurate or inaccurate. But LLMs don’t do what we think they do, rendering our evaluations inaccurate. Similarly, despite humans making these technologies, they are not controllable in the same way other technologies are. They surprise us both with compelling and unnerving things, but also with things that don’t correspond to reality. In short, we need a different mindset.

What can be reasonably asserted is that LLMs—and generative AI overall—are dependably unexpected in how they do what they do. That unexpectedness is a risk and liability in instances and applications when LLMs and other generative AI are expected to mediate circumstances or conditions that humans are far more adept at, or when inherent biases in data trickle into responses. More optimistically, the dependable unexpectedness of LLMs offers us a glimpse at pathways we might never consider because we as humans respond given our history and education, personal and professional expectations, and social mores that may foreclose certain considerations. In other words, many times we expect particular outcomes to questions, and LLMs might surprise us (but caveat emptor).

Considering LLMs as dependably unexpected is maybe more useful in the inverse. When expectedness is critical, LLMs and generative AI are likely not the technology to use. Approaching the application of LLMs in terms of a tolerance of unexpectedness is, then, a way to think of their application as adding value to scenarios, and detracting when safety, stability, and assurance are preferred. Because LLMs are given human-qualities, a prevailing assumption is that they can be taught to be good actors in a human world. Regardless, those using LLMs for tasks or in products have an obligation to think about the human world LLMs might create, and whether that world is one people are eager to expect.