TA6: Using AI for research — LLMs in empirical finance

Expected outcomes

By the end of this TA, you should be able to:

  • explain what a large language model (LLM) is and why it is not a database
  • set up an OpenAI API account and make your first call from R using httr2
  • design structured prompts that return guaranteed-parseable JSON
  • explain why temperature = 0 is required for classification tasks
  • identify the three most important practical constraints on LLM-based research: cost, privacy, and reproducibility
  • use the caching pattern from TA5 to run LLM calls once and never again
  • articulate the difference between a single LLM call and an agent loop
  • critically compare what an LLM and a transparent dictionary tell you about the same text

Download TA6 Code

The full source script is code/07_ta6_llm_research.R. Every result on this page is generated by that script.

What LLMs are — and what they are not

A large language model is a statistical model trained to predict the next token in a sequence. It has no database, no memory between sessions (unless you give it one), and no ground truth. What it does have is an extraordinarily compressed representation of patterns in text — billions of human-written documents — which means it is extremely good at tasks that look like “what would a knowledgeable person write in response to this?”

This makes LLMs useful for:

  • Classification: is this 10-K’s risk-factor section positive, negative, or neutral?
  • Extraction: which named entities appear in this earnings call?
  • Summarisation: give me the main argument of this working paper in two sentences.
  • Generation: write a first draft of the related-literature section given these references.
  • Translation / normalisation: convert these inconsistently formatted firm names into a standard CUSIP-linkable key.

It makes LLMs unreliable for:

  • Factual lookup: “what was JPMorgan’s reported ROE in Q3 2023?” The model will give you a number. It may be wrong. Never use an LLM as a primary source for specific numerical facts.
  • Reproducibility over time: model providers update their models. A call with gpt-4o-mini today may return a different answer in six months even at temperature = 0. Cache your outputs.
  • Confidential data: calling an external API means your data leaves your machine. Never send proprietary, pre-publication, or personally-identifiable data to a commercial API without explicit clearance from your institution’s data governance and legal teams.

Setting up an OpenAI account

Three steps:

  1. Go to platform.openai.com and create an account.
  2. Navigate to API keys in your account dashboard and create a key (sk-...).
  3. Set the key as an environment variable in R — never hard-code it in a script:
Sys.setenv(OPENAI_API_KEY = "sk-...")

# Or add it to your ~/.Renviron so it loads automatically in every R session:
# OPENAI_API_KEY=sk-...
  1. Verify the key is visible:
nzchar(Sys.getenv("OPENAI_API_KEY"))  # should return TRUE

Critical rule: your .Renviron file must be in your .gitignore. A leaked API key is a billing problem — attackers find leaked keys within minutes and use them to generate tokens at your expense.

How the API actually works

An LLM API call is an HTTP POST. The request body contains a list of messages (a conversation history), the model name, and a few parameters. The response body contains the model’s reply. In R, httr2 handles this with no wrapper packages needed:

httr2::request("https://api.openai.com/v1/chat/completions") |>
  httr2::req_auth_bearer_token(Sys.getenv("OPENAI_API_KEY")) |>
  httr2::req_body_json(list(
    model       = "gpt-4o-mini",
    temperature = 0,
    response_format = list(type = "json_object"),
    messages = list(
      list(role = "system", content = "You are a financial analyst. Return JSON."),
      list(role = "user",   content = "Classify the sentiment of: 'record profits'")
    )
  )) |>
  httr2::req_perform() |>
  httr2::resp_body_json()

Three parameters matter most:

  • temperature: controls randomness. 0 = deterministic (same input, same output, every time). 1 = creative. For classification and extraction tasks: always 0.
  • response_format = list(type = "json_object"): tells the model it must return a valid JSON object. Without this, the model sometimes includes prose around the JSON and your parser breaks.
  • model: gpt-4o-mini is the cheap option (roughly 100× cheaper than GPT-4o). Use it for classification and extraction. Use GPT-4o or GPT-o1 when reasoning quality matters.

Prompt engineering

The quality of an LLM output is almost entirely determined by the quality of the prompt. Bad prompts produce bad outputs even with the best model; good prompts get useful results from cheap models.

The anatomy of a good prompt

Every prompt has two parts: the system message (sets the role and rules) and the user message (the specific input). For a classification task you want the system message to do four things:

  1. Assign a role: “You are a financial analyst assistant.”
  2. Specify the output format exactly: list every JSON field, its type, and its allowed values.
  3. Set scope constraints: “Base your assessment ONLY on the provided text.” vs “Use your general knowledge.”
  4. Eliminate ambiguity: if “neutral” means “no strong evidence either way”, say that explicitly.

Here is the system prompt we use for the excerpt-based analysis:

You are a financial analyst assistant. Analyze the provided Wikipedia excerpt about a bank and return a JSON object with EXACTLY these fields:
  sentiment       : one of 'positive', 'neutral', or 'negative'
  sentiment_reason: a single sentence explaining the sentiment
  business_lines  : an array of strings listing main activities mentioned
  summary         : a single sentence describing what the bank does
Base your assessment ONLY on the provided text. Do not use outside knowledge.

And the one for the general-knowledge reputation analysis:

  reputation_assessment: one of 'positive', 'neutral', or 'negative'
  key_controversies    : an array of 2-4 strings, each describing one issue
  summary              : a single sentence summarising the reputation assessment
NA
NA
NA
NA

What changes between prompts

The only difference between the two prompts above is the scope instruction. Prompt A says “base your assessment ONLY on the provided text.” Prompt B says “based on your general knowledge.” This single sentence change produces dramatically different outputs for some banks and identical outputs for others — which is exactly the comparison we are about to run.

Length, multi-turn, and chain of thought

A few practical rules for prompt design:

  • Be specific, not long. A focused 100-token prompt usually outperforms a vague 500-token one.
  • Ask for one thing at a time. If you need sentiment and a list of risk factors and a summary, it is fine to ask for all three in one JSON object — but structure them clearly.
  • Chain of thought: for reasoning tasks, adding “think step by step” before the final answer often improves quality. This is less important at temperature = 0 for simple classification.
  • Multi-turn: for iterative tasks — “now refine your summary to be more formal” — use the conversation history. Each additional turn costs tokens.
  • Few-shot examples: providing two or three example input-output pairs in the user prompt dramatically improves consistency for edge cases.

What to consider before using LLMs in research

Cost

LLMs cost money, but less than most people expect. The gpt-4o-mini model costs roughly $0.15 per million input tokens and $0.60 per million output tokens. This TA exercise — 20 API calls analysing bank excerpts — costs about $0.002 in total.

The numbers that matter for a real project:

Task Cost
100 bank excerpts (~250 tokens each) ~$0.01
100 full 10-K filings (~40,000 tokens each) ~$0.61
1,000 full 10-K filings (~40,000 tokens each) ~$6.09

Two rules: cache every response on disk (so you pay once, not once per pipeline run), and use gpt-4o-mini for classification/extraction tasks — GPT-4o is roughly 15× more expensive per token.

Privacy and data governance

If the data contains material non-public information, pre-publication research findings, or personal data about identifiable individuals, you need explicit clearance from your institution before it leaves your machine. Options when clearance is not available: Azure OpenAI with a private deployment, an open-source model running locally (e.g. llama.cpp, ollama), or restructuring the task so that only non-sensitive metadata leaves your machine.

Reproducibility

Three threats:

  1. Model updates: OpenAI silently updates model weights. Archive the exact model string (e.g. gpt-4o-mini-2024-07-18) in your paper’s data appendix, not just gpt-4o-mini.
  2. Temperature > 0: non-zero temperature means non-deterministic outputs. If you use it, run the same prompt multiple times and report the distribution, not a single draw.
  3. Cache loss: if you lose the cached responses, you cannot reproduce the exact outputs. Treat the cache the way you treat your data — version it, back it up.

Hallucinations

LLMs confidently produce false information. For classification tasks on text you provide, hallucinations are rare (the model is constrained by the text). For factual-lookup tasks (specific numbers, dates, citations), hallucinations are common and dangerous. A standard check: if the answer is specific and numerical, verify it against a primary source.

Time to code

source("code/00_utils.R")
load_finance4_packages()

REFRESH_CACHE <- FALSE   # set to TRUE with OPENAI_API_KEY set to make live calls

The full script is code/07_ta6_llm_research.R. The remainder of this section walks through every result in the same order the script produces them.

Step 1: Loading the TA5 corpus

We pick up exactly where TA5 left off — the Wikipedia excerpt for each of the 15 largest US banks, plus the toy-dictionary tone scores we computed by hand.

The TA5 Wikipedia corpus used as input to the LLM analysis.
Rank Bank Excerpt (first 120 chars)
1 JPMorgan Chase JPMorgan Chase & Co. (stylized as JPMorganChase) is an American multinational banking institution headquartered in New Y
2 Bank of America The Bank of America Corporation (Bank of America; often abbreviated BAC or BofA) is an American multinational investment
3 Citigroup Citigroup Inc. or Citi (stylized as citi) is an American multinational investment bank and financial services company ba
4 Wells Fargo Wells Fargo & Company is an American multinational financial services company.[5][8] The company operates in 35 countrie
5 Goldman Sachs The Goldman Sachs Group, Inc. (/sæks/ SAKS) is an American multinational investment bank and financial services company.
6 Morgan Stanley Morgan Stanley[4] is an American multinational investment bank and financial services company headquartered at 1585 Broa
7 U.S. Bancorp U.S. Bancorp (stylized as us bancorp) is an American multinational banking institution headquartered in Minneapolis, Min
8 Capital One Capital One Financial Corporation is an American bank holding company headquartered in Tysons, Virginia, with operations
9 PNC Financial Services The PNC Financial Services Group, Inc. is an American bank holding company and financial services corporation based in P
10 Truist Financial Truist Financial Corporation (/ˈtruːɪst/[6]) is an American bank holding company headquartered in Charlotte, North Carol
11 TD Bank, N.A. TD Bank, N.A. (previously TD Banknorth) is an American national bank and the United States subsidiary of the Canadian mu
12 Charles Schwab Corporation The Charles Schwab Corporation[1] is an American multinational financial services company. It offers banking, commercial
13 The Bank of New York Mellon The Bank of New York Mellon Corporation, commonly known as BNY or by its prior brand name BNY Mellon, is an American int
14 State Street Corporation State Street Corporation is an American multinational[2]financial services and bank holding company headquartered at One
15 American Express American Express Company (Amex) is an American bank holding company and multinational financial services corporation tha

Step 2: Excerpt-based sentiment analysis — all 15 banks (Prompt A)

For each bank we send a prompt containing only the 600-character excerpt and ask the model for a structured JSON response. Prompt A explicitly says “base your assessment ONLY on the provided text.”

LLM excerpt-based sentiment and business lines for all 15 banks (Prompt A, temperature = 0).
Rank Bank Sentiment Business lines
1 JPMorgan Chase positive retail banking; investment banking; commercial banking; a…
2 Bank of America positive retail banking; investment banking; wealth management; co…
3 Citigroup neutral investment banking; consumer banking; global financial se…
4 Wells Fargo neutral retail banking; commercial banking; consumer finance; mor…
5 Goldman Sachs positive investment banking; securities; asset management; consume…
6 Morgan Stanley positive investment banking; wealth management; investment managem…
7 U.S. Bancorp positive retail banking; commercial banking; payment services; wea…
8 Capital One neutral credit cards; consumer banking; commercial banking; auto …
9 PNC Financial Services positive retail banking; corporate banking; asset management; weal…
10 Truist Financial neutral retail banking; commercial banking; investment banking; i…
11 TD Bank, N.A. neutral retail banking; consumer lending; commercial banking
12 Charles Schwab Corporation positive retail brokerage; investment management; banking; wealth …
13 The Bank of New York Mellon neutral investment management; investment services; custody banki…
14 State Street Corporation neutral investment servicing; investment management; custody bank…
15 American Express positive payment network; credit cards; travel services; financial…

LLM sentiment distribution

Eight banks score positive, seven neutral, and none negative. The positive cases are the banks whose Wikipedia leads emphasise scale, global reach, and market leadership. The neutral cases are banks whose leads focus on structural facts (merger origins, geographic coverage) without strong evaluative language. Nothing in the excerpt text for any of these 15 banks is primarily negative — which is exactly what you would expect from an introductory Wikipedia paragraph.

This is the first lesson: the LLM is only as good as what you give it. If the input text is neutral, so is the output. This is not a bug; it is the correct behaviour of a model following its instructions.

Step 3: Reputation analysis for 5 selected banks (Prompt B)

We switch to Prompt B — general knowledge — for five banks chosen because they illustrate the gap between what a Wikipedia lead paragraph says and what the model knows from its broader training data.

LLM general-knowledge reputation assessment for 5 selected banks (Prompt B, temperature = 0).
Bank Assessment Key controversies (truncated)
Wells Fargo negative 2016 fake accounts scandal: employees opened millions of unauthorised savings and checking accoun…
Goldman Sachs neutral 1MDB scandal: Goldman Sachs paid approximately $5 billion to settle charges related to its role i…
TD Bank, N.A. negative 2024 anti-money laundering (AML) settlement: TD Bank pleaded guilty to Bank Secrecy Act violation…
Citigroup neutral Major recipient of government bailout funds during the 2008 financial crisis, receiving $45 billi…
JPMorgan Chase positive 2013 ‘London Whale’ trading loss of approximately $6 billion due to complex derivatives positions…

Two banks flip to negative: Wells Fargo (the fake-accounts scandal and Federal Reserve asset cap) and TD Bank (the 2024 AML settlement, the largest anti-money-laundering penalty ever imposed on a US bank). Goldman Sachs moves from positive to neutral (the 1MDB scandal and broader post-crisis criticism). JPMorgan stays positive. Citigroup stays neutral.

Step 4: Three-way comparison — toy dictionary, Prompt A, Prompt B

This is the core diagnostic. We line up three methods side by side for the same five banks.

Three-way sentiment comparison: toy dictionary vs LLM excerpt-only vs LLM general knowledge.
Bank Toy dictionary (TA5) LLM excerpt (Prompt A) LLM knowledge (Prompt B) Toy dict score
Wells Fargo neutral neutral negative 0.000
Goldman Sachs positive positive neutral 0.019
TD Bank, N.A. negative neutral negative -0.026
Citigroup positive neutral neutral 0.040
JPMorgan Chase positive positive positive 0.014

Sentiment comparison heatmap

Read the comparison table carefully — four stories emerge:

Wells Fargo: toy dict = neutral (no matching words in excerpt), LLM excerpt = neutral (excerpt contains no negative language), LLM knowledge = negative (model knows the accounts scandal). The LLM adds real information that neither the text nor the dictionary contained.

TD Bank: toy dict = negative (the word “risk” appeared in the excerpt), LLM excerpt = neutral (the excerpt focused on geographic coverage, not risk), LLM knowledge = negative (the AML settlement). Interesting: the toy dictionary was accidentally right for the wrong reason. The word “risk” in the excerpt was not about the AML settlement — it was about regulatory classification language.

Goldman Sachs: toy dict = positive, LLM excerpt = positive, LLM knowledge = neutral. The LLM knows about the 1MDB scandal and the “vampire squid” criticism; neither the excerpt nor the dictionary had any signal for this.

Citigroup: toy dict = positive (the word “investment” appeared, which is in our positive list), LLM excerpt = neutral. The toy dictionary was wrong here — Citigroup’s lead is not positive, it is just structurally descriptive.

JPMorgan: all three agree on positive. This is the easy case.

Step 5: Cost estimate

Estimated API cost for the TA6 exercise (gpt-4o-mini, mid-2025 pricing).
Prompt type API calls Input tokens Output tokens Input cost (\()| Output cost (\)) Total cost ($)
Prompt A — excerpt 15 3627 1337 0.00054 0.00080 0.00135
Prompt B — reputation 5 974 550 0.00015 0.00033 0.00048
TOTAL 20 4601 1887 0.00069 0.00113 0.00182

The entire exercise — 20 API calls, 15 excerpt analyses and 5 reputation analyses — costs about $0.002. At this price point there is no reason to avoid LLMs for small classification tasks. The cost only becomes material when you scale to thousands of long documents (10-K filings, earnings call transcripts).

Step 6: Agents — the next level

A single API call takes one input and returns one output. An agent is a loop: the model generates a tool call, you execute it, feed the result back, and the model generates the next action. This continues until the model signals it is done.

# Conceptual agent loop — pseudocode only, not executed in TA6

# Tools the agent can call:
tools <- list(
  search = function(query) { ... },     # web search or database lookup
  read   = function(url)   { ... },     # fetch and return text
  write  = function(path, text) { ... } # save results to disk
)

# Agent loop:
task <- 'Summarise risk factors for the top 10 US banks from SEC EDGAR'
history <- list(list(role = 'user', content = task))

repeat {
  response <- call_llm(history, tools = tools)
  if (response$type == 'final_answer') break
  # Execute the tool the model requested
  tool_result <- tools[[response$tool]](response$arguments)
  # Feed the result back into the conversation
  history <- append(history, list(list(role = 'tool', content = tool_result)))
}

The practical value of agents for finance research: tasks like “for each of the 500 banks in my sample, find their most recent 10-K on SEC EDGAR, download the risk-factors section, classify the top three risk themes, and write the result to a CSV” can be automated with an agent that chains a search tool, a download tool, and an LLM call. Without agents, you write that pipeline by hand; with agents, you write the task description in English and the model figures out the steps.

R packages that support agent-style loops: ellmer and tidyllm. The OpenAI Responses API supports server-side tool calling directly.

Question for you

You want to classify the sentiment of 10,000 earnings call transcripts. Each transcript is approximately 20,000 tokens. You plan to use gpt-4o-mini at the prices in the cost table above.

  1. Estimate the total input-token cost.
  2. What would you do differently if the transcripts contain material non-public information?
  3. How would you design the caching strategy so the pipeline can be interrupted and resumed without re-paying for calls already made?

What to take away

  • LLMs are pattern-completion engines, not databases. Use them where a knowledgeable person reading the text would be useful; never use them as a primary source for specific numerical facts.
  • Temperature = 0 for classification and extraction. Document the exact model version in your data appendix. Cache every response.
  • Prompt design is the research design. The scope instruction (“based only on this text” vs “based on your general knowledge”) is a methodological choice with real consequences for what you measure.
  • The comparison table is the result. The interesting signal in TA6 is not “what is Wells Fargo’s sentiment?” — it is “why do the toy dictionary, the excerpt LLM, and the knowledge LLM disagree, and what does that tell us about what each method is actually measuring?”
  • Agents are the next step: chaining LLM calls with tool-use lets you automate multi-step research pipelines. The infrastructure is available today in R through ellmer and tidyllm.

A few questions to consolidate:

  • Why does temperature = 0 not guarantee reproducibility across time?
  • You send a 10-K filing to the OpenAI API. Before doing so, what questions should you ask your institution’s data governance team?
  • Why did the toy dictionary call TD Bank negative when the excerpt contained no explicit negative language?
  • What is the difference between an LLM call and an agent? When would you reach for one versus the other?
  • If a journal referee asks you to reproduce an LLM-based result from three years ago, what should your paper’s data appendix contain?

For further reading:

  • Lopez-Lira, T., and Y. Tang (2023), “Can ChatGPT Forecast Stock Price Movements? Return Predictability and Large Language Models”, SSRN Working Paper.
  • Kim, A. G., M. Muhn, and V. V. Nikolaev (2024), “Financial Statement Analysis with Large Language Models”, SSRN Working Paper.
  • Loughran, T., and B. McDonald (2011), “When is a Liability not a Liability? Textual Analysis, Dictionaries, and 10-Ks”, Journal of Finance 66(1), 35–65.
  • OpenAI (2024), Prompt Engineering Guide: platform.openai.com/docs/guides/prompt-engineering.