| Rank | Bank | Headquarters | Total assets (US$ bn) |
|---|---|---|---|
| 1 | JPMorgan Chase | New York City | 4424 |
| 2 | Bank of America | Charlotte, North Carolina | 3411 |
| 3 | Citigroup | New York City | 2657 |
| 4 | Wells Fargo | San Francisco, California | 2148 |
| 5 | Goldman Sachs | New York City | 1809 |
| 6 | Morgan Stanley | New York City | 1420 |
| 7 | U.S. Bancorp | Minneapolis, Minnesota | 692 |
| 8 | Capital One | Tysons, Virginia | 669 |
| 9 | PNC Financial Services | Pittsburgh, Pennsylvania | 573 |
| 10 | Truist Financial | Charlotte, North Carolina | 547 |
| 11 | TD Bank, N.A. | Cherry Hill, New Jersey[a] | 519 |
| 12 | Charles Schwab Corporation | Westlake, Texas | 490 |
| 13 | The Bank of New York Mellon | New York City | 472 |
| 14 | State Street Corporation | Boston, Massachusetts | 366 |
| 15 | American Express | New York City | 300 |
TA5: Introduction to web scraping
Expected outcomes
By the end of this TA, you should be able to explain:
- what a web page actually is — the HTML / DOM tree, CSS selectors, and how a browser turns one into the other
- the difference between a static page (where the HTML response already contains the data) and a dynamic page (where JavaScript renders the data after the page loads)
- the
rvestworkflow:read_html()→html_elements()→html_text2()/html_table() - when you must reach for a real browser instead —
chromoteorRSelenium - the legal and ethical rules that should bind every scraper you write: User-Agent, robots.txt, rate limiting, Terms of Service, copyright, and GDPR
- how to turn raw HTML into a small but useful dataset and run a basic text analysis on it
The full source script is code/06_ta5_webscraping.R. Every result you see on this page is generated by that script and saved into output/tables/, output/figures/, or output/text/.
The main empirical question
This TA is built around a single concrete question:
What can we learn about the largest US banks by scraping public web data, and how do we do it safely, ethically, and reproducibly?
The web is the largest and least-curated dataset humanity has ever assembled. A surprising fraction of recent finance papers — Adams & Mosk (2023) on letters to Congress, Akcigit, Baslandze & Lotti (2023) on political connections in patents, Akey, Heimer & Lewellen (2021) on politicised consumer credit, Babenko, Fedaseyeu & Zhang (2020) on CEO political donations — rely on data that exists only because someone scraped it. None of those papers had a paid Bloomberg-like data feed; they all sat down with HTML pages and parsed them.
For our example we scrape Wikipedia’s List of largest banks in the United States, follow the links to each bank’s individual article, and assemble a small text corpus we can do real analysis on. Wikipedia is an honest teaching example because:
- it is explicitly OK to scrape under the CC-BY-SA licence (with attribution),
- it is stable — the wikitable structure and the URL scheme barely change from year to year,
- it lets us combine structured data (the table) with unstructured data (the article text), which is exactly the workflow most finance scraping projects end up using.
Web scraping
A web page is just a tree of HTML tags. Your browser receives that tree, runs any JavaScript attached to it, and renders the result. A scraper does the first step (download the tree) and skips the second (rendering) — which is why static scraping works only when the data you want is already in the response.
The rvest workflow
In R the canonical static-scraping pipeline is three function calls:
page <- xml2::read_html("https://example.com/some-page")
table <- page |>
rvest::html_elements("table.wikitable") |>
rvest::html_table(fill = TRUE) |>
purrr::pluck(1)
paragraphs <- page |>
rvest::html_elements("div.mw-parser-output > p") |>
rvest::html_text2()Three things to internalise:
- CSS selectors are the language we use to point at HTML elements.
table.wikitablemeans “every<table>whose class includeswikitable”.div.mw-parser-output > pmeans “every<p>that is an immediate child of<div class='mw-parser-output'>”. The browser developer tools give you these selectors for free — right-click any element and choose Inspect. html_elements()returns a set of nodes,html_element()returns a single node. They have the same name minus ansand the difference matters once you start nesting walks.html_text2()is almost always what you want for human-readable text.html_text()keeps every\nand\texactly as they appear in the source, which produces ugly output 90% of the time.
When rvest is enough — and when it is not
rvest is enough whenever the data you want is in the HTML response. You can check this in 30 seconds: open the page in the browser, view the page source (Cmd-U / Ctrl-U), and search the source for the value you can see on the page. If it is there, rvest will find it. If the search returns nothing, the page is rendered dynamically by JavaScript and you need a real browser, which is what chromote and RSelenium are for. We come back to this in the When you need a real browser section below.
Legal and ethical considerations
Scraping is not technically illegal in most jurisdictions, but it can break a contract (the site’s Terms of Service), violate copyright, violate computer-misuse statutes if the target tries to block you and you keep going, or violate privacy law if the data is personal. The rules below are what a careful empirical researcher always follows.
The seven rules of polite scraping
- Identify yourself. Always send a meaningful
User-Agentheader. The script in this TA sendsFinance4-TA5-educational-scraper/1.0 (... contact e-mail ...). A site admin who sees that string can decide whether they care; an anonymous bot just gets blocked. - Respect
robots.txt. Every well-behaved site publishes one at/robots.txt. Wikipedia’s, for instance, lists every directory that bots are not supposed to crawl. Read it and obey it. - Throttle. A single request per second is a safe default for small projects. The
httr2::req_throttle()helper does this for you; the polite-scraping pattern incode/00_utils.Rdoes it viaSys.sleep(1). - Cache everything. Scrape once, analyse forever. The cached HTML is part of your dataset and must be archived alongside your code; otherwise your work is not reproducible.
- Honour the licence. Wikipedia is CC-BY-SA: any derivative dataset must keep the attribution and share-alike requirement. SEC EDGAR is public-domain. Most newspapers are not — they ban scraping in their Terms of Service even when their content is public.
- Mind GDPR (and equivalent laws). As soon as the data you scrape contains personal information about identifiable EU residents, you are processing personal data and you need a lawful basis under GDPR. “It is on the public web” is not a lawful basis on its own.
- Prefer APIs. Wikipedia, GitHub, Twitter (X), the SEC, the Federal Reserve, and most major data sources offer APIs. If an API exists, the API is more stable, more polite, and usually faster. Scraping is the fallback, not the goal.
Examples of where this matters
- The hiQ vs LinkedIn litigation in the US (2017–2022) settled the rough principle that scraping public data from a site you do not have an account with is not a Computer Fraud and Abuse Act violation, but the case explicitly did not address ToS or copyright.
- The Ryanair vs PR Aviation ruling in the EU (2015) confirmed that a site’s ToS can prohibit scraping even when the scraping itself is technically legal.
- Adams & Mosk (2023) scrape letters that constituents send to Congress members — public, government records, free of personal-data issues. That is the kind of source you can scrape safely. A press archive that ToS-prohibits scraping is not.
When you need a real browser
Most financial dashboards (Yahoo Finance ticker pages, Bloomberg-like front-ends, fund factsheet viewers, real-time market depth charts) are JavaScript-rendered. The HTML you receive over HTTP is a near-empty shell, and the actual table you can see in the browser is built client-side after the page loads. rvest cannot see that content because rvest does not run JavaScript.
The two standard R answers are:
chromoteis a thin wrapper around headless Chrome via the DevTools protocol. It is the modern recommendation: no Java required, well maintained, and easy to install.RSeleniumis the older, classic Selenium binding. More universal — there are Selenium drivers for every browser — but heavier, requires Java, and is best run inside a Docker container.
The pattern with chromote looks like this. We do not run it as part of the TA pipeline because it requires a Chromium install on the lab machine, but the snippet is shipped as output/text/ta5_chromote_snippet.R so you can reuse it in your own work.
library(chromote)
b <- ChromoteSession$new()
b$Page$navigate("https://example.com/dashboard")
b$Page$loadEventFired()
rendered <- b$DOM$getDocument() |>
purrr::pluck("root", "nodeId") |>
(\(id) b$DOM$getOuterHTML(nodeId = id))() |>
purrr::pluck("outerHTML")
b$close()
parsed <- xml2::read_html(rendered)Once you have parsed, the rest of the pipeline (html_elements, html_table, …) is identical to the static case.
Where this fits in finance research
Web scraping is rarely the contribution of a paper, but it is often the enabling step without which the paper could not have happened. A few examples directly from the syllabus reading list:
- Adams & Mosk (2023) — Dear Congress Member. Scrape letters from constituents to congress members and use NLP to detect mentions of financial regulation. The paper is identification + text analysis; the data only exists because someone wrote a careful scraper.
- Akcigit, Baslandze & Lotti (2023) — Connecting to Power. Combine patent data with political-connection records. The political-connection panel is hand-built from a mix of scraped and bulk-downloaded sources.
- Babenko, Fedaseyeu & Zhang (2020) — Do CEOs Affect Employees’ Political Choices? Match employee donor records (FEC) to firm rosters. The matching dictionary is built by scraping public profiles.
- Akey, Heimer & Lewellen (2021) — Politicising Consumer Credit. Build a panel of FOMC and political appointments by scraping the relevant agency websites.
Notice the pattern: the scrape produces a small, well-documented dataset that is then merged with a standard panel. The paper’s contribution is the merge and the identification — not the scrape — but without the scrape there is no paper.
Time to code
source("code/00_utils.R")
load_finance4_packages()
REFRESH_CACHE <- FALSE # set to TRUE if you want a fresh scrape
ensure_dir("data/raw_wikipedia")
ensure_dir("data/raw_wikipedia/banks")The full script is code/06_ta5_webscraping.R. The remainder of this section walks through every result the script produces, in the same order it produces them.
Step 1: Robots.txt and the polite scraper
Before we scrape anything, we fetch and cache /robots.txt. The Wikipedia file is long, so we extract just the directives that matter — User-agent, Disallow, Allow, Crawl-delay. The full log is at output/text/ta5_ethics_log.txt.
TA5 polite-scraping log
Built on: 2026-05-06 12:05:23 CEST
User-Agent sent: Finance4-TA5-educational-scraper/1.0 (Bocconi PhD teaching assistant; alejandra.inzunza@phd.unibocconi.it)
Robots.txt cached at: data/raw_wikipedia/robots.txt
First robots.txt directives we matched:
User-agent: MJ12bot
Disallow: /
User-agent: Mediapartners-Google*
Disallow: /
User-agent: IsraBot
Disallow:
User-agent: Orthogaffe
Two things to notice:
- The header records the exact User-Agent we sent. This is good practice for reproducibility and accountability — anyone reading the log later knows who scraped what and how to contact us.
- Wikipedia’s
robots.txtis permissive for normal crawlers but explicitly blocks several aggressive bots (MJ12bot,WebCopier,WebStripper, …). Reading the file yourself before you scrape is part of the contract.
Step 2: Scrape the table of largest US banks
The page contains several <table class="wikitable"> blocks. We extract them all with rvest::html_table(fill = TRUE), keep the first one, and clean the column names. Because Wikipedia tables drift over time (column headings change wording, footnote markers leak in), we normalise headers and parse numeric values defensively.
This is exactly the kind of dataset that is awkward to find pre-packaged but cheap to scrape: 15 rows, four columns, no JavaScript, no login. The total assets in US-dollar billions are the same numbers you would pay $40 for on a “list of largest banks” PDF aggregator.
Step 3: Follow links to each bank’s article
Each row in the wikitable contains a <a href="…"> link to the bank’s individual Wikipedia article. We walk the <tr> elements of the same table, extract the first <a> from the bank-name cell, and build absolute URLs. We then loop over the URLs with purrr::map2(), calling fetch_or_cache() on each one — this is where the polite-scraping helpers from code/00_utils.R earn their keep.
| Rank | Bank | Headquarters | URL |
|---|---|---|---|
| 1 | JPMorgan Chase | New York City | JPMorgan_Chase |
| 2 | Bank of America | Charlotte, North Carolina | Bank_of_America |
| 3 | Citigroup | New York City | Citigroup |
| 4 | Wells Fargo | San Francisco, California | Wells_Fargo |
| 5 | Goldman Sachs | New York City | Goldman_Sachs |
| 6 | Morgan Stanley | New York City | Morgan_Stanley |
| 7 | U.S. Bancorp | Minneapolis, Minnesota | U.S._Bancorp |
| 8 | Capital One | Tysons, Virginia | Capital_One |
| 9 | PNC Financial Services | Pittsburgh, Pennsylvania | PNC_Financial_Services |
| 10 | Truist Financial | Charlotte, North Carolina | Truist |
| 11 | TD Bank, N.A. | Cherry Hill, New Jersey[a] | TD_Bank_(United_States) |
| 12 | Charles Schwab Corporation | Westlake, Texas | Charles_Schwab_Corporation |
| 13 | The Bank of New York Mellon | New York City | BNY |
| 14 | State Street Corporation | Boston, Massachusetts | State_Street_Corporation |
| 15 | American Express | New York City | American_Express |
Once the article HTML is in hand, we extract the first three substantial paragraphs of the article body — that is, the lead text. We deliberately drop very short paragraphs (less than 80 characters) to avoid section headers and stub fragments. The result is a small, coherent corpus.
| Rank | Bank | Excerpt |
|---|---|---|
| 1 | JPMorgan Chase | JPMorgan Chase & Co. (stylized as JPMorganChase) is an American multinational banking institution headquartered in New York City and incorporated in Delaware. It is the largest bank in the United States, and the world’s largest bank by market capitalization as of 2026.[3][4] As the largest of the Big Four banks in America, the firm is considered systemically important by the Financial Stability Board. Its size and scale have often led to enhanced regulatory oversight as well as the development of an internal “Fortress Balance Sheet”.[5][6] The firm has had its global headquarters on 270 Park A… |
| 2 | Bank of America | The Bank of America Corporation (Bank of America; often abbreviated BAC or BofA) is an American multinational investment bank and financial services holding company headquartered at the Bank of America Corporate Center in Charlotte, North Carolina, with investment banking and auxiliary headquarters in Manhattan. The bank was founded by the merger of NationsBank and Bank of America in 1998. It is the second-largest banking institution in the United States and the second-largest bank in the world by market capitalization, both after JPMorgan Chase. Bank of America is one of the Big Four banking … |
| 3 | Citigroup | Citigroup Inc. or Citi (stylized as citi) is an American multinational investment bank and financial services company based in New York City. The company was formed in 1998 by the merger of Citicorp, the bank holding company for Citibank, and Travelers; Travelers was spun off from the company in 2002.[2][3] Citigroup is the third-largest banking institution in the United States by assets; alongside JPMorgan Chase, Bank of America, and Wells Fargo, it is one of the Big Four banking institutions of the United States.[4] It is considered a systemically important bank by the Financial Stability Bo… |
| 4 | Wells Fargo | Wells Fargo & Company is an American multinational financial services company.[5][8] The company operates in 35 countries and serves more than 70 million customers worldwide.[5] It is a systemically important financial institution according to the Financial Stability Board, and is considered one of the “Big Four Banks” in the United States, alongside JPMorgan Chase, Bank of America, and Citigroup.[9] The company’s primary subsidiary is Wells Fargo Bank, N.A., a national bank that designates its Sioux Falls, South Dakota, site as its main office (and therefore is treated by most U.S. federal co… |
| 5 | Goldman Sachs | The Goldman Sachs Group, Inc. (/sæks/ SAKS) is an American multinational investment bank and financial services company. Founded in 1869, Goldman Sachs is headquartered in the Battery Park City neighborhood of Manhattan in New York City, with regional offices in many international financial centers.[1] Goldman Sachs is one of the largest investment banks in the world by revenue[2] and is ranked 32nd on the Fortune 500 list of the largest United States corporations by total revenue.[3] In the Forbes Global 2000 of 2025, Goldman Sachs ranked 20th.[4] It is considered a systemically important fin… |
| 6 | Morgan Stanley | Morgan Stanley[4] is an American multinational investment bank and financial services company headquartered at 1585 Broadway in Midtown Manhattan, New York City. With offices in 42 countries and more than 80,000 employees, the firm’s clients include corporations, governments, institutions, and individuals.[2][4] Morgan Stanley ranked No. 61 in the 2023 Fortune 500 list of the largest United States corporations by total revenue[5] and in the same year ranked No. 30 in the Forbes Global 2000.[6] The original Morgan Stanley, formed by J.P. Morgan & Co. partners Henry Sturgis Morgan (a grandson of… |
| 7 | U.S. Bancorp | U.S. Bancorp (stylized as us bancorp) is an American multinational banking institution headquartered in Minneapolis, Minnesota and incorporated in Delaware. It is the 5th-largest bank in the United States as of 2025. As the largest bank in the Midwestern United States, it is considered systemically important by the Financial Stability Board. It is the parent company of its primary operating entity, U.S. Bank National Association, which does business as U.S. Bank (stylized as us bank). The company provides banking, investment, mortgage, trust, and payment services products to individuals, busin… |
| 8 | Capital One | Capital One Financial Corporation is an American bank holding company headquartered in Tysons, Virginia, with operations in the United States, Canada, and the United Kingdom.[1] One of the largest banks in the United States, it is the largest issuer of credit cards in the United States, and is one of the largest car finance companies in the U.S. It owns the Discover Card, Diners Club, and Pulse payment networks.[1] The company’s three business segments are credit cards, consumer banking, and commercial banking.[1] It has approximately 750 bank branches, including over 60 café style locations,[… |
| 9 | PNC Financial Services | The PNC Financial Services Group, Inc. is an American bank holding company and financial services corporation based in Pittsburgh, Pennsylvania. Its banking subsidiary, PNC Bank, operates in 27 states and the District of Columbia, with 2,629 branches and 9,523 ATMs. PNC Bank is one of the largest banks in the US by assets and by number of branches, deposits, and ATMs. The company also provides financial services such as asset management, wealth management, estate planning, loan servicing, and information processing. PNC is one of the largest Small Business Administration lenders and one of the… |
| 10 | Truist Financial | Truist Financial Corporation (/ˈtruːɪst/[6]) is an American bank holding company headquartered in Charlotte, North Carolina.[7] The company was formed in December 2019 as the result of the merger of BB&T (Branch Banking and Trust Company) and SunTrust.[1][8][9] Its bank operates 1,928 branches in 15 states and Washington, D.C., offering consumer, commercial, and investment banking, securities brokerage, asset management, mortgage, and insurance products and services.[3] It is the tenth-largest bank in the United States, with $523 billion in assets.[10][11] In 1872, Alpheus Branch and Thomas Je… |
| 11 | TD Bank, N.A. | TD Bank, N.A. (previously TD Banknorth) is an American national bank and the United States subsidiary of the Canadian multinational TD Bank Group. It operates primarily across the East Coast, in 15 U.S. states and Washington, D.C. TD Bank is the fifteenth-largest U.S. bank by deposits and the 15th largest bank in the United States by total assets, resulting from a series of several mergers and acquisitions. TD Bank, N.A. is headquartered in Cherry Hill, New Jersey, an inner suburb eight miles (13 km) outside Philadelphia. TD Bank is a federally chartered bank, thus its trading name bears “N.A…. |
| 12 | Charles Schwab Corporation | The Charles Schwab Corporation[1] is an American multinational financial services company. It offers banking, commercial banking, investing, and related services including consulting, and wealth management advisory services to both retail and institutional clients. One of the largest banks in the United States by assets,[1] as of December 31, 2024,[update] it had $10.10 trillion in client assets, 36.5 million active brokerage accounts, 5.4 million workplace retirement plan participant accounts, and 2.0 million banking accounts.[1] It also offers a donor advised fund for clients seeking to dona… |
| 13 | The Bank of New York Mellon | The Bank of New York Mellon Corporation, commonly known as BNY or by its prior brand name BNY Mellon, is an American international financial services company headquartered in New York City with major offices also in Pittsburgh, Pennsylvania. It was established in its current form in July 2007 by the merger of the Bank of New York and Mellon Financial Corporation. Through the lineage of Bank of New York, which was founded in 1784 by a group that included Alexander Hamilton, BNY is regarded as one of the three oldest banks in the United States and among the oldest in the world. It was the first … |
| 14 | State Street Corporation | State Street Corporation is an American multinational[2]financial services and bank holding company headquartered at One Congress Street in Boston. It is the second-oldest continuously operating U.S. bank, tracing its roots to Union Bank, chartered in 1792. As of the third quarter of 2025, State Street is one of the world’s largest asset managers and custodians, with approximately US$5.4 trillion in assets under management and US$51.7 trillion under custody and administration.[3] State Street operates globally through three main divisions: Global Services (custody and fund administration), Glo… |
| 15 | American Express | American Express Company (Amex) is an American bank holding company and multinational financial services corporation that specializes in payment cards. It is headquartered at 200 Vesey Street, also known as American Express Tower, in the Battery Park City neighborhood of Lower Manhattan. Amex is the fourth-largest card network globally based on purchase volume, behind China UnionPay, Visa, and Mastercard.[7][8] 141.2 million Amex cards were in force worldwide as of December 31, 2023, with an average annual spend per card member of US$24,059. That year, Amex handled over $1.7 trillion in purcha… |
Workflow rule. Each individual article is fetched at most once: the very first run downloads it, every subsequent run reads from data/raw_wikipedia/banks/<slug>.html. The cache is part of the dataset, and a serious project commits the cached HTML to the repository (or to a versioned data store) alongside the code.
Step 4: When static scraping is not enough
Stop-and-think. The script writes out a small chromote snippet to output/text/ta5_chromote_snippet.R that you can lift into your own work later:
# Pseudocode: scraping a JavaScript-rendered page with chromote
library(chromote)
b <- ChromoteSession$new()
b$Page$navigate("https://example.com/dashboard")
b$Page$loadEventFired()
rendered <- b$DOM$getDocument() |>
purrr::pluck("root", "nodeId") |>
(\(id) b$DOM$getOuterHTML(nodeId = id))() |>
purrr::pluck("outerHTML")
b$close()
parsed <- xml2::read_html(rendered)The point of this snippet is not the lines of code, it is the mental flowchart:
- Open the target page in a normal browser.
- View the page source. Search for the value you can see on the page.
- If the value is in the source,
rvestis enough. Stop. - If the value is not in the source, the page is JavaScript-rendered. Use
chromote.
We do not run chromote in the TA5 pipeline because installing a Chromium binary on every student’s laptop is a different kind of fight. The snippet is a reference, not a hidden dependency.
Step 5: Tokenisation and term frequencies
Now we turn the lead_text column into one row per word with tidytext::unnest_tokens(), drop standard English stopwords, and remove a small custom list of words that would dominate a corpus made of bank Wikipedia pages (bank, banks, company, incorporated, headquartered, american, …). Customising stopwords is a normal step in finance text analysis: the standard lists are tuned for general English, not for sector-specific corpora.
| # banks | # tokens | # unique words |
|---|---|---|
| 15 | 1750 | 752 |
The most frequent terms across the whole corpus give a quick sanity check on the scrape: if financial, services, investment, management, … dominate, the corpus is doing what we expect. If the top words were references, external, links, that would tell us we accidentally scraped the bibliography sections.
| Word | Frequency |
|---|---|
| financial | 42 |
| services | 37 |
| investment | 24 |
| management | 24 |
| assets | 18 |
| america | 16 |
| ranked | 15 |
| morgan | 14 |
| card | 13 |
| founded | 13 |
| market | 13 |
| wealth | 12 |
| amex | 11 |
| asset | 11 |
| branch | 11 |
| branches | 11 |
| business | 11 |
| commercial | 11 |
| global | 11 |
| multinational | 11 |
| bny | 10 |
| chase | 10 |
| considered | 10 |
| credit | 10 |
| fortune | 10 |
We also save the top three words per bank to output/tables/ta5_top_terms_per_bank.csv. This is a quick eyeball check that each bank’s article is parsed correctly: if all 15 banks share exactly the same top words, the lead-paragraph extraction has gone wrong somewhere.
Step 6: A simple finance tone score
A standard next step in finance text analysis is to score a document with a financial sentiment dictionary — Loughran & McDonald (2011) is the canonical choice. We use a deliberately small, transparent stand-in dictionary defined in code/00_utils.R (15 positive words, 23 negative ones) so the mechanics are visible at a glance. The score per bank is
\[ \text{tone}_b \;=\; \frac{(\text{positive matches})_b - (\text{negative matches})_b}{(\text{total tokens})_b}. \]
| Rank | Bank | Total tokens | Positive | Negative | Tone |
|---|---|---|---|---|---|
| 1 | JPMorgan Chase | 142 | 3 | 1 | 0.0141 |
| 2 | Bank of America | 182 | 7 | 1 | 0.0330 |
| 3 | Citigroup | 75 | 3 | 0 | 0.0400 |
| 4 | Wells Fargo | 100 | 1 | 1 | 0.0000 |
| 5 | Goldman Sachs | 154 | 3 | 0 | 0.0195 |
| 6 | Morgan Stanley | 106 | 3 | 0 | 0.0283 |
| 7 | U.S. Bancorp | 106 | 2 | 0 | 0.0189 |
| 8 | Capital One | 99 | 0 | 1 | -0.0101 |
| 9 | PNC Financial Services | 102 | 0 | 0 | 0.0000 |
| 10 | Truist Financial | 113 | 1 | 1 | 0.0000 |
| 11 | TD Bank, N.A. | 114 | 0 | 3 | -0.0263 |
| 12 | Charles Schwab Corporation | 137 | 4 | 0 | 0.0292 |
| 13 | The Bank of New York Mellon | 128 | 0 | 0 | 0.0000 |
| 14 | State Street Corporation | 70 | 0 | 0 | 0.0000 |
| 15 | American Express | 122 | 0 | 0 | 0.0000 |
A few things:
- The numbers are small — most leads contain only one or two dictionary matches. That is expected from a 600-character excerpt; the absolute score is mainly a function of how the article begins.
- The cross-sectional ordering is not random. Banks whose lead emphasises investment, management, successful, innovative score higher; banks whose lead mentions fines, settlements, losses, risk score lower.
- Real research scales this up. A typical Loughran-McDonald implementation uses a dictionary of several thousand words and applies it to entire 10-K filings rather than 600-character excerpts.
Step 7: LaTeX exports
The script also exports two LaTeX tables — the bank list and the tone scores — to output/tables/ta5_top_banks.tex and output/tables/ta5_tone_scores.tex, ready to drop into a draft with \input{...}. Together with the cached HTML in data/raw_wikipedia/, this is the full reproducible bundle you would attach to a referee report.
Question for you
Suppose you want to extend this exercise to the largest 100 US banks rather than just the top 15.
- Which step (or steps) of the script would you change, and why?
- What would you do to keep the scrape polite — both legally and operationally — at that scale?
- If the lead paragraphs of the smaller banks turn out to be only one or two sentences, which step of the pipeline would you redesign so that the corpus stays comparable across banks?
What to take away
A few things to internalise from this TA:
- A scraper is a research instrument and should be designed like one: identifiable, throttled, cached, documented, and obedient to the source’s licence and terms.
- Static scraping with
rvestcovers the vast majority of finance use cases. Reach forchromote(orRSelenium) only when the data is genuinely JavaScript-rendered — and check by viewing the page source first. - The cache is the dataset. If your colleague cannot reproduce your tables six months from now without the cached HTML, you have not really done reproducible research.
- Once you have the corpus, the rest is small data:
tidytextfor tokenisation, a transparent dictionary for tone, plain dplyr for everything else. The complicated step is getting the data, not analysing it.
A few questions you should be able to answer:
- Why is sending a real
User-Agentthe cheapest favour you can do for both your project and the server you are scraping? - What is the difference between
html_element()andhtml_elements(), and when does it matter? - Why would you cache scraped HTML on disk even on a fast network connection?
- What would it take, operationally and legally, to extend this scrape to a non-Wikipedia source — say, a national newspaper?
- Why is the Loughran-McDonald financial sentiment dictionary preferable to a general-purpose dictionary like AFINN or Bing for finance text?
For the formal references behind everything above:
- Loughran, T., and B. McDonald (2011), “When is a Liability not a Liability? Textual Analysis, Dictionaries, and 10-Ks”, Journal of Finance 66(1), 35–65.
- Adams, R., and T. Mosk (2023), “Dear Congress Member: Letters to Congress and the Shaping of (Financial) Legislation”, working paper, University of Oxford.
- Akcigit, U., S. Baslandze, and F. Lotti (2023), “Connecting to Power: Political Connections, Innovation, and Firm Dynamics”, Econometrica 91(2), 529–564.
- Babenko, I., V. Fedaseyeu, and S. Zhang (2020), “Do CEOs Affect Employees’ Political Choices?”, Review of Financial Studies 33(4), 1781–1817.
- Akey, P., R. Z. Heimer, and S. Lewellen (2021), “Politicizing Consumer Credit”, Journal of Financial Economics 139(2), 627–655.