Get Data From the Web Into Your Automations · Module 2 of 10

APIs vs Scraping vs RSS: Pick the Right Way to Get Web Data

StepthroughBeginner~7 min practice17 steps

Why this matters

Every Monday, someone at a small firm copies competitor prices, industry news, CRM numbers or Reddit questions into a sheet. It takes an hour or two, it is the first job skipped in a busy week, and a slip of the mouse is a wrong number in the report.

All of that can arrive on its own. But most automations that break are not broken by the tool: the data source changed, blocked them or got expensive. Pick the source well first, and the rest of this course is just plumbing.

What you’ll build

You help Sam Rivera, who runs reporting at Copperleaf Marketing, a made-up agency. You compare the four ways web data gets into an automation, open the five parts of an HTTP request, pick the right way in for three of Sam’s Monday jobs and do the cost math on a pay-per-call API.

Every screen is a Stepthrough worksheet, not a real app. Nothing is sent. The modules after this one practice each way in real tools: Coefficient, n8n, Litescrape and ProLevelMax.

What you need

  • NothingThe practice is a worksheet in your browser.
  • One report you fill by hand, to use it for realWrite down where each number comes from. That list is your plan for the rest of the course.

Concepts

3 ideas you’ll use in the practice.

Four ways in, in order of preference

An official API is the app’s own door for software: stable, allowed and documented. An RSS feed is a free list of a site’s newest posts. A third-party data API collects data such as search results and sells it per call. Web scraping reads the page like a person, and breaks when the page changes.

Use them in that order: official API first, RSS when a site offers it, a paid data API for things like search results, scraping only as a last resort.

Five parts of every HTTP request

A method (GET reads, POST sends), a URL (the endpoint), query parameters that narrow it down, headers that say who is asking (the key, as a bearer token), and the response: a status code and JSON.

n8n’s HTTP Request node, Zapier’s webhooks step and Make’s HTTP module all use the same five parts.

Do the cost math, and expect change

Pay-per-call sounds expensive and usually is not: calls per run times runs per month times the price. At $0.003 a call, once a day is about nine cents a month. A loop over thousands of items is a different bill, so multiply before you build. See API usage billing.

Free sources can disappear: Reddit announced it is switching off its RSS feeds on November 13, 2026. Build in steps, so a source change is one node to swap.

Practice

Do every step yourself in a practice copy of the real tool. It checks your work, and “Do it for me” finishes any step you get stuck on.

StepthroughAPIs vs Scraping vs RSS: Pick the Right Way to Get Web Data17 steps · ~7 min · nothing touches your real accounts

Do it for real

Now repeat it in your own account. Tick each step as you go.

Quick check

3 questions to check you’ve got it.

1. A news site publishes a feed of its newest posts. Best way in?

  • Scrape the homepage
  • Read its RSS feed
  • Pay a data API
Show the answer

Read its RSS feed. Free, no key, made for exactly this.

2. Where does the API key go in a request?

  • In the URL
  • In a header, usually as a bearer token
  • In the body of every request
Show the answer

In a header, usually as a bearer token. Headers are where most APIs ask for it, and URLs end up in logs.

3. You need competitor prices from Google Shopping every week. Best way in?

  • Scrape Google yourself
  • A search-results API that charges per call
  • An RSS feed
Show the answer

A search-results API that charges per call. Google blocks scrapers and has no feed for this. A search API returns clean JSON and handles the blocking.

Up next · Module 3

Coefficient: Pull Live Pipeline Data into Google Sheets and Auto-Refresh It Daily

You can pick a source. Next, the easiest way in for your own app data: pull a live CRM pipeline into Google Sheets with Coefficient, no code.

Start module 3 →