UVF IT: Simple Web Scraper Clone with Python

Welcome to another official UVF IT build guide! In this tutorial we’ll create a tiny web‑scraping app that mimics the core idea of Scrapling – fetch a page, parse HTML, and output structured data – using only Python’s standard library and the free requests and beautifulsoup4 packages.

We’ll keep the stack ultra‑light: a single Python script, no database, no external paid APIs. In ~45 minutes you’ll have a working CLI tool that you can extend to your own projects.

Before you start

You need Python 3.9+ and pip installed – older versions may miss typing features used in the example.
The clone only handles static HTML; JavaScript‑rendered sites require a headless browser, which is out of scope here.
Network errors are not fully handled – the script will raise an exception if the target URL is unreachable.

Minimal Scraper Architecture

Phase 1: Scaffold the project

Create a new folder and initialise a virtual environment – this isolates dependencies for the beginner-friendly setup.

mkdir simple_scraper && cd simple_scraper
python3 -m venv venv
source venv/bin/activate
pip install --upgrade pip
pip install requests beautifulsoup4

Phase 2: Core – fetch and parse HTML

Write a tiny function that downloads a page and extracts the title and all links using BeautifulSoup.

scraper.py

import argparse
import requests
from bs4 import BeautifulSoup
from typing import List, Tuple

def fetch_and_parse(url: str) -> Tuple[str, List[str]]:
    """Download *url* and return the page title and a list of hrefs.
    Raises ``requests.RequestException`` on network failure.
    """
    resp = requests.get(url, timeout=10)
    resp.raise_for_status()
    soup = BeautifulSoup(resp.text, "html.parser")
    title = soup.title.string.strip() if soup.title else "(no title)"
    links = [a.get('href') for a in soup.find_all('a', href=True)]
    return title, links

def main():
    parser = argparse.ArgumentParser(description="Minimal web scraper clone")
    parser.add_argument('url', help='The page to fetch')
    args = parser.parse_args()
    title, links = fetch_and_parse(args.url)
    print(f"Page title: {title}\nLinks found ({len(links)}):")
    for link in links:
        print(f" - {link}")

if __name__ == "__main__":
    main()

Phase 3: Simple CLI wrapper

In this step we give our scraper a command‑line interface so it can be called from a terminal or other tools. The script will accept a URL argument, fetch the page, extract all anchor hrefs, and emit a JSON array. Keeping the output pure JSON makes it easy to pipe into jq, log files, or downstream programs. We'll also add basic error handling so the user gets a helpful message if something goes wrong.

scraper.py

import sys, json, requests
from bs4 import BeautifulSoup

def fetch_links(url):
    try:
        resp = requests.get(url, timeout=10)
        resp.raise_for_status()
    except Exception as e:
        sys.stderr.write(f'Error fetching {url}: {e}\n')
        sys.exit(1)
    soup = BeautifulSoup(resp.text, 'html.parser')
    links = [a.get('href') for a in soup.find_all('a', href=True)]
    return links

if __name__ == '__main__':
    if len(sys.argv) != 2:
        sys.stderr.write('Usage: python scraper.py <URL>\n')
        sys.exit(1)
    url = sys.argv[1]
    result = {'url': url, 'links': fetch_links(url)}
    print(json.dumps(result, ensure_ascii=False, indent=2))

Phase 4: Run locally

Now that the CLI is ready, activate the virtual environment you created earlier and run the script against a real page. We'll use Python's official homepage as a simple test because it contains a handful of links and loads quickly. Watching the JSON printed to the console confirms that our scraper is pulling data correctly.

source venv/bin/activate
python scraper.py https://www.python.org

Phase 5: Verify & extend

Inspect the printed JSON; you should see the requested URL and an array of href strings. If the output looks good, try redirecting it to a file for later analysis or further processing. As a quick extension, you can filter the list to only external links (those not starting with the original domain) or write the results to a JSON file using a simple shell redirection.

# Save raw output to a file
python scraper.py https://www.python.org > python_links.json
# Show only external links using jq (install jq if needed)
python scraper.py https://www.python.org | jq '.links[] | select(test("^https?://") and (test("python.org") | not))'

Verifying the Stack

Check	Command / Why it matters
Virtual environment activated	source venv/bin/activate && echo $VIRTUAL_ENV
`requests` and `beautifulsoup4` installed	pip list \| grep -E 'requests\|beautifulsoup4'
Script returns JSON	python scraper.py https://example.com \| python -m json.tool

What’s next?

You now have a functional clone of Scrapling’s core fetch‑and‑parse feature. Experiment by adding CSS selectors, pagination loops, or storing results in SQLite for a richer mini‑scraper.