UVF IT: Simple Website‑to‑API Service with FastAPI

Welcome to another official UVF IT guide. In the next 45 minutes you’ll build a tiny service that fetches a web page, extracts its title and body text, and returns the data as JSON – just like Pomerado does, but with a free‑tier stack.

We’ll use Python, FastAPI, and Playwright (headless Chromium) with an in‑memory SQLite cache for repeated requests. No paid APIs, no cloud services – everything runs locally.

Before you start…

You need Python 3.9+ and Node 16+ (Playwright installs a bundled Chromium).
The clone only returns the page’s <title> and visible text – no CSS selectors, authentication, or rate‑limiting that the real product provides.
Running headless browsers can be heavy; on low‑RAM machines increase the timeout or use --disable-gpu.

Architecture Overview

Phase 1: Scaffold the project

Create a fresh virtual environment so our dependencies stay isolated. Then install FastAPI for the web framework, Uvicorn as the ASGI server, Playwright for headless browsing, and a tiny SQLite wrapper to handle our cache. Finally, let Playwright download the bundled Chromium binary.

All commands are runnable from your project root. After this step you’ll have a ready‑to‑code Python environment and a working browser ready for automation.

python3 -m venv .venv
source .venv/bin/activate
pip install --upgrade pip
pip install fastapi uvicorn playwright aiosqlite
playwright install chromium

Phase 2: Data model & cache layer

We need a lightweight way to remember results for URLs we’ve already scraped. A single SQLite table mapping a URL string to a JSON blob does the trick, and a tiny helper module will hide the SQL details behind async functions.

The cache helper will be imported by the FastAPI route later, allowing us to check the cache first and fall back to Playwright only when needed.

cache.py

import aiosqlite
import json
from pathlib import Path

DB_PATH = Path('cache.db')

async def init_db():
    async with aiosqlite.connect(DB_PATH) as db:
        await db.execute(
            """CREATE TABLE IF NOT EXISTS page_cache (
                url TEXT PRIMARY KEY,
                response TEXT NOT NULL,
                fetched_at REAL NOT NULL
            )"""
        )
        await db.commit()

async def get_cached(url: str):
    async with aiosqlite.connect(DB_PATH) as db:
        async with db.execute('SELECT response FROM page_cache WHERE url = ?', (url,)) as cur:
            row = await cur.fetchone()
            return json.loads(row[0]) if row else None

async def set_cached(url: str, data: dict, timestamp: float):
    async with aiosqlite.connect(DB_PATH) as db:
        await db.execute(
            'INSERT OR REPLACE INTO page_cache (url, response, fetched_at) VALUES (?, ?, ?)',
            (url, json.dumps(data), timestamp)
        )
        await db.commit()

Phase 3: Core fetch API

In this step we create the heart of our clone: an endpoint that receives a URL, spins up Playwright, grabs the page title and all visible text, and then returns a tidy JSON payload. We also add an in‑memory cache so repeated calls for the same URL are lightning fast, avoiding the heavy browser startup each time. The cache lives for the lifetime of the process – perfect for a minimal demo.

We use FastAPI for routing and Pydantic models for a clean response schema. Playwright runs headlessly, extracts the <title> tag and the innerText of the <body> element, then we strip excess whitespace. If the URL is already cached we skip Playwright entirely and serve the stored result.

main.py

import asyncio
from fastapi import FastAPI, HTTPException, Query
from pydantic import BaseModel
from typing import Dict
from playwright.async_api import async_playwright

app = FastAPI()

# Simple in‑memory cache: url -> result
cache: Dict[str, dict] = {}

class FetchResult(BaseModel):
    url: str
    title: str
    text: str

async def scrape(url: str) -> dict:
    try:
        async with async_playwright() as p:
            browser = await p.chromium.launch(headless=True)
            page = await browser.new_page()
            await page.goto(url, wait_until="domcontentloaded", timeout=15000)
            title = await page.title()
            # Grab visible text from body
            body_handle = await page.query_selector("body")
            raw_text = await body_handle.inner_text() if body_handle else ""
            await browser.close()
            # Collapse whitespace
            clean_text = " ".join(raw_text.split())
            return {"title": title, "text": clean_text}
    except Exception as e:
        raise HTTPException(status_code=400, detail=str(e))

@app.get("/api/fetch", response_model=FetchResult)
async def fetch(url: str = Query(..., description="Full URL to fetch")):
    if url in cache:
        result = cache[url]
    else:
        result = await scrape(url)
        cache[url] = result
    return {"url": url, **result}

Phase 4: Run and verify locally

Now we spin up the FastAPI server with Uvicorn. The --reload flag watches our files and restarts automatically – handy while we iterate. Once the server is listening, we can hit the new /api/fetch endpoint with curl or a browser to see the JSON output.

If everything is wired correctly you’ll receive a JSON object containing the original URL, the page title, and a cleaned block of visible text. Try a couple of different sites to confirm the cache works – the second call should return instantly.

# Install dependencies (run once)
python -m pip install fastapi uvicorn playwright
python -m playwright install chromium

# Start the server
uvicorn main:app --reload

Phase 5: Extend (optional)

A useful enhancement is a limit query parameter that caps the amount of text returned, preventing huge payloads for very long articles. We simply slice the cleaned text before caching or returning it. You could also add a tiny HTML front‑end that calls this API via fetch() and displays the result, but the code below focuses on the limit feature.

The change is tiny: we read an optional integer, apply it to the text, and store the truncated version in the cache. This keeps the example simple while showing how easy it is to evolve the service.

main.py

import asyncio
from fastapi import FastAPI, HTTPException, Query
from pydantic import BaseModel
from typing import Dict
from playwright.async_api import async_playwright

app = FastAPI()
cache: Dict[str, dict] = {}

class FetchResult(BaseModel):
    url: str
    title: str
    text: str

async def scrape(url: str) -> dict:
    async with async_playwright() as p:
        browser = await p.chromium.launch(headless=True)
        page = await browser.new_page()
        await page.goto(url, wait_until="domcontentloaded", timeout=15000)
        title = await page.title()
        body = await page.query_selector("body")
        raw = await body.inner_text() if body else ""
        await browser.close()
        clean = " ".join(raw.split())
        return {"title": title, "text": clean}

@app.get("/api/fetch", response_model=FetchResult)
async def fetch(
    url: str = Query(..., description="Full URL to fetch"),
    limit: int = Query(0, ge=0, description="Maximum number of characters to return; 0 means no limit"),
):
    if url in cache:
        result = cache[url]
    else:
        result = await scrape(url)
        cache[url] = result
    text = result["text"]
    if limit:
        text = text[:limit]
    return {"url": url, "title": result["title"], "text": text}

Verifying the Stack

Check	Command / Why it matters
Server starts without errors	uvicorn main:app --reload → should show 'Uvicorn running on http://127.0.0.1:8000'
API returns JSON for a public page	curl 'http://127.0.0.1:8000/api/fetch?url=https://example.com' → JSON with title and text
Second request hits cache (fast)	Repeat curl; response time drops dramatically (no browser launch)

You did it!

Your minimal Pomerado‑style service is up and running. Feel free to add more selectors, authentication headers, or deploy to a free tier like Render or Fly.io.