UVF IT: Minimal Fastbrowse‑style Browser Agent

Welcome to another official UVF IT guide. In this tutorial you’ll spin up a tiny browser‑agent inspired by Fastbrowse, using only Node.js, Puppeteer and a local SQLite DB – all free and open‑source.

We’ll keep the feature set tiny: fetch a page, extract its title, store the result, and expose a one‑line HTTP endpoint to trigger the crawl. By the end you’ll have a working prototype you can run locally and extend as you wish.

Before you start…

Node.js >=18 and npm must be installed; older versions may lack ES‑module support used below.
The real Fastbrowse offers authentication, rate‑limiting and multi‑page crawling – our clone only grabs a single page title.
Puppeteer downloads a recent Chromium binary (~100 MB) on first install; ensure you have ~200 MB of free disk space.

Minimal Fastbrowse Clone Architecture

Phase 1: Scaffold the project

Create a fresh npm project and install Express, Puppeteer and better‑sqlite3 – all run locally. This gives us a lightweight web server, a headless browser for scraping, and a tiny embedded database to persist results. We'll keep everything in the project folder so you can run the clone with a single command.

mkdir fastbrowse-clone && cd fastbrowse-clone
npm init -y
npm install express puppeteer better-sqlite3
# Verify installation
npm list --depth=0

Phase 2: Set up SQLite model

Create a tiny DB with a single table to store URLs and their page titles. Using better‑sqlite3 lets us work synchronously, which keeps the example easy to read for beginners. The helper module will expose a function to insert a record and another to fetch all saved pages.

db.js

import Database from 'better-sqlite3';

// Open (or create) a local SQLite file named pages.db
const db = new Database('pages.db');

// Ensure the table exists – id, url, title, fetched_at
const init = `CREATE TABLE IF NOT EXISTS pages (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  url TEXT NOT NULL,
  title TEXT,
  fetched_at TEXT NOT NULL
);`;
db.exec(init);

export function savePage(url, title) {
  const stmt = db.prepare('INSERT INTO pages (url, title, fetched_at) VALUES (?, ?, datetime())');
  stmt.run(url, title);
}

export function listPages() {
  const stmt = db.prepare('SELECT id, url, title, fetched_at FROM pages ORDER BY fetched_at DESC');
  return stmt.all();
}

Phase 3: Build the crawling API

In this step we create a tiny Express server that will accept crawl requests. The POST /crawl endpoint receives a JSON body with a url, launches Puppeteer, grabs the page title, and saves the result in an in‑memory array. We also add a GET /pages route so we can inspect what has been stored. Keeping everything in memory keeps the example simple and fast to iterate on.

server.js

import express from 'express';
import puppeteer from 'puppeteer';

const app = express();
app.use(express.json());

// simple in‑memory store
const pages = [];

app.post('/crawl', async (req, res) => {
  const {url} = req.body;
  if (!url) return res.status(400).json({error: 'url required'});
  try {
    const browser = await puppeteer.launch();
    const page = await browser.newPage();
    await page.goto(url, {waitUntil: 'domcontentloaded'});
    const title = await page.title();
    await browser.close();
    const record = {url, title, timestamp: Date.now()};
    pages.push(record);
    res.json(record);
  } catch (e) {
    res.status(500).json({error: e.message});
  }
});

app.get('/pages', (req, res) => res.json(pages));

const PORT = process.env.PORT || 3000;
app.listen(PORT, () => console.log(`🚀 Server listening on http://localhost:${PORT}`));

Phase 4: Run locally

Now that the server code is ready, we just need to start it. Because we used ES modules (import syntax) we either run Node with the --experimental-modules flag or tell Node it’s a module via package.json. The simplest approach is to add "type":"module" to your package file and then launch with node server.js. The console will show a listening message when everything is good.

# add module type if not present
jq '.type = "module"' package.json > tmp.$$.json && mv tmp.$$.json package.json
# install dependencies (run once)
npm install express puppeteer
# start the server
node server.js

Phase 5: Verify & extend

With the server running, we can verify the API using curl. A POST to /crawl with a target URL should return a JSON object containing the page title. Afterwards, a GET to /pages will list all crawled entries. From here you can extend the API – add screenshot capture, store data in a real database, or build a tiny React UI that calls these endpoints.

# test the crawl endpoint (example.com)
curl -X POST http://localhost:3000/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com"}'

# view stored pages
curl http://localhost:3000/pages

Verifying the Stack

Check	Command / Why it matters
Server starts without errors	node server.js → should log listening message
POST /crawl returns JSON with title	curl -X POST http://localhost:3000/crawl -H 'Content-Type: application/json' -d '{"url":"https://example.com"}' → response contains "title":"Example Domain"
GET /pages lists the stored record	curl http://localhost:3000/pages → array with the example.com entry

What’s next?

You now have a functional browser‑agent core. Consider adding rate‑limiting, a simple React front‑end, or persisting screenshots. Happy hacking!