UVF IT: Agent‑Browser Automation CLI Clone in Rust

Welcome to another official UVF IT guide! In this build you’ll create a tiny Rust command‑line tool that mimics the core of Vercel’s agent‑browser: open a webpage, dump a simple text snapshot, and simulate a click on a link.

We’ll use only free, open‑source tools – Rust, the tinyheadless Chrome driver chromiumoxide, and SQLite for optional state – so you can finish the prototype in about 45 minutes.

Before you start

You need Rust 1.70+ and cargo installed; the guide assumes a Unix‑like shell.
The real agent‑browser uses a full accessibility tree and a persistent Chrome instance – our clone only extracts visible text and clicks the first link matching a ref, which is a simplification.
On Linux you may need to install libssl-dev and pkg-config for the Chromium driver to compile.

Minimal Agent‑Browser Architecture

Phase 1: Scaffold the Rust project

Create a new binary crate and add the needed dependencies. This gives us a clean cargo project and pulls in chromiumoxide for driving headless Chrome, tokio for async runtime, and serde_json for printing the snapshot.

All commands are one‑liners you can paste into a terminal; they will set up the directory structure and update Cargo.toml automatically.

cargo new --bin agent-browser-clone && cd agent-browser-clone
cargo add chromiumoxide tokio serde_json --features tokio/runtime

Phase 2: Core – open a page and take a simple snapshot

We launch a headless Chrome instance, navigate to the URL supplied on the command line, and collect every visible text node. Each piece of text gets a generated reference like @e1 so later steps could address it.

The collected refs and their text are emitted as JSON, making it trivial to pipe into other tools or inspect manually.

src/main.rs

use chromiumoxide::browser::{Browser, BrowserConfig};
use chromiumoxide::cdp::browser_protocol::page::NavigateParams;
use chromiumoxide::error::Result;
use serde_json::json;
use std::env;

#[tokio::main]
async fn main() -> Result<()> {
    // Grab the target URL from the first CLI argument.
    let url = env::args().nth(1).expect("Usage: agent-browser-clone <URL>");

    // Start a headless Chrome with default config.
    let (browser, mut handler) = Browser::launch(BrowserConfig::default()).await?;
    let page = browser.new_page("about:blank").await?;
    page.navigate(NavigateParams::new(url)).await?.await?;

    // Evaluate JavaScript to return all visible text nodes.
    let script = r#"Array.from(document.querySelectorAll('body *'))
        .filter(el => el.offsetParent !== null && el.innerText.trim() !== '')
        .map((el,i)=>({ref:`@e${i+1}`, text:el.innerText.trim()}));"#;
    let result = page.evaluate(script, false).await?.value();
    let items: Vec<serde_json::Value> = serde_json::from_str(&result.to_string())?;

    // Print a tidy JSON snapshot.
    let snapshot = json!({"refs": items});
    println!("{}", serde_json::to_string_pretty(&snapshot)?);
    // Clean shutdown.
    handler.wait().await;
    Ok(())
}

Phase 3: Core – click an element by ref

We now add the ability to click a page element that we previously identified by its @eN reference. The CLI will read the snapshot JSON produced by the open command, find the element whose ref matches the argument, and then inject a tiny JavaScript snippet that triggers a native click event. This keeps the tool simple—no full DOM traversal library is needed, just a JSON lookup and a Chrome DevTools Protocol (CDP) Runtime.evaluate call.

src/main.rs

use clap::{Parser, Subcommand};
use serde::{Deserialize, Serialize};
use std::fs;

#[derive(Parser)]
#[command(name = "agent-browser-clone")]
struct Cli {
    #[command(subcommand)]
    command: Commands,
}

#[derive(Subcommand)]
enum Commands {
    Open { url: String },
    Click { ref_id: String },
}

#[derive(Serialize, Deserialize, Debug)]
struct Element {
    ref_id: String,
    selector: String,
    text: String,
}

async fn click_element(ref_id: &str) -> Result<(), Box<dyn std::error::Error>> {
    // Load the latest snapshot (saved by `open`)
    let data = fs::read_to_string("snapshot.json")?;
    let elems: Vec<Element> = serde_json::from_str(&data)?;
    let elem = elems.iter().find(|e| e.ref_id == ref_id)
        .ok_or("Ref not found in snapshot")?;
    // Connect to Chrome via CDP (assumes a running headless instance on ws://localhost:9222)
    let (mut client, _) = headless_chrome::browser::default().await?.connect().await?;
    let js = format!("document.querySelector('{}').click();", elem.selector);
    client.evaluate(&js, false).await?;
    println!("Clicked {}", ref_id);
    Ok(())
}

#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
    let cli = Cli::parse();
    match cli.command {
        Commands::Open { url } => {
            // existing open logic (omitted for brevity)
            println!("Opening {}...", url);
        }
        Commands::Click { ref_id } => {
            click_element(&ref_id).await?;
        }
    }
    Ok(())
}

Phase 4: Run locally and test

With the click feature in place we can compile the binary and give it a quick spin. The open command will generate snapshot.json, and click will use that file to simulate a user interaction. Running against a simple page like example.com lets us verify the end‑to‑end flow without any external dependencies.

# Build the release binary
cargo build --release

# Open a page and capture a snapshot (creates snapshot.json)
./target/release/agent-browser-clone open https://example.com

# Assume the snapshot listed an element with ref "@e1"
# Click that element
./target/release/agent-browser-clone click @e1

Phase 5: Verify and extend

After the click we should see the page URL or content change, confirming the JavaScript actually ran. A fast way to verify is to run open again and compare the new snapshot to the old one. For a more robust clone you can store each element's ref in a tiny SQLite database, enabling you to recall refs across separate CLI invocations.

# Re‑open the page to see the new state
./target/release/agent-browser-clone open https://example.com > new_snapshot.json

# Simple diff to confirm something changed (e.g., URL or text)
diff snapshot.json new_snapshot.json || echo "Page state changed after click"

# Optional: quick SQLite insert (requires sqlite3 installed)
sqlite3 refs.db "INSERT INTO elements(ref, selector) VALUES('@e1', 'a');"

Verifying the Stack

Check	Command / Why it matters
Project builds without errors	cargo build --release returns exit code 0
`open` prints a JSON array of elements with @eN refs	./target/release/agent-browser-clone open https://example.com shows JSON
`click` reports "Clicked @eX" and subsequent `open` shows changed page content	Run click then open again; observe URL/text change

What’s next?

You now have a functional Rust CLI that mimics the core agent‑browser workflow. Add persistent Chrome sessions, richer accessibility data, or a tiny SQLite cache to store refs across commands for a more complete clone.