VoiceBox Clone: AI Voice Studio

Build a local voice cloning & TTS studio with Node.js and Whisper

Welcome to another official UVF IT build. Today we are creating a minimal, working clone of VoiceBox, the open-source AI voice studio that lets you clone voices and generate speech locally.

We will use a free-tier stack: Node.js, Express, and Whisper (via whisper.cpp or a local API) for transcription, and a simple TTS engine for synthesis. No paid APIs required—everything runs on your machine.

Prerequisites & Simplifications

VoiceBox Clone Architecture
LOCAL RUNTIME HTTP Transcribe Synthesize Store/Load HTML UI Browser Express API Node.js Whisper Transcription TTS Engine Speech Synthesis SQLite Voice Profiles
1

Phase 1: Scaffold Project

We start by initializing a Node.js project with TypeScript, Express, and SQLite. This setup gives us a type-safe backend that can handle file uploads and database operations without external services.

We also create the necessary directories for storing uploaded audio files and generated speech outputs. This keeps our project organized and ensures the server knows where to look for media assets.

mkdir voicebox-clone && cd voicebox-clone
npm init -y
npm install express multer better-sqlite3 uuid
npm install -D typescript @types/node @types/express @types/multer @types/better-sqlite3 ts-node
mkdir -p src/routes public uploads generated
npx tsc --init
# Configure tsconfig.json for Node.js
node -e "const fs=require('fs'); const cfg=JSON.parse(fs.readFileSync('tsconfig.json','utf8')); cfg.compilerOptions={module:'commonjs',target:'es2016',esModuleInterop:true,strict:true,skipLibCheck:true,outDir:'./dist'}; fs.writeFileSync('tsconfig.json',JSON.stringify(cfg,null,2));"
# Update package.json scripts
node -e "const fs=require('fs'); const pkg=JSON.parse(fs.readFileSync('package.json','utf8')); pkg.scripts={dev:'ts-node src/index.ts',build:'tsc',start:'node dist/index.js'}; fs.writeFileSync('package.json',JSON.stringify(pkg,null,2));"
2

Phase 2: Data Model & SQLite Setup

Next, we define our data layer using SQLite via the better-sqlite3 package. This is a synchronous, fast database that is perfect for local applications.

We create a voice_profiles table to store metadata about each cloned voice, including the name, the path to the sample audio, and the transcription text. We also write a helper function to initialize the database schema and insert a sample profile for testing.

src/db.ts

import Database from 'better-sqlite3';
import path from 'path';

const dbPath = path.join(__dirname, '..', 'voicebox.db');
const db = new Database(dbPath);

// Enable WAL mode for better performance
pragma(db);

function pragma(db: Database.Database) {
  db.pragma('journal_mode = WAL');
}

// Initialize schema
export function initDb() {
  db.exec(`
    CREATE TABLE IF NOT EXISTS voice_profiles (
      id TEXT PRIMARY KEY,
      name TEXT NOT NULL,
      audio_path TEXT NOT NULL,
      transcription TEXT NOT NULL,
      created_at DATETIME DEFAULT CURRENT_TIMESTAMP
    );
  `);
  
  // Insert sample profile if table is empty
  const count = db.prepare('SELECT COUNT(*) as count FROM voice_profiles').get() as { count: number };
  if (count.count === 0) {
    db.prepare('INSERT INTO voice_profiles (id, name, audio_path, transcription) VALUES (?, ?, ?, ?)')
      .run('sample-1', 'Sample Voice', 'uploads/sample.mp3', 'This is a sample voice profile for testing.');
  }
}

export function getDb() {
  return db;
}

export function getVoiceProfiles() {
  return db.prepare('SELECT * FROM voice_profiles ORDER BY created_at DESC').all();
}

export function getVoiceProfileById(id: string) {
  return db.prepare('SELECT * FROM voice_profiles WHERE id = ?').get(id);
}

export function insertVoiceProfile(id: string, name: string, audioPath: string, transcription: string) {
  db.prepare('INSERT INTO voice_profiles (id, name, audio_path, transcription) VALUES (?, ?, ?, ?)')
    .run(id, name, audioPath, transcription);
}
3

Phase 3: Core Feature 1 – Voice Upload & Transcription

Now we build the core upload functionality. We create an Express route that accepts audio files using multer for multipart form data handling.

Upon receiving an audio file, we simulate the Whisper transcription process. In a full implementation, you would call a local whisper.cpp binary or API. For this clone, we generate a mock transcription to demonstrate the flow and store the result in our SQLite database.

src/routes/voice.ts

import { Router, Request, Response } from 'express';
import multer from 'multer';
import path from 'path';
import { v4 as uuidv4 } from 'uuid';
import { insertVoiceProfile, getVoiceProfiles } from '../db';

const router = Router();

// Configure Multer for file uploads
const storage = multer.diskStorage({
  destination: (req, file, cb) => {
    cb(null, 'uploads/');
  },
  filename: (req, file, cb) => {
    const uniqueName = `${uuidv4()}${path.extname(file.originalname)}`;
    cb(null, uniqueName);
  }
});

const upload = multer({ 
  storage: storage,
  fileFilter: (req, file, cb) => {
    if (file.mimetype === 'audio/mpeg' || file.mimetype === 'audio/wav') {
      cb(null, true);
    } else {
      cb(new Error('Only MP3 and WAV files are allowed'), false);
    }
  }
});

// GET /api/voices - List all voice profiles
router.get('/', (req: Request, res: Response) => {
  const profiles = getVoiceProfiles();
  res.json(profiles);
});

// POST /api/voices - Upload a new voice profile
router.post('/', upload.single('audio'), (req: Request, res: Response) => {
  if (!req.file) {
    return res.status(400).json({ error: 'No audio file provided' });
  }

  const id = uuidv4();
  const name = req.body.name || 'Unnamed Voice';
  const audioPath = req.file.path;
  
  // Simulate Whisper Transcription
  // In a real app, you would call whisper.cpp here:
  // const { stdout } = await exec(`whisper-cli -m models/base.en -f ${audioPath} -oj -of output`);
  const mockTranscription = `Transcription of ${req.file.originalname} at ${new Date().toISOString()}`;

  insertVoiceProfile(id, name, audioPath, mockTranscription);

  res.status(201).json({
    id,
    name,
    audio_path: audioPath,
    transcription: mockTranscription
  });
});

export default router;
4

Phase 4: Core Feature 2 – TTS Synthesis API

Now that we can transcribe audio, let's build the reverse: turning text back into speech. We will create a /synthesize endpoint that takes a text string and a voice profile ID.

For this minimal clone, we will use espeak-ng on Linux or say on macOS as our TTS engine. In a production VoiceBox clone, you would swap this out for a neural TTS model like Coqui TTS, but for now, we want a working pipeline that proves the architecture.

src/routes/tts.ts

import { Router, Request, Response } from 'express';
import { exec } from 'child_process';
import { promisify } from 'util';
import path from 'path';
import fs from 'fs';
import { getVoiceProfile } from '../db';

const execAsync = promisify(exec);
const router = Router();

// Configuration for TTS engine
const TTS_ENGINE = process.platform === 'darwin' ? 'say' : 'espeak-ng';
const OUTPUT_DIR = path.join(process.cwd(), 'generated');

// Ensure output directory exists
if (!fs.existsSync(OUTPUT_DIR)) {
  fs.mkdirSync(OUTPUT_DIR, { recursive: true });
}

router.post('/synthesize', async (req: Request, res: Response) => {
  const { text, voiceId } = req.body;

  // Validate input
  if (!text || !voiceId) {
    return res.status(400).json({ error: 'Text and voiceId are required' });
  }

  // Fetch voice profile to ensure it exists
  const profile = getVoiceProfile(voiceId);
  if (!profile) {
    return res.status(404).json({ error: 'Voice profile not found' });
  }

  // Generate a unique filename
  const filename = `tts_${Date.now()}.wav`;
  const outputFilePath = path.join(OUTPUT_DIR, filename);

  try {
    // Sanitize text to prevent command injection
    const safeText = text.replace(/[^a-zA-Z0-9 ,.!?]/g, '');

    let command = '';
    if (process.platform === 'darwin') {
      // macOS say command
      command = `say -o "${outputFilePath}" -f /dev/stdin <<< "${safeText}"`;
    } else {
      // Linux espeak-ng command
      command = `espeak-ng -w "${outputFilePath}" "${safeText}"`;
    }

    await execAsync(command);

    // Return the audio file as a download
    res.download(outputFilePath, filename, (err) => {
      if (err) {
        console.error('Error downloading file:', err);
        res.status(500).json({ error: 'Failed to send audio file' });
      }
    });

  } catch (error) {
    console.error('TTS generation error:', error);
    res.status(500).json({ error: 'Failed to generate speech' });
  }
});

export default router;
5

Phase 5: Minimal HTML UI

With our API endpoints ready, we need a simple interface to interact with them. We will create a single index.html file that serves as the frontend for our VoiceBox clone.

This UI will have two main sections: a file upload form for voice samples and a text input area for TTS generation. We will use vanilla JavaScript with fetch to communicate with the Express server, keeping the setup lightweight and dependency-free.

public/index.html

<!DOCTYPE html>
<html lang="en">
<head>
  <meta charset="UTF-8">
  <meta name="viewport" content="width=device-width, initial-scale=1.0">
  <title>VoiceBox Clone</title>
  <style>
    body { font-family: Arial, sans-serif; max-width: 800px; margin: 0 auto; padding: 20px; }
    h1 { color: #333; }
    section { margin-bottom: 30px; padding: 20px; border: 1px solid #ddd; border-radius: 8px; }
    label { display: block; margin-bottom: 5px; font-weight: bold; }
    input[type="file"], input[type="text"], textarea { width: 100%; padding: 10px; margin-bottom: 10px; box-sizing: border-box; }
    button { background-color: #007bff; color: white; padding: 10px 15px; border: none; border-radius: 4px; cursor: pointer; }
    button:hover { background-color: #0056b3; }
    #results { margin-top: 10px; padding: 10px; background-color: #f9f9f9; border-radius: 4px; }
    audio { width: 100%; margin-top: 10px; }
  </style>
</head>
<body>
  <h1>VoiceBox Clone</h1>

  <section>
    <h2>Upload Voice Sample</h2>
    <form id="uploadForm">
      <label for="audioFile">Select Audio File (MP3/WAV):</label>
      <input type="file" id="audioFile" accept=".mp3,.wav" required>
      <button type="submit">Upload & Transcribe</button>
    </form>
    <div id="uploadResults"></div>
  </section>

  <section>
    <h2>Text-to-Speech</h2>
    <form id="ttsForm">
      <label for="voiceId">Voice Profile ID:</label>
      <input type="text" id="voiceId" placeholder="Enter voice ID" required>
      <label for="ttsText">Text to Synthesize:</label>
      <textarea id="ttsText" rows="4" placeholder="Enter text here" required></textarea>
      <button type="submit">Generate Speech</button>
    </form>
    <div id="ttsResults"></div>
  </section>

  <script>
    // Handle Voice Upload
    document.getElementById('uploadForm').addEventListener('submit', async (e) => {
      e.preventDefault();
      const fileInput = document.getElementById('audioFile');
      const resultsDiv = document.getElementById('uploadResults');
      
      if (!fileInput.files.length) return;
      
      const formData = new FormData();
      formData.append('audio', fileInput.files[0]);
      
      resultsDiv.innerHTML = 'Uploading and transcribing...';
      
      try {
        const response = await fetch('/api/voice/upload', { method: 'POST', body: formData });
        const data = await response.json();
        
        if (response.ok) {
          resultsDiv.innerHTML = `
            <strong>Transcription:</strong> ${data.transcription}<br>
            <strong>Voice ID:</strong> ${data.voiceId}
          `;
        } else {
          resultsDiv.innerHTML = `Error: ${data.error}`;
        }
      } catch (error) {
        resultsDiv.innerHTML = `Upload failed: ${error.message}`;
      }
    });

    // Handle TTS Generation
    document.getElementById('ttsForm').addEventListener('submit', async (e) => {
      e.preventDefault();
      const voiceId = document.getElementById('voiceId').value;
      const text = document.getElementById('ttsText').value;
      const resultsDiv = document.getElementById('ttsResults');
      
      resultsDiv.innerHTML = 'Generating speech...';
      
      try {
        const response = await fetch('/api/tts/synthesize', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ text, voiceId })
        });
        
        if (response.ok) {
          const blob = await response.blob();
          const url = URL.createObjectURL(blob);
          resultsDiv.innerHTML = `<audio controls src="${url}"></audio>`;
        } else {
          const data = await response.json();
          resultsDiv.innerHTML = `Error: ${data.error}`;
        }
      } catch (error) {
        resultsDiv.innerHTML = `TTS failed: ${error.message}`;
      }
    });
  </script>
</body>
</html>
6

Phase 6: Run Locally & Verify

Finally, let's start our server and test the full workflow. We will use ts-node-dev to automatically restart the server when we make changes, making development smoother.

Once the server is running, open your browser to http://localhost:3000. You should see the minimal UI. Upload a short audio clip to test transcription, then use the TTS section to generate speech from text.

npm run dev

Verifying the Stack

CheckCommand / Why it matters
Server starts without errorsRun npm run dev and confirm http://localhost:3000 loads the HTML UI.
Voice upload worksUpload a 10-second MP3 via the UI; verify the transcription text appears in the response.
TTS synthesis worksEnter text in the UI, click Generate, and confirm an audio file is downloaded and playable.
SQLite persistence worksRestart the server and verify the uploaded voice profile is still listed in the UI.

Next Steps

Extend this clone by adding real-time voice cloning with a neural TTS model (e.g., Coqui TTS), implementing a full Electron GUI, or adding voice profile management features. Explore the VoiceBox GitHub repo for advanced features like agent integration and multi-voice mixing.