Build a private, full-text search index for your local files in 45 minutes.
Welcome to another official UVF IT build guide. Today we are constructing a minimal, self-hosted search engine inspired by Hister, focusing on privacy and full-text indexing of local documents.
You will build a Python-based service that indexes text files and serves search results via a simple web interface, allowing you to retrieve information from your own data without relying on external cloud services.
whoosh library for full-text search, which is lightweight and runs entirely locally.Create a project directory and initialize a virtual environment to isolate dependencies. This ensures that the specific versions of Flask and Whoosh we use do not conflict with other Python projects on your system.
Install Flask for the web server and Whoosh for full-text search capabilities. Flask will handle the HTTP requests from your browser, while Whoosh provides the underlying engine to store and query text data efficiently.
mkdir minimal-search-clone
cd minimal-search-clone
python3 -m venv venv
source venv/bin/activate
pip install flask whooshCreate a Python module that defines the search schema (title, content, path) and handles indexing of text files. We use Whoosh's Schema class to define the fields we want to store and index, ensuring that both the file path and its content are searchable.
Implement functions to add documents to the index and perform search queries using Whoosh. The index_documents function scans a directory for .txt files, reads their content, and adds them to the index, while search queries the index and returns matching documents with snippets.
search_engine.py
import os
from whoosh.index import create_in, open_dir
from whoosh.fields import Schema, TEXT, ID
from whoosh.qparser import QueryParser
INDEX_DIR = "whoosh_index"
SCHEMA = Schema(
path=ID(stored=True),
title=TEXT(stored=True),
content=TEXT(stored=True)
)
def get_index():
if not os.path.exists(INDEX_DIR):
os.mkdir(INDEX_DIR)
return create_in(INDEX_DIR, SCHEMA)
return open_dir(INDEX_DIR)
def index_documents(docs_dir):
ix = get_index()
writer = ix.writer()
for filename in os.listdir(docs_dir):
if filename.endswith(".txt"):
filepath = os.path.join(docs_dir, filename)
with open(filepath, 'r', encoding='utf-8') as f:
content = f.read()
writer.add_document(
path=filepath,
title=filename,
content=content
)
writer.commit()
return True
def search(query_string):
ix = get_index()
with ix.searcher() as searcher:
query = QueryParser("content", schema=ix.schema).parse(query_string)
results = searcher.search(query)
return [
{
"title": hit["title"],
"path": hit["path"],
"snippet": hit["content"][:200] + "..."
} for hit in results
]Create a Flask application that exposes endpoints for searching and indexing. We define two routes: one for the main search interface and another API endpoint that accepts search queries via POST requests.
Add a simple HTML template for the search interface. This template includes a search bar and a results area, using basic JavaScript to send fetch requests to our Flask backend and display the results dynamically.
app.py
from flask import Flask, render_template, request, jsonify
from search_engine import index_documents, search
app = Flask(__name__)
@app.route('/')
def index():
return render_template('index.html')
@app.route('/api/search', methods=['POST'])
def api_search():
data = request.json
query = data.get('query', '')
if not query:
return jsonify({"results": []})
results = search(query)
return jsonify({"results": results})
@app.route('/api/reindex', methods=['POST'])
def api_reindex():
docs_dir = request.json.get('docs_dir', 'documents')
try:
index_documents(docs_dir)
return jsonify({"status": "success", "message": "Indexing complete"})
except Exception as e:
return jsonify({"status": "error", "message": str(e)}), 500
if __name__ == '__main__':
app.run(debug=True)Before launching the server, we need some data to index so the search isn't empty. Create a directory named docs and populate it with a few simple text files containing relevant keywords like 'Python' or 'search'.
Once the sample data is in place, start the Flask development server. This will initialize the Whoosh index if it doesn't exist and begin listening for HTTP requests on port 5000.
mkdir -p docs
echo "This is a sample Python document for testing." > docs/sample1.txt
echo "Another file about search engines and indexing." > docs/sample2.txt
python app.pyOpen your web browser and navigate to http://127.0.0.1:5000. You should see the minimal search interface. Try searching for 'Python' to verify that the indexer picked up the content from sample1.txt.
To test dynamic indexing, create a new file in the docs folder, then click the 'Re-index Documents' button on the web interface. Search for a keyword unique to the new file to confirm the index updates correctly.
echo "This is a new document about UVF IT guides." > docs/new_sample.txt
# Then in browser: Click 'Re-index Documents' and search for 'UVF'| Check | Command / Why it matters |
|---|---|
| Dependencies installed | pip list | grep whoosh |
| Server running | Check terminal for 'Running on http://127.0.0.1:5000' |
| Indexing works | Click 'Re-index Documents' and see 'Indexing complete' |
| Search returns results | Search for 'Python' and see a result with snippet |
To extend this clone, consider adding support for PDF files using PyPDF2, implementing pagination for large result sets, or adding a browser extension to automatically feed URLs into the indexer.