← All projects
WIP November 2026

ShoppingScraperHelper

A home price tracker for the family: you put an article on a list, it finds the shops that sell it, watches their prices and warns when one drops. Feasibility study done, six of seven milestones built, the last one under trial.

PythonDockerPlaywrightSearXNGGeminiSelf-hosted

The idea

Something like the well-known price-history sites for Amazon, but for several shops at once and only for the family: you put an article on a list, the system finds who sells it, checks the prices every few hours and tells me by email and Telegram when one falls under a threshold or there’s a real offer. It’s for internal use only, on the mini PC at home.

First, a feasibility study

Before writing the application I spent a day checking whether it could work, because the idea has one easy half and one hard half.

How the matching works

Amazon doesn’t publish the EAN code of a product, but the details table has the manufacturer’s code. That became the key: normalized (dots, dashes and spaces removed, so EXAM440.55.G equals EXAM44055G) it lets plain rules decide, with no AI. On the coffee machine, all eleven candidates read were classified correctly: six yes, the second-choice version at a lower price rejected (words like “used”, “refurbished” and “second choice” win over the code), and four relatives with a different code rejected.

When there’s no code (“any RX 7900 XTX 24 GB”, or a mouse described by name) a model decides. I chose Gemini on its free tier as a judge: it receives the article and the page read and answers yes or no with a reason, in a fixed format. On the tests it was right seventeen times out of seventeen, with readable reasons (“20 GB, not XTX 24 GB”, “previous model”, “business version”). What the free tier does not include is Google search by the model, so the search for candidates is done by SearXNG, a self-hosted meta-search engine that gives about twenty candidates in a couple of seconds, almost all real Italian shops once filtered.

In the test the cheapest price of the coffee machine was in a shop I wouldn’t have thought of, sixteen euros less than Amazon: exactly the case the project is for. Even so, the system’s match stays a proposal that someone confirms with a click.

What I decided not to do

A curiosity of the network: from the mini PC, which goes out on the other fibre line, Amazon shows its CAPTCHA to every plain request, while the headless browser reads the page without a problem. The rule for that shop is therefore “browser, ninety seconds between requests”.

How it’s built

A web application and a worker in two Docker containers on the mini PC, under WSL. Data in SQLite inside a Docker volume (never on a mounted Windows disk, where SQLite’s locking isn’t reliable). The worker keeps a queue per domain and, before telling anyone that a price has dropped, checks it twice. There are lists you can share, articles added from a link or a name, manual sources, a price-history chart and a Problems page that gathers the sources that stopped answering. The code reaches the mini PC through a git bundle, because that machine can’t reach the private repository.

Search in practice

Milestone 4 put all of the above into the application. “Search other shops” looks for candidates with the meta-search engine and with Amazon’s own search, filters out comparison sites, review pages, search pages, duplicates and sources already known (even the discarded ones), and reads at most ten pages per search, always respecting each domain’s pace. Then the matching: currency, used or second-choice words, EAN, the manufacturer’s code (also when it’s in the title, which tells EXAM440.55.W apart), and only at the end one call to Gemini, which gets no Google search. Without a key, or if Gemini fails, the proposals stay “to be checked” instead of being invented.

The “yes” and the “maybe” become proposals with Confirm and Discard buttons (and “Confirm all”); the “no” are discarded with the reason shown, and can still be turned on with a click if I disagree. A family article (“any RX 7900 XTX”) works differently: its “yes” become active at once and the search is repeated every 24 hours. I tried it on the mini PC: for the coffee machine, with the manufacturer’s code the rules alone sorted eight yes and six no; without a code, Gemini correctly told the XT from the XTX, the 3S from the 4, the Business version, the Mac one and the bundles. A technical lesson from this milestone: the worker’s “heartbeat” now lives in a thread of its own, because a long search made the worker look stopped. The temporary search engine from the test phase was removed straight away, since the application has its own.

Alerts, HTTPS and backups

Milestone 5 gave the application a voice. An alert fires in three cases: the price falls below the threshold I set (it re-arms when the price goes back above it, and once disarmed it fires again only after another 1% drop); the price falls well under the 30-day median of that source; or it hits a new all-time low, which counts only with at least 14 days of history. If several reasons hold in the same reading, they become one message. Only active sources count, in euro and from the accepted seller (“Amazon only”). Messages go by email and Telegram, on the channels each person picks in their profile; Telegram is linked with a one-time code valid for 30 minutes. They leave through an outgoing queue in the worker, with three attempts five minutes apart, and the Alerts page shows the outcome of every send. Malfunctions (a suspended source, the search engine, Amazon or Gemini not answering) go only to the administrators, at most one message a day per problem.

Milestone 6 made it fit to live on the mini PC. The application is no longer exposed directly: in front of it sits a web server with HTTPS on an internal name, with a Let’s Encrypt certificate obtained through the same DNS-01 delegation I already use for the other services at home, renewed every night. The name resolves only inside the house and over the VPN, and I checked that the bare IP address is refused. A backup at 02:00 makes a consistent copy of the database, keeps 14 days of them and copies the latest to a shared folder on the NAS; a failure becomes an alert to the administrators, and a command restores from any copy. Updates come from GitHub with a script that makes a backup, rebuilds, checks that the application is healthy and, if not, goes back to the previous code, image and database and tells me. One small lesson: Let’s Encrypt’s test servers are unreachable from that machine, so the certificate was requested straight from production.

The first real articles, and a migration that wiped everything

Milestone 7 started with real articles, and the trial showed bugs the tests hadn’t. Two shops put the price without VAT in their structured data: now the page’s “VAT included” price counts, or 22% is added when only “VAT excluded” is shown. The manufacturer’s code is compared as a word on its own, so a switch no longer matches the model that merely starts with the same code; for an extended code Gemini decides. Short codes like the Wi-Fi standard in a router’s name are no longer enough for an automatic “yes”. When Gemini answers 503 the app tries three times, and proposals left without a verdict are re-evaluated at the next search. For the family there is now an Administration page (create an account with a temporary password shown once and a forced change at first login, deactivate, reset; state of worker, backup and latest searches) and a Guide page.

The serious one: a migration that rebuilt the users table, with foreign keys on, deleted lists, articles, sources and history on the mini PC by cascade. I got everything back from the backup the update script takes just before updating, so nothing was lost, but only by luck of design. The cause is plain: the tests started from empty databases and the health check only asked whether the app answered, so neither could notice. Now migrations run with foreign keys off, a test runs every migration on a database with data (and fails without the fix), and the update script compares the counts before and after and rolls back if the data shrinks. A smaller side fix: the web server looks up the application container on every request, so an update no longer leaves a 502.

Status

Work in progress. Feasibility study and test phase done; milestones 1 to 6 of 7 built and installed, and the seventh, the family trial, begun: fixes from the first real articles, the Administration page and the Guide are in. Only my account exists so far. Next I create the family’s accounts and we use it for a week; then a report on the week (right alerts, no duplicates) decides whether to close it. The update script’s rollback path has still never been exercised for real, and I still have to check access from the phone over the VPN.

Stack

Python · Playwright · SearXNG · Gemini · SQLite · Docker · WSL