Compare commits

...

10 Commits

Author SHA1 Message Date
Sjoerd de Vries 74f487486f Fix final-review findings for Playwright fallback
- server.js: catch browser-launch failures inside the injected
  fetchAndExtractPriceViaBrowser wrapper so they route through
  checkProduct's normal error classification (recordCheckFailure)
  instead of escaping as a thrown error that bypasses the DB write.
- server.js: add an in-flight guard around runCheck so overlapping
  callers (cron + check-now) share one run instead of each launching
  its own Chromium.
- Dockerfile: copy only package.json/package-lock.json/node_modules
  before installing Chromium so the expensive install layer is keyed
  to dependency changes, not every commit; copy the rest of the app
  afterward.
- Dockerfile: clean up apt package lists after the Playwright install
  step, matching the build stage's existing cleanup.
- README.md: note that exercising the fallback locally (not just
  running the test suite) needs `npx playwright install chromium`.
- README.md: correct the fallback description — domcontentloaded
  navigation reads the page before most client-side JS finishes, so
  the fallback mainly helps with HTTP-client blocking, not JS-rendered
  prices; reworded to drop the inaccurate claim.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 18:06:14 +02:00
Sjoerd de Vries 7a117e7a46 chore: install Chromium in the Docker image for the fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:50:25 +02:00
Sjoerd de Vries 966a80c690 fix: don't let a failed browser launch override a successful runCheck result
Guard the finally block's re-await of browserPromise so a rejected
launch (e.g. Playwright can't start Chromium) doesn't rethrow out of
finally and override the try block's already-successful return value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:44:29 +02:00
Sjoerd de Vries fc8dadc8a3 feat: launch a lazy per-run browser for the fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:39:03 +02:00
Sjoerd de Vries 358347e8ba feat: fall back to the browser scraper when the primary fetch fails
- Extract fetchAndExtractPriceViaBrowser from deps (optional)
- After primary fetch, if it errors or returns no price and fallback exists, try fallback
- Fallback result replaces primary result for further processing
- Retry-safety logic unchanged: last_price/last_checked_at only updated on success
- All 4 new tests pass, all existing tests unchanged

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:34:02 +02:00
Sjoerd de Vries df3d3250e5 feat: add Playwright-based fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:30:18 +02:00
Sjoerd de Vries dac7893cd2 Add implementation plan for Playwright fallback
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:24:06 +02:00
Sjoerd de Vries 957555bdbe Refine spec: lazy browser launch for Playwright fallback
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:21:30 +02:00
Sjoerd de Vries 4fee15173b chore: ignore local .claude harness state directory
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:18:29 +02:00
Sjoerd de Vries 1d3d1caeb1 Add design spec for Playwright fallback scraping
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:18:23 +02:00
12 changed files with 1026 additions and 17 deletions
+1
View File
@@ -1,3 +1,4 @@
node_modules/
data/
.env
.claude/
+4
View File
@@ -8,6 +8,10 @@ COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app/package.json /app/package-lock.json ./
COPY --from=build /app/node_modules ./node_modules
RUN npx --yes playwright install --with-deps chromium \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app .
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
+12
View File
@@ -19,6 +19,11 @@ app bij het opstarten met een melding.
De UI staat achter basic auth (`BASIC_AUTH_USER` / `BASIC_AUTH_PASS`) op
`http://localhost:3000`.
Wil je de headless-browser-fallback ook echt lokaal uitproberen (de
testsuite gebruikt fakes en heeft hier geen echte browser voor nodig), run
dan eerst `npx playwright install chromium` om de Chromium-binary te
downloaden.
## Environment variables
| Variabele | Omschrijving |
@@ -39,3 +44,10 @@ check. Een mislukte poging (pagina onbereikbaar, geen prijs gevonden,
of de mail kon niet verstuurd worden) verandert de opgeslagen prijs
niet, zodat de volgende dag opnieuw wordt geprobeerd. Via de "Nu
controleren"-knop in de UI kan dit ook handmatig getriggerd worden.
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
die eenvoudige HTTP-clients blokkeren. Geen garantie tegen elke vorm van
bot-detectie, maar dekt een reëel deel van de gevallen die de gewone
aanpak mist.
@@ -0,0 +1,633 @@
# Korting Playwright-fallback Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** When the existing plain-fetch price scraper fails (network error or no price found), retry once with a real headless-Chromium page load (Playwright) before recording the check as failed.
**Architecture:** A new, self-contained `src/browserScraper.js` module wraps Playwright and reuses `extractPrice()` from the existing `src/scraper.js` (no duplicated parsing logic). `checkProducts.js` gains one new *optional* injected dependency that it calls only when the primary fetch already failed — `scraper.js` itself is untouched. `server.js` lazily launches one browser per `runCheck()` run (only if a fallback is actually needed), reuses it for the rest of that run, and always closes it in a `finally`.
**Tech Stack:** Playwright (`chromium`), same Node.js 22 / CommonJS / `node --test` stack as the rest of the project.
**Spec:** `docs/superpowers/specs/2026-09-05-korting-playwright-fallback-design.md` (builds on `docs/superpowers/specs/2026-09-05-korting-prijstracker-design.md`)
## Global Constraints
- Node.js 22, CommonJS modules, no TypeScript — matches the existing codebase.
- `src/scraper.js` and its tests are **not modified** by this plan — the fallback is a separate module, invoked from `checkProducts.js`, not from inside `fetchAndExtractPrice`.
- The browser fallback is invoked **only** when the primary `fetchAndExtractPrice` result has `error` or `price === null` — never when it already found a price.
- Every existing test must keep passing unmodified in behavior (the new dependency is optional; omitting it from `deps` must not change any existing test's outcome).
- `playwright` is a production dependency (not `devDependencies`) — the Docker runtime stage needs it, not just the build stage.
- **Local `npm install` must not attempt to download the actual Chromium binary** — set `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1` in the environment before running `npm install` so the postinstall step is skipped. The real browser binary is only installed inside the Docker image (Task 4) via `npx playwright install --with-deps chromium`. Tasks 1-3 never need a real Chromium binary in the dev/test sandbox — all automated tests use fake browser/page objects (dependency injection), and Task 3's manual check deliberately avoids exercising the fallback path for this reason. Task 4's Docker build/run verification is the first (and only) point in this plan where a real browser actually launches.
- Timeout for a fallback page load: 20000ms (`timeoutMs` default), passed to Playwright's `page.goto` timeout option.
---
### Task 1: Browser-based scraper module
**Files:**
- Modify: `package.json` (add `playwright` dependency)
- Create: `src/browserScraper.js`
- Create: `test/browserScraper.test.js`
**Interfaces:**
- Consumes: `extractPrice` from `src/scraper.js` (already exported, unchanged signature: `extractPrice(html) -> { price: number|null, method: string|null }`).
- Produces:
- `launchBrowser() -> Promise<Browser>` — thin wrapper around Playwright's `chromium.launch({ headless: true })`.
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) -> Promise<{ price: number|null, method: string|null, error: Error|null }>` — same return shape as `scraper.js`'s `fetchAndExtractPrice`, so later tasks can treat both interchangeably.
- [ ] **Step 1: Add the `playwright` dependency**
Run (this environment must not attempt to download the Chromium binary — see Global Constraints):
```bash
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install playwright
```
Expected: `package.json`'s `dependencies` gains a `playwright` entry (not `devDependencies`), `package-lock.json` updates, and the command completes without trying to download a browser binary.
- [ ] **Step 2: Write the failing tests**
Create `test/browserScraper.test.js`:
```js
const test = require('node:test');
const assert = require('node:assert/strict');
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
let closed = false;
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
const page = {
goto: async () => {},
content: async () => html,
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
content: async () => '',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.equal(result.price, null);
assert.equal(result.method, null);
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => '<html><body><p>Geen prijs</p></body></html>',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: null, method: null, error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => { throw new Error('content failed'); },
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
let receivedOptions;
const page = {
goto: async (url, options) => { receivedOptions = options; },
content: async () => '<html><body></body></html>',
close: async () => {},
};
const browser = { newPage: async () => page };
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
assert.equal(receivedOptions.timeout, 5000);
});
```
- [ ] **Step 3: Run tests to verify they fail**
Run: `npm test`
Expected: FAIL with "Cannot find module '../src/browserScraper'"
- [ ] **Step 4: Write `src/browserScraper.js`**
```js
const { chromium } = require('playwright');
const { extractPrice } = require('./scraper');
async function launchBrowser() {
return chromium.launch({ headless: true });
}
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
let page;
try {
page = await browser.newPage();
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
const html = await page.content();
const { price, method } = extractPrice(html);
return { price, method, error: null };
} catch (err) {
return { price: null, method: null, error: err };
} finally {
if (page) {
await page.close().catch(() => {});
}
}
}
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
```
- [ ] **Step 5: Run tests to verify they pass**
Run: `npm test`
Expected: PASS (all `browserScraper.test.js` tests + every previously-passing test, unchanged)
- [ ] **Step 6: Commit**
```bash
git add package.json package-lock.json src/browserScraper.js test/browserScraper.test.js
git commit -m "feat: add Playwright-based fallback scraper"
```
---
### Task 2: Wire the fallback into the price-check orchestration
**Files:**
- Modify: `src/checkProducts.js`
- Modify: `test/checkProducts.test.js`
**Interfaces:**
- Consumes: `fetchAndExtractPriceViaBrowser`-shaped function (Task 1's return shape), passed as the new optional `deps.fetchAndExtractPriceViaBrowser(url) -> Promise<{price, method, error}>`. `checkProducts.js` does not `require` Task 1's module directly — the composition root (Task 3) curries in the real browser instance.
- Produces: `checkProduct`/`checkAllProducts` — same exported names and signatures as before; the only change is that `checkProduct`'s deps object now recognizes one more optional key.
The current `src/checkProducts.js` (for reference — this is what you are editing):
```js
async function checkProduct(product, deps) {
const { fetchAndExtractPrice, sendMail, recordCheckSuccess, recordCheckFailure, now } = deps;
const result = await fetchAndExtractPrice(product.url);
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
recordCheckFailure(product.id, 'fetch_error');
return { status: 'fetch_error' };
}
if (result.price == null) {
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
recordCheckFailure(product.id, 'not_found');
return { status: 'not_found' };
}
const isBaseline = product.last_price == null;
const changed = !isBaseline && result.price !== product.last_price;
if (changed) {
try {
await sendMail({
name: product.name,
url: product.url,
oldPrice: product.last_price,
newPrice: result.price,
});
} catch (err) {
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
recordCheckFailure(product.id, 'mail_error');
return { status: 'mail_error' };
}
}
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
return { status: 'ok', changed };
}
async function checkAllProducts(products, deps) {
const results = [];
for (const product of products) {
try {
const result = await checkProduct(product, deps);
results.push({ productId: product.id, ...result });
} catch (err) {
console.error(`[check] error "${product.name}" (${product.url}): ${err && err.message}`);
results.push({ productId: product.id, status: 'error', error: err });
}
}
return results;
}
module.exports = { checkProduct, checkAllProducts };
```
The current `test/checkProducts.test.js` (for reference — you are appending new tests to this file, not replacing the existing ones):
```js
const test = require('node:test');
const assert = require('node:assert/strict');
const { checkProduct, checkAllProducts } = require('../src/checkProducts');
function makeDeps(overrides = {}) {
const failures = [];
const successes = [];
const mailCalls = [];
const deps = {
fetchAndExtractPrice: async () => ({ price: 10, method: 'meta', error: null }),
sendMail: async (payload) => { mailCalls.push(payload); },
recordCheckFailure: (id, status) => failures.push({ id, status }),
recordCheckSuccess: (id, data) => successes.push({ id, ...data }),
now: () => '2026-09-05T08:00:00.000Z',
...overrides,
};
return { deps, failures, successes, mailCalls };
}
// Failure paths log to stderr; capture it so the test output stays clean
// while still proving the logging happened.
function captureErrorLog(t) {
const logged = [];
t.mock.method(console, 'error', (...args) => { logged.push(args.join(' ')); });
return logged;
}
// ... (existing tests, unchanged — do not modify or remove any of them)
```
- [ ] **Step 1: Write the failing tests**
Append these tests to the end of `test/checkProducts.test.js` (after the existing `checkAllProducts processes every product independently...` test, before the final closing of the file):
```js
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes, mailCalls } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 10, method: 'json-ld', error: null };
},
});
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
assert.equal(mailCalls.length, 0);
});
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 12, method: 'regex', error: null };
},
});
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
});
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
const browserCalls = [];
const { deps } = makeDeps({
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 999, method: 'meta', error: null };
},
});
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
await checkProduct(product, deps);
assert.equal(browserCalls.length, 0);
});
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
});
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'fetch_error');
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /fetch_error.*still down/);
});
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `npm test`
Expected: FAIL — the 4 new tests fail because `checkProduct` doesn't yet call `fetchAndExtractPriceViaBrowser` (e.g. the "tries the browser fallback" tests fail their `browserCalls` assertions with an empty array; the "never calls" test passes vacuously already, which is fine — it's a guard for the next step, not expected to fail).
- [ ] **Step 3: Modify `src/checkProducts.js`**
Change only the first few lines of `checkProduct` (everything from the baseline/changed logic onward stays exactly as it is):
```js
async function checkProduct(product, deps) {
const {
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
sendMail, recordCheckSuccess, recordCheckFailure, now,
} = deps;
let result = await fetchAndExtractPrice(product.url);
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
result = await fetchAndExtractPriceViaBrowser(product.url);
}
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
recordCheckFailure(product.id, 'fetch_error');
return { status: 'fetch_error' };
}
if (result.price == null) {
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
recordCheckFailure(product.id, 'not_found');
return { status: 'not_found' };
}
const isBaseline = product.last_price == null;
const changed = !isBaseline && result.price !== product.last_price;
if (changed) {
try {
await sendMail({
name: product.name,
url: product.url,
oldPrice: product.last_price,
newPrice: result.price,
});
} catch (err) {
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
recordCheckFailure(product.id, 'mail_error');
return { status: 'mail_error' };
}
}
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
return { status: 'ok', changed };
}
```
(`checkAllProducts` and `module.exports` are unchanged — leave them exactly as they are.)
- [ ] **Step 4: Run tests to verify they pass**
Run: `npm test`
Expected: PASS (all new tests + every previously-passing test, unchanged)
- [ ] **Step 5: Commit**
```bash
git add src/checkProducts.js test/checkProducts.test.js
git commit -m "feat: fall back to the browser scraper when the primary fetch fails"
```
---
### Task 3: Wire the browser lifecycle into the composition root
**Files:**
- Modify: `server.js`
**Interfaces:**
- Consumes: `launchBrowser`, `fetchAndExtractPriceViaBrowser` (Task 1); `checkProduct`'s new optional `fetchAndExtractPriceViaBrowser` deps key (Task 2).
- Produces: nothing new for other tasks to consume — `server.js` is the top of the dependency graph.
No new automated test — this is a composition-root change, verified with a manual smoke check (same pattern as the original server.js task). **This step does not require a real Chromium binary**: the smoke check below only exercises a product whose plain fetch succeeds, so the lazy browser launch never actually triggers.
- [ ] **Step 1: Modify `server.js`**
Add the import (alongside the other `require`s near the top):
```js
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
```
Replace the `runCheck` function with a version that lazily launches one browser per run, only if a fallback is actually needed, and always closes it:
```js
async function runCheck() {
const products = listProducts(db);
let browserPromise = null;
function getBrowser() {
if (!browserPromise) {
browserPromise = launchBrowser();
}
return browserPromise;
}
try {
const deps = {
fetchAndExtractPrice,
fetchAndExtractPriceViaBrowser: async (url) => {
const browser = await getBrowser();
return fetchAndExtractPriceViaBrowser(browser, url);
},
sendMail: (payload) => sendPriceChangeEmail(transport, {
from: process.env.SMTP_FROM,
to: process.env.NOTIFY_EMAIL,
...payload,
}),
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
now: () => new Date().toISOString(),
};
const results = await checkAllProducts(products, deps);
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
return results;
} finally {
if (browserPromise) {
const browser = await browserPromise;
await browser.close().catch((err) => {
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
});
}
}
}
```
(Everything else in `server.js` — the required-env check, `openDb`, `createTransport`, `summarize`, `startScheduler(runCheck)`, `createApp({...})`, `app.listen(...)` — stays exactly as it is.)
- [ ] **Step 2: Manual smoke check (no real browser needed for this check)**
Deliberately **do not** call `/check-now` in this step: with the Task 2
change, any URL whose plain fetch doesn't yield a price (which is most
URLs without a real product price, e.g. `https://example.com`) now
triggers the browser fallback — and no real Chromium binary is installed
in this dev sandbox (see Global Constraints). Exercising the actual
fallback path with a real browser is Task 4's job, once Chromium is
installed in the Docker image. This step only proves `server.js` still
starts and its non-check routes still work after the edit — i.e. that the
new `require('./src/browserScraper')` and the rewritten `runCheck` are
syntactically sound and don't break anything at startup.
Run:
```bash
cp .env.example .env
# edit .env: fill in real BASIC_AUTH_USER/PASS and SMTP_FROM/NOTIFY_EMAIL at minimum
set -a; source .env; set +a
node server.js
```
In another terminal:
```bash
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" -X POST http://localhost:3000/products \
-d "name=Testproduct" -d "url=https://example.com"
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" http://localhost:3000/
```
Expected: both succeed (302, then 200 with "Testproduct" visible in the
list). Stop the server with Ctrl-C — do not proceed to `/check-now` here.
- [ ] **Step 3: Commit**
```bash
git add server.js
git commit -m "feat: launch a lazy per-run browser for the fallback scraper"
```
---
### Task 4: Docker image — install Chromium for the fallback
**Files:**
- Modify: `Dockerfile`
- Modify: `README.md`
**Interfaces:** none — this task only changes the deployment image and documentation.
The current `Dockerfile` (for reference):
```dockerfile
FROM node:22-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app .
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
# en bij elke redeploy alle data verliezen.
ENV DB_PATH=/data/korting.db
VOLUME /data
EXPOSE 3000
CMD ["node", "server.js"]
```
- [ ] **Step 1: Modify the `Dockerfile`'s runtime stage**
Add the Chromium install line right after `COPY --from=build /app .` (this is the first point where `playwright` from `node_modules` is available to `npx`, and it's the stage that actually runs the browser, so the browser binary + its system libraries belong here, not in the build stage):
```dockerfile
FROM node:22-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app .
RUN npx --yes playwright install --with-deps chromium
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
# en bij elke redeploy alle data verliezen.
ENV DB_PATH=/data/korting.db
VOLUME /data
EXPOSE 3000
CMD ["node", "server.js"]
```
- [ ] **Step 2: Update `README.md`**
Add one short paragraph after the existing "Hoe het werkt" section explaining the fallback (read the current `README.md` first to place it consistently with the existing tone/structure):
```markdown
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
die de prijs pas met JavaScript renderen, of die eenvoudige HTTP-clients
blokkeren. Geen garantie tegen elke vorm van bot-detectie, maar dekt een
reëel deel van de gevallen die de gewone aanpak mist.
```
- [ ] **Step 3: Manual build/run verification (this is the first task that needs a real browser)**
Run:
```bash
docker build -t korting .
```
Expected: build succeeds; watch for the `playwright install --with-deps chromium` step completing without error (it will print apt package installation output — that's expected).
Run:
```bash
docker run --rm -d --name korting-fallback-smoketest -p 3002:3000 \
-e BASIC_AUTH_USER=admin -e BASIC_AUTH_PASS=test \
-e SMTP_HOST=mail.sjoerd.app -e SMTP_PORT=587 \
-e SMTP_FROM=korting@sjoerd.app -e NOTIFY_EMAIL=test@example.com \
-e DB_PATH=/data/korting.db -v korting-fallback-smoketest-data:/data \
korting
sleep 2
curl -u admin:test -X POST http://localhost:3002/products \
-d "name=Fallback test" -d "url=https://example.com"
curl -u admin:test -X POST http://localhost:3002/check-now
docker logs korting-fallback-smoketest 2>&1 | tail -20
```
Expected: the `check-now` request succeeds (302); the container logs show the `[check] 1 product(en) gecontroleerd: ...` summary line. This URL (`https://example.com`) has no parseable price, so the primary fetch will return `not_found`, which **does** trigger the fallback — confirm the logs show no crash and no unhandled Playwright error (a `not_found` after the fallback also finds nothing is the expected, correct outcome here — the goal of this check is proving the browser actually launches and runs inside the container without error, not that this particular blank test page has a price).
Clean up:
```bash
docker stop korting-fallback-smoketest
docker volume rm korting-fallback-smoketest-data
docker rmi korting
```
- [ ] **Step 4: Commit**
```bash
git add Dockerfile README.md
git commit -m "chore: install Chromium in the Docker image for the fallback scraper"
```
---
## After this plan
Deploy the merged result to Coolify (redeploy the existing `korting.sjoerd.app` application — same app, no new resources needed) via the `deploying-to-coolify` skill, and verify against the actual reported failing URL
(`https://www.iciparisxl.nl/chanel/bleu-de-chanel/parfum-verstuiver/p/BP_967866`)
using Coolify's deployment/application logs — not a direct request from this
machine, since the user has since rotated the app's basic-auth credentials
and testing now happens on their side.
@@ -0,0 +1,128 @@
# Korting — Playwright-fallback voor prijs-extractie
**Datum:** 2026-09-05
**Status:** Approved
**Bouwt voort op:** `2026-09-05-korting-prijstracker-design.md`
## Aanleiding
Een gebruiker meldde dat `fetchAndExtractPrice` faalt voor
`https://www.iciparisxl.nl/...` met `fetch_error HTTP 403`. Onderzoek
(systematic-debugging) toonde aan dat dit domein achter **Akamai Bot
Manager** draait: elk verzoek dat niet als "echte browser" wordt herkend
krijgt al op de edge een `403 Access Denied` (`server: AkamaiGHost`),
vóórdat de eigenlijke website bereikt wordt. Bevestigd zowel met een
directe `curl`-test (ook met browser-achtige User-Agent) als via de live
app zelf (identieke `fetch_error HTTP 403` in de logs). Dit is geen bug in
de bestaande scraper-logica — die blijft correct voor de meeste shops —
maar een structurele blokkade die headers alleen niet oplossen.
**Belangrijk:** een headless-browser fallback is geen garantie tegen
Akamai specifiek (die kan ook headless browsers detecteren), maar is de
enige realistische route voor sites die de prijs pas client-side
renderen, en verbetert de dekking voor dat bredere geval.
## Doel
Wanneer de bestaande `fetchAndExtractPrice` geen prijs oplevert (fout of
`price: null`), probeert het systeem het opnieuw met een echte headless
Chromium-browser (Playwright) voordat de check als mislukt wordt
geregistreerd.
## Architectuur
Nieuwe module `src/browserScraper.js`, losstaand van `src/scraper.js`
(die blijft ongewijzigd — alle bestaande tests blijven geldig):
- `launchBrowser() -> Promise<Browser>` — start één Chromium-instance
(Playwright).
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 }) ->
Promise<{ price: number|null, method: string|null, error: Error|null
}>` — opent een nieuwe pagina op de meegegeven browser, navigeert naar
`url` (timeout `timeoutMs`), leest de gerenderde HTML via
`page.content()`, en hergebruikt `extractPrice()` uit `src/scraper.js`
(JSON-LD → meta → regex, exact dezelfde tiers) om de prijs eruit te
halen. Sluit de pagina altijd (try/finally), nooit de browser zelf.
## Wijziging in `checkProducts.js`
`checkProduct(product, deps)` krijgt een **optionele** extra dependency:
`deps.fetchAndExtractPriceViaBrowser(url) ->
Promise<{price,method,error}>` (curried met de browser-instance door de
composition root — `checkProducts.js` weet niets van Playwright zelf,
blijft puur dependency-injected zoals nu).
Logica: eerst zoals nu `fetchAndExtractPrice(product.url)` aanroepen. Als
dat resultaat een `error` heeft **of** `price === null`, én
`deps.fetchAndExtractPriceViaBrowser` is meegegeven, probeer de fallback
en gebruik dát resultaat verder (ook als de fallback zelf ook faalt — de
bestaande classificatie hieronder werkt op wat dan ook het laatste
resultaat is). **Fallback wordt nooit aangeroepen als de eerste poging al
een prijs opleverde.**
De rest van de functie (baseline/unchanged/changed+mail/mail_error/
fetch_error/not_found, en de regel dat `last_price`/`last_checked_at`
alleen bij een volledig geslaagde check worden bijgewerkt) blijft
ongewijzigd.
## Lifecycle (in `server.js`)
Eén browser per `runCheck()`-aanroep (zowel de dagelijkse cron als de
"nu controleren"-knop delen dit patroon, want beide roepen `runCheck()`
aan) — **lazy**: de browser start pas op het moment dat de eerste
fallback binnen die run daadwerkelijk nodig is (dus niet als alle
producten al gewoon ophaalbaar zijn via de normale fetch). Eenmaal
gestart wordt diezelfde instance hergebruikt voor de rest van die run
(nieuwe pagina per product), en aan het eind van de run altijd gesloten
in een `finally` — nooit een browser die de hele dag blijft openstaan, en
geen onnodige browser-start op runs die 'm niet nodig hebben.
## Docker
De runtime-stage van de Dockerfile installeert Chromium + de benodigde
systeemlibraries via `npx playwright install --with-deps chromium`, ná
het kopiëren van de app (zodat de juiste Playwright-versie uit
`node_modules` gebruikt wordt). `playwright` wordt een production
dependency (niet dev) zodat `npm ci --omit=dev` in de build-stage 'm wél
meeneemt naar de runtime-stage. Dit vergroot de image en de build-tijd
merkbaar (geaccepteerd trade-off, al gecommuniceerd).
## Error handling
- `page.goto` faalt of timet uit (`timeoutMs`, default 20000ms) → fallback
retourneert `{price: null, method: null, error}`, exact zelfde vorm als
de bestaande `fetchAndExtractPrice`-fouten. Geen crash van de run.
- Pagina wordt altijd gesloten, ook bij een gooiende `page.goto`
(try/finally), zodat er geen paginas/geheugen lekken binnen een run met
meerdere fallback-pogingen.
- Browser wordt altijd gesloten aan het eind van `runCheck()`, ook als er
onderweg een fout optreedt (try/finally rond de hele check-run).
## Testing
- `test/browserScraper.test.js`: test tegen een **nep** browser/pagina-
object (zelfde dependency-injection-patroon als de rest van de
codebase — geen echte Chromium nodig in de testrun): succesvolle
extractie, `page.goto` die faalt/timet uit, en dat `page.close()`
altijd wordt aangeroepen (ook bij een fout).
- `test/checkProducts.test.js`: nieuwe tests die bevestigen dat de
fallback (a) wordt aangeroepen wanneer de eerste poging faalt, (b)
**niet** wordt aangeroepen wanneer de eerste poging al een prijs
oplevert, en (c) dat bestaande tests zonder
`fetchAndExtractPriceViaBrowser` in de deps ongewijzigd blijven werken
(de dependency is optioneel, achterwaarts compatibel).
- Geen geautomatiseerde test voor de Dockerfile-wijziging zelf — net als
bij de oorspronkelijke Dockerfile-taak, een handmatige
build+run-verificatie (ditmaal ook een handmatige check dat Chromium
daadwerkelijk een pagina kan laden binnen de container).
## Out of scope
- Geen garantie dat dit specifieke Akamai-domein (`iciparisxl.nl`)
hierdoor gaat werken — dat hangt af van Akamai's detectie-niveau voor
dat domein op dat moment. Dit wordt niet apart getest tegen dat
specifieke domein als onderdeel van de geautomatiseerde testsuite (te
broos/afhankelijk van een externe, veranderlijke anti-bot-dienst); wel
handmatig te proberen na deploy.
- Playwright-stealth-plugins of andere anti-detectie-technieken — niet
gevraagd, voegt complexiteit toe zonder gegarandeerd resultaat.
+29 -1
View File
@@ -14,7 +14,8 @@
"express": "^4.19.2",
"express-basic-auth": "^1.2.1",
"node-cron": "^3.0.3",
"nodemailer": "^6.9.14"
"nodemailer": "^6.9.14",
"playwright": "^1.63.0"
},
"devDependencies": {
"supertest": "^7.0.0"
@@ -1285,6 +1286,33 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
},
"node_modules/playwright": {
"version": "1.63.0",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz",
"integrity": "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==",
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.63.0"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/playwright-core": {
"version": "1.63.0",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.63.0.tgz",
"integrity": "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==",
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/prebuild-install": {
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
+2 -1
View File
@@ -15,7 +15,8 @@
"express": "^4.19.2",
"express-basic-auth": "^1.2.1",
"node-cron": "^3.0.3",
"nodemailer": "^6.9.14"
"nodemailer": "^6.9.14",
"playwright": "^1.63.0"
},
"devDependencies": {
"supertest": "^7.0.0"
+50 -13
View File
@@ -8,6 +8,7 @@ const {
recordCheckSuccess, recordCheckFailure,
} = require('./src/db');
const { fetchAndExtractPrice } = require('./src/scraper');
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
const { createTransport, sendPriceChangeEmail } = require('./src/mailer');
const { checkAllProducts } = require('./src/checkProducts');
const { start: startScheduler } = require('./src/scheduler');
@@ -36,23 +37,59 @@ function summarize(results) {
return parts.length > 0 ? parts.join(' ') : 'geen producten';
}
let inFlightCheck = null;
function runCheck() {
if (inFlightCheck) {
return inFlightCheck;
}
inFlightCheck = runCheckOnce().finally(() => {
inFlightCheck = null;
});
return inFlightCheck;
}
async function runCheckOnce() {
const products = listProducts(db);
const deps = {
fetchAndExtractPrice,
sendMail: (payload) => sendPriceChangeEmail(transport, {
from: process.env.SMTP_FROM,
to: process.env.NOTIFY_EMAIL,
...payload,
}),
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
now: () => new Date().toISOString(),
};
return checkAllProducts(products, deps).then((results) => {
let browserPromise = null;
function getBrowser() {
if (!browserPromise) {
browserPromise = launchBrowser();
}
return browserPromise;
}
try {
const deps = {
fetchAndExtractPrice,
fetchAndExtractPriceViaBrowser: async (url) => {
try {
const browser = await getBrowser();
return await fetchAndExtractPriceViaBrowser(browser, url);
} catch (err) {
return { price: null, method: null, error: err };
}
},
sendMail: (payload) => sendPriceChangeEmail(transport, {
from: process.env.SMTP_FROM,
to: process.env.NOTIFY_EMAIL,
...payload,
}),
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
now: () => new Date().toISOString(),
};
const results = await checkAllProducts(products, deps);
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
return results;
});
} finally {
if (browserPromise) {
const browser = await browserPromise.catch(() => null);
if (browser) {
await browser.close().catch((err) => {
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
});
}
}
}
}
startScheduler(runCheck);
+25
View File
@@ -0,0 +1,25 @@
const { chromium } = require('playwright');
const { extractPrice } = require('./scraper');
async function launchBrowser() {
return chromium.launch({ headless: true });
}
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
let page;
try {
page = await browser.newPage();
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
const html = await page.content();
const { price, method } = extractPrice(html);
return { price, method, error: null };
} catch (err) {
return { price: null, method: null, error: err };
} finally {
if (page) {
await page.close().catch(() => {});
}
}
}
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
+9 -2
View File
@@ -1,6 +1,13 @@
async function checkProduct(product, deps) {
const { fetchAndExtractPrice, sendMail, recordCheckSuccess, recordCheckFailure, now } = deps;
const result = await fetchAndExtractPrice(product.url);
const {
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
sendMail, recordCheckSuccess, recordCheckFailure, now,
} = deps;
let result = await fetchAndExtractPrice(product.url);
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
result = await fetchAndExtractPriceViaBrowser(product.url);
}
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
+70
View File
@@ -0,0 +1,70 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
let closed = false;
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
const page = {
goto: async () => {},
content: async () => html,
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
content: async () => '',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.equal(result.price, null);
assert.equal(result.method, null);
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => '<html><body><p>Geen prijs</p></body></html>',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: null, method: null, error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => { throw new Error('content failed'); },
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
let receivedOptions;
const page = {
goto: async (url, options) => { receivedOptions = options; },
content: async () => '<html><body></body></html>',
close: async () => {},
};
const browser = { newPage: async () => page };
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
assert.equal(receivedOptions.timeout, 5000);
});
+63
View File
@@ -119,3 +119,66 @@ test('checkAllProducts processes every product independently, isolating failures
assert.equal(results[0].status, 'error');
assert.equal(results[1].status, 'ok');
});
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes, mailCalls } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 10, method: 'json-ld', error: null };
},
});
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
assert.equal(mailCalls.length, 0);
});
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 12, method: 'regex', error: null };
},
});
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
});
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
const browserCalls = [];
const { deps } = makeDeps({
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 999, method: 'meta', error: null };
},
});
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
await checkProduct(product, deps);
assert.equal(browserCalls.length, 0);
});
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
});
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'fetch_error');
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /fetch_error.*still down/);
});