Compare commits

...

20 Commits

Author SHA1 Message Date
Sjoerd de Vries 74f487486f Fix final-review findings for Playwright fallback
- server.js: catch browser-launch failures inside the injected
  fetchAndExtractPriceViaBrowser wrapper so they route through
  checkProduct's normal error classification (recordCheckFailure)
  instead of escaping as a thrown error that bypasses the DB write.
- server.js: add an in-flight guard around runCheck so overlapping
  callers (cron + check-now) share one run instead of each launching
  its own Chromium.
- Dockerfile: copy only package.json/package-lock.json/node_modules
  before installing Chromium so the expensive install layer is keyed
  to dependency changes, not every commit; copy the rest of the app
  afterward.
- Dockerfile: clean up apt package lists after the Playwright install
  step, matching the build stage's existing cleanup.
- README.md: note that exercising the fallback locally (not just
  running the test suite) needs `npx playwright install chromium`.
- README.md: correct the fallback description — domcontentloaded
  navigation reads the page before most client-side JS finishes, so
  the fallback mainly helps with HTTP-client blocking, not JS-rendered
  prices; reworded to drop the inaccurate claim.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 18:06:14 +02:00
Sjoerd de Vries 7a117e7a46 chore: install Chromium in the Docker image for the fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:50:25 +02:00
Sjoerd de Vries 966a80c690 fix: don't let a failed browser launch override a successful runCheck result
Guard the finally block's re-await of browserPromise so a rejected
launch (e.g. Playwright can't start Chromium) doesn't rethrow out of
finally and override the try block's already-successful return value.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:44:29 +02:00
Sjoerd de Vries fc8dadc8a3 feat: launch a lazy per-run browser for the fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:39:03 +02:00
Sjoerd de Vries 358347e8ba feat: fall back to the browser scraper when the primary fetch fails
- Extract fetchAndExtractPriceViaBrowser from deps (optional)
- After primary fetch, if it errors or returns no price and fallback exists, try fallback
- Fallback result replaces primary result for further processing
- Retry-safety logic unchanged: last_price/last_checked_at only updated on success
- All 4 new tests pass, all existing tests unchanged

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:34:02 +02:00
Sjoerd de Vries df3d3250e5 feat: add Playwright-based fallback scraper
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:30:18 +02:00
Sjoerd de Vries dac7893cd2 Add implementation plan for Playwright fallback
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:24:06 +02:00
Sjoerd de Vries 957555bdbe Refine spec: lazy browser launch for Playwright fallback
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:21:30 +02:00
Sjoerd de Vries 4fee15173b chore: ignore local .claude harness state directory
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:18:29 +02:00
Sjoerd de Vries 1d3d1caeb1 Add design spec for Playwright fallback scraping
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 17:18:23 +02:00
Sjoerd de Vries 7f378115b5 fix: guard the async /check-now handler against crashing (finding 8)
Express 4 geeft een afgewezen promise uit een async handler niet door aan
de error-handling; met Node 22 crashte daardoor het hele proces. De
handler vangt de fout nu zelf af, logt hem en antwoordt met een 500.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:53:20 +02:00
Sjoerd de Vries d07cda48c6 chore: default DB_PATH to the mounted volume in Docker (finding 5)
Zonder ENV DB_PATH schreef een vergeten env var stilletjes naar de
container-filesystem, waardoor alle data bij de volgende redeploy weg
was. VOLUME /data documenteert het verwachte mountpoint.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:53:08 +02:00
Sjoerd de Vries fb3fc62b4b feat: log failures, validate env at startup, load .env (findings 3, 4, 7)
Finding 3: fetch_error, not_found, mail_error and de onverwachte-fout
branch loggen nu naam/url plus reden naar stderr; runCheck logt na elke
run een regel met de tellingen per status, zodat een cron-run zichtbaar
is in de container-logs. De recordCheckSuccess/recordCheckFailure-condities
en alle return-waarden zijn ongewijzigd.

Finding 4: server.js controleert bij het opstarten dat BASIC_AUTH_USER,
BASIC_AUTH_PASS, SMTP_FROM en NOTIFY_EMAIL gezet zijn en stopt anders met
exit code 1 in plaats van later opake 500's te geven.

Finding 7: dotenv toegevoegd en als eerste regel in server.js geladen,
zodat het gedocumenteerde `cp .env.example .env && npm start` echt werkt.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:52:57 +02:00
Sjoerd de Vries 52c9947267 fix: harden price scraping (findings 1, 2, 6)
Finding 1: parsePriceValue mishandled thousands separators — "1.299,00"
and "1,299.00" both parsed to 1.3. Both the meta-tag path and the
text-regex path now funnel through a single normalizeAmount() helper
that picks the last separator as the decimal one, and the text regex
also matches thousands-grouped amounts.

Finding 2: the text fallback read <script>/<style> contents via
$('body').text(); those elements are now stripped from a clone first.

Finding 6: the default fetch now uses a 15s AbortSignal.timeout and a
browser-like User-Agent, so one unresponsive host cannot stall the
whole sequential run. The fetchImpl injection point is unchanged.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:49:56 +02:00
Sjoerd de Vries 862699d35b docs: add README 2026-09-05 15:40:08 +02:00
Sjoerd de Vries 195dff6fe3 chore: add Dockerfile for deployment 2026-09-05 15:28:31 +02:00
Sjoerd de Vries 5b6d672506 feat: wire modules together into the running server
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:21:15 +02:00
Sjoerd de Vries f73ea41a6c feat: add web UI for managing products
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:17:42 +02:00
Sjoerd de Vries 3b0e824c5b feat: add daily cron scheduler for price checks
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-09-05 15:15:09 +02:00
Sjoerd de Vries 68cb8d0c08 feat: add price-check orchestration with retry-safe error handling
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_013gwc3aCiyQxmCWQke8RvEz
2026-09-05 15:12:16 +02:00
23 changed files with 1658 additions and 9 deletions
+6
View File
@@ -0,0 +1,6 @@
node_modules
data
.env
.git
docs
test
+1
View File
@@ -1,3 +1,4 @@
node_modules/
data/
.env
.claude/
+23
View File
@@ -0,0 +1,23 @@
FROM node:22-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app/package.json /app/package-lock.json ./
COPY --from=build /app/node_modules ./node_modules
RUN npx --yes playwright install --with-deps chromium \
&& rm -rf /var/lib/apt/lists/*
COPY --from=build /app .
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
# en bij elke redeploy alle data verliezen.
ENV DB_PATH=/data/korting.db
VOLUME /data
EXPOSE 3000
CMD ["node", "server.js"]
+53
View File
@@ -0,0 +1,53 @@
# Korting — prijstracker
Houdt een lijst productpagina's dagelijks in de gaten en mailt wanneer
de prijs verandert.
## Lokaal draaien
```bash
npm install
cp .env.example .env # env vars invullen
npm test
npm start
```
De app leest `.env` in via `dotenv`. `BASIC_AUTH_USER`, `BASIC_AUTH_PASS`,
`SMTP_FROM` en `NOTIFY_EMAIL` zijn verplicht; ontbreken ze, dan stopt de
app bij het opstarten met een melding.
De UI staat achter basic auth (`BASIC_AUTH_USER` / `BASIC_AUTH_PASS`) op
`http://localhost:3000`.
Wil je de headless-browser-fallback ook echt lokaal uitproberen (de
testsuite gebruikt fakes en heeft hier geen echte browser voor nodig), run
dan eerst `npx playwright install chromium` om de Chromium-binary te
downloaden.
## Environment variables
| Variabele | Omschrijving |
|---|---|
| `PORT` | HTTP-poort (default `3000`) |
| `DB_PATH` | Pad naar het SQLite-bestand (default `./data/korting.db`) |
| `BASIC_AUTH_USER` / `BASIC_AUTH_PASS` | Login voor de web-UI |
| `SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASS` | SMTP-server (Postal op `mail.sjoerd.app`) |
| `SMTP_FROM` | Afzenderadres van de notificatiemails |
| `NOTIFY_EMAIL` | Ontvangeradres van de notificatiemails |
## Hoe het werkt
Elke dag om 08:00 (Europe/Amsterdam) doorloopt de app alle producten,
haalt de huidige prijs op (JSON-LD → meta tags → regex-fallback), en
stuurt een email als de prijs is veranderd sinds de vorige geslaagde
check. Een mislukte poging (pagina onbereikbaar, geen prijs gevonden,
of de mail kon niet verstuurd worden) verandert de opgeslagen prijs
niet, zodat de volgende dag opnieuw wordt geprobeerd. Via de "Nu
controleren"-knop in de UI kan dit ook handmatig getriggerd worden.
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
die eenvoudige HTTP-clients blokkeren. Geen garantie tegen elke vorm van
bot-detectie, maar dekt een reëel deel van de gevallen die de gewone
aanpak mist.
@@ -0,0 +1,633 @@
# Korting Playwright-fallback Implementation Plan
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
**Goal:** When the existing plain-fetch price scraper fails (network error or no price found), retry once with a real headless-Chromium page load (Playwright) before recording the check as failed.
**Architecture:** A new, self-contained `src/browserScraper.js` module wraps Playwright and reuses `extractPrice()` from the existing `src/scraper.js` (no duplicated parsing logic). `checkProducts.js` gains one new *optional* injected dependency that it calls only when the primary fetch already failed — `scraper.js` itself is untouched. `server.js` lazily launches one browser per `runCheck()` run (only if a fallback is actually needed), reuses it for the rest of that run, and always closes it in a `finally`.
**Tech Stack:** Playwright (`chromium`), same Node.js 22 / CommonJS / `node --test` stack as the rest of the project.
**Spec:** `docs/superpowers/specs/2026-09-05-korting-playwright-fallback-design.md` (builds on `docs/superpowers/specs/2026-09-05-korting-prijstracker-design.md`)
## Global Constraints
- Node.js 22, CommonJS modules, no TypeScript — matches the existing codebase.
- `src/scraper.js` and its tests are **not modified** by this plan — the fallback is a separate module, invoked from `checkProducts.js`, not from inside `fetchAndExtractPrice`.
- The browser fallback is invoked **only** when the primary `fetchAndExtractPrice` result has `error` or `price === null` — never when it already found a price.
- Every existing test must keep passing unmodified in behavior (the new dependency is optional; omitting it from `deps` must not change any existing test's outcome).
- `playwright` is a production dependency (not `devDependencies`) — the Docker runtime stage needs it, not just the build stage.
- **Local `npm install` must not attempt to download the actual Chromium binary** — set `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1` in the environment before running `npm install` so the postinstall step is skipped. The real browser binary is only installed inside the Docker image (Task 4) via `npx playwright install --with-deps chromium`. Tasks 1-3 never need a real Chromium binary in the dev/test sandbox — all automated tests use fake browser/page objects (dependency injection), and Task 3's manual check deliberately avoids exercising the fallback path for this reason. Task 4's Docker build/run verification is the first (and only) point in this plan where a real browser actually launches.
- Timeout for a fallback page load: 20000ms (`timeoutMs` default), passed to Playwright's `page.goto` timeout option.
---
### Task 1: Browser-based scraper module
**Files:**
- Modify: `package.json` (add `playwright` dependency)
- Create: `src/browserScraper.js`
- Create: `test/browserScraper.test.js`
**Interfaces:**
- Consumes: `extractPrice` from `src/scraper.js` (already exported, unchanged signature: `extractPrice(html) -> { price: number|null, method: string|null }`).
- Produces:
- `launchBrowser() -> Promise<Browser>` — thin wrapper around Playwright's `chromium.launch({ headless: true })`.
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) -> Promise<{ price: number|null, method: string|null, error: Error|null }>` — same return shape as `scraper.js`'s `fetchAndExtractPrice`, so later tasks can treat both interchangeably.
- [ ] **Step 1: Add the `playwright` dependency**
Run (this environment must not attempt to download the Chromium binary — see Global Constraints):
```bash
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install playwright
```
Expected: `package.json`'s `dependencies` gains a `playwright` entry (not `devDependencies`), `package-lock.json` updates, and the command completes without trying to download a browser binary.
- [ ] **Step 2: Write the failing tests**
Create `test/browserScraper.test.js`:
```js
const test = require('node:test');
const assert = require('node:assert/strict');
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
let closed = false;
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
const page = {
goto: async () => {},
content: async () => html,
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
content: async () => '',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.equal(result.price, null);
assert.equal(result.method, null);
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => '<html><body><p>Geen prijs</p></body></html>',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: null, method: null, error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => { throw new Error('content failed'); },
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
let receivedOptions;
const page = {
goto: async (url, options) => { receivedOptions = options; },
content: async () => '<html><body></body></html>',
close: async () => {},
};
const browser = { newPage: async () => page };
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
assert.equal(receivedOptions.timeout, 5000);
});
```
- [ ] **Step 3: Run tests to verify they fail**
Run: `npm test`
Expected: FAIL with "Cannot find module '../src/browserScraper'"
- [ ] **Step 4: Write `src/browserScraper.js`**
```js
const { chromium } = require('playwright');
const { extractPrice } = require('./scraper');
async function launchBrowser() {
return chromium.launch({ headless: true });
}
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
let page;
try {
page = await browser.newPage();
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
const html = await page.content();
const { price, method } = extractPrice(html);
return { price, method, error: null };
} catch (err) {
return { price: null, method: null, error: err };
} finally {
if (page) {
await page.close().catch(() => {});
}
}
}
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
```
- [ ] **Step 5: Run tests to verify they pass**
Run: `npm test`
Expected: PASS (all `browserScraper.test.js` tests + every previously-passing test, unchanged)
- [ ] **Step 6: Commit**
```bash
git add package.json package-lock.json src/browserScraper.js test/browserScraper.test.js
git commit -m "feat: add Playwright-based fallback scraper"
```
---
### Task 2: Wire the fallback into the price-check orchestration
**Files:**
- Modify: `src/checkProducts.js`
- Modify: `test/checkProducts.test.js`
**Interfaces:**
- Consumes: `fetchAndExtractPriceViaBrowser`-shaped function (Task 1's return shape), passed as the new optional `deps.fetchAndExtractPriceViaBrowser(url) -> Promise<{price, method, error}>`. `checkProducts.js` does not `require` Task 1's module directly — the composition root (Task 3) curries in the real browser instance.
- Produces: `checkProduct`/`checkAllProducts` — same exported names and signatures as before; the only change is that `checkProduct`'s deps object now recognizes one more optional key.
The current `src/checkProducts.js` (for reference — this is what you are editing):
```js
async function checkProduct(product, deps) {
const { fetchAndExtractPrice, sendMail, recordCheckSuccess, recordCheckFailure, now } = deps;
const result = await fetchAndExtractPrice(product.url);
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
recordCheckFailure(product.id, 'fetch_error');
return { status: 'fetch_error' };
}
if (result.price == null) {
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
recordCheckFailure(product.id, 'not_found');
return { status: 'not_found' };
}
const isBaseline = product.last_price == null;
const changed = !isBaseline && result.price !== product.last_price;
if (changed) {
try {
await sendMail({
name: product.name,
url: product.url,
oldPrice: product.last_price,
newPrice: result.price,
});
} catch (err) {
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
recordCheckFailure(product.id, 'mail_error');
return { status: 'mail_error' };
}
}
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
return { status: 'ok', changed };
}
async function checkAllProducts(products, deps) {
const results = [];
for (const product of products) {
try {
const result = await checkProduct(product, deps);
results.push({ productId: product.id, ...result });
} catch (err) {
console.error(`[check] error "${product.name}" (${product.url}): ${err && err.message}`);
results.push({ productId: product.id, status: 'error', error: err });
}
}
return results;
}
module.exports = { checkProduct, checkAllProducts };
```
The current `test/checkProducts.test.js` (for reference — you are appending new tests to this file, not replacing the existing ones):
```js
const test = require('node:test');
const assert = require('node:assert/strict');
const { checkProduct, checkAllProducts } = require('../src/checkProducts');
function makeDeps(overrides = {}) {
const failures = [];
const successes = [];
const mailCalls = [];
const deps = {
fetchAndExtractPrice: async () => ({ price: 10, method: 'meta', error: null }),
sendMail: async (payload) => { mailCalls.push(payload); },
recordCheckFailure: (id, status) => failures.push({ id, status }),
recordCheckSuccess: (id, data) => successes.push({ id, ...data }),
now: () => '2026-09-05T08:00:00.000Z',
...overrides,
};
return { deps, failures, successes, mailCalls };
}
// Failure paths log to stderr; capture it so the test output stays clean
// while still proving the logging happened.
function captureErrorLog(t) {
const logged = [];
t.mock.method(console, 'error', (...args) => { logged.push(args.join(' ')); });
return logged;
}
// ... (existing tests, unchanged — do not modify or remove any of them)
```
- [ ] **Step 1: Write the failing tests**
Append these tests to the end of `test/checkProducts.test.js` (after the existing `checkAllProducts processes every product independently...` test, before the final closing of the file):
```js
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes, mailCalls } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 10, method: 'json-ld', error: null };
},
});
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
assert.equal(mailCalls.length, 0);
});
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 12, method: 'regex', error: null };
},
});
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
});
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
const browserCalls = [];
const { deps } = makeDeps({
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 999, method: 'meta', error: null };
},
});
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
await checkProduct(product, deps);
assert.equal(browserCalls.length, 0);
});
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
});
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'fetch_error');
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /fetch_error.*still down/);
});
```
- [ ] **Step 2: Run tests to verify they fail**
Run: `npm test`
Expected: FAIL — the 4 new tests fail because `checkProduct` doesn't yet call `fetchAndExtractPriceViaBrowser` (e.g. the "tries the browser fallback" tests fail their `browserCalls` assertions with an empty array; the "never calls" test passes vacuously already, which is fine — it's a guard for the next step, not expected to fail).
- [ ] **Step 3: Modify `src/checkProducts.js`**
Change only the first few lines of `checkProduct` (everything from the baseline/changed logic onward stays exactly as it is):
```js
async function checkProduct(product, deps) {
const {
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
sendMail, recordCheckSuccess, recordCheckFailure, now,
} = deps;
let result = await fetchAndExtractPrice(product.url);
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
result = await fetchAndExtractPriceViaBrowser(product.url);
}
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
recordCheckFailure(product.id, 'fetch_error');
return { status: 'fetch_error' };
}
if (result.price == null) {
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
recordCheckFailure(product.id, 'not_found');
return { status: 'not_found' };
}
const isBaseline = product.last_price == null;
const changed = !isBaseline && result.price !== product.last_price;
if (changed) {
try {
await sendMail({
name: product.name,
url: product.url,
oldPrice: product.last_price,
newPrice: result.price,
});
} catch (err) {
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
recordCheckFailure(product.id, 'mail_error');
return { status: 'mail_error' };
}
}
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
return { status: 'ok', changed };
}
```
(`checkAllProducts` and `module.exports` are unchanged — leave them exactly as they are.)
- [ ] **Step 4: Run tests to verify they pass**
Run: `npm test`
Expected: PASS (all new tests + every previously-passing test, unchanged)
- [ ] **Step 5: Commit**
```bash
git add src/checkProducts.js test/checkProducts.test.js
git commit -m "feat: fall back to the browser scraper when the primary fetch fails"
```
---
### Task 3: Wire the browser lifecycle into the composition root
**Files:**
- Modify: `server.js`
**Interfaces:**
- Consumes: `launchBrowser`, `fetchAndExtractPriceViaBrowser` (Task 1); `checkProduct`'s new optional `fetchAndExtractPriceViaBrowser` deps key (Task 2).
- Produces: nothing new for other tasks to consume — `server.js` is the top of the dependency graph.
No new automated test — this is a composition-root change, verified with a manual smoke check (same pattern as the original server.js task). **This step does not require a real Chromium binary**: the smoke check below only exercises a product whose plain fetch succeeds, so the lazy browser launch never actually triggers.
- [ ] **Step 1: Modify `server.js`**
Add the import (alongside the other `require`s near the top):
```js
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
```
Replace the `runCheck` function with a version that lazily launches one browser per run, only if a fallback is actually needed, and always closes it:
```js
async function runCheck() {
const products = listProducts(db);
let browserPromise = null;
function getBrowser() {
if (!browserPromise) {
browserPromise = launchBrowser();
}
return browserPromise;
}
try {
const deps = {
fetchAndExtractPrice,
fetchAndExtractPriceViaBrowser: async (url) => {
const browser = await getBrowser();
return fetchAndExtractPriceViaBrowser(browser, url);
},
sendMail: (payload) => sendPriceChangeEmail(transport, {
from: process.env.SMTP_FROM,
to: process.env.NOTIFY_EMAIL,
...payload,
}),
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
now: () => new Date().toISOString(),
};
const results = await checkAllProducts(products, deps);
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
return results;
} finally {
if (browserPromise) {
const browser = await browserPromise;
await browser.close().catch((err) => {
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
});
}
}
}
```
(Everything else in `server.js` — the required-env check, `openDb`, `createTransport`, `summarize`, `startScheduler(runCheck)`, `createApp({...})`, `app.listen(...)` — stays exactly as it is.)
- [ ] **Step 2: Manual smoke check (no real browser needed for this check)**
Deliberately **do not** call `/check-now` in this step: with the Task 2
change, any URL whose plain fetch doesn't yield a price (which is most
URLs without a real product price, e.g. `https://example.com`) now
triggers the browser fallback — and no real Chromium binary is installed
in this dev sandbox (see Global Constraints). Exercising the actual
fallback path with a real browser is Task 4's job, once Chromium is
installed in the Docker image. This step only proves `server.js` still
starts and its non-check routes still work after the edit — i.e. that the
new `require('./src/browserScraper')` and the rewritten `runCheck` are
syntactically sound and don't break anything at startup.
Run:
```bash
cp .env.example .env
# edit .env: fill in real BASIC_AUTH_USER/PASS and SMTP_FROM/NOTIFY_EMAIL at minimum
set -a; source .env; set +a
node server.js
```
In another terminal:
```bash
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" -X POST http://localhost:3000/products \
-d "name=Testproduct" -d "url=https://example.com"
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" http://localhost:3000/
```
Expected: both succeed (302, then 200 with "Testproduct" visible in the
list). Stop the server with Ctrl-C — do not proceed to `/check-now` here.
- [ ] **Step 3: Commit**
```bash
git add server.js
git commit -m "feat: launch a lazy per-run browser for the fallback scraper"
```
---
### Task 4: Docker image — install Chromium for the fallback
**Files:**
- Modify: `Dockerfile`
- Modify: `README.md`
**Interfaces:** none — this task only changes the deployment image and documentation.
The current `Dockerfile` (for reference):
```dockerfile
FROM node:22-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app .
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
# en bij elke redeploy alle data verliezen.
ENV DB_PATH=/data/korting.db
VOLUME /data
EXPOSE 3000
CMD ["node", "server.js"]
```
- [ ] **Step 1: Modify the `Dockerfile`'s runtime stage**
Add the Chromium install line right after `COPY --from=build /app .` (this is the first point where `playwright` from `node_modules` is available to `npx`, and it's the stage that actually runs the browser, so the browser binary + its system libraries belong here, not in the build stage):
```dockerfile
FROM node:22-slim AS build
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
&& rm -rf /var/lib/apt/lists/*
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
FROM node:22-slim
WORKDIR /app
COPY --from=build /app .
RUN npx --yes playwright install --with-deps chromium
ENV NODE_ENV=production
# De SQLite-database hoort op het persistente volume te staan; zonder deze
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
# en bij elke redeploy alle data verliezen.
ENV DB_PATH=/data/korting.db
VOLUME /data
EXPOSE 3000
CMD ["node", "server.js"]
```
- [ ] **Step 2: Update `README.md`**
Add one short paragraph after the existing "Hoe het werkt" section explaining the fallback (read the current `README.md` first to place it consistently with the existing tone/structure):
```markdown
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
die de prijs pas met JavaScript renderen, of die eenvoudige HTTP-clients
blokkeren. Geen garantie tegen elke vorm van bot-detectie, maar dekt een
reëel deel van de gevallen die de gewone aanpak mist.
```
- [ ] **Step 3: Manual build/run verification (this is the first task that needs a real browser)**
Run:
```bash
docker build -t korting .
```
Expected: build succeeds; watch for the `playwright install --with-deps chromium` step completing without error (it will print apt package installation output — that's expected).
Run:
```bash
docker run --rm -d --name korting-fallback-smoketest -p 3002:3000 \
-e BASIC_AUTH_USER=admin -e BASIC_AUTH_PASS=test \
-e SMTP_HOST=mail.sjoerd.app -e SMTP_PORT=587 \
-e SMTP_FROM=korting@sjoerd.app -e NOTIFY_EMAIL=test@example.com \
-e DB_PATH=/data/korting.db -v korting-fallback-smoketest-data:/data \
korting
sleep 2
curl -u admin:test -X POST http://localhost:3002/products \
-d "name=Fallback test" -d "url=https://example.com"
curl -u admin:test -X POST http://localhost:3002/check-now
docker logs korting-fallback-smoketest 2>&1 | tail -20
```
Expected: the `check-now` request succeeds (302); the container logs show the `[check] 1 product(en) gecontroleerd: ...` summary line. This URL (`https://example.com`) has no parseable price, so the primary fetch will return `not_found`, which **does** trigger the fallback — confirm the logs show no crash and no unhandled Playwright error (a `not_found` after the fallback also finds nothing is the expected, correct outcome here — the goal of this check is proving the browser actually launches and runs inside the container without error, not that this particular blank test page has a price).
Clean up:
```bash
docker stop korting-fallback-smoketest
docker volume rm korting-fallback-smoketest-data
docker rmi korting
```
- [ ] **Step 4: Commit**
```bash
git add Dockerfile README.md
git commit -m "chore: install Chromium in the Docker image for the fallback scraper"
```
---
## After this plan
Deploy the merged result to Coolify (redeploy the existing `korting.sjoerd.app` application — same app, no new resources needed) via the `deploying-to-coolify` skill, and verify against the actual reported failing URL
(`https://www.iciparisxl.nl/chanel/bleu-de-chanel/parfum-verstuiver/p/BP_967866`)
using Coolify's deployment/application logs — not a direct request from this
machine, since the user has since rotated the app's basic-auth credentials
and testing now happens on their side.
@@ -0,0 +1,128 @@
# Korting — Playwright-fallback voor prijs-extractie
**Datum:** 2026-09-05
**Status:** Approved
**Bouwt voort op:** `2026-09-05-korting-prijstracker-design.md`
## Aanleiding
Een gebruiker meldde dat `fetchAndExtractPrice` faalt voor
`https://www.iciparisxl.nl/...` met `fetch_error HTTP 403`. Onderzoek
(systematic-debugging) toonde aan dat dit domein achter **Akamai Bot
Manager** draait: elk verzoek dat niet als "echte browser" wordt herkend
krijgt al op de edge een `403 Access Denied` (`server: AkamaiGHost`),
vóórdat de eigenlijke website bereikt wordt. Bevestigd zowel met een
directe `curl`-test (ook met browser-achtige User-Agent) als via de live
app zelf (identieke `fetch_error HTTP 403` in de logs). Dit is geen bug in
de bestaande scraper-logica — die blijft correct voor de meeste shops —
maar een structurele blokkade die headers alleen niet oplossen.
**Belangrijk:** een headless-browser fallback is geen garantie tegen
Akamai specifiek (die kan ook headless browsers detecteren), maar is de
enige realistische route voor sites die de prijs pas client-side
renderen, en verbetert de dekking voor dat bredere geval.
## Doel
Wanneer de bestaande `fetchAndExtractPrice` geen prijs oplevert (fout of
`price: null`), probeert het systeem het opnieuw met een echte headless
Chromium-browser (Playwright) voordat de check als mislukt wordt
geregistreerd.
## Architectuur
Nieuwe module `src/browserScraper.js`, losstaand van `src/scraper.js`
(die blijft ongewijzigd — alle bestaande tests blijven geldig):
- `launchBrowser() -> Promise<Browser>` — start één Chromium-instance
(Playwright).
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 }) ->
Promise<{ price: number|null, method: string|null, error: Error|null
}>` — opent een nieuwe pagina op de meegegeven browser, navigeert naar
`url` (timeout `timeoutMs`), leest de gerenderde HTML via
`page.content()`, en hergebruikt `extractPrice()` uit `src/scraper.js`
(JSON-LD → meta → regex, exact dezelfde tiers) om de prijs eruit te
halen. Sluit de pagina altijd (try/finally), nooit de browser zelf.
## Wijziging in `checkProducts.js`
`checkProduct(product, deps)` krijgt een **optionele** extra dependency:
`deps.fetchAndExtractPriceViaBrowser(url) ->
Promise<{price,method,error}>` (curried met de browser-instance door de
composition root — `checkProducts.js` weet niets van Playwright zelf,
blijft puur dependency-injected zoals nu).
Logica: eerst zoals nu `fetchAndExtractPrice(product.url)` aanroepen. Als
dat resultaat een `error` heeft **of** `price === null`, én
`deps.fetchAndExtractPriceViaBrowser` is meegegeven, probeer de fallback
en gebruik dát resultaat verder (ook als de fallback zelf ook faalt — de
bestaande classificatie hieronder werkt op wat dan ook het laatste
resultaat is). **Fallback wordt nooit aangeroepen als de eerste poging al
een prijs opleverde.**
De rest van de functie (baseline/unchanged/changed+mail/mail_error/
fetch_error/not_found, en de regel dat `last_price`/`last_checked_at`
alleen bij een volledig geslaagde check worden bijgewerkt) blijft
ongewijzigd.
## Lifecycle (in `server.js`)
Eén browser per `runCheck()`-aanroep (zowel de dagelijkse cron als de
"nu controleren"-knop delen dit patroon, want beide roepen `runCheck()`
aan) — **lazy**: de browser start pas op het moment dat de eerste
fallback binnen die run daadwerkelijk nodig is (dus niet als alle
producten al gewoon ophaalbaar zijn via de normale fetch). Eenmaal
gestart wordt diezelfde instance hergebruikt voor de rest van die run
(nieuwe pagina per product), en aan het eind van de run altijd gesloten
in een `finally` — nooit een browser die de hele dag blijft openstaan, en
geen onnodige browser-start op runs die 'm niet nodig hebben.
## Docker
De runtime-stage van de Dockerfile installeert Chromium + de benodigde
systeemlibraries via `npx playwright install --with-deps chromium`, ná
het kopiëren van de app (zodat de juiste Playwright-versie uit
`node_modules` gebruikt wordt). `playwright` wordt een production
dependency (niet dev) zodat `npm ci --omit=dev` in de build-stage 'm wél
meeneemt naar de runtime-stage. Dit vergroot de image en de build-tijd
merkbaar (geaccepteerd trade-off, al gecommuniceerd).
## Error handling
- `page.goto` faalt of timet uit (`timeoutMs`, default 20000ms) → fallback
retourneert `{price: null, method: null, error}`, exact zelfde vorm als
de bestaande `fetchAndExtractPrice`-fouten. Geen crash van de run.
- Pagina wordt altijd gesloten, ook bij een gooiende `page.goto`
(try/finally), zodat er geen paginas/geheugen lekken binnen een run met
meerdere fallback-pogingen.
- Browser wordt altijd gesloten aan het eind van `runCheck()`, ook als er
onderweg een fout optreedt (try/finally rond de hele check-run).
## Testing
- `test/browserScraper.test.js`: test tegen een **nep** browser/pagina-
object (zelfde dependency-injection-patroon als de rest van de
codebase — geen echte Chromium nodig in de testrun): succesvolle
extractie, `page.goto` die faalt/timet uit, en dat `page.close()`
altijd wordt aangeroepen (ook bij een fout).
- `test/checkProducts.test.js`: nieuwe tests die bevestigen dat de
fallback (a) wordt aangeroepen wanneer de eerste poging faalt, (b)
**niet** wordt aangeroepen wanneer de eerste poging al een prijs
oplevert, en (c) dat bestaande tests zonder
`fetchAndExtractPriceViaBrowser` in de deps ongewijzigd blijven werken
(de dependency is optioneel, achterwaarts compatibel).
- Geen geautomatiseerde test voor de Dockerfile-wijziging zelf — net als
bij de oorspronkelijke Dockerfile-taak, een handmatige
build+run-verificatie (ditmaal ook een handmatige check dat Chromium
daadwerkelijk een pagina kan laden binnen de container).
## Out of scope
- Geen garantie dat dit specifieke Akamai-domein (`iciparisxl.nl`)
hierdoor gaat werken — dat hangt af van Akamai's detectie-niveau voor
dat domein op dat moment. Dit wordt niet apart getest tegen dat
specifieke domein als onderdeel van de geautomatiseerde testsuite (te
broos/afhankelijk van een externe, veranderlijke anti-bot-dienst); wel
handmatig te proberen na deploy.
- Playwright-stealth-plugins of andere anti-detectie-technieken — niet
gevraagd, voegt complexiteit toe zonder gegarandeerd resultaat.
+42 -1
View File
@@ -10,10 +10,12 @@
"dependencies": {
"better-sqlite3": "^11.3.0",
"cheerio": "^1.0.0",
"dotenv": "^17.4.2",
"express": "^4.19.2",
"express-basic-auth": "^1.2.1",
"node-cron": "^3.0.3",
"nodemailer": "^6.9.14"
"nodemailer": "^6.9.14",
"playwright": "^1.63.0"
},
"devDependencies": {
"supertest": "^7.0.0"
@@ -527,6 +529,18 @@
"url": "https://github.com/fb55/domutils?sponsor=1"
}
},
"node_modules/dotenv": {
"version": "17.4.2",
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
"integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
"license": "BSD-2-Clause",
"engines": {
"node": ">=12"
},
"funding": {
"url": "https://dotenvx.com"
}
},
"node_modules/dunder-proto": {
"version": "1.0.1",
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
@@ -1272,6 +1286,33 @@
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
"license": "MIT"
},
"node_modules/playwright": {
"version": "1.63.0",
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz",
"integrity": "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==",
"license": "Apache-2.0",
"dependencies": {
"playwright-core": "1.63.0"
},
"bin": {
"playwright": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/playwright-core": {
"version": "1.63.0",
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.63.0.tgz",
"integrity": "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==",
"license": "Apache-2.0",
"bin": {
"playwright-core": "cli.js"
},
"engines": {
"node": ">=20"
}
},
"node_modules/prebuild-install": {
"version": "7.1.3",
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
+3 -1
View File
@@ -11,10 +11,12 @@
"dependencies": {
"better-sqlite3": "^11.3.0",
"cheerio": "^1.0.0",
"dotenv": "^17.4.2",
"express": "^4.19.2",
"express-basic-auth": "^1.2.1",
"node-cron": "^3.0.3",
"nodemailer": "^6.9.14"
"nodemailer": "^6.9.14",
"playwright": "^1.63.0"
},
"devDependencies": {
"supertest": "^7.0.0"
+108
View File
@@ -0,0 +1,108 @@
// Leest een .env-bestand in voor lokale ontwikkeling; in productie komen de
// env vars van Coolify. quiet houdt de opstartlogging schoon.
require('dotenv').config({ quiet: true });
const path = require('node:path');
const {
openDb, listProducts, addProduct, deleteProduct,
recordCheckSuccess, recordCheckFailure,
} = require('./src/db');
const { fetchAndExtractPrice } = require('./src/scraper');
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
const { createTransport, sendPriceChangeEmail } = require('./src/mailer');
const { checkAllProducts } = require('./src/checkProducts');
const { start: startScheduler } = require('./src/scheduler');
const { createApp } = require('./src/app');
const REQUIRED_ENV = ['BASIC_AUTH_USER', 'BASIC_AUTH_PASS', 'SMTP_FROM', 'NOTIFY_EMAIL'];
const missing = REQUIRED_ENV.filter((name) => !process.env[name]);
if (missing.length > 0) {
console.error(`Ontbrekende verplichte environment variables: ${missing.join(', ')}`);
console.error('Zet ze (bijvoorbeeld in .env, zie .env.example) en start opnieuw.');
process.exit(1);
}
const PORT = process.env.PORT || 3000;
const DB_PATH = process.env.DB_PATH || path.join(__dirname, 'data', 'korting.db');
const db = openDb(DB_PATH);
const transport = createTransport(process.env);
function summarize(results) {
const counts = {};
for (const result of results) {
counts[result.status] = (counts[result.status] || 0) + 1;
}
const parts = Object.entries(counts).map(([status, count]) => `${status}=${count}`);
return parts.length > 0 ? parts.join(' ') : 'geen producten';
}
let inFlightCheck = null;
function runCheck() {
if (inFlightCheck) {
return inFlightCheck;
}
inFlightCheck = runCheckOnce().finally(() => {
inFlightCheck = null;
});
return inFlightCheck;
}
async function runCheckOnce() {
const products = listProducts(db);
let browserPromise = null;
function getBrowser() {
if (!browserPromise) {
browserPromise = launchBrowser();
}
return browserPromise;
}
try {
const deps = {
fetchAndExtractPrice,
fetchAndExtractPriceViaBrowser: async (url) => {
try {
const browser = await getBrowser();
return await fetchAndExtractPriceViaBrowser(browser, url);
} catch (err) {
return { price: null, method: null, error: err };
}
},
sendMail: (payload) => sendPriceChangeEmail(transport, {
from: process.env.SMTP_FROM,
to: process.env.NOTIFY_EMAIL,
...payload,
}),
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
now: () => new Date().toISOString(),
};
const results = await checkAllProducts(products, deps);
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
return results;
} finally {
if (browserPromise) {
const browser = await browserPromise.catch(() => null);
if (browser) {
await browser.close().catch((err) => {
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
});
}
}
}
}
startScheduler(runCheck);
const app = createApp({
listProducts: () => listProducts(db),
addProduct: (data) => addProduct(db, data),
deleteProduct: (id) => deleteProduct(db, id),
checkAllProducts: runCheck,
authUser: process.env.BASIC_AUTH_USER,
authPass: process.env.BASIC_AUTH_PASS,
});
app.listen(PORT, () => {
console.log(`Korting draait op poort ${PORT}`);
});
+50
View File
@@ -0,0 +1,50 @@
const express = require('express');
const basicAuth = require('express-basic-auth');
const { renderIndex } = require('./views');
function createApp(deps) {
const {
listProducts, addProduct, deleteProduct,
checkAllProducts, authUser, authPass,
} = deps;
const app = express();
app.use(express.urlencoded({ extended: false }));
app.use(basicAuth({ users: { [authUser]: authPass }, challenge: true }));
app.get('/', (req, res) => {
const products = listProducts();
const message = req.query.checked ? 'Controle uitgevoerd.' : undefined;
res.send(renderIndex({ products, message }));
});
app.post('/products', (req, res) => {
const { name, url } = req.body;
if (name && url) {
addProduct({ name, url });
}
res.redirect('/');
});
app.post('/products/:id/delete', (req, res) => {
deleteProduct(Number(req.params.id));
res.redirect('/');
});
// Express 4 vangt een afgewezen promise uit een async handler niet af, en
// Node 22 laat het proces daarop crashen — dus zelf afvangen.
app.post('/check-now', async (req, res) => {
try {
const products = listProducts();
await checkAllProducts(products);
res.redirect('/?checked=1');
} catch (err) {
console.error(`[check-now] mislukt: ${err && err.message}`);
res.status(500).send('Er ging iets mis bij het controleren van de prijzen.');
}
});
return app;
}
module.exports = { createApp };
+25
View File
@@ -0,0 +1,25 @@
const { chromium } = require('playwright');
const { extractPrice } = require('./scraper');
async function launchBrowser() {
return chromium.launch({ headless: true });
}
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
let page;
try {
page = await browser.newPage();
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
const html = await page.content();
const { price, method } = extractPrice(html);
return { price, method, error: null };
} catch (err) {
return { price: null, method: null, error: err };
} finally {
if (page) {
await page.close().catch(() => {});
}
}
}
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
+59
View File
@@ -0,0 +1,59 @@
async function checkProduct(product, deps) {
const {
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
sendMail, recordCheckSuccess, recordCheckFailure, now,
} = deps;
let result = await fetchAndExtractPrice(product.url);
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
result = await fetchAndExtractPriceViaBrowser(product.url);
}
if (result.error) {
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
recordCheckFailure(product.id, 'fetch_error');
return { status: 'fetch_error' };
}
if (result.price == null) {
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
recordCheckFailure(product.id, 'not_found');
return { status: 'not_found' };
}
const isBaseline = product.last_price == null;
const changed = !isBaseline && result.price !== product.last_price;
if (changed) {
try {
await sendMail({
name: product.name,
url: product.url,
oldPrice: product.last_price,
newPrice: result.price,
});
} catch (err) {
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
recordCheckFailure(product.id, 'mail_error');
return { status: 'mail_error' };
}
}
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
return { status: 'ok', changed };
}
async function checkAllProducts(products, deps) {
const results = [];
for (const product of products) {
try {
const result = await checkProduct(product, deps);
results.push({ productId: product.id, ...result });
} catch (err) {
console.error(`[check] error "${product.name}" (${product.url}): ${err && err.message}`);
results.push({ productId: product.id, status: 'error', error: err });
}
}
return results;
}
module.exports = { checkProduct, checkAllProducts };
+10
View File
@@ -0,0 +1,10 @@
const cronDefault = require('node-cron');
const SCHEDULE = '0 8 * * *';
const TIMEZONE = 'Europe/Amsterdam';
function start(runCheck, { cron = cronDefault } = {}) {
return cron.schedule(SCHEDULE, runCheck, { timezone: TIMEZONE });
}
module.exports = { start, SCHEDULE, TIMEZONE };
+44 -6
View File
@@ -4,10 +4,32 @@ function roundMoney(n) {
return Math.round(n * 100) / 100;
}
function parsePriceValue(raw) {
// Turns a human-written amount into a plain JS-parsable number string.
// When both separators occur, the last one is the decimal separator
// ("1.299,00" and "1,299.00" both mean 1299.00); a lone comma is decimal
// only when exactly two digits follow it, otherwise it groups thousands.
function normalizeAmount(raw) {
if (raw == null) return null;
const cleaned = String(raw).trim().replace(',', '.');
const value = parseFloat(cleaned);
const match = String(raw).trim().match(/-?\d[\d.,]*/);
if (!match) return null;
const str = match[0].replace(/[.,]+$/, '');
const lastDot = str.lastIndexOf('.');
const lastComma = str.lastIndexOf(',');
if (lastDot !== -1 && lastComma !== -1) {
return lastComma > lastDot
? str.replace(/\./g, '').replace(',', '.')
: str.replace(/,/g, '');
}
if (lastComma !== -1) {
return /,\d{2}$/.test(str) ? str.replace(',', '.') : str.replace(/,/g, '');
}
return str;
}
function parsePriceValue(raw) {
const normalized = normalizeAmount(raw);
if (normalized == null) return null;
const value = parseFloat(normalized);
return Number.isFinite(value) ? roundMoney(value) : null;
}
@@ -59,9 +81,14 @@ function findPriceInMeta($) {
return null;
}
// Thousands-grouped amounts first, then a plain amount, so "€1.299,00"
// is not truncated to "€1.29".
const TEXT_PRICE_RE = /€\s?(\d{1,3}(?:[.,]\d{3})+[.,]\d{2}|\d+[.,]\d{2})/;
function findPriceInText($) {
const text = $('body').text();
const match = text.match(/€\s?(\d+[.,]\d{2})/);
const text = $('body').clone().find('script, style, noscript').remove().end()
.text();
const match = text.match(TEXT_PRICE_RE);
return match ? parsePriceValue(match[1]) : null;
}
@@ -76,7 +103,18 @@ function extractPrice(html) {
return { price: null, method: null };
}
async function fetchAndExtractPrice(url, { fetchImpl = fetch } = {}) {
const FETCH_TIMEOUT_MS = 15000;
const USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 '
+ '(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36';
function defaultFetch(url) {
return fetch(url, {
signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
headers: { 'User-Agent': USER_AGENT },
});
}
async function fetchAndExtractPrice(url, { fetchImpl = defaultFetch } = {}) {
let response;
try {
response = await fetchImpl(url);
+63
View File
@@ -0,0 +1,63 @@
function escapeHtml(str) {
return String(str).replace(/[&<>"']/g, (c) => ({
'&': '&amp;', '<': '&lt;', '>': '&gt;', '"': '&quot;', "'": '&#39;',
}[c]));
}
function formatPrice(price) {
return price == null ? '—' : `${Number(price).toFixed(2).replace('.', ',')}`;
}
function renderProductRow(product) {
return `
<tr>
<td>${escapeHtml(product.name)}</td>
<td><a href="${escapeHtml(product.url)}" target="_blank" rel="noopener">bekijken</a></td>
<td>${formatPrice(product.last_price)}</td>
<td>${product.last_checked_at ? escapeHtml(product.last_checked_at) : '—'}</td>
<td>${escapeHtml(product.last_check_status || '—')}</td>
<td>
<form method="post" action="/products/${product.id}/delete" onsubmit="return confirm('Product verwijderen?');">
<button type="submit">Verwijder</button>
</form>
</td>
</tr>`;
}
function renderIndex({ products, message }) {
return `<!doctype html>
<html lang="nl">
<head>
<meta charset="utf-8">
<title>Korting — prijstracker</title>
<style>
body { font-family: system-ui, sans-serif; max-width: 800px; margin: 2rem auto; padding: 0 1rem; }
table { width: 100%; border-collapse: collapse; margin-top: 1rem; }
th, td { text-align: left; padding: 0.5rem; border-bottom: 1px solid #ddd; }
.message { background: #eef; padding: 0.5rem 1rem; border-radius: 4px; }
</style>
</head>
<body>
<h1>Korting — prijstracker</h1>
${message ? `<p class="message">${escapeHtml(message)}</p>` : ''}
<form method="post" action="/products">
<input name="name" placeholder="Naam" required>
<input name="url" type="url" placeholder="https://..." required>
<button type="submit">Toevoegen</button>
</form>
<form method="post" action="/check-now">
<button type="submit">Nu controleren</button>
</form>
<table>
<thead>
<tr><th>Naam</th><th>URL</th><th>Prijs</th><th>Laatst gecontroleerd</th><th>Status</th><th></th></tr>
</thead>
<tbody>
${products.map(renderProductRow).join('')}
</tbody>
</table>
</body>
</html>`;
}
module.exports = { renderIndex, escapeHtml, formatPrice };
+87
View File
@@ -0,0 +1,87 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const request = require('supertest');
const { createApp } = require('../src/app');
function makeDeps(overrides = {}) {
let products = [];
let nextId = 1;
return {
listProducts: () => products,
addProduct: ({ name, url }) => {
const product = { id: nextId++, name, url, last_price: null, last_checked_at: null, last_check_status: null };
products.push(product);
return product;
},
deleteProduct: (id) => { products = products.filter((p) => p.id !== id); },
checkAllProducts: async () => [],
authUser: 'admin',
authPass: 'secret',
...overrides,
};
}
test('rejects requests without basic auth credentials', async () => {
const app = createApp(makeDeps());
const res = await request(app).get('/');
assert.equal(res.status, 401);
});
test('lists products for authenticated requests', async () => {
const deps = makeDeps();
deps.addProduct({ name: 'Voorbeeld', url: 'https://example.com' });
const app = createApp(deps);
const res = await request(app).get('/').auth('admin', 'secret');
assert.equal(res.status, 200);
assert.match(res.text, /Voorbeeld/);
});
test('adding a product redirects to the list', async () => {
const deps = makeDeps();
const app = createApp(deps);
const res = await request(app)
.post('/products')
.auth('admin', 'secret')
.type('form')
.send({ name: 'Nieuw product', url: 'https://shop.example/p' });
assert.equal(res.status, 302);
assert.equal(res.headers.location, '/');
const list = await request(app).get('/').auth('admin', 'secret');
assert.match(list.text, /Nieuw product/);
});
test('deleting a product removes it from the list', async () => {
const deps = makeDeps();
const product = deps.addProduct({ name: 'Weg ermee', url: 'https://shop.example/p2' });
const app = createApp(deps);
const res = await request(app).post(`/products/${product.id}/delete`).auth('admin', 'secret');
assert.equal(res.status, 302);
const list = await request(app).get('/').auth('admin', 'secret');
assert.doesNotMatch(list.text, /Weg ermee/);
});
test('check-now triggers checkAllProducts and redirects with confirmation', async () => {
let called = false;
const deps = makeDeps({ checkAllProducts: async () => { called = true; return []; } });
const app = createApp(deps);
const res = await request(app).post('/check-now').auth('admin', 'secret');
assert.equal(res.status, 302);
assert.equal(res.headers.location, '/?checked=1');
assert.equal(called, true);
});
test('check-now answers 500 when the check fails, without crashing the process', async (t) => {
t.mock.method(console, 'error', () => {});
const deps = makeDeps({
checkAllProducts: async () => { throw new Error('db weg'); },
});
deps.addProduct({ name: 'Nog steeds hier', url: 'https://shop.example/p3' });
const app = createApp(deps);
const res = await request(app).post('/check-now').auth('admin', 'secret');
assert.equal(res.status, 500);
assert.match(res.text, /Er ging iets mis/);
// Same process, same app instance: proves nothing crashed.
const list = await request(app).get('/').auth('admin', 'secret');
assert.equal(list.status, 200);
assert.match(list.text, /Nog steeds hier/);
});
+70
View File
@@ -0,0 +1,70 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
let closed = false;
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
const page = {
goto: async () => {},
content: async () => html,
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
content: async () => '',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.equal(result.price, null);
assert.equal(result.method, null);
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => '<html><body><p>Geen prijs</p></body></html>',
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.deepEqual(result, { price: null, method: null, error: null });
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
let closed = false;
const page = {
goto: async () => {},
content: async () => { throw new Error('content failed'); },
close: async () => { closed = true; },
};
const browser = { newPage: async () => page };
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
assert.ok(result.error instanceof Error);
assert.equal(closed, true);
});
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
let receivedOptions;
const page = {
goto: async (url, options) => { receivedOptions = options; },
content: async () => '<html><body></body></html>',
close: async () => {},
};
const browser = { newPage: async () => page };
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
assert.equal(receivedOptions.timeout, 5000);
});
+184
View File
@@ -0,0 +1,184 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const { checkProduct, checkAllProducts } = require('../src/checkProducts');
function makeDeps(overrides = {}) {
const failures = [];
const successes = [];
const mailCalls = [];
const deps = {
fetchAndExtractPrice: async () => ({ price: 10, method: 'meta', error: null }),
sendMail: async (payload) => { mailCalls.push(payload); },
recordCheckFailure: (id, status) => failures.push({ id, status }),
recordCheckSuccess: (id, data) => successes.push({ id, ...data }),
now: () => '2026-09-05T08:00:00.000Z',
...overrides,
};
return { deps, failures, successes, mailCalls };
}
// Failure paths log to stderr; capture it so the test output stays clean
// while still proving the logging happened.
function captureErrorLog(t) {
const logged = [];
t.mock.method(console, 'error', (...args) => { logged.push(args.join(' ')); });
return logged;
}
test('checkProduct stores baseline price without sending mail', async () => {
const { deps, successes, mailCalls } = makeDeps();
const product = { id: 1, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.equal(result.changed, false);
assert.equal(mailCalls.length, 0);
assert.deepEqual(successes, [{ id: 1, price: 10, checkedAt: '2026-09-05T08:00:00.000Z' }]);
});
test('checkProduct sends mail and updates price when price changed', async () => {
const { deps, successes, mailCalls } = makeDeps();
const product = { id: 2, name: 'X', url: 'https://x', last_price: 15 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.equal(result.changed, true);
assert.equal(mailCalls.length, 1);
assert.deepEqual(mailCalls[0], { name: 'X', url: 'https://x', oldPrice: 15, newPrice: 10 });
assert.equal(successes.length, 1);
});
test('checkProduct does not send mail or update when price unchanged', async () => {
const { deps, successes, mailCalls } = makeDeps();
const product = { id: 3, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.equal(result.changed, false);
assert.equal(mailCalls.length, 0);
assert.equal(successes.length, 1);
});
test('checkProduct records fetch_error and does not touch price on network failure', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
});
const product = { id: 4, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'fetch_error');
assert.deepEqual(failures, [{ id: 4, status: 'fetch_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /fetch_error.*https:\/\/x.*down/);
});
test('checkProduct records not_found when no price could be extracted', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
});
const product = { id: 5, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'not_found');
assert.deepEqual(failures, [{ id: 5, status: 'not_found' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /not_found.*https:\/\/x/);
});
test('checkProduct records mail_error and keeps old price when sending mail fails', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
sendMail: async () => { throw new Error('smtp down'); },
});
const product = { id: 6, name: 'X', url: 'https://x', last_price: 15 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'mail_error');
assert.deepEqual(failures, [{ id: 6, status: 'mail_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /mail_error.*https:\/\/x.*smtp down/);
});
test('checkAllProducts processes every product independently, isolating failures', async (t) => {
captureErrorLog(t);
const { deps } = makeDeps();
let calls = 0;
const throwingDeps = {
...deps,
fetchAndExtractPrice: async () => {
calls += 1;
if (calls === 1) throw new Error('boom');
return { price: 10, method: 'meta', error: null };
},
};
const products = [
{ id: 1, name: 'A', url: 'https://a', last_price: null },
{ id: 2, name: 'B', url: 'https://b', last_price: null },
];
const results = await checkAllProducts(products, throwingDeps);
assert.equal(results.length, 2);
assert.equal(results[0].status, 'error');
assert.equal(results[1].status, 'ok');
});
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes, mailCalls } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 10, method: 'json-ld', error: null };
},
});
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
assert.equal(mailCalls.length, 0);
});
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
captureErrorLog(t);
const browserCalls = [];
const { deps, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 12, method: 'regex', error: null };
},
});
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'ok');
assert.deepEqual(browserCalls, ['https://x']);
assert.equal(successes.length, 1);
});
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
const browserCalls = [];
const { deps } = makeDeps({
fetchAndExtractPriceViaBrowser: async (url) => {
browserCalls.push(url);
return { price: 999, method: 'meta', error: null };
},
});
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
await checkProduct(product, deps);
assert.equal(browserCalls.length, 0);
});
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
const logged = captureErrorLog(t);
const { deps, failures, successes } = makeDeps({
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
});
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
const result = await checkProduct(product, deps);
assert.equal(result.status, 'fetch_error');
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
assert.equal(successes.length, 0);
assert.equal(logged.length, 1);
assert.match(logged[0], /fetch_error.*still down/);
});
+5
View File
@@ -0,0 +1,5 @@
<html><body>
<script>var config = {"fallbackPrice": "€9,99"};</script>
<style>.price:after { content: "€1,11"; }</style>
<p>Onze prijs: &euro;49,95 vandaag</p>
</body></html>
+3
View File
@@ -0,0 +1,3 @@
<html><head>
<meta property="product:price:amount" content="1.299,00" />
</head><body><p>Duur product</p></body></html>
+1
View File
@@ -0,0 +1 @@
<html><body><p>Nu voor &euro;1.299,00 incl. btw</p></body></html>
+21
View File
@@ -0,0 +1,21 @@
const test = require('node:test');
const assert = require('node:assert/strict');
const { start, SCHEDULE, TIMEZONE } = require('../src/scheduler');
test('start schedules the daily check with the expected cron expression and timezone', () => {
const calls = [];
const fakeCron = {
schedule: (expression, fn, options) => {
calls.push({ expression, fn, options });
return { stop: () => {} };
},
};
const runCheck = async () => {};
start(runCheck, { cron: fakeCron });
assert.equal(calls.length, 1);
assert.equal(calls[0].expression, SCHEDULE);
assert.equal(SCHEDULE, '0 8 * * *');
assert.equal(calls[0].options.timezone, TIMEZONE);
assert.equal(TIMEZONE, 'Europe/Amsterdam');
assert.equal(calls[0].fn, runCheck);
});
+39 -1
View File
@@ -2,7 +2,7 @@ const test = require('node:test');
const assert = require('node:assert/strict');
const fs = require('node:fs');
const path = require('node:path');
const { extractPrice, fetchAndExtractPrice } = require('../src/scraper');
const { extractPrice, fetchAndExtractPrice, parsePriceValue } = require('../src/scraper');
function loadFixture(name) {
return fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');
@@ -28,6 +28,31 @@ test('extractPrice returns null price when nothing matches', () => {
assert.deepEqual(result, { price: null, method: null });
});
test('parsePriceValue handles both thousands-separator conventions', () => {
assert.equal(parsePriceValue('1.299,00'), 1299.00);
assert.equal(parsePriceValue('1,299.00'), 1299.00);
assert.equal(parsePriceValue('19,99'), 19.99);
assert.equal(parsePriceValue('19.99'), 19.99);
assert.equal(parsePriceValue('1.234.567,89'), 1234567.89);
assert.equal(parsePriceValue(null), null);
assert.equal(parsePriceValue('geen prijs'), null);
});
test('extractPrice reads a thousands-grouped price from a meta tag', () => {
const result = extractPrice(loadFixture('thousands-meta.html'));
assert.deepEqual(result, { price: 1299.00, method: 'meta' });
});
test('extractPrice reads a thousands-grouped price from page text', () => {
const result = extractPrice(loadFixture('thousands-regex.html'));
assert.deepEqual(result, { price: 1299.00, method: 'regex' });
});
test('extractPrice ignores euro amounts inside script and style tags', () => {
const result = extractPrice(loadFixture('script-decoy.html'));
assert.deepEqual(result, { price: 49.95, method: 'regex' });
});
test('fetchAndExtractPrice returns error when fetch rejects', async () => {
const fetchImpl = async () => { throw new Error('network down'); };
const result = await fetchAndExtractPrice('https://example.invalid', { fetchImpl });
@@ -42,6 +67,19 @@ test('fetchAndExtractPrice returns error on non-OK response', async () => {
assert.ok(result.error instanceof Error);
});
test('fetchAndExtractPrice returns error when the request times out', async () => {
const fetchImpl = async () => {
const err = new Error('The operation was aborted due to timeout');
err.name = 'TimeoutError';
throw err;
};
const result = await fetchAndExtractPrice('https://example.invalid', { fetchImpl });
assert.deepEqual(result.price, null);
assert.equal(result.method, null);
assert.ok(result.error instanceof Error);
assert.equal(result.error.name, 'TimeoutError');
});
test('fetchAndExtractPrice extracts price from fetched HTML', async () => {
const html = loadFixture('meta-tag.html');
const fetchImpl = async () => ({ ok: true, status: 200, text: async () => html });