Compare commits
20 Commits
e1f8576b95
...
main
| Author | SHA1 | Date | |
|---|---|---|---|
| 74f487486f | |||
| 7a117e7a46 | |||
| 966a80c690 | |||
| fc8dadc8a3 | |||
| 358347e8ba | |||
| df3d3250e5 | |||
| dac7893cd2 | |||
| 957555bdbe | |||
| 4fee15173b | |||
| 1d3d1caeb1 | |||
| 7f378115b5 | |||
| d07cda48c6 | |||
| fb3fc62b4b | |||
| 52c9947267 | |||
| 862699d35b | |||
| 195dff6fe3 | |||
| 5b6d672506 | |||
| f73ea41a6c | |||
| 3b0e824c5b | |||
| 68cb8d0c08 |
@@ -0,0 +1,6 @@
|
|||||||
|
node_modules
|
||||||
|
data
|
||||||
|
.env
|
||||||
|
.git
|
||||||
|
docs
|
||||||
|
test
|
||||||
@@ -1,3 +1,4 @@
|
|||||||
node_modules/
|
node_modules/
|
||||||
data/
|
data/
|
||||||
.env
|
.env
|
||||||
|
.claude/
|
||||||
|
|||||||
+23
@@ -0,0 +1,23 @@
|
|||||||
|
FROM node:22-slim AS build
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
WORKDIR /app
|
||||||
|
COPY package.json package-lock.json ./
|
||||||
|
RUN npm ci --omit=dev
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
FROM node:22-slim
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=build /app/package.json /app/package-lock.json ./
|
||||||
|
COPY --from=build /app/node_modules ./node_modules
|
||||||
|
RUN npx --yes playwright install --with-deps chromium \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
COPY --from=build /app .
|
||||||
|
ENV NODE_ENV=production
|
||||||
|
# De SQLite-database hoort op het persistente volume te staan; zonder deze
|
||||||
|
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
|
||||||
|
# en bij elke redeploy alle data verliezen.
|
||||||
|
ENV DB_PATH=/data/korting.db
|
||||||
|
VOLUME /data
|
||||||
|
EXPOSE 3000
|
||||||
|
CMD ["node", "server.js"]
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Korting — prijstracker
|
||||||
|
|
||||||
|
Houdt een lijst productpagina's dagelijks in de gaten en mailt wanneer
|
||||||
|
de prijs verandert.
|
||||||
|
|
||||||
|
## Lokaal draaien
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install
|
||||||
|
cp .env.example .env # env vars invullen
|
||||||
|
npm test
|
||||||
|
npm start
|
||||||
|
```
|
||||||
|
|
||||||
|
De app leest `.env` in via `dotenv`. `BASIC_AUTH_USER`, `BASIC_AUTH_PASS`,
|
||||||
|
`SMTP_FROM` en `NOTIFY_EMAIL` zijn verplicht; ontbreken ze, dan stopt de
|
||||||
|
app bij het opstarten met een melding.
|
||||||
|
|
||||||
|
De UI staat achter basic auth (`BASIC_AUTH_USER` / `BASIC_AUTH_PASS`) op
|
||||||
|
`http://localhost:3000`.
|
||||||
|
|
||||||
|
Wil je de headless-browser-fallback ook echt lokaal uitproberen (de
|
||||||
|
testsuite gebruikt fakes en heeft hier geen echte browser voor nodig), run
|
||||||
|
dan eerst `npx playwright install chromium` om de Chromium-binary te
|
||||||
|
downloaden.
|
||||||
|
|
||||||
|
## Environment variables
|
||||||
|
|
||||||
|
| Variabele | Omschrijving |
|
||||||
|
|---|---|
|
||||||
|
| `PORT` | HTTP-poort (default `3000`) |
|
||||||
|
| `DB_PATH` | Pad naar het SQLite-bestand (default `./data/korting.db`) |
|
||||||
|
| `BASIC_AUTH_USER` / `BASIC_AUTH_PASS` | Login voor de web-UI |
|
||||||
|
| `SMTP_HOST` / `SMTP_PORT` / `SMTP_USER` / `SMTP_PASS` | SMTP-server (Postal op `mail.sjoerd.app`) |
|
||||||
|
| `SMTP_FROM` | Afzenderadres van de notificatiemails |
|
||||||
|
| `NOTIFY_EMAIL` | Ontvangeradres van de notificatiemails |
|
||||||
|
|
||||||
|
## Hoe het werkt
|
||||||
|
|
||||||
|
Elke dag om 08:00 (Europe/Amsterdam) doorloopt de app alle producten,
|
||||||
|
haalt de huidige prijs op (JSON-LD → meta tags → regex-fallback), en
|
||||||
|
stuurt een email als de prijs is veranderd sinds de vorige geslaagde
|
||||||
|
check. Een mislukte poging (pagina onbereikbaar, geen prijs gevonden,
|
||||||
|
of de mail kon niet verstuurd worden) verandert de opgeslagen prijs
|
||||||
|
niet, zodat de volgende dag opnieuw wordt geprobeerd. Via de "Nu
|
||||||
|
controleren"-knop in de UI kan dit ook handmatig getriggerd worden.
|
||||||
|
|
||||||
|
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
|
||||||
|
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
|
||||||
|
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
|
||||||
|
die eenvoudige HTTP-clients blokkeren. Geen garantie tegen elke vorm van
|
||||||
|
bot-detectie, maar dekt een reëel deel van de gevallen die de gewone
|
||||||
|
aanpak mist.
|
||||||
@@ -0,0 +1,633 @@
|
|||||||
|
# Korting Playwright-fallback Implementation Plan
|
||||||
|
|
||||||
|
> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking.
|
||||||
|
|
||||||
|
**Goal:** When the existing plain-fetch price scraper fails (network error or no price found), retry once with a real headless-Chromium page load (Playwright) before recording the check as failed.
|
||||||
|
|
||||||
|
**Architecture:** A new, self-contained `src/browserScraper.js` module wraps Playwright and reuses `extractPrice()` from the existing `src/scraper.js` (no duplicated parsing logic). `checkProducts.js` gains one new *optional* injected dependency that it calls only when the primary fetch already failed — `scraper.js` itself is untouched. `server.js` lazily launches one browser per `runCheck()` run (only if a fallback is actually needed), reuses it for the rest of that run, and always closes it in a `finally`.
|
||||||
|
|
||||||
|
**Tech Stack:** Playwright (`chromium`), same Node.js 22 / CommonJS / `node --test` stack as the rest of the project.
|
||||||
|
|
||||||
|
**Spec:** `docs/superpowers/specs/2026-09-05-korting-playwright-fallback-design.md` (builds on `docs/superpowers/specs/2026-09-05-korting-prijstracker-design.md`)
|
||||||
|
|
||||||
|
## Global Constraints
|
||||||
|
|
||||||
|
- Node.js 22, CommonJS modules, no TypeScript — matches the existing codebase.
|
||||||
|
- `src/scraper.js` and its tests are **not modified** by this plan — the fallback is a separate module, invoked from `checkProducts.js`, not from inside `fetchAndExtractPrice`.
|
||||||
|
- The browser fallback is invoked **only** when the primary `fetchAndExtractPrice` result has `error` or `price === null` — never when it already found a price.
|
||||||
|
- Every existing test must keep passing unmodified in behavior (the new dependency is optional; omitting it from `deps` must not change any existing test's outcome).
|
||||||
|
- `playwright` is a production dependency (not `devDependencies`) — the Docker runtime stage needs it, not just the build stage.
|
||||||
|
- **Local `npm install` must not attempt to download the actual Chromium binary** — set `PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1` in the environment before running `npm install` so the postinstall step is skipped. The real browser binary is only installed inside the Docker image (Task 4) via `npx playwright install --with-deps chromium`. Tasks 1-3 never need a real Chromium binary in the dev/test sandbox — all automated tests use fake browser/page objects (dependency injection), and Task 3's manual check deliberately avoids exercising the fallback path for this reason. Task 4's Docker build/run verification is the first (and only) point in this plan where a real browser actually launches.
|
||||||
|
- Timeout for a fallback page load: 20000ms (`timeoutMs` default), passed to Playwright's `page.goto` timeout option.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 1: Browser-based scraper module
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `package.json` (add `playwright` dependency)
|
||||||
|
- Create: `src/browserScraper.js`
|
||||||
|
- Create: `test/browserScraper.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `extractPrice` from `src/scraper.js` (already exported, unchanged signature: `extractPrice(html) -> { price: number|null, method: string|null }`).
|
||||||
|
- Produces:
|
||||||
|
- `launchBrowser() -> Promise<Browser>` — thin wrapper around Playwright's `chromium.launch({ headless: true })`.
|
||||||
|
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) -> Promise<{ price: number|null, method: string|null, error: Error|null }>` — same return shape as `scraper.js`'s `fetchAndExtractPrice`, so later tasks can treat both interchangeably.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Add the `playwright` dependency**
|
||||||
|
|
||||||
|
Run (this environment must not attempt to download the Chromium binary — see Global Constraints):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
PLAYWRIGHT_SKIP_BROWSER_DOWNLOAD=1 npm install playwright
|
||||||
|
```
|
||||||
|
|
||||||
|
Expected: `package.json`'s `dependencies` gains a `playwright` entry (not `devDependencies`), `package-lock.json` updates, and the command completes without trying to download a browser binary.
|
||||||
|
|
||||||
|
- [ ] **Step 2: Write the failing tests**
|
||||||
|
|
||||||
|
Create `test/browserScraper.test.js`:
|
||||||
|
|
||||||
|
```js
|
||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => html,
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
|
||||||
|
content: async () => '',
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.equal(result.price, null);
|
||||||
|
assert.equal(result.method, null);
|
||||||
|
assert.ok(result.error instanceof Error);
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => '<html><body><p>Geen prijs</p></body></html>',
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.deepEqual(result, { price: null, method: null, error: null });
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => { throw new Error('content failed'); },
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.ok(result.error instanceof Error);
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
|
||||||
|
let receivedOptions;
|
||||||
|
const page = {
|
||||||
|
goto: async (url, options) => { receivedOptions = options; },
|
||||||
|
content: async () => '<html><body></body></html>',
|
||||||
|
close: async () => {},
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
|
||||||
|
assert.equal(receivedOptions.timeout, 5000);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Run tests to verify they fail**
|
||||||
|
|
||||||
|
Run: `npm test`
|
||||||
|
Expected: FAIL with "Cannot find module '../src/browserScraper'"
|
||||||
|
|
||||||
|
- [ ] **Step 4: Write `src/browserScraper.js`**
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { chromium } = require('playwright');
|
||||||
|
const { extractPrice } = require('./scraper');
|
||||||
|
|
||||||
|
async function launchBrowser() {
|
||||||
|
return chromium.launch({ headless: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
|
||||||
|
let page;
|
||||||
|
try {
|
||||||
|
page = await browser.newPage();
|
||||||
|
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
|
||||||
|
const html = await page.content();
|
||||||
|
const { price, method } = extractPrice(html);
|
||||||
|
return { price, method, error: null };
|
||||||
|
} catch (err) {
|
||||||
|
return { price: null, method: null, error: err };
|
||||||
|
} finally {
|
||||||
|
if (page) {
|
||||||
|
await page.close().catch(() => {});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 5: Run tests to verify they pass**
|
||||||
|
|
||||||
|
Run: `npm test`
|
||||||
|
Expected: PASS (all `browserScraper.test.js` tests + every previously-passing test, unchanged)
|
||||||
|
|
||||||
|
- [ ] **Step 6: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add package.json package-lock.json src/browserScraper.js test/browserScraper.test.js
|
||||||
|
git commit -m "feat: add Playwright-based fallback scraper"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 2: Wire the fallback into the price-check orchestration
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `src/checkProducts.js`
|
||||||
|
- Modify: `test/checkProducts.test.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `fetchAndExtractPriceViaBrowser`-shaped function (Task 1's return shape), passed as the new optional `deps.fetchAndExtractPriceViaBrowser(url) -> Promise<{price, method, error}>`. `checkProducts.js` does not `require` Task 1's module directly — the composition root (Task 3) curries in the real browser instance.
|
||||||
|
- Produces: `checkProduct`/`checkAllProducts` — same exported names and signatures as before; the only change is that `checkProduct`'s deps object now recognizes one more optional key.
|
||||||
|
|
||||||
|
The current `src/checkProducts.js` (for reference — this is what you are editing):
|
||||||
|
|
||||||
|
```js
|
||||||
|
async function checkProduct(product, deps) {
|
||||||
|
const { fetchAndExtractPrice, sendMail, recordCheckSuccess, recordCheckFailure, now } = deps;
|
||||||
|
const result = await fetchAndExtractPrice(product.url);
|
||||||
|
|
||||||
|
if (result.error) {
|
||||||
|
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
|
||||||
|
recordCheckFailure(product.id, 'fetch_error');
|
||||||
|
return { status: 'fetch_error' };
|
||||||
|
}
|
||||||
|
if (result.price == null) {
|
||||||
|
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
|
||||||
|
recordCheckFailure(product.id, 'not_found');
|
||||||
|
return { status: 'not_found' };
|
||||||
|
}
|
||||||
|
|
||||||
|
const isBaseline = product.last_price == null;
|
||||||
|
const changed = !isBaseline && result.price !== product.last_price;
|
||||||
|
|
||||||
|
if (changed) {
|
||||||
|
try {
|
||||||
|
await sendMail({
|
||||||
|
name: product.name,
|
||||||
|
url: product.url,
|
||||||
|
oldPrice: product.last_price,
|
||||||
|
newPrice: result.price,
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
|
||||||
|
recordCheckFailure(product.id, 'mail_error');
|
||||||
|
return { status: 'mail_error' };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
|
||||||
|
return { status: 'ok', changed };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkAllProducts(products, deps) {
|
||||||
|
const results = [];
|
||||||
|
for (const product of products) {
|
||||||
|
try {
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
results.push({ productId: product.id, ...result });
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check] error "${product.name}" (${product.url}): ${err && err.message}`);
|
||||||
|
results.push({ productId: product.id, status: 'error', error: err });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { checkProduct, checkAllProducts };
|
||||||
|
```
|
||||||
|
|
||||||
|
The current `test/checkProducts.test.js` (for reference — you are appending new tests to this file, not replacing the existing ones):
|
||||||
|
|
||||||
|
```js
|
||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { checkProduct, checkAllProducts } = require('../src/checkProducts');
|
||||||
|
|
||||||
|
function makeDeps(overrides = {}) {
|
||||||
|
const failures = [];
|
||||||
|
const successes = [];
|
||||||
|
const mailCalls = [];
|
||||||
|
const deps = {
|
||||||
|
fetchAndExtractPrice: async () => ({ price: 10, method: 'meta', error: null }),
|
||||||
|
sendMail: async (payload) => { mailCalls.push(payload); },
|
||||||
|
recordCheckFailure: (id, status) => failures.push({ id, status }),
|
||||||
|
recordCheckSuccess: (id, data) => successes.push({ id, ...data }),
|
||||||
|
now: () => '2026-09-05T08:00:00.000Z',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
return { deps, failures, successes, mailCalls };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Failure paths log to stderr; capture it so the test output stays clean
|
||||||
|
// while still proving the logging happened.
|
||||||
|
function captureErrorLog(t) {
|
||||||
|
const logged = [];
|
||||||
|
t.mock.method(console, 'error', (...args) => { logged.push(args.join(' ')); });
|
||||||
|
return logged;
|
||||||
|
}
|
||||||
|
|
||||||
|
// ... (existing tests, unchanged — do not modify or remove any of them)
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 1: Write the failing tests**
|
||||||
|
|
||||||
|
Append these tests to the end of `test/checkProducts.test.js` (after the existing `checkAllProducts processes every product independently...` test, before the final closing of the file):
|
||||||
|
|
||||||
|
```js
|
||||||
|
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
|
||||||
|
captureErrorLog(t);
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps, successes, mailCalls } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 10, method: 'json-ld', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.deepEqual(browserCalls, ['https://x']);
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
assert.equal(mailCalls.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
|
||||||
|
captureErrorLog(t);
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 12, method: 'regex', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.deepEqual(browserCalls, ['https://x']);
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps } = makeDeps({
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 999, method: 'meta', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
await checkProduct(product, deps);
|
||||||
|
assert.equal(browserCalls.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
|
||||||
|
const logged = captureErrorLog(t);
|
||||||
|
const { deps, failures, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
|
||||||
|
});
|
||||||
|
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'fetch_error');
|
||||||
|
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
|
||||||
|
assert.equal(successes.length, 0);
|
||||||
|
assert.equal(logged.length, 1);
|
||||||
|
assert.match(logged[0], /fetch_error.*still down/);
|
||||||
|
});
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Run tests to verify they fail**
|
||||||
|
|
||||||
|
Run: `npm test`
|
||||||
|
Expected: FAIL — the 4 new tests fail because `checkProduct` doesn't yet call `fetchAndExtractPriceViaBrowser` (e.g. the "tries the browser fallback" tests fail their `browserCalls` assertions with an empty array; the "never calls" test passes vacuously already, which is fine — it's a guard for the next step, not expected to fail).
|
||||||
|
|
||||||
|
- [ ] **Step 3: Modify `src/checkProducts.js`**
|
||||||
|
|
||||||
|
Change only the first few lines of `checkProduct` (everything from the baseline/changed logic onward stays exactly as it is):
|
||||||
|
|
||||||
|
```js
|
||||||
|
async function checkProduct(product, deps) {
|
||||||
|
const {
|
||||||
|
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
|
||||||
|
sendMail, recordCheckSuccess, recordCheckFailure, now,
|
||||||
|
} = deps;
|
||||||
|
let result = await fetchAndExtractPrice(product.url);
|
||||||
|
|
||||||
|
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
|
||||||
|
result = await fetchAndExtractPriceViaBrowser(product.url);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (result.error) {
|
||||||
|
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
|
||||||
|
recordCheckFailure(product.id, 'fetch_error');
|
||||||
|
return { status: 'fetch_error' };
|
||||||
|
}
|
||||||
|
if (result.price == null) {
|
||||||
|
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
|
||||||
|
recordCheckFailure(product.id, 'not_found');
|
||||||
|
return { status: 'not_found' };
|
||||||
|
}
|
||||||
|
|
||||||
|
const isBaseline = product.last_price == null;
|
||||||
|
const changed = !isBaseline && result.price !== product.last_price;
|
||||||
|
|
||||||
|
if (changed) {
|
||||||
|
try {
|
||||||
|
await sendMail({
|
||||||
|
name: product.name,
|
||||||
|
url: product.url,
|
||||||
|
oldPrice: product.last_price,
|
||||||
|
newPrice: result.price,
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
|
||||||
|
recordCheckFailure(product.id, 'mail_error');
|
||||||
|
return { status: 'mail_error' };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
|
||||||
|
return { status: 'ok', changed };
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
(`checkAllProducts` and `module.exports` are unchanged — leave them exactly as they are.)
|
||||||
|
|
||||||
|
- [ ] **Step 4: Run tests to verify they pass**
|
||||||
|
|
||||||
|
Run: `npm test`
|
||||||
|
Expected: PASS (all new tests + every previously-passing test, unchanged)
|
||||||
|
|
||||||
|
- [ ] **Step 5: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add src/checkProducts.js test/checkProducts.test.js
|
||||||
|
git commit -m "feat: fall back to the browser scraper when the primary fetch fails"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 3: Wire the browser lifecycle into the composition root
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `server.js`
|
||||||
|
|
||||||
|
**Interfaces:**
|
||||||
|
- Consumes: `launchBrowser`, `fetchAndExtractPriceViaBrowser` (Task 1); `checkProduct`'s new optional `fetchAndExtractPriceViaBrowser` deps key (Task 2).
|
||||||
|
- Produces: nothing new for other tasks to consume — `server.js` is the top of the dependency graph.
|
||||||
|
|
||||||
|
No new automated test — this is a composition-root change, verified with a manual smoke check (same pattern as the original server.js task). **This step does not require a real Chromium binary**: the smoke check below only exercises a product whose plain fetch succeeds, so the lazy browser launch never actually triggers.
|
||||||
|
|
||||||
|
- [ ] **Step 1: Modify `server.js`**
|
||||||
|
|
||||||
|
Add the import (alongside the other `require`s near the top):
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
|
||||||
|
```
|
||||||
|
|
||||||
|
Replace the `runCheck` function with a version that lazily launches one browser per run, only if a fallback is actually needed, and always closes it:
|
||||||
|
|
||||||
|
```js
|
||||||
|
async function runCheck() {
|
||||||
|
const products = listProducts(db);
|
||||||
|
let browserPromise = null;
|
||||||
|
function getBrowser() {
|
||||||
|
if (!browserPromise) {
|
||||||
|
browserPromise = launchBrowser();
|
||||||
|
}
|
||||||
|
return browserPromise;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const deps = {
|
||||||
|
fetchAndExtractPrice,
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
const browser = await getBrowser();
|
||||||
|
return fetchAndExtractPriceViaBrowser(browser, url);
|
||||||
|
},
|
||||||
|
sendMail: (payload) => sendPriceChangeEmail(transport, {
|
||||||
|
from: process.env.SMTP_FROM,
|
||||||
|
to: process.env.NOTIFY_EMAIL,
|
||||||
|
...payload,
|
||||||
|
}),
|
||||||
|
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
|
||||||
|
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
|
||||||
|
now: () => new Date().toISOString(),
|
||||||
|
};
|
||||||
|
const results = await checkAllProducts(products, deps);
|
||||||
|
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
|
||||||
|
return results;
|
||||||
|
} finally {
|
||||||
|
if (browserPromise) {
|
||||||
|
const browser = await browserPromise;
|
||||||
|
await browser.close().catch((err) => {
|
||||||
|
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
(Everything else in `server.js` — the required-env check, `openDb`, `createTransport`, `summarize`, `startScheduler(runCheck)`, `createApp({...})`, `app.listen(...)` — stays exactly as it is.)
|
||||||
|
|
||||||
|
- [ ] **Step 2: Manual smoke check (no real browser needed for this check)**
|
||||||
|
|
||||||
|
Deliberately **do not** call `/check-now` in this step: with the Task 2
|
||||||
|
change, any URL whose plain fetch doesn't yield a price (which is most
|
||||||
|
URLs without a real product price, e.g. `https://example.com`) now
|
||||||
|
triggers the browser fallback — and no real Chromium binary is installed
|
||||||
|
in this dev sandbox (see Global Constraints). Exercising the actual
|
||||||
|
fallback path with a real browser is Task 4's job, once Chromium is
|
||||||
|
installed in the Docker image. This step only proves `server.js` still
|
||||||
|
starts and its non-check routes still work after the edit — i.e. that the
|
||||||
|
new `require('./src/browserScraper')` and the rewritten `runCheck` are
|
||||||
|
syntactically sound and don't break anything at startup.
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
cp .env.example .env
|
||||||
|
# edit .env: fill in real BASIC_AUTH_USER/PASS and SMTP_FROM/NOTIFY_EMAIL at minimum
|
||||||
|
set -a; source .env; set +a
|
||||||
|
node server.js
|
||||||
|
```
|
||||||
|
In another terminal:
|
||||||
|
```bash
|
||||||
|
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" -X POST http://localhost:3000/products \
|
||||||
|
-d "name=Testproduct" -d "url=https://example.com"
|
||||||
|
curl -u "$BASIC_AUTH_USER:$BASIC_AUTH_PASS" http://localhost:3000/
|
||||||
|
```
|
||||||
|
Expected: both succeed (302, then 200 with "Testproduct" visible in the
|
||||||
|
list). Stop the server with Ctrl-C — do not proceed to `/check-now` here.
|
||||||
|
|
||||||
|
- [ ] **Step 3: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add server.js
|
||||||
|
git commit -m "feat: launch a lazy per-run browser for the fallback scraper"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
### Task 4: Docker image — install Chromium for the fallback
|
||||||
|
|
||||||
|
**Files:**
|
||||||
|
- Modify: `Dockerfile`
|
||||||
|
- Modify: `README.md`
|
||||||
|
|
||||||
|
**Interfaces:** none — this task only changes the deployment image and documentation.
|
||||||
|
|
||||||
|
The current `Dockerfile` (for reference):
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM node:22-slim AS build
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
WORKDIR /app
|
||||||
|
COPY package.json package-lock.json ./
|
||||||
|
RUN npm ci --omit=dev
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
FROM node:22-slim
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=build /app .
|
||||||
|
ENV NODE_ENV=production
|
||||||
|
# De SQLite-database hoort op het persistente volume te staan; zonder deze
|
||||||
|
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
|
||||||
|
# en bij elke redeploy alle data verliezen.
|
||||||
|
ENV DB_PATH=/data/korting.db
|
||||||
|
VOLUME /data
|
||||||
|
EXPOSE 3000
|
||||||
|
CMD ["node", "server.js"]
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 1: Modify the `Dockerfile`'s runtime stage**
|
||||||
|
|
||||||
|
Add the Chromium install line right after `COPY --from=build /app .` (this is the first point where `playwright` from `node_modules` is available to `npx`, and it's the stage that actually runs the browser, so the browser binary + its system libraries belong here, not in the build stage):
|
||||||
|
|
||||||
|
```dockerfile
|
||||||
|
FROM node:22-slim AS build
|
||||||
|
RUN apt-get update && apt-get install -y --no-install-recommends python3 build-essential \
|
||||||
|
&& rm -rf /var/lib/apt/lists/*
|
||||||
|
WORKDIR /app
|
||||||
|
COPY package.json package-lock.json ./
|
||||||
|
RUN npm ci --omit=dev
|
||||||
|
COPY . .
|
||||||
|
|
||||||
|
FROM node:22-slim
|
||||||
|
WORKDIR /app
|
||||||
|
COPY --from=build /app .
|
||||||
|
RUN npx --yes playwright install --with-deps chromium
|
||||||
|
ENV NODE_ENV=production
|
||||||
|
# De SQLite-database hoort op het persistente volume te staan; zonder deze
|
||||||
|
# default zou een vergeten DB_PATH stilletjes naar de container-fs schrijven
|
||||||
|
# en bij elke redeploy alle data verliezen.
|
||||||
|
ENV DB_PATH=/data/korting.db
|
||||||
|
VOLUME /data
|
||||||
|
EXPOSE 3000
|
||||||
|
CMD ["node", "server.js"]
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 2: Update `README.md`**
|
||||||
|
|
||||||
|
Add one short paragraph after the existing "Hoe het werkt" section explaining the fallback (read the current `README.md` first to place it consistently with the existing tone/structure):
|
||||||
|
|
||||||
|
```markdown
|
||||||
|
Als het ophalen van de prijs op de gewone manier mislukt (netwerkfout, of
|
||||||
|
geen prijs gevonden in de HTML), probeert de app het één keer opnieuw met
|
||||||
|
een echte headless-browser (Playwright/Chromium) — nuttig voor pagina's
|
||||||
|
die de prijs pas met JavaScript renderen, of die eenvoudige HTTP-clients
|
||||||
|
blokkeren. Geen garantie tegen elke vorm van bot-detectie, maar dekt een
|
||||||
|
reëel deel van de gevallen die de gewone aanpak mist.
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 3: Manual build/run verification (this is the first task that needs a real browser)**
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
docker build -t korting .
|
||||||
|
```
|
||||||
|
Expected: build succeeds; watch for the `playwright install --with-deps chromium` step completing without error (it will print apt package installation output — that's expected).
|
||||||
|
|
||||||
|
Run:
|
||||||
|
```bash
|
||||||
|
docker run --rm -d --name korting-fallback-smoketest -p 3002:3000 \
|
||||||
|
-e BASIC_AUTH_USER=admin -e BASIC_AUTH_PASS=test \
|
||||||
|
-e SMTP_HOST=mail.sjoerd.app -e SMTP_PORT=587 \
|
||||||
|
-e SMTP_FROM=korting@sjoerd.app -e NOTIFY_EMAIL=test@example.com \
|
||||||
|
-e DB_PATH=/data/korting.db -v korting-fallback-smoketest-data:/data \
|
||||||
|
korting
|
||||||
|
sleep 2
|
||||||
|
curl -u admin:test -X POST http://localhost:3002/products \
|
||||||
|
-d "name=Fallback test" -d "url=https://example.com"
|
||||||
|
curl -u admin:test -X POST http://localhost:3002/check-now
|
||||||
|
docker logs korting-fallback-smoketest 2>&1 | tail -20
|
||||||
|
```
|
||||||
|
Expected: the `check-now` request succeeds (302); the container logs show the `[check] 1 product(en) gecontroleerd: ...` summary line. This URL (`https://example.com`) has no parseable price, so the primary fetch will return `not_found`, which **does** trigger the fallback — confirm the logs show no crash and no unhandled Playwright error (a `not_found` after the fallback also finds nothing is the expected, correct outcome here — the goal of this check is proving the browser actually launches and runs inside the container without error, not that this particular blank test page has a price).
|
||||||
|
|
||||||
|
Clean up:
|
||||||
|
```bash
|
||||||
|
docker stop korting-fallback-smoketest
|
||||||
|
docker volume rm korting-fallback-smoketest-data
|
||||||
|
docker rmi korting
|
||||||
|
```
|
||||||
|
|
||||||
|
- [ ] **Step 4: Commit**
|
||||||
|
|
||||||
|
```bash
|
||||||
|
git add Dockerfile README.md
|
||||||
|
git commit -m "chore: install Chromium in the Docker image for the fallback scraper"
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## After this plan
|
||||||
|
|
||||||
|
Deploy the merged result to Coolify (redeploy the existing `korting.sjoerd.app` application — same app, no new resources needed) via the `deploying-to-coolify` skill, and verify against the actual reported failing URL
|
||||||
|
(`https://www.iciparisxl.nl/chanel/bleu-de-chanel/parfum-verstuiver/p/BP_967866`)
|
||||||
|
using Coolify's deployment/application logs — not a direct request from this
|
||||||
|
machine, since the user has since rotated the app's basic-auth credentials
|
||||||
|
and testing now happens on their side.
|
||||||
@@ -0,0 +1,128 @@
|
|||||||
|
# Korting — Playwright-fallback voor prijs-extractie
|
||||||
|
|
||||||
|
**Datum:** 2026-09-05
|
||||||
|
**Status:** Approved
|
||||||
|
**Bouwt voort op:** `2026-09-05-korting-prijstracker-design.md`
|
||||||
|
|
||||||
|
## Aanleiding
|
||||||
|
|
||||||
|
Een gebruiker meldde dat `fetchAndExtractPrice` faalt voor
|
||||||
|
`https://www.iciparisxl.nl/...` met `fetch_error HTTP 403`. Onderzoek
|
||||||
|
(systematic-debugging) toonde aan dat dit domein achter **Akamai Bot
|
||||||
|
Manager** draait: elk verzoek dat niet als "echte browser" wordt herkend
|
||||||
|
krijgt al op de edge een `403 Access Denied` (`server: AkamaiGHost`),
|
||||||
|
vóórdat de eigenlijke website bereikt wordt. Bevestigd zowel met een
|
||||||
|
directe `curl`-test (ook met browser-achtige User-Agent) als via de live
|
||||||
|
app zelf (identieke `fetch_error HTTP 403` in de logs). Dit is geen bug in
|
||||||
|
de bestaande scraper-logica — die blijft correct voor de meeste shops —
|
||||||
|
maar een structurele blokkade die headers alleen niet oplossen.
|
||||||
|
|
||||||
|
**Belangrijk:** een headless-browser fallback is geen garantie tegen
|
||||||
|
Akamai specifiek (die kan ook headless browsers detecteren), maar is de
|
||||||
|
enige realistische route voor sites die de prijs pas client-side
|
||||||
|
renderen, en verbetert de dekking voor dat bredere geval.
|
||||||
|
|
||||||
|
## Doel
|
||||||
|
|
||||||
|
Wanneer de bestaande `fetchAndExtractPrice` geen prijs oplevert (fout of
|
||||||
|
`price: null`), probeert het systeem het opnieuw met een echte headless
|
||||||
|
Chromium-browser (Playwright) voordat de check als mislukt wordt
|
||||||
|
geregistreerd.
|
||||||
|
|
||||||
|
## Architectuur
|
||||||
|
|
||||||
|
Nieuwe module `src/browserScraper.js`, losstaand van `src/scraper.js`
|
||||||
|
(die blijft ongewijzigd — alle bestaande tests blijven geldig):
|
||||||
|
|
||||||
|
- `launchBrowser() -> Promise<Browser>` — start één Chromium-instance
|
||||||
|
(Playwright).
|
||||||
|
- `fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 }) ->
|
||||||
|
Promise<{ price: number|null, method: string|null, error: Error|null
|
||||||
|
}>` — opent een nieuwe pagina op de meegegeven browser, navigeert naar
|
||||||
|
`url` (timeout `timeoutMs`), leest de gerenderde HTML via
|
||||||
|
`page.content()`, en hergebruikt `extractPrice()` uit `src/scraper.js`
|
||||||
|
(JSON-LD → meta → regex, exact dezelfde tiers) om de prijs eruit te
|
||||||
|
halen. Sluit de pagina altijd (try/finally), nooit de browser zelf.
|
||||||
|
|
||||||
|
## Wijziging in `checkProducts.js`
|
||||||
|
|
||||||
|
`checkProduct(product, deps)` krijgt een **optionele** extra dependency:
|
||||||
|
`deps.fetchAndExtractPriceViaBrowser(url) ->
|
||||||
|
Promise<{price,method,error}>` (curried met de browser-instance door de
|
||||||
|
composition root — `checkProducts.js` weet niets van Playwright zelf,
|
||||||
|
blijft puur dependency-injected zoals nu).
|
||||||
|
|
||||||
|
Logica: eerst zoals nu `fetchAndExtractPrice(product.url)` aanroepen. Als
|
||||||
|
dat resultaat een `error` heeft **of** `price === null`, én
|
||||||
|
`deps.fetchAndExtractPriceViaBrowser` is meegegeven, probeer de fallback
|
||||||
|
en gebruik dát resultaat verder (ook als de fallback zelf ook faalt — de
|
||||||
|
bestaande classificatie hieronder werkt op wat dan ook het laatste
|
||||||
|
resultaat is). **Fallback wordt nooit aangeroepen als de eerste poging al
|
||||||
|
een prijs opleverde.**
|
||||||
|
|
||||||
|
De rest van de functie (baseline/unchanged/changed+mail/mail_error/
|
||||||
|
fetch_error/not_found, en de regel dat `last_price`/`last_checked_at`
|
||||||
|
alleen bij een volledig geslaagde check worden bijgewerkt) blijft
|
||||||
|
ongewijzigd.
|
||||||
|
|
||||||
|
## Lifecycle (in `server.js`)
|
||||||
|
|
||||||
|
Eén browser per `runCheck()`-aanroep (zowel de dagelijkse cron als de
|
||||||
|
"nu controleren"-knop delen dit patroon, want beide roepen `runCheck()`
|
||||||
|
aan) — **lazy**: de browser start pas op het moment dat de eerste
|
||||||
|
fallback binnen die run daadwerkelijk nodig is (dus niet als alle
|
||||||
|
producten al gewoon ophaalbaar zijn via de normale fetch). Eenmaal
|
||||||
|
gestart wordt diezelfde instance hergebruikt voor de rest van die run
|
||||||
|
(nieuwe pagina per product), en aan het eind van de run altijd gesloten
|
||||||
|
in een `finally` — nooit een browser die de hele dag blijft openstaan, en
|
||||||
|
geen onnodige browser-start op runs die 'm niet nodig hebben.
|
||||||
|
|
||||||
|
## Docker
|
||||||
|
|
||||||
|
De runtime-stage van de Dockerfile installeert Chromium + de benodigde
|
||||||
|
systeemlibraries via `npx playwright install --with-deps chromium`, ná
|
||||||
|
het kopiëren van de app (zodat de juiste Playwright-versie uit
|
||||||
|
`node_modules` gebruikt wordt). `playwright` wordt een production
|
||||||
|
dependency (niet dev) zodat `npm ci --omit=dev` in de build-stage 'm wél
|
||||||
|
meeneemt naar de runtime-stage. Dit vergroot de image en de build-tijd
|
||||||
|
merkbaar (geaccepteerd trade-off, al gecommuniceerd).
|
||||||
|
|
||||||
|
## Error handling
|
||||||
|
|
||||||
|
- `page.goto` faalt of timet uit (`timeoutMs`, default 20000ms) → fallback
|
||||||
|
retourneert `{price: null, method: null, error}`, exact zelfde vorm als
|
||||||
|
de bestaande `fetchAndExtractPrice`-fouten. Geen crash van de run.
|
||||||
|
- Pagina wordt altijd gesloten, ook bij een gooiende `page.goto`
|
||||||
|
(try/finally), zodat er geen paginas/geheugen lekken binnen een run met
|
||||||
|
meerdere fallback-pogingen.
|
||||||
|
- Browser wordt altijd gesloten aan het eind van `runCheck()`, ook als er
|
||||||
|
onderweg een fout optreedt (try/finally rond de hele check-run).
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
- `test/browserScraper.test.js`: test tegen een **nep** browser/pagina-
|
||||||
|
object (zelfde dependency-injection-patroon als de rest van de
|
||||||
|
codebase — geen echte Chromium nodig in de testrun): succesvolle
|
||||||
|
extractie, `page.goto` die faalt/timet uit, en dat `page.close()`
|
||||||
|
altijd wordt aangeroepen (ook bij een fout).
|
||||||
|
- `test/checkProducts.test.js`: nieuwe tests die bevestigen dat de
|
||||||
|
fallback (a) wordt aangeroepen wanneer de eerste poging faalt, (b)
|
||||||
|
**niet** wordt aangeroepen wanneer de eerste poging al een prijs
|
||||||
|
oplevert, en (c) dat bestaande tests zonder
|
||||||
|
`fetchAndExtractPriceViaBrowser` in de deps ongewijzigd blijven werken
|
||||||
|
(de dependency is optioneel, achterwaarts compatibel).
|
||||||
|
- Geen geautomatiseerde test voor de Dockerfile-wijziging zelf — net als
|
||||||
|
bij de oorspronkelijke Dockerfile-taak, een handmatige
|
||||||
|
build+run-verificatie (ditmaal ook een handmatige check dat Chromium
|
||||||
|
daadwerkelijk een pagina kan laden binnen de container).
|
||||||
|
|
||||||
|
## Out of scope
|
||||||
|
|
||||||
|
- Geen garantie dat dit specifieke Akamai-domein (`iciparisxl.nl`)
|
||||||
|
hierdoor gaat werken — dat hangt af van Akamai's detectie-niveau voor
|
||||||
|
dat domein op dat moment. Dit wordt niet apart getest tegen dat
|
||||||
|
specifieke domein als onderdeel van de geautomatiseerde testsuite (te
|
||||||
|
broos/afhankelijk van een externe, veranderlijke anti-bot-dienst); wel
|
||||||
|
handmatig te proberen na deploy.
|
||||||
|
- Playwright-stealth-plugins of andere anti-detectie-technieken — niet
|
||||||
|
gevraagd, voegt complexiteit toe zonder gegarandeerd resultaat.
|
||||||
Generated
+42
-1
@@ -10,10 +10,12 @@
|
|||||||
"dependencies": {
|
"dependencies": {
|
||||||
"better-sqlite3": "^11.3.0",
|
"better-sqlite3": "^11.3.0",
|
||||||
"cheerio": "^1.0.0",
|
"cheerio": "^1.0.0",
|
||||||
|
"dotenv": "^17.4.2",
|
||||||
"express": "^4.19.2",
|
"express": "^4.19.2",
|
||||||
"express-basic-auth": "^1.2.1",
|
"express-basic-auth": "^1.2.1",
|
||||||
"node-cron": "^3.0.3",
|
"node-cron": "^3.0.3",
|
||||||
"nodemailer": "^6.9.14"
|
"nodemailer": "^6.9.14",
|
||||||
|
"playwright": "^1.63.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"supertest": "^7.0.0"
|
"supertest": "^7.0.0"
|
||||||
@@ -527,6 +529,18 @@
|
|||||||
"url": "https://github.com/fb55/domutils?sponsor=1"
|
"url": "https://github.com/fb55/domutils?sponsor=1"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/dotenv": {
|
||||||
|
"version": "17.4.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/dotenv/-/dotenv-17.4.2.tgz",
|
||||||
|
"integrity": "sha512-nI4U3TottKAcAD9LLud4Cb7b2QztQMUEfHbvhTH09bqXTxnSie8WnjPALV/WMCrJZ6UV/qHJ6L03OqO3LcdYZw==",
|
||||||
|
"license": "BSD-2-Clause",
|
||||||
|
"engines": {
|
||||||
|
"node": ">=12"
|
||||||
|
},
|
||||||
|
"funding": {
|
||||||
|
"url": "https://dotenvx.com"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/dunder-proto": {
|
"node_modules/dunder-proto": {
|
||||||
"version": "1.0.1",
|
"version": "1.0.1",
|
||||||
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
|
"resolved": "https://registry.npmjs.org/dunder-proto/-/dunder-proto-1.0.1.tgz",
|
||||||
@@ -1272,6 +1286,33 @@
|
|||||||
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
|
"integrity": "sha512-A/AGNMFN3c8bOlvV9RreMdrv7jsmF9XIfDeCd87+I8RNg6s78BhJxMu69NEMHBSJFxKidViTEdruRwEk/WIKqA==",
|
||||||
"license": "MIT"
|
"license": "MIT"
|
||||||
},
|
},
|
||||||
|
"node_modules/playwright": {
|
||||||
|
"version": "1.63.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/playwright/-/playwright-1.63.0.tgz",
|
||||||
|
"integrity": "sha512-+7ziBLidS4NaNCdt57SUDT+wYmmd5fmiQejUic/kb+YsYSCPyOOE9sebzMjNmQrsnNpDJqd4WHvV/8lfKfUDUg==",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"dependencies": {
|
||||||
|
"playwright-core": "1.63.0"
|
||||||
|
},
|
||||||
|
"bin": {
|
||||||
|
"playwright": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"node_modules/playwright-core": {
|
||||||
|
"version": "1.63.0",
|
||||||
|
"resolved": "https://registry.npmjs.org/playwright-core/-/playwright-core-1.63.0.tgz",
|
||||||
|
"integrity": "sha512-rYCsBF/M5HjUch52bbtVONEFjv6Xu8sm8h72dNlR5bzIE1fvC/bxgspzkjSfU+MweEMmPM8KJebG6nnyxo5mCg==",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"bin": {
|
||||||
|
"playwright-core": "cli.js"
|
||||||
|
},
|
||||||
|
"engines": {
|
||||||
|
"node": ">=20"
|
||||||
|
}
|
||||||
|
},
|
||||||
"node_modules/prebuild-install": {
|
"node_modules/prebuild-install": {
|
||||||
"version": "7.1.3",
|
"version": "7.1.3",
|
||||||
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
|
"resolved": "https://registry.npmjs.org/prebuild-install/-/prebuild-install-7.1.3.tgz",
|
||||||
|
|||||||
+3
-1
@@ -11,10 +11,12 @@
|
|||||||
"dependencies": {
|
"dependencies": {
|
||||||
"better-sqlite3": "^11.3.0",
|
"better-sqlite3": "^11.3.0",
|
||||||
"cheerio": "^1.0.0",
|
"cheerio": "^1.0.0",
|
||||||
|
"dotenv": "^17.4.2",
|
||||||
"express": "^4.19.2",
|
"express": "^4.19.2",
|
||||||
"express-basic-auth": "^1.2.1",
|
"express-basic-auth": "^1.2.1",
|
||||||
"node-cron": "^3.0.3",
|
"node-cron": "^3.0.3",
|
||||||
"nodemailer": "^6.9.14"
|
"nodemailer": "^6.9.14",
|
||||||
|
"playwright": "^1.63.0"
|
||||||
},
|
},
|
||||||
"devDependencies": {
|
"devDependencies": {
|
||||||
"supertest": "^7.0.0"
|
"supertest": "^7.0.0"
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
// Leest een .env-bestand in voor lokale ontwikkeling; in productie komen de
|
||||||
|
// env vars van Coolify. quiet houdt de opstartlogging schoon.
|
||||||
|
require('dotenv').config({ quiet: true });
|
||||||
|
|
||||||
|
const path = require('node:path');
|
||||||
|
const {
|
||||||
|
openDb, listProducts, addProduct, deleteProduct,
|
||||||
|
recordCheckSuccess, recordCheckFailure,
|
||||||
|
} = require('./src/db');
|
||||||
|
const { fetchAndExtractPrice } = require('./src/scraper');
|
||||||
|
const { launchBrowser, fetchAndExtractPriceViaBrowser } = require('./src/browserScraper');
|
||||||
|
const { createTransport, sendPriceChangeEmail } = require('./src/mailer');
|
||||||
|
const { checkAllProducts } = require('./src/checkProducts');
|
||||||
|
const { start: startScheduler } = require('./src/scheduler');
|
||||||
|
const { createApp } = require('./src/app');
|
||||||
|
|
||||||
|
const REQUIRED_ENV = ['BASIC_AUTH_USER', 'BASIC_AUTH_PASS', 'SMTP_FROM', 'NOTIFY_EMAIL'];
|
||||||
|
const missing = REQUIRED_ENV.filter((name) => !process.env[name]);
|
||||||
|
if (missing.length > 0) {
|
||||||
|
console.error(`Ontbrekende verplichte environment variables: ${missing.join(', ')}`);
|
||||||
|
console.error('Zet ze (bijvoorbeeld in .env, zie .env.example) en start opnieuw.');
|
||||||
|
process.exit(1);
|
||||||
|
}
|
||||||
|
|
||||||
|
const PORT = process.env.PORT || 3000;
|
||||||
|
const DB_PATH = process.env.DB_PATH || path.join(__dirname, 'data', 'korting.db');
|
||||||
|
|
||||||
|
const db = openDb(DB_PATH);
|
||||||
|
const transport = createTransport(process.env);
|
||||||
|
|
||||||
|
function summarize(results) {
|
||||||
|
const counts = {};
|
||||||
|
for (const result of results) {
|
||||||
|
counts[result.status] = (counts[result.status] || 0) + 1;
|
||||||
|
}
|
||||||
|
const parts = Object.entries(counts).map(([status, count]) => `${status}=${count}`);
|
||||||
|
return parts.length > 0 ? parts.join(' ') : 'geen producten';
|
||||||
|
}
|
||||||
|
|
||||||
|
let inFlightCheck = null;
|
||||||
|
function runCheck() {
|
||||||
|
if (inFlightCheck) {
|
||||||
|
return inFlightCheck;
|
||||||
|
}
|
||||||
|
inFlightCheck = runCheckOnce().finally(() => {
|
||||||
|
inFlightCheck = null;
|
||||||
|
});
|
||||||
|
return inFlightCheck;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function runCheckOnce() {
|
||||||
|
const products = listProducts(db);
|
||||||
|
let browserPromise = null;
|
||||||
|
function getBrowser() {
|
||||||
|
if (!browserPromise) {
|
||||||
|
browserPromise = launchBrowser();
|
||||||
|
}
|
||||||
|
return browserPromise;
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const deps = {
|
||||||
|
fetchAndExtractPrice,
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
try {
|
||||||
|
const browser = await getBrowser();
|
||||||
|
return await fetchAndExtractPriceViaBrowser(browser, url);
|
||||||
|
} catch (err) {
|
||||||
|
return { price: null, method: null, error: err };
|
||||||
|
}
|
||||||
|
},
|
||||||
|
sendMail: (payload) => sendPriceChangeEmail(transport, {
|
||||||
|
from: process.env.SMTP_FROM,
|
||||||
|
to: process.env.NOTIFY_EMAIL,
|
||||||
|
...payload,
|
||||||
|
}),
|
||||||
|
recordCheckFailure: (id, status) => recordCheckFailure(db, id, status),
|
||||||
|
recordCheckSuccess: (id, data) => recordCheckSuccess(db, id, data),
|
||||||
|
now: () => new Date().toISOString(),
|
||||||
|
};
|
||||||
|
const results = await checkAllProducts(products, deps);
|
||||||
|
console.log(`[check] ${products.length} product(en) gecontroleerd: ${summarize(results)}`);
|
||||||
|
return results;
|
||||||
|
} finally {
|
||||||
|
if (browserPromise) {
|
||||||
|
const browser = await browserPromise.catch(() => null);
|
||||||
|
if (browser) {
|
||||||
|
await browser.close().catch((err) => {
|
||||||
|
console.error(`[check] kon de browser niet netjes sluiten: ${err && err.message}`);
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
startScheduler(runCheck);
|
||||||
|
|
||||||
|
const app = createApp({
|
||||||
|
listProducts: () => listProducts(db),
|
||||||
|
addProduct: (data) => addProduct(db, data),
|
||||||
|
deleteProduct: (id) => deleteProduct(db, id),
|
||||||
|
checkAllProducts: runCheck,
|
||||||
|
authUser: process.env.BASIC_AUTH_USER,
|
||||||
|
authPass: process.env.BASIC_AUTH_PASS,
|
||||||
|
});
|
||||||
|
|
||||||
|
app.listen(PORT, () => {
|
||||||
|
console.log(`Korting draait op poort ${PORT}`);
|
||||||
|
});
|
||||||
+50
@@ -0,0 +1,50 @@
|
|||||||
|
const express = require('express');
|
||||||
|
const basicAuth = require('express-basic-auth');
|
||||||
|
const { renderIndex } = require('./views');
|
||||||
|
|
||||||
|
function createApp(deps) {
|
||||||
|
const {
|
||||||
|
listProducts, addProduct, deleteProduct,
|
||||||
|
checkAllProducts, authUser, authPass,
|
||||||
|
} = deps;
|
||||||
|
|
||||||
|
const app = express();
|
||||||
|
app.use(express.urlencoded({ extended: false }));
|
||||||
|
app.use(basicAuth({ users: { [authUser]: authPass }, challenge: true }));
|
||||||
|
|
||||||
|
app.get('/', (req, res) => {
|
||||||
|
const products = listProducts();
|
||||||
|
const message = req.query.checked ? 'Controle uitgevoerd.' : undefined;
|
||||||
|
res.send(renderIndex({ products, message }));
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post('/products', (req, res) => {
|
||||||
|
const { name, url } = req.body;
|
||||||
|
if (name && url) {
|
||||||
|
addProduct({ name, url });
|
||||||
|
}
|
||||||
|
res.redirect('/');
|
||||||
|
});
|
||||||
|
|
||||||
|
app.post('/products/:id/delete', (req, res) => {
|
||||||
|
deleteProduct(Number(req.params.id));
|
||||||
|
res.redirect('/');
|
||||||
|
});
|
||||||
|
|
||||||
|
// Express 4 vangt een afgewezen promise uit een async handler niet af, en
|
||||||
|
// Node 22 laat het proces daarop crashen — dus zelf afvangen.
|
||||||
|
app.post('/check-now', async (req, res) => {
|
||||||
|
try {
|
||||||
|
const products = listProducts();
|
||||||
|
await checkAllProducts(products);
|
||||||
|
res.redirect('/?checked=1');
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check-now] mislukt: ${err && err.message}`);
|
||||||
|
res.status(500).send('Er ging iets mis bij het controleren van de prijzen.');
|
||||||
|
}
|
||||||
|
});
|
||||||
|
|
||||||
|
return app;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { createApp };
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
const { chromium } = require('playwright');
|
||||||
|
const { extractPrice } = require('./scraper');
|
||||||
|
|
||||||
|
async function launchBrowser() {
|
||||||
|
return chromium.launch({ headless: true });
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchAndExtractPriceViaBrowser(browser, url, { timeoutMs = 20000 } = {}) {
|
||||||
|
let page;
|
||||||
|
try {
|
||||||
|
page = await browser.newPage();
|
||||||
|
await page.goto(url, { waitUntil: 'domcontentloaded', timeout: timeoutMs });
|
||||||
|
const html = await page.content();
|
||||||
|
const { price, method } = extractPrice(html);
|
||||||
|
return { price, method, error: null };
|
||||||
|
} catch (err) {
|
||||||
|
return { price: null, method: null, error: err };
|
||||||
|
} finally {
|
||||||
|
if (page) {
|
||||||
|
await page.close().catch(() => {});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { launchBrowser, fetchAndExtractPriceViaBrowser };
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
async function checkProduct(product, deps) {
|
||||||
|
const {
|
||||||
|
fetchAndExtractPrice, fetchAndExtractPriceViaBrowser,
|
||||||
|
sendMail, recordCheckSuccess, recordCheckFailure, now,
|
||||||
|
} = deps;
|
||||||
|
let result = await fetchAndExtractPrice(product.url);
|
||||||
|
|
||||||
|
if ((result.error || result.price == null) && fetchAndExtractPriceViaBrowser) {
|
||||||
|
result = await fetchAndExtractPriceViaBrowser(product.url);
|
||||||
|
}
|
||||||
|
|
||||||
|
if (result.error) {
|
||||||
|
console.error(`[check] fetch_error "${product.name}" (${product.url}): ${result.error.message}`);
|
||||||
|
recordCheckFailure(product.id, 'fetch_error');
|
||||||
|
return { status: 'fetch_error' };
|
||||||
|
}
|
||||||
|
if (result.price == null) {
|
||||||
|
console.error(`[check] not_found "${product.name}" (${product.url}): geen prijs gevonden op de pagina`);
|
||||||
|
recordCheckFailure(product.id, 'not_found');
|
||||||
|
return { status: 'not_found' };
|
||||||
|
}
|
||||||
|
|
||||||
|
const isBaseline = product.last_price == null;
|
||||||
|
const changed = !isBaseline && result.price !== product.last_price;
|
||||||
|
|
||||||
|
if (changed) {
|
||||||
|
try {
|
||||||
|
await sendMail({
|
||||||
|
name: product.name,
|
||||||
|
url: product.url,
|
||||||
|
oldPrice: product.last_price,
|
||||||
|
newPrice: result.price,
|
||||||
|
});
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check] mail_error "${product.name}" (${product.url}): ${err && err.message}`);
|
||||||
|
recordCheckFailure(product.id, 'mail_error');
|
||||||
|
return { status: 'mail_error' };
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
recordCheckSuccess(product.id, { price: result.price, checkedAt: now() });
|
||||||
|
return { status: 'ok', changed };
|
||||||
|
}
|
||||||
|
|
||||||
|
async function checkAllProducts(products, deps) {
|
||||||
|
const results = [];
|
||||||
|
for (const product of products) {
|
||||||
|
try {
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
results.push({ productId: product.id, ...result });
|
||||||
|
} catch (err) {
|
||||||
|
console.error(`[check] error "${product.name}" (${product.url}): ${err && err.message}`);
|
||||||
|
results.push({ productId: product.id, status: 'error', error: err });
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return results;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { checkProduct, checkAllProducts };
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
const cronDefault = require('node-cron');
|
||||||
|
|
||||||
|
const SCHEDULE = '0 8 * * *';
|
||||||
|
const TIMEZONE = 'Europe/Amsterdam';
|
||||||
|
|
||||||
|
function start(runCheck, { cron = cronDefault } = {}) {
|
||||||
|
return cron.schedule(SCHEDULE, runCheck, { timezone: TIMEZONE });
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { start, SCHEDULE, TIMEZONE };
|
||||||
+44
-6
@@ -4,10 +4,32 @@ function roundMoney(n) {
|
|||||||
return Math.round(n * 100) / 100;
|
return Math.round(n * 100) / 100;
|
||||||
}
|
}
|
||||||
|
|
||||||
function parsePriceValue(raw) {
|
// Turns a human-written amount into a plain JS-parsable number string.
|
||||||
|
// When both separators occur, the last one is the decimal separator
|
||||||
|
// ("1.299,00" and "1,299.00" both mean 1299.00); a lone comma is decimal
|
||||||
|
// only when exactly two digits follow it, otherwise it groups thousands.
|
||||||
|
function normalizeAmount(raw) {
|
||||||
if (raw == null) return null;
|
if (raw == null) return null;
|
||||||
const cleaned = String(raw).trim().replace(',', '.');
|
const match = String(raw).trim().match(/-?\d[\d.,]*/);
|
||||||
const value = parseFloat(cleaned);
|
if (!match) return null;
|
||||||
|
const str = match[0].replace(/[.,]+$/, '');
|
||||||
|
const lastDot = str.lastIndexOf('.');
|
||||||
|
const lastComma = str.lastIndexOf(',');
|
||||||
|
if (lastDot !== -1 && lastComma !== -1) {
|
||||||
|
return lastComma > lastDot
|
||||||
|
? str.replace(/\./g, '').replace(',', '.')
|
||||||
|
: str.replace(/,/g, '');
|
||||||
|
}
|
||||||
|
if (lastComma !== -1) {
|
||||||
|
return /,\d{2}$/.test(str) ? str.replace(',', '.') : str.replace(/,/g, '');
|
||||||
|
}
|
||||||
|
return str;
|
||||||
|
}
|
||||||
|
|
||||||
|
function parsePriceValue(raw) {
|
||||||
|
const normalized = normalizeAmount(raw);
|
||||||
|
if (normalized == null) return null;
|
||||||
|
const value = parseFloat(normalized);
|
||||||
return Number.isFinite(value) ? roundMoney(value) : null;
|
return Number.isFinite(value) ? roundMoney(value) : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -59,9 +81,14 @@ function findPriceInMeta($) {
|
|||||||
return null;
|
return null;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// Thousands-grouped amounts first, then a plain amount, so "€1.299,00"
|
||||||
|
// is not truncated to "€1.29".
|
||||||
|
const TEXT_PRICE_RE = /€\s?(\d{1,3}(?:[.,]\d{3})+[.,]\d{2}|\d+[.,]\d{2})/;
|
||||||
|
|
||||||
function findPriceInText($) {
|
function findPriceInText($) {
|
||||||
const text = $('body').text();
|
const text = $('body').clone().find('script, style, noscript').remove().end()
|
||||||
const match = text.match(/€\s?(\d+[.,]\d{2})/);
|
.text();
|
||||||
|
const match = text.match(TEXT_PRICE_RE);
|
||||||
return match ? parsePriceValue(match[1]) : null;
|
return match ? parsePriceValue(match[1]) : null;
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -76,7 +103,18 @@ function extractPrice(html) {
|
|||||||
return { price: null, method: null };
|
return { price: null, method: null };
|
||||||
}
|
}
|
||||||
|
|
||||||
async function fetchAndExtractPrice(url, { fetchImpl = fetch } = {}) {
|
const FETCH_TIMEOUT_MS = 15000;
|
||||||
|
const USER_AGENT = 'Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 '
|
||||||
|
+ '(KHTML, like Gecko) Chrome/128.0.0.0 Safari/537.36';
|
||||||
|
|
||||||
|
function defaultFetch(url) {
|
||||||
|
return fetch(url, {
|
||||||
|
signal: AbortSignal.timeout(FETCH_TIMEOUT_MS),
|
||||||
|
headers: { 'User-Agent': USER_AGENT },
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
async function fetchAndExtractPrice(url, { fetchImpl = defaultFetch } = {}) {
|
||||||
let response;
|
let response;
|
||||||
try {
|
try {
|
||||||
response = await fetchImpl(url);
|
response = await fetchImpl(url);
|
||||||
|
|||||||
@@ -0,0 +1,63 @@
|
|||||||
|
function escapeHtml(str) {
|
||||||
|
return String(str).replace(/[&<>"']/g, (c) => ({
|
||||||
|
'&': '&', '<': '<', '>': '>', '"': '"', "'": ''',
|
||||||
|
}[c]));
|
||||||
|
}
|
||||||
|
|
||||||
|
function formatPrice(price) {
|
||||||
|
return price == null ? '—' : `€${Number(price).toFixed(2).replace('.', ',')}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderProductRow(product) {
|
||||||
|
return `
|
||||||
|
<tr>
|
||||||
|
<td>${escapeHtml(product.name)}</td>
|
||||||
|
<td><a href="${escapeHtml(product.url)}" target="_blank" rel="noopener">bekijken</a></td>
|
||||||
|
<td>${formatPrice(product.last_price)}</td>
|
||||||
|
<td>${product.last_checked_at ? escapeHtml(product.last_checked_at) : '—'}</td>
|
||||||
|
<td>${escapeHtml(product.last_check_status || '—')}</td>
|
||||||
|
<td>
|
||||||
|
<form method="post" action="/products/${product.id}/delete" onsubmit="return confirm('Product verwijderen?');">
|
||||||
|
<button type="submit">Verwijder</button>
|
||||||
|
</form>
|
||||||
|
</td>
|
||||||
|
</tr>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
function renderIndex({ products, message }) {
|
||||||
|
return `<!doctype html>
|
||||||
|
<html lang="nl">
|
||||||
|
<head>
|
||||||
|
<meta charset="utf-8">
|
||||||
|
<title>Korting — prijstracker</title>
|
||||||
|
<style>
|
||||||
|
body { font-family: system-ui, sans-serif; max-width: 800px; margin: 2rem auto; padding: 0 1rem; }
|
||||||
|
table { width: 100%; border-collapse: collapse; margin-top: 1rem; }
|
||||||
|
th, td { text-align: left; padding: 0.5rem; border-bottom: 1px solid #ddd; }
|
||||||
|
.message { background: #eef; padding: 0.5rem 1rem; border-radius: 4px; }
|
||||||
|
</style>
|
||||||
|
</head>
|
||||||
|
<body>
|
||||||
|
<h1>Korting — prijstracker</h1>
|
||||||
|
${message ? `<p class="message">${escapeHtml(message)}</p>` : ''}
|
||||||
|
<form method="post" action="/products">
|
||||||
|
<input name="name" placeholder="Naam" required>
|
||||||
|
<input name="url" type="url" placeholder="https://..." required>
|
||||||
|
<button type="submit">Toevoegen</button>
|
||||||
|
</form>
|
||||||
|
<form method="post" action="/check-now">
|
||||||
|
<button type="submit">Nu controleren</button>
|
||||||
|
</form>
|
||||||
|
<table>
|
||||||
|
<thead>
|
||||||
|
<tr><th>Naam</th><th>URL</th><th>Prijs</th><th>Laatst gecontroleerd</th><th>Status</th><th></th></tr>
|
||||||
|
</thead>
|
||||||
|
<tbody>
|
||||||
|
${products.map(renderProductRow).join('')}
|
||||||
|
</tbody>
|
||||||
|
</table>
|
||||||
|
</body>
|
||||||
|
</html>`;
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { renderIndex, escapeHtml, formatPrice };
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const request = require('supertest');
|
||||||
|
const { createApp } = require('../src/app');
|
||||||
|
|
||||||
|
function makeDeps(overrides = {}) {
|
||||||
|
let products = [];
|
||||||
|
let nextId = 1;
|
||||||
|
return {
|
||||||
|
listProducts: () => products,
|
||||||
|
addProduct: ({ name, url }) => {
|
||||||
|
const product = { id: nextId++, name, url, last_price: null, last_checked_at: null, last_check_status: null };
|
||||||
|
products.push(product);
|
||||||
|
return product;
|
||||||
|
},
|
||||||
|
deleteProduct: (id) => { products = products.filter((p) => p.id !== id); },
|
||||||
|
checkAllProducts: async () => [],
|
||||||
|
authUser: 'admin',
|
||||||
|
authPass: 'secret',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
test('rejects requests without basic auth credentials', async () => {
|
||||||
|
const app = createApp(makeDeps());
|
||||||
|
const res = await request(app).get('/');
|
||||||
|
assert.equal(res.status, 401);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('lists products for authenticated requests', async () => {
|
||||||
|
const deps = makeDeps();
|
||||||
|
deps.addProduct({ name: 'Voorbeeld', url: 'https://example.com' });
|
||||||
|
const app = createApp(deps);
|
||||||
|
const res = await request(app).get('/').auth('admin', 'secret');
|
||||||
|
assert.equal(res.status, 200);
|
||||||
|
assert.match(res.text, /Voorbeeld/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('adding a product redirects to the list', async () => {
|
||||||
|
const deps = makeDeps();
|
||||||
|
const app = createApp(deps);
|
||||||
|
const res = await request(app)
|
||||||
|
.post('/products')
|
||||||
|
.auth('admin', 'secret')
|
||||||
|
.type('form')
|
||||||
|
.send({ name: 'Nieuw product', url: 'https://shop.example/p' });
|
||||||
|
assert.equal(res.status, 302);
|
||||||
|
assert.equal(res.headers.location, '/');
|
||||||
|
const list = await request(app).get('/').auth('admin', 'secret');
|
||||||
|
assert.match(list.text, /Nieuw product/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('deleting a product removes it from the list', async () => {
|
||||||
|
const deps = makeDeps();
|
||||||
|
const product = deps.addProduct({ name: 'Weg ermee', url: 'https://shop.example/p2' });
|
||||||
|
const app = createApp(deps);
|
||||||
|
const res = await request(app).post(`/products/${product.id}/delete`).auth('admin', 'secret');
|
||||||
|
assert.equal(res.status, 302);
|
||||||
|
const list = await request(app).get('/').auth('admin', 'secret');
|
||||||
|
assert.doesNotMatch(list.text, /Weg ermee/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('check-now triggers checkAllProducts and redirects with confirmation', async () => {
|
||||||
|
let called = false;
|
||||||
|
const deps = makeDeps({ checkAllProducts: async () => { called = true; return []; } });
|
||||||
|
const app = createApp(deps);
|
||||||
|
const res = await request(app).post('/check-now').auth('admin', 'secret');
|
||||||
|
assert.equal(res.status, 302);
|
||||||
|
assert.equal(res.headers.location, '/?checked=1');
|
||||||
|
assert.equal(called, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('check-now answers 500 when the check fails, without crashing the process', async (t) => {
|
||||||
|
t.mock.method(console, 'error', () => {});
|
||||||
|
const deps = makeDeps({
|
||||||
|
checkAllProducts: async () => { throw new Error('db weg'); },
|
||||||
|
});
|
||||||
|
deps.addProduct({ name: 'Nog steeds hier', url: 'https://shop.example/p3' });
|
||||||
|
const app = createApp(deps);
|
||||||
|
const res = await request(app).post('/check-now').auth('admin', 'secret');
|
||||||
|
assert.equal(res.status, 500);
|
||||||
|
assert.match(res.text, /Er ging iets mis/);
|
||||||
|
// Same process, same app instance: proves nothing crashed.
|
||||||
|
const list = await request(app).get('/').auth('admin', 'secret');
|
||||||
|
assert.equal(list.status, 200);
|
||||||
|
assert.match(list.text, /Nog steeds hier/);
|
||||||
|
});
|
||||||
@@ -0,0 +1,70 @@
|
|||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { fetchAndExtractPriceViaBrowser } = require('../src/browserScraper');
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser extracts a price from the rendered page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const html = '<html><head><meta property="og:price:amount" content="42.00" /></head><body></body></html>';
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => html,
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.deepEqual(result, { price: 42, method: 'meta', error: null });
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser returns an error when navigation fails, and still closes the page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => { throw new Error('Timeout 20000ms exceeded'); },
|
||||||
|
content: async () => '',
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.equal(result.price, null);
|
||||||
|
assert.equal(result.method, null);
|
||||||
|
assert.ok(result.error instanceof Error);
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser returns null price when nothing matches, and still closes the page', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => '<html><body><p>Geen prijs</p></body></html>',
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.deepEqual(result, { price: null, method: null, error: null });
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser closes the page even if reading its content throws', async () => {
|
||||||
|
let closed = false;
|
||||||
|
const page = {
|
||||||
|
goto: async () => {},
|
||||||
|
content: async () => { throw new Error('content failed'); },
|
||||||
|
close: async () => { closed = true; },
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
const result = await fetchAndExtractPriceViaBrowser(browser, 'https://example.com');
|
||||||
|
assert.ok(result.error instanceof Error);
|
||||||
|
assert.equal(closed, true);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPriceViaBrowser passes the timeout option through to page.goto', async () => {
|
||||||
|
let receivedOptions;
|
||||||
|
const page = {
|
||||||
|
goto: async (url, options) => { receivedOptions = options; },
|
||||||
|
content: async () => '<html><body></body></html>',
|
||||||
|
close: async () => {},
|
||||||
|
};
|
||||||
|
const browser = { newPage: async () => page };
|
||||||
|
await fetchAndExtractPriceViaBrowser(browser, 'https://example.com', { timeoutMs: 5000 });
|
||||||
|
assert.equal(receivedOptions.timeout, 5000);
|
||||||
|
});
|
||||||
@@ -0,0 +1,184 @@
|
|||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { checkProduct, checkAllProducts } = require('../src/checkProducts');
|
||||||
|
|
||||||
|
function makeDeps(overrides = {}) {
|
||||||
|
const failures = [];
|
||||||
|
const successes = [];
|
||||||
|
const mailCalls = [];
|
||||||
|
const deps = {
|
||||||
|
fetchAndExtractPrice: async () => ({ price: 10, method: 'meta', error: null }),
|
||||||
|
sendMail: async (payload) => { mailCalls.push(payload); },
|
||||||
|
recordCheckFailure: (id, status) => failures.push({ id, status }),
|
||||||
|
recordCheckSuccess: (id, data) => successes.push({ id, ...data }),
|
||||||
|
now: () => '2026-09-05T08:00:00.000Z',
|
||||||
|
...overrides,
|
||||||
|
};
|
||||||
|
return { deps, failures, successes, mailCalls };
|
||||||
|
}
|
||||||
|
|
||||||
|
// Failure paths log to stderr; capture it so the test output stays clean
|
||||||
|
// while still proving the logging happened.
|
||||||
|
function captureErrorLog(t) {
|
||||||
|
const logged = [];
|
||||||
|
t.mock.method(console, 'error', (...args) => { logged.push(args.join(' ')); });
|
||||||
|
return logged;
|
||||||
|
}
|
||||||
|
|
||||||
|
test('checkProduct stores baseline price without sending mail', async () => {
|
||||||
|
const { deps, successes, mailCalls } = makeDeps();
|
||||||
|
const product = { id: 1, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.equal(result.changed, false);
|
||||||
|
assert.equal(mailCalls.length, 0);
|
||||||
|
assert.deepEqual(successes, [{ id: 1, price: 10, checkedAt: '2026-09-05T08:00:00.000Z' }]);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct sends mail and updates price when price changed', async () => {
|
||||||
|
const { deps, successes, mailCalls } = makeDeps();
|
||||||
|
const product = { id: 2, name: 'X', url: 'https://x', last_price: 15 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.equal(result.changed, true);
|
||||||
|
assert.equal(mailCalls.length, 1);
|
||||||
|
assert.deepEqual(mailCalls[0], { name: 'X', url: 'https://x', oldPrice: 15, newPrice: 10 });
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct does not send mail or update when price unchanged', async () => {
|
||||||
|
const { deps, successes, mailCalls } = makeDeps();
|
||||||
|
const product = { id: 3, name: 'X', url: 'https://x', last_price: 10 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.equal(result.changed, false);
|
||||||
|
assert.equal(mailCalls.length, 0);
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct records fetch_error and does not touch price on network failure', async (t) => {
|
||||||
|
const logged = captureErrorLog(t);
|
||||||
|
const { deps, failures, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
|
||||||
|
});
|
||||||
|
const product = { id: 4, name: 'X', url: 'https://x', last_price: 10 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'fetch_error');
|
||||||
|
assert.deepEqual(failures, [{ id: 4, status: 'fetch_error' }]);
|
||||||
|
assert.equal(successes.length, 0);
|
||||||
|
assert.equal(logged.length, 1);
|
||||||
|
assert.match(logged[0], /fetch_error.*https:\/\/x.*down/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct records not_found when no price could be extracted', async (t) => {
|
||||||
|
const logged = captureErrorLog(t);
|
||||||
|
const { deps, failures, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
|
||||||
|
});
|
||||||
|
const product = { id: 5, name: 'X', url: 'https://x', last_price: 10 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'not_found');
|
||||||
|
assert.deepEqual(failures, [{ id: 5, status: 'not_found' }]);
|
||||||
|
assert.equal(successes.length, 0);
|
||||||
|
assert.equal(logged.length, 1);
|
||||||
|
assert.match(logged[0], /not_found.*https:\/\/x/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct records mail_error and keeps old price when sending mail fails', async (t) => {
|
||||||
|
const logged = captureErrorLog(t);
|
||||||
|
const { deps, failures, successes } = makeDeps({
|
||||||
|
sendMail: async () => { throw new Error('smtp down'); },
|
||||||
|
});
|
||||||
|
const product = { id: 6, name: 'X', url: 'https://x', last_price: 15 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'mail_error');
|
||||||
|
assert.deepEqual(failures, [{ id: 6, status: 'mail_error' }]);
|
||||||
|
assert.equal(successes.length, 0);
|
||||||
|
assert.equal(logged.length, 1);
|
||||||
|
assert.match(logged[0], /mail_error.*https:\/\/x.*smtp down/);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkAllProducts processes every product independently, isolating failures', async (t) => {
|
||||||
|
captureErrorLog(t);
|
||||||
|
const { deps } = makeDeps();
|
||||||
|
let calls = 0;
|
||||||
|
const throwingDeps = {
|
||||||
|
...deps,
|
||||||
|
fetchAndExtractPrice: async () => {
|
||||||
|
calls += 1;
|
||||||
|
if (calls === 1) throw new Error('boom');
|
||||||
|
return { price: 10, method: 'meta', error: null };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const products = [
|
||||||
|
{ id: 1, name: 'A', url: 'https://a', last_price: null },
|
||||||
|
{ id: 2, name: 'B', url: 'https://b', last_price: null },
|
||||||
|
];
|
||||||
|
const results = await checkAllProducts(products, throwingDeps);
|
||||||
|
assert.equal(results.length, 2);
|
||||||
|
assert.equal(results[0].status, 'error');
|
||||||
|
assert.equal(results[1].status, 'ok');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct tries the browser fallback when the primary fetch errors, and uses its result', async (t) => {
|
||||||
|
captureErrorLog(t);
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps, successes, mailCalls } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 10, method: 'json-ld', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 7, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.deepEqual(browserCalls, ['https://x']);
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
assert.equal(mailCalls.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct tries the browser fallback when the primary fetch finds no price', async (t) => {
|
||||||
|
captureErrorLog(t);
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: null }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 12, method: 'regex', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 8, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'ok');
|
||||||
|
assert.deepEqual(browserCalls, ['https://x']);
|
||||||
|
assert.equal(successes.length, 1);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct never calls the browser fallback when the primary fetch already succeeds', async () => {
|
||||||
|
const browserCalls = [];
|
||||||
|
const { deps } = makeDeps({
|
||||||
|
fetchAndExtractPriceViaBrowser: async (url) => {
|
||||||
|
browserCalls.push(url);
|
||||||
|
return { price: 999, method: 'meta', error: null };
|
||||||
|
},
|
||||||
|
});
|
||||||
|
const product = { id: 9, name: 'X', url: 'https://x', last_price: null };
|
||||||
|
await checkProduct(product, deps);
|
||||||
|
assert.equal(browserCalls.length, 0);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('checkProduct records fetch_error when both the primary fetch and the browser fallback fail', async (t) => {
|
||||||
|
const logged = captureErrorLog(t);
|
||||||
|
const { deps, failures, successes } = makeDeps({
|
||||||
|
fetchAndExtractPrice: async () => ({ price: null, method: null, error: new Error('down') }),
|
||||||
|
fetchAndExtractPriceViaBrowser: async () => ({ price: null, method: null, error: new Error('still down') }),
|
||||||
|
});
|
||||||
|
const product = { id: 10, name: 'X', url: 'https://x', last_price: 10 };
|
||||||
|
const result = await checkProduct(product, deps);
|
||||||
|
assert.equal(result.status, 'fetch_error');
|
||||||
|
assert.deepEqual(failures, [{ id: 10, status: 'fetch_error' }]);
|
||||||
|
assert.equal(successes.length, 0);
|
||||||
|
assert.equal(logged.length, 1);
|
||||||
|
assert.match(logged[0], /fetch_error.*still down/);
|
||||||
|
});
|
||||||
Vendored
+5
@@ -0,0 +1,5 @@
|
|||||||
|
<html><body>
|
||||||
|
<script>var config = {"fallbackPrice": "€9,99"};</script>
|
||||||
|
<style>.price:after { content: "€1,11"; }</style>
|
||||||
|
<p>Onze prijs: €49,95 vandaag</p>
|
||||||
|
</body></html>
|
||||||
Vendored
+3
@@ -0,0 +1,3 @@
|
|||||||
|
<html><head>
|
||||||
|
<meta property="product:price:amount" content="1.299,00" />
|
||||||
|
</head><body><p>Duur product</p></body></html>
|
||||||
Vendored
+1
@@ -0,0 +1 @@
|
|||||||
|
<html><body><p>Nu voor €1.299,00 incl. btw</p></body></html>
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
const test = require('node:test');
|
||||||
|
const assert = require('node:assert/strict');
|
||||||
|
const { start, SCHEDULE, TIMEZONE } = require('../src/scheduler');
|
||||||
|
|
||||||
|
test('start schedules the daily check with the expected cron expression and timezone', () => {
|
||||||
|
const calls = [];
|
||||||
|
const fakeCron = {
|
||||||
|
schedule: (expression, fn, options) => {
|
||||||
|
calls.push({ expression, fn, options });
|
||||||
|
return { stop: () => {} };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
const runCheck = async () => {};
|
||||||
|
start(runCheck, { cron: fakeCron });
|
||||||
|
assert.equal(calls.length, 1);
|
||||||
|
assert.equal(calls[0].expression, SCHEDULE);
|
||||||
|
assert.equal(SCHEDULE, '0 8 * * *');
|
||||||
|
assert.equal(calls[0].options.timezone, TIMEZONE);
|
||||||
|
assert.equal(TIMEZONE, 'Europe/Amsterdam');
|
||||||
|
assert.equal(calls[0].fn, runCheck);
|
||||||
|
});
|
||||||
+39
-1
@@ -2,7 +2,7 @@ const test = require('node:test');
|
|||||||
const assert = require('node:assert/strict');
|
const assert = require('node:assert/strict');
|
||||||
const fs = require('node:fs');
|
const fs = require('node:fs');
|
||||||
const path = require('node:path');
|
const path = require('node:path');
|
||||||
const { extractPrice, fetchAndExtractPrice } = require('../src/scraper');
|
const { extractPrice, fetchAndExtractPrice, parsePriceValue } = require('../src/scraper');
|
||||||
|
|
||||||
function loadFixture(name) {
|
function loadFixture(name) {
|
||||||
return fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');
|
return fs.readFileSync(path.join(__dirname, 'fixtures', name), 'utf8');
|
||||||
@@ -28,6 +28,31 @@ test('extractPrice returns null price when nothing matches', () => {
|
|||||||
assert.deepEqual(result, { price: null, method: null });
|
assert.deepEqual(result, { price: null, method: null });
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('parsePriceValue handles both thousands-separator conventions', () => {
|
||||||
|
assert.equal(parsePriceValue('1.299,00'), 1299.00);
|
||||||
|
assert.equal(parsePriceValue('1,299.00'), 1299.00);
|
||||||
|
assert.equal(parsePriceValue('19,99'), 19.99);
|
||||||
|
assert.equal(parsePriceValue('19.99'), 19.99);
|
||||||
|
assert.equal(parsePriceValue('1.234.567,89'), 1234567.89);
|
||||||
|
assert.equal(parsePriceValue(null), null);
|
||||||
|
assert.equal(parsePriceValue('geen prijs'), null);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('extractPrice reads a thousands-grouped price from a meta tag', () => {
|
||||||
|
const result = extractPrice(loadFixture('thousands-meta.html'));
|
||||||
|
assert.deepEqual(result, { price: 1299.00, method: 'meta' });
|
||||||
|
});
|
||||||
|
|
||||||
|
test('extractPrice reads a thousands-grouped price from page text', () => {
|
||||||
|
const result = extractPrice(loadFixture('thousands-regex.html'));
|
||||||
|
assert.deepEqual(result, { price: 1299.00, method: 'regex' });
|
||||||
|
});
|
||||||
|
|
||||||
|
test('extractPrice ignores euro amounts inside script and style tags', () => {
|
||||||
|
const result = extractPrice(loadFixture('script-decoy.html'));
|
||||||
|
assert.deepEqual(result, { price: 49.95, method: 'regex' });
|
||||||
|
});
|
||||||
|
|
||||||
test('fetchAndExtractPrice returns error when fetch rejects', async () => {
|
test('fetchAndExtractPrice returns error when fetch rejects', async () => {
|
||||||
const fetchImpl = async () => { throw new Error('network down'); };
|
const fetchImpl = async () => { throw new Error('network down'); };
|
||||||
const result = await fetchAndExtractPrice('https://example.invalid', { fetchImpl });
|
const result = await fetchAndExtractPrice('https://example.invalid', { fetchImpl });
|
||||||
@@ -42,6 +67,19 @@ test('fetchAndExtractPrice returns error on non-OK response', async () => {
|
|||||||
assert.ok(result.error instanceof Error);
|
assert.ok(result.error instanceof Error);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
test('fetchAndExtractPrice returns error when the request times out', async () => {
|
||||||
|
const fetchImpl = async () => {
|
||||||
|
const err = new Error('The operation was aborted due to timeout');
|
||||||
|
err.name = 'TimeoutError';
|
||||||
|
throw err;
|
||||||
|
};
|
||||||
|
const result = await fetchAndExtractPrice('https://example.invalid', { fetchImpl });
|
||||||
|
assert.deepEqual(result.price, null);
|
||||||
|
assert.equal(result.method, null);
|
||||||
|
assert.ok(result.error instanceof Error);
|
||||||
|
assert.equal(result.error.name, 'TimeoutError');
|
||||||
|
});
|
||||||
|
|
||||||
test('fetchAndExtractPrice extracts price from fetched HTML', async () => {
|
test('fetchAndExtractPrice extracts price from fetched HTML', async () => {
|
||||||
const html = loadFixture('meta-tag.html');
|
const html = loadFixture('meta-tag.html');
|
||||||
const fetchImpl = async () => ({ ok: true, status: 200, text: async () => html });
|
const fetchImpl = async () => ({ ok: true, status: 200, text: async () => html });
|
||||||
|
|||||||
Reference in New Issue
Block a user