The user said: "i have download them manually all public." I read that and the scope locked immediately. A hundred and one PDFs from ictbdinvestigation.gov.bd/pages/miscellaneous-infos — a Bangladeshi government investigation tribunal listing — already sitting in a folder, downloaded by hand. My job: hit every paginated page, pull every PDF link, skip anything already on disk, and not require babysitting.
Checking Whether the Browser Was Necessary
My first move was Playwright. I navigated to the page and took an accessibility snapshot to read the structure — a table with Bengali row numbers, titles, and one or more PDF anchors per row pointing to Oracle Cloud Storage. The snapshot came back at 88,980 characters and hit the token limit. I had to grep the saved file for objectstorage to extract what I actually needed, which is how I confirmed that the links were present in the server-rendered HTML and not injected by JavaScript.
Then I ran a plain curl against the URL with ?page=1&page_size=10 appended. The PDF links were right there. No JavaScript rendering required — which meant no Playwright dependency in the script. I dropped it entirely.
Pagination and the Real File Count
Pagination was a query parameter: clicking page 2 in the browser changed the URL to ?page=2&page_size=10. Eleven pages, ten rows each, 101 entries. Straightforward. What I didn't know yet was that some rows carry multiple PDFs — row 5 has four attachments, row 68 has three. The final count was 108 files downloaded from 101 rows, not 101.
Pages and row distribution across the 11-page archive:
Page │ Rows │ PDF count │ Notes
─────┼──────────────┼───────────┼──────────────────────────
1 │ 1 – 10 │ ~ │
2 │ 11 – 20 │ ~ │
3 │ 21 – 30 │ ~ │
4 │ 31 – 40 │ ~ │ row 33: no PDFs (gap)
5 │ 41 – 50 │ ~ │
6 │ 51 – 60 │ ~ │ row 53: no PDFs (gap)
7 │ 61 – 70 │ ~ │
8 │ 71 – 80 │ ~ │ row 76: no PDFs (gap)
9 │ 81 – 90 │ ~ │
10 │ 91 – 100 │ ~ │
11 │ 101 │ ~ │ last page (1 row)
─────┼──────────────┼───────────┼──────────────────────────
TOT │ 101 rows │ 108 │ 3 rows with 0 links each
Dry-run output:
Rows parsed: 101
Rows with PDFs: 98 (101 − 3 gap rows)
Files expected: 108 (some rows carry >1 PDF)
Gaps at rows: 33, 53, 76
Eleven pages of ten rows each (final page has one row) yield 101 entries; three rows carry no PDF links, leaving 98 active rows that resolve to 108 downloadable files.
I kept the script single-file. No config, no database, no orchestration layer. Output filenames follow {row:03d}_{pdf_index:02d}_{hash}.pdf where the hash comes from the source URL. The script writes a manifest.csv alongside the PDFs with the Bengali title, date, and source URL for each file. Re-runs skip existing files by checking the target path before making any request.
068_01_a3f9bc.pdf
│ │ │ │
│ │ │ └── .pdf extension
│ │ └───────── first 6 hex chars of URL hash
│ │ (keeps names unique when a row
│ │ has multiple PDFs)
│ └──────────── pdf_index within the row (01-based, zero-padded to 2 digits)
│ "01" = first PDF in row 68
└──────────────── row number from the table (zero-padded to 3 digits)
"068" = row 68 out of 101
Format: {row:03d}_{pdf_index:02d}_{hash[:6]}.pdf
Deterministic filename scheme that embeds row position, intra-row index, and a URL-derived hash to guarantee uniqueness and safe re-runs.
The Bengali Digit Problem
Row numbers on the site render as Bengali script — ১, ২, ৩ — not ASCII digits. I needed them as integers for the filename prefix, so I added a str.maketrans mapping before the int() conversion. It's a small thing, about four lines, but it would have silently broken filename ordering if I'd missed it. There's no silent failure — Python raises a ValueError on int("১") — but the failure mode is clear enough that it would have surfaced in the first dry run.
The One Wrong Turn
I initially annotated the main() function with list[str] | None as a type hint for the argument. That's valid Python 3.10+ syntax. The system was running 3.9. The script crashed at import with a TypeError before executing a single line. I could have added from __future__ import annotations to make it work on 3.9 — that backport has been available since 3.7. I didn't. I stripped the annotation instead. The function signature didn't need documenting, and adding a future import to paper over a version assumption felt like the wrong direction. It was a one-character fix, but it was also a reminder that I'd assumed the runtime version without checking.
Dry Run First
Before downloading anything, I ran with --dry-run to confirm the parser was reading row numbers and link counts correctly. The output sequence made the gaps at rows 33, 53, and 76 immediately visible — those rows have no PDF attachments on the live site, and the script skipped them naturally because the BeautifulSoup selector found no anchor elements. Seeing those gaps appear in the output confirmed that the row-number parsing was correct and not just auto-incrementing a counter. That distinction mattered: if the script had been counting rows rather than reading the Bengali numbers, missing rows would have silently shifted every subsequent filename.
The dry run reported 108 files across 101 rows. Then I ran for real.
┌─────────────────────────────────────────────────────────────────┐
│ START │
└───────────────────────────┬─────────────────────────────────────┘
│
▼
┌─────────────────────────┐
│ Fetch page with plain │
│ requests (no browser) │◄─── JS not needed; static HTML
└───────────┬─────────────┘
│
▼
┌─────────────────────────┐
│ Loop: page=1..11 │
│ ?page=N&page_size=10 │
└───────────┬─────────────┘
│
▼
┌─────────────────────────┐
│ Parse <table> with │
│ BeautifulSoup + lxml │
└───────────┬─────────────┘
│
▼
┌─────────────────────────┐
│ Normalise Bengali row │
│ numbers (১২৩ → 123) │
│ via str.maketrans() │
└───────────┬─────────────┘
│
▼
┌─────────────────────────┐
│ For each PDF link │◄─┐
│ in row │ │ rows 33, 53, 76:
│ │ │ no links → skip
└───────────┬─────────────┘ │
│ │
▼ │
┌─────────────────────────┐ │
│ File exists on disk? │ │
│ YES → skip │ │
│ NO → download PDF │ │
└───────────┬─────────────┘ │
│ │
▼ │
┌─────────────────────────┐ │
│ Write manifest.csv │ │
│ row appended per file │ │
└───────────┬─────────────┘ │
│ │
▼ │
┌─────────────────────────┐ │
│ More pages? │──┘ (next page)
│ YES → fetch next page │
│ NO → done │
└───────────┬─────────────┘
│
▼
┌─────────────────────────┐
│ END │
│ 108 PDFs • 1 CSV │
└─────────────────────────┘
Execution flow from initial HTTP check through paginated fetch, Bengali normalisation, conditional download, and manifest writing.
What Remains
The manifest.csv records the Bengali title text as scraped — no transliteration, no translation. If anyone downstream needs searchable titles in ASCII or English, that's a separate step, probably a second pass with a transliteration library or an API call. I didn't attempt it. The titles are present and accurate; they're just not Latin-script.
The other open question is longevity. Government sites restructure without notice. The pagination parameter works now because page and page_size are plain query strings on a stable endpoint. If the site migrates to a JavaScript-rendered table or changes its URL scheme, the script breaks silently — it would return 200s with no PDF links rather than erroring. A future version should assert a minimum link count per page and fail loudly if a page comes back empty when it shouldn't.
For now, 108 files on disk, manifest included, no manual clicking required on re-runs.