<?xml version="1.0" encoding="UTF-8"?><rss xmlns:dc="http://purl.org/dc/elements/1.1/" xmlns:content="http://purl.org/rss/1.0/modules/content/" xmlns:atom="http://www.w3.org/2005/Atom" version="2.0"><channel><title><![CDATA[Pandapi Blog]]></title><description><![CDATA[Cute name. Serious data APIs. 🐼 Clean web data in one call: Markdown, screenshots, PDFs to Excel, contacts, tenders and more.]]></description><link>https://rabroubx.hashnode.dev</link><image><url>https://cdn.hashnode.com/uploads/logos/6af65fcd10781d16c2adef36/7e0de33b-cdc9-4dd3-af95-057a593b243e.png</url><title>Pandapi Blog</title><link>https://rabroubx.hashnode.dev</link></image><generator>RSS for Node</generator><lastBuildDate>Fri, 20 Nov 2026 19:37:24 GMT</lastBuildDate><atom:link href="https://rabroubx.hashnode.dev/rss.xml" rel="self" type="application/rss+xml"/><language><![CDATA[en]]></language><ttl>60</ttl><item><title><![CDATA[Keep your RAG index in sync with a docs site (re-embed only the pages that changed)]]></title><description><![CDATA[Re-embed only the docs pages that changed: diff sitemap lastmod, hash chunks, delete removed pages. Real runs on the Docker Compose docs.]]></description><link>https://rabroubx.hashnode.dev/keep-your-rag-index-in-sync-with-a-docs-site-re-embed-only-the-pages-that-changed</link><guid isPermaLink="true">https://rabroubx.hashnode.dev/keep-your-rag-index-in-sync-with-a-docs-site-re-embed-only-the-pages-that-changed</guid><category><![CDATA[RAG ]]></category><category><![CDATA[Python]]></category><category><![CDATA[llm]]></category><category><![CDATA[vector database]]></category><dc:creator><![CDATA[Rabii Slimane]]></dc:creator><pubDate>Wed, 18 Nov 2026 00:08:49 GMT</pubDate><enclosure url="https://cdn.hashnode.com/uploads/covers/6af65fcd10781d16c2adef36/9b6735f3-34a7-4dd6-a0b0-99a39ab093c8.png" length="0" type="image/jpeg"/><content:encoded><![CDATA[<p>Most "chat with the docs" tutorials end after the first crawl. The index looks great on day one. Three weeks later the docs have a new page, two renamed options and a deleted guide, and your bot is still quoting the deleted guide with full confidence.</p>
<p>Re-crawling everything every night works, but it re-embeds thousands of chunks that didn't change, churns every id in your vector store, and still doesn't remove the pages that are gone. This post shows a smaller loop: find out which pages changed, convert only those, re-embed only the chunks whose text changed, and delete what vanished. I ran it on the Docker Compose docs; the numbers below are from those runs.</p>
<p>The full script (about 150 lines) is in <a href="https://github.com/rabroubx-coder/MoneyAPI/tree/main/examples/rag-docs-sync">MoneyAPI/examples/rag-docs-sync</a>.</p>
<h2>Step 1: Which pages changed? Ask the sitemap, but don't trust it blindly</h2>
<p>A sitemap can carry a <code>&lt;lastmod&gt;</code> date for each page. When it's real, it's the cheapest change signal there is: one HTTP request tells you which pages changed since your last run.</p>
<p>So I checked how many docs sites actually publish it:</p>
<table>
<thead>
<tr>
<th>Docs site</th>
<th>Pages in sitemap</th>
<th>With <code>&lt;lastmod&gt;</code></th>
<th>Changed in the last 7 days</th>
<th>Last 30 days</th>
</tr>
</thead>
<tbody><tr>
<td>Kubernetes</td>
<td>1,860</td>
<td>all</td>
<td>12</td>
<td>61</td>
</tr>
<tr>
<td>Docker</td>
<td>2,207</td>
<td>1,178</td>
<td>102</td>
<td>340</td>
</tr>
<tr>
<td>FastAPI</td>
<td>153</td>
<td>none</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Playwright</td>
<td>359</td>
<td>none</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Pydantic</td>
<td>2,029</td>
<td>none</td>
<td></td>
<td></td>
</tr>
<tr>
<td>Supabase</td>
<td>2,548</td>
<td>none</td>
<td></td>
<td></td>
</tr>
</tbody></table>
<p>Two things stand out. When <code>lastmod</code> exists, the share of pages that change in a week is small (12 of 1,860 on Kubernetes), so skipping the rest saves almost all the work. And many popular docs sites don't publish it at all.</p>
<p>What about HTTP <code>ETag</code> and <code>Last-Modified</code> headers for the sites without it? I tried. docs.docker.com returned <code>Last-Modified: Tue, 17 Nov 2026 20:53:46 GMT</code> for a page that hadn't changed in weeks; playwright.dev said 18:40 the same day. Static docs sites rebuild every page on each deploy, so the headers tell you when the site was deployed, not when the page changed. Don't use them for this.</p>
<p>So the rule in the script is:</p>
<ul>
<li>page is new → convert it</li>
<li><code>lastmod</code> changed since the last run → convert it</li>
<li>no <code>lastmod</code> → convert it again every 7 days (configurable)</li>
<li>page in the last run but not in the sitemap now → delete its chunks</li>
</ul>
<pre><code class="language-python">for u in urls:
    old = known.get(u)
    if old is None:
        todo.append(u)                                    # new page
    elif sitemap[u] and sitemap[u] != old.get("lastmod"):
        todo.append(u)                                    # lastmod changed
    elif not sitemap[u] and now - old.get("checked", 0) &gt; args.recheck_days * 86400:
        todo.append(u)                                    # no lastmod: periodic recheck
gone = [u for u in known if u not in sitemap]             # removed from the site
</code></pre>
<p><code>known</code> is a small JSON state file: for each URL, the last <code>lastmod</code>, when it was checked, and a hash per chunk.</p>
<h2>Step 2: Convert only those pages to Markdown chunks</h2>
<p>Any HTML-to-Markdown converter works here as long as it drops navigation and footers and gives you chunks with their heading path. I use <a href="https://apify.com/blueberry_rainstorm/web-page-to-markdown">Website to Markdown for RAG &amp; LLMs</a> (disclosure: I built it), which returns chunks with a <code>headingPath</code> like <code>Control startup and shutdown order in Compose &gt; Control startup</code> and switches to a headless browser only for pages that need JavaScript.</p>
<pre><code class="language-python">run = client.actor("blueberry_rainstorm/web-page-to-markdown").call(run_input={
    "urls": [{"url": u} for u in todo],
    "contentMode": "main",      # drop menus, sidebars, footers
    "chunkSize": 400,           # ~tokens per chunk, split on headings
    "chunkOverlap": 50,
})
</code></pre>
<h2>Step 3: Re-embed only the chunks whose text changed</h2>
<p>Even when a page did change, most of it usually didn't. So the script hashes each chunk's text and compares it with the hash from the last run:</p>
<pre><code class="language-python">for cid, c in zip(chunk_ids(url, chunks), chunks):
    text = f'{c["headingPath"]}\n\n{c["text"]}'
    h = sha(text)
    new[cid] = h
    if old.get(cid) == h:
        continue                      # same text: keep the existing embedding
    up_ids.append(cid); up_docs.append(text)
col.upsert(ids=up_ids, documents=up_docs, metadatas=up_meta)
col.delete(ids=[cid for cid in old if cid not in new])   # sections that were removed
</code></pre>
<p>The chunk id matters more than it looks. If you use <code>url#0</code>, <code>url#1</code>, ..., adding one paragraph near the top of a page shifts every id below it, and everything gets re-embedded. The script uses <code>url#hash(heading path)-n</code> instead, so a new section only touches the chunks under its own heading.</p>
<p>Prepending the heading path to the chunk text before embedding is a cheap retrieval win too: a chunk that starts with "Use <code>depends_on</code> with <code>condition: service_healthy</code>..." is far easier to find when the embedding also knows it sits under "Control startup and shutdown order".</p>
<h2>The runs</h2>
<p>All on the Docker Compose section of docs.docker.com (sitemap URLs containing <code>/compose/</code>), with a local Chroma index:</p>
<table>
<thead>
<tr>
<th>Run</th>
<th>Pages converted</th>
<th>Chunks re-embedded</th>
<th>Chunks deleted</th>
<th>Cost</th>
</tr>
</thead>
<tbody><tr>
<td>1. First index</td>
<td>87 (86 OK, 1 empty page)</td>
<td>525</td>
<td>0</td>
<td>$0.086</td>
</tr>
<tr>
<td>2. Same day, force a recheck of pages without <code>lastmod</code></td>
<td>45</td>
<td><strong>0</strong> (156 compared, all unchanged)</td>
<td>0</td>
<td>$0.044</td>
</tr>
<tr>
<td>3. Five pages with a new <code>lastmod</code>, one page removed from the sitemap</td>
<td>6</td>
<td><strong>0</strong> (46 compared, text unchanged)</td>
<td>3 (the removed page)</td>
<td>$0.006</td>
</tr>
</tbody></table>
<p>(For run 3 I edited the state file to give five pages an older <code>lastmod</code> and to add a page that no longer exists, since I couldn't make Docker edit their docs on cue.)</p>
<p>Run 3 shows why step 3 is worth having: <code>lastmod</code> said five pages changed, but their text was the same, so nothing was re-embedded. Sitemap generators often bump <code>lastmod</code> for template or front-matter changes.</p>
<p>The "1 empty page" is the Compose release notes page, which on docs.docker.com today really is an empty stub. Failed pages are reported and not charged, and the script keeps whatever it had indexed for them.</p>
<p>A sanity query against the index:</p>
<pre><code class="language-python">col.query(query_texts=["How do I make one service wait until the database is ready?"], n_results=3)
</code></pre>
<p>returned "Control startup and shutdown order in Compose", "Run jobs in Compose &gt; When to use jobs" and "Use init containers in Compose", with their URLs.</p>
<h2>What I learned the hard way: check your converter is deterministic</h2>
<p>On my first attempt, run 2 reported two CLI reference pages as "changed" even though nothing had changed. The cause: docs.docker.com serves Markdown instead of HTML to clients that accept <code>text/markdown</code>, and for generated pages like <code>docker compose images</code> that Markdown is an almost empty stub. My converter saw a near-empty page, fell back to a headless browser, and the browser render picked up slightly different page furniture each time (a chat widget one run, nav links the next).</p>
<p>I fixed it in the converter (it now asks for the HTML page before reaching for a browser), and both pages have been stable since. The general lesson: <strong>if a page shows up as changed on every run, your pipeline is non-deterministic, not the docs.</strong> Log which pages change (the script prints them) and look at any page that changes twice in a row.</p>
<h2>Running it on a schedule</h2>
<pre><code class="language-bash">pip install apify-client chromadb requests
export APIFY_TOKEN=...      # free Apify account
python rag_docs_sync.py https://docs.docker.com/sitemap.xml --include /compose/
</code></pre>
<p>Put that in cron or a GitHub Actions schedule. Keep <code>docs-index/</code> and <code>sync-state.json</code> together; if you lose the state file, the next run simply re-indexes everything once. To use Pinecone, Qdrant, pgvector or anything else, change the <code>upsert</code> and <code>delete</code> calls; the change detection doesn't care.</p>
<h2>Cost</h2>
<p>Conversion is $1 per 1,000 pages converted, failed pages free. For a 2,000-page docs site with <code>lastmod</code> and ~100 changes a week (Docker's rate), that's about $0.10 a week. Without <code>lastmod</code>, the weekly recheck converts every page: $2 a week for 2,000 pages, and still only the changed chunks get re-embedded. A free Apify account's monthly credit covers the Docker example many times over.</p>
<p>The free route for the conversion step: <code>requests</code> plus <code>trafilatura</code> or <code>markdownify</code>, with your own heading-aware chunking and your own fallback for JavaScript-only pages. The change-detection part of the script works the same either way.</p>
<hr />
<p>If you run this against another docs site and a page keeps showing up as changed, send me the URL (the actor's Issues tab works). That's exactly the kind of bug I want to find.</p>
<p><em>Disclosure: I make the Markdown converter used here at Pandapi. 🐼</em></p>
]]></content:encoded></item><item><title><![CDATA[One Python script for government tenders from SAM.gov, TED, the UK, Canada and the World Bank]]></title><description><![CDATA[Open government tenders from SAM.gov, TED (EU), UK, Canada and the World Bank in one list with Python: the free API way and a one-feed way.]]></description><link>https://rabroubx.hashnode.dev/one-python-script-for-government-tenders-from-sam-gov-ted-the-uk-canada-and-the-world-bank</link><guid isPermaLink="true">https://rabroubx.hashnode.dev/one-python-script-for-government-tenders-from-sam-gov-ted-the-uk-canada-and-the-world-bank</guid><category><![CDATA[Python]]></category><category><![CDATA[api]]></category><category><![CDATA[open data]]></category><category><![CDATA[data-engineering]]></category><dc:creator><![CDATA[Rabii Slimane]]></dc:creator><pubDate>Mon, 16 Nov 2026 01:40:08 GMT</pubDate><content:encoded><![CDATA[<p>If you sell to governments, the work starts with finding the tender. Then you find out every country publishes them somewhere different, in a different format, with a different search that matches different things.</p>
<p>I wanted one list: every open cybersecurity tender from the last two weeks, from the US, the EU, the UK, Canada and World Bank projects, with a deadline at least a week away. Here's the free way, where it hurts, and what I ended up using.</p>
<h2>The free way: call each portal yourself</h2>
<p>All of these portals have official open-data APIs. Some are pleasant to use and some are not.</p>
<p><strong>TED (EU)</strong> is the nicest one. You don't need a key, and the expert query language is powerful:</p>
<pre><code class="language-python">import requests

r = requests.post("https://api.ted.europa.eu/v3/notices/search", json={
    "query": '(notice-title ~ "cybersecurity" OR description-proc ~ "cybersecurity") AND publication-date&gt;=20261102',
    "fields": ["notice-title", "buyer-name", "deadline-receipt-tender-date-lot", "buyer-country"],
    "limit": 50,
})
for n in r.json()["notices"]:
    print(n["notice-title"].get("eng"), n["buyer-name"])
</code></pre>
<p>What comes back is multilingual on purpose. <code>buyer-name</code> is <code>{"fra": ["Infrabel sa"], "nld": ["Infrabel nv"]}</code>, deadlines are per lot, and values sit in different fields depending on the notice form. That's fine, it's just another parser to write.</p>
<p><a href="http://SAM.gov"><strong>SAM.gov</strong></a> <strong>(US)</strong> needs an API key from your <a href="http://SAM.gov">SAM.gov</a> account. A personal key for a non-federal user with no role is limited to <a href="https://open.gsa.gov/api/get-opportunities-public-api/">10 requests a day</a>. Its keyword search also matches attachments and the standard contract clauses that almost every defense notice carries. In my test window, <strong>164 of 234</strong> <a href="http://SAM.gov">SAM.gov</a> notices matched "cybersecurity" only because of boilerplate like "applicable cybersecurity standards" deep inside the clause list, and they weren't cybersecurity work at all.</p>
<p><strong>The UK</strong> has two systems: Find a Tender (above threshold, OCDS JSON) and Contracts Finder (below threshold, a different API). Find a Tender rate-limits hard. Ask for too much and you get HTTP 429 with "try again in 2 minutes".</p>
<p><strong>CanadaBuys</strong> publishes a daily open-data file rather than a search API. The <strong>World Bank</strong> has its own procurement notices API.</p>
<p>That makes five APIs, five schemas, several date formats and four kinds of industry code (CPV, NAICS, UNSPSC, GSIN), and you still have to dedupe, convert currencies and filter on deadlines. If you only care about one country, use its API directly. It's free and you'll learn the data.</p>
<h2>The lazy way: one feed</h2>
<p>I make a feed on Apify that does that plumbing. Disclosure: I make this at Pandapi.</p>
<pre><code class="language-python"># pip install "apify-client&gt;=3"
import os
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])
run = client.actor("blueberry_rainstorm/public-tenders-feed").call(run_input={
    "keywords": ["cybersecurity", "penetration testing"],
    "postedWithinDays": 14,
    "minDaysLeft": 7,
})

for t in client.dataset(run.default_dataset_id).iterate_items():
    print(f'{t["countryName"]:&lt;15} {str(t["daysLeft"] or "-"):&gt;3}d  {t["valueEur"] or "":&gt;10}  {t["title"][:70]}')
</code></pre>
<p>It returned <strong>22 notices from 4 of the 6 portals</strong> (TED 11, World Bank 6, <a href="http://SAM.gov">SAM.gov</a> 3, CanadaBuys 2; the UK had none that matched). Every notice has the same fields: title, buyer, country, deadline, days left, value in the original currency plus USD and EUR, classification codes and the link to the official notice. A few I'd actually look at:</p>
<table>
<thead>
<tr>
<th>Buyer</th>
<th>Country</th>
<th>What</th>
<th>Deadline</th>
<th>Value (EUR)</th>
</tr>
</thead>
<tbody><tr>
<td>ENISA (EU Agency for Cybersecurity)</td>
<td>Greece</td>
<td>Security assessment services</td>
<td>14 Dec</td>
<td>3,000,000</td>
</tr>
<tr>
<td>HIQA</td>
<td>Ireland</td>
<td>Continuous pen test tool suite and services</td>
<td>18 Dec</td>
<td>550,000</td>
</tr>
<tr>
<td>Aeroporti di Puglia</td>
<td>Italy</td>
<td>Cybersecurity monitoring services</td>
<td>21 Dec</td>
<td>600,000</td>
</tr>
<tr>
<td>EU Agency for the Space Programme</td>
<td>Spain</td>
<td>PKI L2/L3 maintenance</td>
<td>16 Dec</td>
<td>1,100,000</td>
</tr>
<tr>
<td>Danmarks Nationalbank</td>
<td>Denmark</td>
<td>Framework for cybersecurity advisory and assessment</td>
<td>n/a</td>
<td>1,471,454</td>
</tr>
<tr>
<td>Ministry of Telecom (World Bank project)</td>
<td>Palestinian Territories</td>
<td>Cybersecurity maturity assessment</td>
<td>12 Dec</td>
<td>n/a</td>
</tr>
</tbody></table>
<p>Two things I care about in practice:</p>
<ul>
<li><a href="http://SAM.gov"><strong>SAM.gov</strong></a> <strong>boilerplate is filtered.</strong> By default a <a href="http://SAM.gov">SAM.gov</a> notice is kept only if the keyword is in the title, the buyer, or the opening of the real description with the clause lists removed. That's how 234 notices became 3. If you really want deep mentions too, set <code>strictKeywordMatch</code> to <code>false</code>.</li>
<li><code>matchedInTitle</code> marks the strongest hits. Sort on it to get a short list for the bid/no-bid meeting.</li>
</ul>
<h2>What still needs a human</h2>
<ul>
<li>Some notices mention your keyword only as one requirement in a bigger project. A Romanian substation rebuild and a Canadian drone procurement both list cybersecurity in their scope. They're correct matches, but they aren't cybersecurity tenders. Read the titles.</li>
<li>English keywords find many non-English EU notices because TED translates titles, but not all of them. A Croatian SOC tender written only in Croatian didn't match. For the EU, CPV codes are more precise: <code>"cpvCodes": ["72220000", "79417000"]</code>.</li>
<li>Some TED notices publish no deadline or value at the first stage. The feed keeps them so you don't miss them, and shows the fields as empty.</li>
<li>The UK's Find a Tender rate limit makes runs that include it take a few minutes instead of seconds.</li>
</ul>
<h2>Cost</h2>
<p>$2 per 1,000 notices, no start fee, and nothing is charged for notices that are filtered out. This run returned 22 notices, so it cost <strong>$0.044</strong>. With <strong>Only new notices</strong> on and a daily schedule, a 30-notices-a-day alert costs about $0.06 a day, and a free Apify account's $5 monthly credit covers that.</p>
<h2>Which one should you use?</h2>
<ul>
<li>One country, or you want to learn the data: call that portal's API directly. TED's is a pleasure to use.</li>
<li>Several countries, a daily alert into Slack, email or a sheet, or a feed for an AI bid-triage agent: a combined feed saves the parsers. Mine is here: <a href="https://apify.com/blueberry_rainstorm/public-tenders-feed">Government Tenders &amp; Public Procurement API</a>. There are others on Apify Store, so compare them on your own keywords.</li>
</ul>
<p>The full script and more runnable examples are in <a href="https://github.com/rabroubx-coder/MoneyAPI/tree/main/examples/government-tenders-feed">MoneyAPI/examples</a> on GitHub.</p>
<p><em>Disclosure: I make this at Pandapi. 🐼</em></p>
]]></content:encoded></item><item><title><![CDATA[Pull every table out of a pile of PDFs into one Excel file (borderless tables too)]]></title><description><![CDATA[Get ruled and borderless tables out of a batch of PDFs into one Excel workbook, with numbers you can sum. Python and hosted options.]]></description><link>https://rabroubx.hashnode.dev/pull-every-table-out-of-a-pile-of-pdfs-into-one-excel-file-borderless-tables-too</link><guid isPermaLink="true">https://rabroubx.hashnode.dev/pull-every-table-out-of-a-pile-of-pdfs-into-one-excel-file-borderless-tables-too</guid><category><![CDATA[Python]]></category><category><![CDATA[excel]]></category><category><![CDATA[pdf]]></category><category><![CDATA[data extraction]]></category><dc:creator><![CDATA[Rabii Slimane]]></dc:creator><pubDate>Fri, 13 Nov 2026 01:04:10 GMT</pubDate><content:encoded><![CDATA[<p>Someone hands you a folder of PDFs and asks for "the numbers in a spreadsheet". Copy and paste works for one file. By the third file you notice the columns slide around, <code>$1,250.00</code> lands in Excel as text, and the table that runs over four pages comes out as four tables with the header repeated each time.</p>
<p>This post covers how to get tables out of many PDFs at once and into a single Excel workbook, with the free do-it-yourself route first and then a hosted route that needs about 15 lines of Python (or none, if you use the web console).</p>
<h2>Why PDF tables are hard</h2>
<p>A PDF does not contain a "table". It contains characters placed at x/y positions, and sometimes some lines drawn between them. Every extractor has to guess the table back from that. The guess goes wrong in a few predictable places:</p>
<ul>
<li><strong>Borderless tables.</strong> Bank statements and annual reports often have no grid lines, only text lined up in columns. Tools that look for ruled lines see nothing.</li>
<li><strong>Tables that continue on the next page.</strong> You want one table with one header, not one table per page.</li>
<li><strong>Wrapped cells.</strong> A long description that wraps onto a second line becomes a second, mostly empty row.</li>
<li><strong>Numbers as text.</strong> <code>(300)</code> means minus 300 in accounting, <code>12.5%</code> should be 0.125, and <code>$1,250.00</code> should be 1250. Paste them into Excel and <code>SUM()</code> returns 0.</li>
<li><strong>Scanned pages</strong> have no text at all until someone runs OCR.</li>
</ul>
<h2>Option 1: do it yourself in Python (free)</h2>
<p>If you have a handful of PDFs with ruled tables, a library is the right answer:</p>
<pre><code class="language-python">import pdfplumber, pandas as pd

with pdfplumber.open("statement.pdf") as pdf:
    frames = []
    for page in pdf.pages:
        for table in page.extract_tables():
            frames.append(pd.DataFrame(table[1:], columns=table[0]))

pd.concat(frames).to_excel("statement.xlsx", index=False)
</code></pre>
<p><code>camelot</code> is the other common choice: its <code>lattice</code> mode is for ruled tables and its <code>stream</code> mode for borderless ones. Both libraries are good, and you should try them first. Where people usually give up is the rest of the list above: borderless layouts that need per-document tuning, joining tables across pages, cleaning numbers, OCR, and then running all of it over 200 files on a schedule.</p>
<h2>Option 2: a hosted extractor that returns one workbook for the whole batch</h2>
<p>I built an Apify Actor for the bulk case: <a href="https://apify.com/blueberry_rainstorm/pdf-to-text-markdown">PDF to Text, Markdown &amp; Excel</a>. Disclosure: it is my tool. You give it PDF links (or uploads), and it returns:</p>
<ul>
<li>an <strong>Excel workbook per PDF</strong>, one sheet per table plus an index sheet,</li>
<li>a <strong><code>tables.xlsx</code> with every table from every PDF in the run</strong>,</li>
<li>optional CSV per table, plus Markdown, plain text and JSON for each PDF.</li>
</ul>
<p>It costs $3 per 1,000 PDFs, plus $5 per 1,000 pages only for scanned pages that need OCR. PDFs that fail (404, not a PDF, wrong password) are not charged. Apify's free plan includes monthly credit, which covers a test like the one below many times over.</p>
<h3>What it did on real documents</h3>
<p>Here is part of one test run of 7 PDFs (107 pages in total). It found 65 tables, and the platform usage for the whole run came to about $0.02.</p>
<table>
<thead>
<tr>
<th>PDF</th>
<th>What came out</th>
</tr>
</thead>
<tbody><tr>
<td>IRS Publication 15-T (federal income tax withholding tables, 68 pages)</td>
<td>56 tables. The longest one ran from <strong>page 37 to page 49</strong> and came out as <strong>one sheet of 589 rows</strong>, with the repeated page header removed. Others spanned pages 14-19 (359 rows) and 29-35 (314 rows).</td>
</tr>
<tr>
<td>A 2-page bank statement</td>
<td>One table of 49 rows across both pages, with Date, Description, Debit, Credit and Balance as real numbers.</td>
</tr>
<tr>
<td>"Attention Is All You Need" (arXiv 1706.03762)</td>
<td>The 3 results tables from pages 8 to 10, including the 13-column one on page 9.</td>
</tr>
<tr>
<td>A government budget summary (borderless, 9 columns)</td>
<td>41 rows, numbers as numbers, and <code>...</code> kept as text where the PDF had no value.</td>
</tr>
</tbody></table>
<p>All 56 IRS tables were found as borderless tables: the publication has no grid lines, only aligned columns. That is the case where line-based extraction finds nothing.</p>
<h2>Step by step</h2>
<h3>1. Run it</h3>
<p>You can paste the PDF links into the Actor's input form on Apify and click Start. To do it from code:</p>
<pre><code class="language-bash">pip install apify-client pandas openpyxl
</code></pre>
<pre><code class="language-python">import io, os
import pandas as pd
from apify_client import ApifyClient

client = ApifyClient(os.environ["APIFY_TOKEN"])

pdf_urls = [
    "https://www.irs.gov/pub/irs-pdf/p15t.pdf",
    # ...add as many as you like
]

run = client.actor("blueberry_rainstorm/pdf-to-text-markdown").call(run_input={
    "urls": [{"url": u} for u in pdf_urls],
    "tableDetection": "all",          # ruled AND borderless tables
    "spreadsheetOutput": "xlsx",      # or "csv" / "both"
    "mergeMultiPageTables": True,     # one table, one header, across pages
    "numbersAsNumbers": True,         # "$1,250.00" -&gt; 1250, "(300)" -&gt; -300
})
</code></pre>
<h3>2. Check which PDFs worked</h3>
<p>Each PDF is one item in the run's dataset, with a <code>status</code>, a <code>tableCount</code> and a link to its own workbook:</p>
<pre><code class="language-python">for item in client.dataset(run["defaultDatasetId"]).iterate_items():
    if item["status"] == "ok":
        print(item["fileName"], item["tableCount"], "tables -&gt;", item["excelUrl"])
    else:
        print(item.get("url"), "FAILED:", item["error"])   # not charged
</code></pre>
<p>Each item also has a <code>tables</code> field with every table's page range, title, header and rows as JSON, if you would rather not touch Excel at all.</p>
<h3>3. Get the combined workbook into pandas</h3>
<pre><code class="language-python">store = client.key_value_store(run["defaultKeyValueStoreId"])
xlsx = store.get_record("tables.xlsx")["value"]
open("tables.xlsx", "wb").write(xlsx)

sheets = pd.read_excel(io.BytesIO(xlsx), sheet_name=None)   # {sheet name: DataFrame}
print(sheets["Index"])           # file, table no., pages, title, rows, columns, sheet name
</code></pre>
<p>The <code>Index</code> sheet tells you which sheet holds which table (sheet names look like <code>p15t p37-49 t26</code>: file, pages, table number). Because the numbers were written as numbers, totals work straight away:</p>
<pre><code class="language-python">stmt = sheets["sample-bank-stat p1-2 t1"]   # sheet name from the Index sheet
print(stmt["Debit"].sum(), stmt["Credit"].sum())
</code></pre>
<p>If you would rather have one long sheet for the whole batch (handy for 200 monthly statements with the same layout), set <code>"spreadsheetLayout": "oneSheet"</code>.</p>
<h2>Tips from the test runs</h2>
<ul>
<li><strong>Multi-row headers.</strong> Tax tables and financial reports often have headers spread over two or three lines ("At least / But less than / The Tentative Withholding Amount is:"). Those lines come out as the first rows of the sheet. In the JSON, <code>headerRows</code> tells you how many there are, so you can skip or merge them in one line of pandas.</li>
<li><strong>Forms with lots of aligned numbers that are not tables.</strong> Switch <code>tableDetection</code> to <code>"ruled"</code> so only tables drawn with lines are picked up.</li>
<li><strong>Only some pages matter?</strong> Use <code>pageRange</code>, for example <code>"37-49"</code> or <code>"-2"</code> for the last two pages. It is faster and keeps the workbook small.</li>
<li><strong>Scanned PDFs.</strong> OCR (12 languages) gives you the text, but table detection needs a text layer, so for scanned tables you still want a dedicated OCR table service. This is the main gap I know of.</li>
<li><strong>Leading zeros stay text.</strong> Account numbers and ZIP codes like <code>00501</code> are not turned into numbers.</li>
<li><strong>Run it on a schedule.</strong> Apify can run it daily or weekly on a fixed list of URLs (for example a report that is republished monthly) and send the result to Make, n8n, Zapier or a webhook.</li>
</ul>
<h2>Which option to pick</h2>
<ul>
<li><strong>A few PDFs, ruled tables, you write Python:</strong> pdfplumber or camelot, free.</li>
<li><strong>One-off, non-technical, a couple of files:</strong> an online converter is fine.</li>
<li><strong>Dozens to thousands of PDFs, borderless tables, multi-page tables, or a recurring job:</strong> a hosted batch extractor saves you the cleanup. That is the case I built <a href="https://apify.com/blueberry_rainstorm/pdf-to-text-markdown">PDF to Text, Markdown &amp; Excel</a> for.</li>
</ul>
<p>If you run it on a PDF and a table comes out wrong, open an issue on the Actor's Issues tab with the link to the PDF. Every report gets looked at, and messy real documents are the best test cases I can get.</p>
<hr />
<p><em>Written with help from an AI assistant; the test results, numbers and code were checked against real runs of the tool.</em></p>
]]></content:encoded></item></channel></rss>