A personalized feed of recent medical literature for one clinician at a time —
ranked, explained, and safe to read as continuing education. This page explains the
whole thing twice: once in plain words, then stage-by-stage with the engineering underneath.
PubMed publishes 2,000–4,000 new papers every day. No clinician can keep up, and generic
email alerts dump everything unfiltered, unranked, unexplained — so they get ignored.
"Which recent papers deserve my attention right now, why, and how solid are they for my practice?"
MedPulse answers that question. It reads PubMed for you and does four things:
1 · FILTERKeeps only papers in your field (here: fertility & endometriosis), even brand-new ones.
2 · RANKOrders them by how well they match your practice, not by popularity.
3 · EXPLAINFor each paper: a plain summary, five scores, and why each score — never "do this to a patient."
The result is a short weekly digest you can scan in minutes — and it gets sharper the more you use it.
The pipeline each stage: what it does, then how
1
Ingest curl + md5
DoesPulls the raw library of medicine onto disk — the authoritative source, no middleman.
HowDownloads the PubMed annual baseline (.xml.gz) from the NCBI FTP, verifies MD5 checksums. Data lives in ~/data/pubmed/, never in git.
NCBI FTP~30k records / file
2
Parse & filter parse_articles.py
DoesThrows away everything outside the specialty, keeping fresh papers even before they're tagged.
HowStream-parses each file; keeps an article on a MeSH tag match OR a repro-anchored keyword match on title+abstract. High-recall on purpose — ~79% of fresh articles have no MeSH yet, so keywords are the safety net.
Python · iterparseMeSH ∪ lexical→ JSONL
3
Embed embed.py
DoesTurns each paper into a "meaning fingerprint" so similar topics sit close together — that's how relevance is measured.
HowMedCPT-Article-Encoder (NCBI's medical retrieval model) → a 768-number vector per title+abstract. Local CPU, parallelized across worker processes, scoped to recent articles.
MedCPT (local)torch CPU · --workers768-d vector
4
Persist db.py · schema.sql
DoesStores papers, profiles, past appraisals, and feedback so the system is fast and remembers.
HowPostgres + pgvector with an HNSW index for fast nearest-vector search. Tables: articles (+vector), profiles (+vector), appraisals (with model/prompt provenance, cached), feedback.
Postgres 16pgvector · HNSWpsycopg
5
Retrieve db.py
DoesFinds the recent papers closest to your practice — by meaning, so it catches synonyms and new drugs keyword search misses.
HowEmbeds the clinician profile with MedCPT-Query-Encoder (jointly trained with the article side; the encoder lives in search.py), then ranks by cosine distance in Postgres — the embedding <=> operator, which the planner runs through the HNSW index (verified: ~2 ms). Restricted to the recency window — load-bearing, or evergreen reviews bury fresh papers. search.py also offers a standalone in-memory (numpy) equivalent for quick CLI checks.
DoesReads each top paper and writes a plain summary + five reasoned scores — the part that makes the feed trustworthy, not just a list.
HowAn LLM (Codex CLI — no paid API) returns strict JSON under a regulatory-safe rubric: relevance, evidence (est.), practice relevance, novelty, analysis confidence, each with a reason. Cached by (pmid, profile, prompt-version).
relevanceevidence (est.)practicenoveltyconfidence
7
Digest digest.py
DoesAssembles the finished feed the clinician actually reads.
HowRanks by refined relevance; renders each item with its explained scores, a six-field summary, a source link, and the mandated footer ("clinical watch, not medical advice").
ranked markdowndisclaimersource links
Handling missing MeSH the freshness problem, and the fix
The catch with a "fresh" feed: PubMed's curated MeSH topic tags are the natural
thing to filter on — but the NLM assigns them with a lag, so the newest papers don't have them
yet. We measured it on the latest data: ~79% of just-published articles arrive with no MeSH
at all. Filtering on MeSH alone would silently drop the majority of the freshest, most
valuable papers — the exact opposite of what a current-awareness feed should do.
So the wedge filter works in layers, recall first:
NOW · MeSH matchWhen tags exist, match curated MeSH descriptors. Precise, but only covers already-indexed articles.
NOW · lexical fallbackWhen MeSH is absent, match repro-specific keywords in the title + abstract. This is what rescues fresh, untagged papers.
NEXT · learned taggerTrain a classifier on MedCPT embeddings — using PubMed's millions of MeSH-labelled articles as training data — to predict the wedge for untagged papers. Better precision than keywords; or reuse an existing tagger (NLM MTI, BERTMeSH).
Why this is safe: the filter only needs high recall —
it casts a wide net on purpose. Precision is the embedding's job: MedCPT ranks genuinely
relevant papers up and pushes keyword false-positives down. We verified the failure mode too — an
over-loose keyword set once pulled in 83% cross-specialty noise (crab genetics, knee surgery),
so the keywords are anchored to genuinely reproductive terms.
Closing the loop what makes it improve and how we test it
↺
Feedback feedback.py
DoesYour taps (useful / not / wrong specialty…) retune the feed to you, instantly.
HowRocchio relevance feedback shifts the profile's query vector toward liked papers, away from disliked ones. No model training — pgvector just re-ranks against the adjusted vector.
A/B
Validation compare.py
DoesTests whether MedPulse is actually better than a normal PubMed alert — judged blind by a human.
HowRenders the same week two ways (MedPulse vs a recency baseline), anonymized to Feed A/B with a scoring sheet; answer key kept separate. The question: "which would you rather receive Monday?"
External components the four things it talks to
PubMed FTP (NCBI)
The source. Annual baseline + daily updates. Gives abstracts + MeSH tags, not full text.
MedCPT (HuggingFace)
NCBI's medical retrieval encoders (article + query). Runs locally on CPU — no API cost.
Codex CLI
The LLM that writes scores + summaries, via strict --output-schema. Deliberately not the paid Anthropic API.
Postgres 16 + pgvector
Vector store + HNSW index, in a Docker container on host port 5433.
Why it's built this way the load-bearing decisions
Wide net, then precisionCheap high-recall filter (MeSH ∪ keywords); the medical embedding does the relevance sorting.
Explained, not just scoredEvery score carries its reasoning — the differentiator vs a generic summarizer or alert.
Regulatory-safe by construction"Estimated, not GRADE"; literature triage, never clinical advice. Enforced in the rubric.
Local & freeMedCPT on CPU + Codex CLI for scoring — no per-article paid-API cost at scale.
Feedback is the moatPrecise feedback is the training signal that sharpens ranking per clinician over time.
Single-specialty wedgeProve quality narrow (PMA / endometriosis) before going broad.
Glossary the jargon, defined
Wedge
The narrow starting market — here, fertility (PMA) & endometriosis — chosen to prove quality before going multi-specialty.
MeSH
Medical Subject Headings — NLM's curated topic tags on PubMed articles. Precise, but missing on freshly published papers.
Embedding / vector
A list of numbers representing a text's meaning. Texts with similar meaning have nearby vectors.
MedCPT
NCBI's biomedical retrieval model. Two halves: one embeds articles, one embeds queries — trained together to match.
pgvector / HNSW
A Postgres extension that stores vectors and finds nearest ones fast (HNSW is the index that makes it quick).
Recency window
Only ranking the last weeks' articles — so a fresh feed, not a re-rank of all history.
Evidence level
A rough study-strength estimate (RCT & meta-analysis high, case report low). An estimate, not a formal GRADE rating.
Rocchio feedback
A classic technique: nudge the query toward things you liked and away from things you didn't — no model retraining.