Work / PEP Health

PEP Health: reading patient comments for meaning, not just keywords

Navi Insights built AI natural language processing models that read every patient comment automatically and decide which service lines it belongs to, so that the feedback for each service line can be pulled into its own report.

Client: PEP Health
Sector: Healthcare, patient experience analytics
Status: Developed and validated; production thresholds being finalised

The problem

PEP Health's hospital clients generate tens of millions of written patient comments about their care. These are unsolicited feedback comments from sources such as Google, Yelp and RateMDs.

Within that feedback are comments about specific services such as digestive health, women's health, children's health and high-risk pregnancy. PEP Health needed to know which comments were about which service, rather than only an overall score. At that volume, the comments could not be triaged by hand.

The obvious first attempt is a list of keywords for each service line. But patients rarely write like a textbook. They describe real symptoms in their own words, which a keyword list would miss.

So Navi Insights trained AI models that learn the meaning behind a comment.

The scope

What the work covered

01Classifiers for four service lines: digestive health, women's health, children's health and high-risk pregnancy
02Every comment in the full dataset scored
03Each classifier checked against independent clinical review, not just against its own training data

How it was solved

What Navi Insights built

The build layers a few proven pieces of technology on top of each other

Part 1 of 3 · Training a small classifier with a large model

Navi Insights started with metabolic health as the pilot service line, using large language model (LLM) label distillation. Llama 3.3 70B scores a sample of comments for relevance, and those scores train a smaller classifier, rather than running the large model over the full dataset.

Llama 3.3 70B is self-hosted via vLLM on PEP Health's own AWS infrastructure, so patient data never leaves a controlled environment.

Part 2 of 3 · Checking the keyword pre-filter

Navi Insights used a keyword list as a rough first pass, to pre-filter the comments for labelling, and then checked that pre-filter against real, independently reviewed data.

For metabolic health, 86 per cent of genuinely relevant comments had no keyword match at all. For digestive health, the figure was 62 per cent. The pre-filter was discarding relevant comments, so labelling moved to a broad random sample rather than a keyword-gated one.

Everyday, indirect phrasing: a lay term for the oesophagus, an ear infection described through behaviour, and a high-risk pregnancy indicator described without using the term ‘high risk’.

Part 3 of 3 · Four classifiers that read for meaning

Navi Insights generalised the same method to digestive health, women's health, children's health and high-risk pregnancy, and trained four independent binary classifiers.

MPNet, a sentence-embedding model, converts each comment into a form that captures its meaning rather than its exact wording, which is what lets ‘food pipe’ and ‘oesophagus’ land in the same place. XGBoost, a well-established machine learning classifier, takes those meaning-based representations and, for each service line, makes the final call on whether a comment belongs to it.

The keyword match is kept only as a minor supporting signal, never the deciding factor. SHAP was used to check that the keyword-match feature was not dominating the classifiers.

The result

Four classifiers applied across the full dataset

The classifiers have been developed and validated. Navi Insights is finalising the decision thresholds for each model before the classifiers are rolled out into production.

93%High-risk pregnancy classifier: agreement with independent clinical review, in testing

23.3M+All four classifiers: patient comments scored across the full dataset, with zero processing errors

For high-risk pregnancy, the classifier agrees with independent clinical review 93 per cent of the time. This figure was measured against a human-reviewed sample independent of the model's own training labels.

Across the four service lines, Navi Insights is working through the remaining edge cases with PEP Health's clinical reviewers.

Once comments are reliably sorted by service line, PEP Health can show its hospital clients a clear picture of how patients feel about each service. That gives PEP Health additional service-line-level reporting to sell, built on feedback that already existed, with no extra effort for the hospital.

Results

High-risk pregnancy classifier, agreement with independent clinical review, in testing: 93%
All four classifiers, patient comments scored across the full dataset: 23.3M+
Processing errors across the full dataset: zero

Stack

Llama 3.3 70B
vLLM
MPNet embeddings
XGBoost
SHAP
Amazon SageMaker
S3
Redshift

The pipeline runs on Amazon SageMaker for building, training and running the models, with S3 for storing datasets and results, and Redshift for data infrastructure.

A separate deliverable for PEP Health, a reporting agent, is described on the agents page.

Next case

Yaspa

Yaspa: identifying gambling operators by behaviour, not by name

Read the Yaspa case →

Get in touch

Reach out and tell us what you're working on. Every message gets a personal reply within two working days.

Prefer email? Write straight to hello@naviins.com.