The Beauty Of 'Small Data' In Medicine, From Measuring Kids To Tumor Mutations

The 9-year-old boy had been referred to our Children's Hospital clinic for short stature. He didn't seem all that short to me, but a family photo showed that he was shorter than his 7-year-old brother. His parents and pediatrician were worried, and I was the fourth-year medical student learning endocrinology and charged with his care.

We saw many such patients, and most often we would plot their growth, see that it followed a normal pattern, and occasionally obtain an X-ray and a blood test to screen for unlikely disorders. We’d then send the family home with reassurance that the child was growing normally, and write a summary note to them and their pediatrician, asking them to follow up with the results of any tests.

I presented the patient's history and findings to my supervisor, Dr. John F. Crigler, a well-known endocrinologist with the rare honor of having a genetic disease named after him: Crigler-Najjar syndrome, a cause of jaundice so severe it can damage a baby's brain.

Dr. Crigler's reaction became my first encounter with medical magic — an experience that, though it occurred over 30 years ago, informs my work analyzing patient data — big and small — to this day.

He interrupted me when I showed him the growth chart. It did not look any different to me from dozens I'd seen that month of slow-growing prepubertal boys. But he pointed out to me a subtle inflection of the height curve that was not mirrored in the weight curve on the chart.

After checking the most recent height measurement for himself — he never trusted the measurements made by trainees — he sat down with me, and reviewed how we were going to tell the family that the child mostly likely had acquired growth hormone deficiency, that it had started at least two years ago, and that an MRI of his head was strongly recommended.

Two weeks later, the MRI confirmed the presence of an abnormal mass in the area around the pituitary gland — the part of the brain that regulates multiple hormones, including growth hormone. Neurosurgery was successful in removing that mass; it turned out not to be malignant, but it had been steadily growing for years.

All that from a growth curve.

I was so impressed by this diagnostic feat, its implications stretching back years into the child's past and forward into his future, that I decided then and there to train as a pediatric endocrinologist.

That case also marked the beginning of a long education on the value of small data — that is, the clinical impact of a small number of reliable measurements on a single patient.

Small data is not in style these days, and I'm a big data practitioner myself, working at the intersection of data, computing and medicine as chair of Harvard Medical School's Department of Biomedical Informatics. So much the more reason, I think, to remember how much can be done with careful, meticulous consideration of data coming from a single patient.

Over the next six years, Dr. Crigler taught me how dozens of different childhood diseases could be picked up from careful examination and verification of a child’s height and weight chart.

He also emphasized the importance of comparing a child’s growth to the standards of the right reference population. Children from Central America do not have the same growth curves as immigrants from Northern Europe.

He understood that many young doctors in training prize their ability to come up with impressively long and comprehensive lists of possible diagnoses — the so-called differential diagnosis — to account for what we had measured and observed in the patient. He therefore pointedly taught us humility by showing us, again and again, that by revisiting our original measurements, and remeasuring them when there was cause, the differential diagnosis could completely change.

This was indeed humbling. We thought we were learning how to diagnose a patient as a prelude to treatment. By comparison, getting the height measured correctly to within 1/16 of an inch seemed pedestrian, and certainly not intellectually demanding.

Even now, attention to these simple, cheap measurements remains underwhelming. A recent study documented frequent height measurement errors that can affect osteoporosis care. In children, incorrect height measurements may mean missing disorders such as Celiac disease or hypothyroidism, which sometimes first show up as a growth problem. They may also throw off dosages of drugs that depend on height and weight. But that still doesn't make these small data sexy.

From Low-Tech To High-Tech

All the way at the other end of the technological spectrum, in the use of sophisticated and marvelous biotechnology to measure the mutations present in tumors from a blood sample alone, we find an equally large, Criglerian (to give him an adjectival eponym) challenge.

That is, there's a disturbing gap between the theoretical use of these measurements and the accuracy with which those measurements are made.

Measuring mutations in tumors is one of the more dramatic potential medical benefits of the human genome sequencing revolution. For a small but rapidly growing subset of cancer types, knowing that the tumor has a specific mutation type allows the selection of a particularly effective treatment.

Unfortunately, tumors acquire additional mutations over time — which allows mutated cells that are resistant to the current therapy to proliferate.

Being able to measure these mutations in the tumor at multiple points in time would be very helpful. That’s why many of us have been so excited about the development of "liquid biopsy" technology. Liquid biopsies allow us to fish bits of tumor DNA out of the bloodstream and then figure out which mutations are present and which are new relative to the last measurement. No biopsy or surgery needed; just a blood draw will do.

Given that promise, valuations for liquid biopsy companies in the hundreds of millions of dollars make sense, as would paying thousands of dollars for a liquid biopsy test on a patient with a malignancy.

Except that, at present, these tests may not be very accurate at all.

One of the simplest ways to compare two clinical tests is to perform them on the same patient. If the height I measured on a patient was different by one inch from that measured by Dr. Crigler, we’d know one of us was wrong (you know who).

In that light, a recent study of liquid biopsies was very concerning. It demonstrated that when two liquid biopsy tests were both tried in the same 40 patients with metastatic prostate cancer, the two tests were discordant for 40 percent of the patients — they did not agree on any of the mutations found.

There are many technical and biological reasons for such a poor agreement rate between these two tests, but if clinical decisions are to be made based on these costly tests, then in 40 percent of cases, the data driving decisions would be different.

The accuracy of liquid biopsy tests will likely improve. At the moment, however, given the high rate of discordance in these commercially marketed tests, what are we — patients and doctors — to do?

There seems to be little short-term hope for regulatory oversight. The FDA approves these tests one by one; it does not run “bakeoffs” as, for instance, Consumer Reports does for blood pressure monitors or cars. The agency is innovating in many ways to keep up with scientific advances, but I see no immediate solution there. Similarly, the Centers for Medicaid and Medicare Services have no head-to-head process to compare the performance of competing tests.

Dr. Crigler passed away this year, and the many people he mentored now mourn him. Perhaps a fitting memorial to him would be a renewed focus for medical trainees on their attention to the accuracy and meaning of individual patient data. We should ensure that all doctors can stand behind their patient measurements — their meaning and the clinical decisions that they entail — whether a cutting-edge genomic test or an old-fashioned reading on a ruler.

Dr. Isaac Kohane is the inaugural chair of the Department of Biomedical Informatics at Harvard Medical School.