Hospitals say structural issues compromise value study, while employers stand by it

Hospitals are questioning the methodology behind a study sponsored by eastern Wisconsin employers that found businesses could save hundreds of millions of dollars by steering consumers toward high-quality, low-cost doctors and helping develop similar high-value providers.

The Business Health Care Group stands by the analysis, conducted by Massachusetts-based GNS Healthcare and funded by the employer coalition and the Greater Milwaukee Business Foundation on Health.

That study, released in December, analyzed the performance of 3,760 primary care doctors throughout the state and found $394.5 million in savings if patients were steered toward high-performing providers and if doctors performing in the bottom half of the distribution were able to move to the top half.

Similar efforts for cardiologists performing angioplasty procedures, orthopedic surgeons performing hip and knee replacements, and obstetricians delivering babies could save $100 million.

The Wisconsin Hospital Association brought on Wisconsin-based BSG Analytics to look at the study. In its response, the firm warned that the study has structural issues that undermine its use for evaluating provider performance and where to direct patients, arguing that it focused on primary care and excluded the majority of care provided within integrated health systems.

“While the intent is good, the study’s methodology is so fundamentally flawed that we think it’s impossible to really draw meaningful conclusions,” said Dr. Mark Kaufman, WHA chief medical officer.

Kaufman said the study looked at around half of the doctors in the state, due to its exclusion criteria.

According to the BSG Analytics report, there were 7,457 unique primary care provider names with at least one 2017 claim in the Wisconsin Health Information Organization’s database, which the study used. The study limited its analysis to doctors with more than 100 observations.

BSG Analytics noted that the study couldn’t determine whether 72 percent analyzed had above or below average scores with 80 percent confidence when it came to costs. On quality, the study wasn’t able to determine above or below average scores with that level of confidence for 48 percent.

The study is based on 26 conditions during a single-year of claims data, which Kaufman said was a small sample size of the care that primary care physicians provide and a limited time frame.

He said that chronic diseases could have catastrophic events that are expensive in one year, so it would make sense to look at multiple years to make judgments about the cost efficiency of doctors.

Kaufman said the study doesn’t look at the totality of care provided by health systems to patients. The claims data it uses focus mostly on process measures and don’t provide much information about outcomes, he said.

And he said that the study’s risk adjustment methodology, which involves identifying sick patients that may need expensive care, isn’t transparent. It should be, so that experts can evaluate it and doctors can learn how to improve their performance, he said.

The BSG Analytics report described the study as using “‘black box’ technology” based on machine-learning algorithms that haven’t been made available for evaluation.

The Business Health Care Group in August pushed back against the hospitals’ response, saying they believe that evaluation of individual physician performance provides actionable, useful information.

Dave Osterndorf, the group’s strategic consultant, said they welcome discussion surrounding the methodology and data, but there’s no such thing as a perfect study.

“There is a very good and robust study that we’ve put together,” he said. “So if the goal is to refute the study by saying it isn’t perfect, there isn’t a lot of value to us in that kind of input. If the goal is to look at ways to improve the study and use the results in a mutually beneficial way, we’re always open to that kind of dialogue.”

Osterndorf said they looked at primary care doctors with 100 observations because a number of physicians in the database they used had less than 10 observations. The threshold gave them enough data for credible evaluation, he said.

When it comes to confidence levels, the Business Health Care Group’s response said they should be “applauded, rather than criticized” for reporting on the doctors they could and could not adequately evaluate.

Osterndorf said that the one-year timeframe is used by several governmental agencies and warned that looking at quality over multiple years could “wash out improvement.” They plan to repeat the study at regular intervals, so they can compare it year to year.

They looked at the 26 conditions because they make up the majority of care provided by primary care physicians, he said.

Osterndorf said they used machine learning and artificial intelligence, which take into account data elements that influence the study’s outcomes and build them into the evaluation process.

“If you don’t really understand machine learning, I’m sure it feels like a black box,” he said. “If you understand the process, it’s pretty straightforward.”

Osterndorf said they would love to use a dataset with a comprehensive understanding of patients’ health status changes but that doesn’t exist. And the Business Health Care Group said it’s not feasible to evaluate the totality of care provided by a health system.

“We can’t think of a reason that an evaluation of the totality of medical care needs to be performed in order for an evaluation of physician performance to be valid or credible,” the group said in its response. “Suggesting that it does likely would preclude any provider performance evaluation from ever being performed.”

Dave Jensen, senior consultant at BSG Analytics, said they’ve been doing studies looking at the totality of care for a decade, using data from the Wisconsin Health Information Organization, the Centers for Medicare and Medicaid Services and the Wisconsin Collaborative for Healthcare Quality.

He noted that the approach hasn’t been perfected, and that “you can argue back and forth about it at cocktail parties.” Such approaches are necessary for health systems to leverage their resources, he said.

“We understand that businesses want and need good data to make decisions. I think the goal is laudable,” Joanne Alig, WHA senior vice president of public policy, said of the study. “We’re just concerned that’s not going to move us forward and drive that improvement that we’re all looking for.”

This article first appeared in the Wisconsin Health News daily email newsletter. Sign up for your free trial here.