- The best way to rate an mHealth app these days? Try it out yourself.
That’s the decidedly-negative conclusion reached by a team of researchers at Boston’s Brigham and Women’s Hospital. In a study published on Feb. 10 in JMIR mHealth and uHealth, the group of six mHealth experts found that they couldn’t agree on the effectiveness of any apps, even when using the same set of guidelines to rate them.
“There’s not a lot of guidance on which apps are recommended,” Dr. Adam Landman, the hospital’s chief medical information officer for health information innovation and integration, told the Boston Globe.
The quest to standardize and evaluate mHealth apps isn't new, and it doesn't seem to be getting any easier. While the B&W researchers noted there are many different methods for rating an app available to the physician and consumer, even those don't produce similar results. One company's analysis of an app might therefore vary dramatically from another's. As a result, providers have been wary of prescribing apps, and patients are often skeptical of their value.
For the study, the research group – four physicians, a nurse practitioner and a health economist – reviewed 20 apps targeting smoking cessation and depression over a three-month period in early 2015. That’s a tiny speck compared to the estimated 165,000 mobile health apps now on the market, but it gave the group an idea, minus any demographic limitations, of what to look for and analyze.
According to the study, the group was rarely in agreement on the most clinically useful aspects of an app, such as effectiveness and ease of use. They also differed in assessing an app’s privacy policies and software support capabilities.
“Overall, the results suggest a need for great caution when assigning and interpreting app ratings,” the study reported. “Organizations rating apps should create robust measures with clearly defined criteria to ensure consistency. They should then test the interrater reliability and validity of their measures and utilize only ones with high levels of agreement.”
And for physicians looking to recommend an app to a patient, the study suggests that the best evaluation may be personal experience.
“Clinicians may wish to use reviews as a mechanism for selecting potentially beneficial apps and then carefully test apps before recommending them to patients to ensure that the reviewers’ definitions of quality are consistent with their own,” the report said. “There may likewise be value in discussing apps with colleagues before recommending them, as it appears that clinicians may not identify quality issues consistently. Even in this era of digital health, clinician judgment may still be the best available tool for evaluating apps.”