Model-informed precision dosing platforms, like InsightRX Nova, enable easy application of precision dosing in clinical care, primarily for antibiotics, immunomodulatory drugs, and drugs used in transplantation medicine. Practically all dosing modules in InsightRX Nova are based on pharmacokinetic models, which describe the time-course of drug concentration in the body over time based on the dosing regimen and on known patient characteristics such as weight, age, and renal function. The quality and predictive ability of such models is key to successful use in patient care.
For any given drug there are usually multiple pharmacokinetic models available in the medical literature, ranging from a handful of options to over a hundred for extensively studied drugs like vancomycin! A question that comes up regularly from our users is how the InsightRX team decides which of these models to make available in Nova. In Figure 1 I’ve depicted a “model selection funnel”, which is helpful in describing how we think about this selection process. In this blog post, we’ll explore why model selection is important, review how models are selected and implemented at InsightRX, and wrap up by covering how the final point-of-care model selection is done in InsightRX Nova.
Figure 1. Model selection funnel.
Essentially, our team carefully selects models because our collective experience and extensive internal comparison studies show that not all models are sufficiently predictive for clinical use. This point was also made recently in a research paper [Gandia 2024], which showed that predictions from 18 different vancomycin pharmacokinetic (PK) models from literature vary wildly; in their simulations for a “typical” patient, the predicted probability of target attainment (PTA) – in this case, achieving an AUC>400 – for a chosen dosing regimen varied from 9% to 94% between models! The grey lines in Figure 1 show the PTA for these 18 models, for a range of AUC thresholds. The 18 models in this paper were selected simply based on the fact that they were described in literature as suitable for a general population. Importantly, however, these models were not vetted for their quality or expected predictive performance.
For the curated set of models that we provide in our platform, we would not expect such large discrepancies. In fact, Figure 2 shows that for the same “typical” patient we see a six times smaller range of PTA (57 - 70%) for the models that have passed our quality checks and are recommended for this patient type (teal lines). It is clear that the curation of models for use in real world patient care is of crucial importance for precision dosing.
Figure 2. Variability in predicted probability of target attainment (PTA) for different vancomycin models selected naively from literature (grey lines), and curated models available in InsightRX Nova recommended for a typical patient (teal lines).
Intuitively, one might think that pharmacokinetic models specifically developed on patient data from a narrow subpopulation would be most predictive for patients with similar characteristics. This is true to some extent. For example, we recently built a vancomycin model internally [Hughes] (in collaboration with the University of Vermont Medical Center) on only obese patients with a BMI of over 40. In subsequent validation analyses, we’ve seen that this model unsurprisingly performs exceptionally well in its intended population. However, we’ve also found that it often outperforms models developed for non-obese populations in non-obese patients.
You may be asking yourself: “How is this possible? The pharmacokinetics of vancomycin in a patient with a BMI of 50 is surely very different compared to a patient with a BMI of 25?” It definitely is. But while patient covariates like body size and kidney function influence drug clearance, the relationships in a model can allow it to generalize beyond its original population. For instance, this new model incorporates fat-free mass using an approach called allometric scaling, which adjusts for how metabolic rates and drug clearance changes with size. This enables it to predict vancomycin pharmacokinetics also fairly accurately in patients with lower BMIs, despite being built on data from obese patients.
Alternatively, use of “generic” models developed on diverse patient populations can also perform well across many subpopulations. For example, the Colin model [Colin 2019] for vancomycin, built on a joint dataset of many different patient populations with a large range in age, BMI, and renal function, performs well across age groups, often outperforming models specifically built for a smaller age range. [Hughes 2020, Hughes 2023]
These observations and others have led us to conclude that naively selecting pharmacokinetic models because of matching patient characteristics is not a good strategy for selecting the model for the patient. The reason for this? One reason could be that such narrow-population models were developed on too small of a patient dataset to properly characterize the covariate relationships, or perhaps were developed from data from a single hospital that offered a less diverse set of parameters in the final build. We often see significant differences between hospitals in the predictive performance of models. [Hughes 2022/ACoP] This could be due to minor differences in the assay for vancomycin, or even the formulation of vancomycin [Gandia 2024]. It could also be due to differences in the assay for creatinine [Murthy 2005], which is a covariate in most models of vancomycin.
So how do we select the models that we make available in Nova to our users? Let me walk you through the model selection funnel that I introduced at the start of this blog post.
For a new dosing module for a particular drug (or when refreshing the models for an existing drug module) we typically start with a literature search to identify candidate models from the literature. We generally limit our search to parametric population pharmacokinetic models published in the last 15-20 years*1. After the initial literature selection, we review each model for implementation based on its description. Nowadays, the model code is often available in the supplementary to the article.
Sometimes, at this stage we can already disqualify a model from being considered for implementation in InsightRX Nova. For example, a model may have an inappropriate structural model, or include covariates that are not commonly measured. If the model is described ambiguously and the authors also are not able to share the model code, the model is disqualified since we need to be absolutely certain about the model details.
At this point, if more than a handful candidate models still remain, we might have a deeper look at the research articles and factor in things like:
During this phase we also perform sanity checks. For example, we may confirm that the pharmacokinetic concentration curve for particular patients broadly conforms to expectations and compares reasonably to predictions from existing models, if these are available. We also definitely have a preference for models that are developed on data from multiple sites, and have been validated in a hold-out data set or other external data source. Overall though, weighing all these factors cannot easily be captured in a strict procedure and is quite often a judgement call.
Of course, besides the model selection performed by our team, we also often rely on knowledgeable users of InsightRX Nova and scientific collaborators in academia to recommend candidate models for implementation. Sometimes this could be models that were developed by users themselves, or sometimes models that were published but are not yet available in Nova. Needless to say we always welcome such user input and recommendations!
Population pharmacokinetic models are most often developed in NONMEM. This is the gold standard software for development of the type of statistical model used in MIPD, and also more broadly in the pharmaceutical industry for model-informed drug development (MIDD). One challenge, however, is that the InsightRX Nova platform does not rely on NONMEM for its pharmacokinetic predictions: all computations are performed using the open source statistical software R and in-house developed R libraries.
To be able to use the models we identified in Nova, we therefore first have to translate the NONMEM code to R. Obviously, this could easily result in mistakes and inaccuracies, so we perform an automated numerical verification step to confirm that the model in R performs exactly similar to the original NONMEM version. Since numerical performance and stability is key to our platform, we do this for every model in the platform and repeat these checks for each new release of Nova to make sure we didn’t make an unintended change.
Once the model has been selected and the implementation verified, and we deem it ready for use in routine practice, we deploy it in InsightRX Nova and make it available to users for clinical decision support (CDS). Models can be released platform-wide, or only to a limited set of hospitals or users, for example those that initially requested the model. If, for example, we’re unsure of the predictive performance of a model we might release it just to a few expert users initially, pending further evaluation in practice. The availability of new models in Nova is usually communicated in release notes and other announcements that we send out via email. If you are not yet enrolled in our newsletter(s) please let us know and we’ll add you.
After implementation of a model and release, we do not sit back and forget about it. At regular intervals we check the performance of new and existing models in Nova, and compare their performance against others within the same drug module. This could be through our analytics platform InsightRX Apollo, or through internal data science tools where we take a deeper dive into model performance or target attainment for specific models. We commonly set the best performing model as the default model for that drug module, and deactivate models that do not perform as well or otherwise have problems. This de-activation of models can again be either platform-wide or site-specific, as we sometimes see differences in model performance between different hospitals or hospital types. [Hughes 2022/ACoP]
Additionally, post-release model evaluations may reveal that a model is performing sub-optimally, but could be improved by making minor modifications to it or by retraining the model on new data.
For example, a few years ago we noticed that the vancomycin pharmacokinetic model by [Goti 2018], which included a capped serum creatinine level as a covariate, actually performed better when that cap was removed, even without retraining the model. [Tong 2021] The uncapped model was then also made available in Nova and users were advised to use the revised model. This continuous learning principle is a core part of our success model and part of what separates our solution from other Bayesian-based applications. We are currently working on several other improvements and aim to publish as many of these modifications and improvements in the peer-reviewed literature and on this blog, so that anyone can benefit from these scientific advancements.
At InsightRX we aim to have the most predictive models available in Nova - for any patient, for any drug. That means that for most drug modules we will have more than a single pharmacokinetic model available, because model performance often depends on patient characteristics. That means that even though the available models are all expected to perform fairly well through our model selection process, for truly optimal performance a user still may have to decide between a handful of models, often even before a TDM level is obtained. This individual model selection is the last part of the funnel, and it can sometimes be a hurdle in clinical practice.
We try to help users of InsightRX Nova in several ways:
That wraps up this blog post on how we choose models at InsightRX for delivery at the point-of-care. I hope I was able to drive home the point that we take model selection seriously, and we apply a scientific and data-driven approach to this process. We are always looking for ways to improve the predictive ability of models in Nova so that it performs and users will get better outcomes for their patients.
Want to learn more about implementing MIPD at your organization, or find out more about our continuous learning framework? Click below to learn more!
Footnotes
*1 “Parametric” models assume a certain distribution of the model parameters, most often a log-normal distribution. “Non-parametric” (NP) models, in contrast, are more flexible in their assumptions on the distribution of model parameters and therefore provide some theoretical advantages over parametric models, although no evidence is available that it actually is more predictive in practice. [Goutelle 2022] Technically InsightRX Nova does have the capability to support NP models. However, articles presenting NP models are more scarce in literature, and most often do not contain enough detail to implement the model. Therefore we commonly only make parametric models available. We focus on models from the past 15-20 years or so since the field of pharmacometrics has matured quite a lot in the past two decades, and older models may have been developed with more crude algorithms, and with poorer diagnostic tools and development practices.