Improving efficiency of a rheumatology outpatient clinic through classification of newly referred patients

Authors: Sander Dijkstra, Yaël A de Man, Aleida Braaksma, Maurits de Graaf, Hein J Bernelot Moens, Richard J Boucherie

Abstract

Objectives

Diagnosing a rheumatological disease in patients newly referred by their general practitioner requires assessment by a rheumatologist and often diagnostic tests. Ideally, these tests are performed prior to the patient’s first consultation with the rheumatologist, aiming for quick diagnosis and fewer visits. We retrospectively studied whether a pre-first visit digital patient questionnaire can lead to fewer consultations and faster diagnosis.

Methods

We applied machine learning-based binary classification algorithms to questionnaire data of newly referred patients to classify a patient’s diagnostic class (inflammatory vs. non-inflammatory). Afterwards, we quantified the rheumatology clinic’s benefit when all patients classified to be non-inflammatory are planned for specific diagnostic tests at their first visit.

Results

Classification for inflammatory vs. non-inflammatory disease could be done with accuracy of 0.771, sensitivity (recall) of 0.809 and precision of 0.833. When non-inflammatory classified patients receive a blood test before first consultation, on average 0.1 in-person consultations and 0.26 teleconsultations per patient are avoided at the cost of having overdiagnostics for 44% of all newly referred patients. If, in addition to a blood test, the first consultation is also preceded by a radiology examination, on average 0.21 in-person consultations and 0.49 teleconsultations per patient are avoided, at the cost of having overdiagnostics (e.g. only the blood test or the combination of the blood test and radiology examination) for 71% of all newly referred patients.

Conclusion

Classification algorithms based on pre-first visit patient questionnaires may shorten the patient journey in a rheumatology outpatient clinic and may therefore improve efficiency.

Introduction

In the Netherlands, five out of 1000 adults were referred to a rheumatologist in 2021 (approximately 75 000 newly referred patients yearly) [1]. As the population is ageing and health services are becoming scarce, efficient use of healthcare resources becomes inevitable, also in rheumatology outpatient clinics. In particular, for both patients and rheumatologists, it is desirable to diagnose with an efficient combination of consultations and diagnostic tests, preferably in a minimum number of visits. Making a diagnosis often requires diagnostic tests, such as blood tests, a musculoskeletal ultrasound or an X-ray. Ideally, all required tests are performed at the first visit, prior to the consultation with the rheumatologist, as in this way the diagnosis can be made in a so-called one-stop shop. Rheumatic diseases fall into two diagnostic inflammatory and non-inflammatory rheumatic diseases. For patients in the second class, performing all required diagnostic tests prior to the first consultation might be easier to achieve than for patients in the first class, as patients with non-inflammatory rheumatic diseases usually do not need follow-up tests or consultations (e.g. for monitoring their conditions over time). We study whether distinguishing between the two diagnostic classes prior to the first clinic visit could improve the efficiency of rheumatology outpatient care.

From the literature, we know that teleconsultation before a first visit reduced visits to the outpatient clinic for non-inflammatory patients but did not reduce time spent by a rheumatologist [2]. In this study, patients for whom it was not certain that they had inflammatory rheumatic disease (if this was certain, they were immediately called to the outpatient clinic) were (video)called to distinguish between the diagnostic classes. Likewise, a first-visit questionnaire may discriminate between inflammatory and non-inflammatory rheumatic diseases [3], which potentially saves time for rheumatologists or nurse practitioners.

This study aims to contribute to efficient use of resources in the rheumatology clinic by two means. First, we apply machine learning-based binary classification algorithms to classify a newly referred patient’s diagnostic class (inflammatory vs. non-inflammatory disease). We use data from a pre-first visit digital questionnaire that is available for all newly referred patients. The answers to this questionnaire are collected before the first clinic visit and presented to an algorithm, which classifies (predicts) a patient into one of the two diagnostic classes. This classification supports the rheumatologist prior to the first contact with the patient. Second, we quantify the impact of planning pre-specified diagnostic tests at the first clinic visit for all patients classified as experiencing non-inflammatory rheumatic disease. This impact is expressed by means of a defined cost function that weighs the advantage of reduction in the number of consultations against the disadvantage of the number of (over)diagnostics.

Material and methods

Data

We collected retrospective data from patients’ first visit and the follow-up visits in the subsequent 4 months at the Outpatient Department of Rheumatology, Ziekenhuisgroep Twente (ZGT), in Almelo, the Netherlands, between 1 January 2017 and 1 November 2021. Throughout that time, the medical team consisted of nine rheumatologists and two trainees in rheumatology. Four-month follow-up has been chosen since after this interval a definite diagnosis has usually been made.

The collected data comprised information about patients’ diagnoses and logistical data of patients’ journeys. A patient journey is defined as the sequence of all performed care activities (diagnostic tests and consultations) initiated by the treating rheumatologist. For analysis, diagnoses were aggregated into two diagnostic inflammatory and non-inflammatory rheumatic diseases. Activities in a patient journey an in-person consultation, a teleconsultation, a blood test, a musculoskeletal ultrasound and radiological counting plain radiographs and CT scans combined and MRI separately. For each activity, the day on which it took place was used.

The pre-first visit digital patient questionnaire consisted of 10 questions, possibly extended with a maximum of nine additional questions depending on answers given to selected questions and became available since February 2020 (see Supplementary Data S1 for the questionnaire, available at Rheumatology Advances in Practice online). New patients were invited to complete this questionnaire after referral by their general practitioner. For this study, questionnaire data were obtained between February 2020 and July 2021.

The MEC-U Nieuwegein, the Netherlands, considered the study (W21.296) as not subject to the Medical Research Involving Human Subjects Act, and it was approved by ZGT Almelo that no consent was needed nor received for this retrospective study.

Methods

Binary classification of inflammatory vs. non-inflammatory disease

The first objective was to accurately classify patients into inflammatory or non-inflammatory rheumatic diseases based on the questionnaire. Python 3.8 with package PyCaret 2.3.10 [4] was used to compare 15 built-in binary classification models, where the non-inflammatory class was labelled as positive (i.e. 1) and the inflammatory class as negative (i.e. 0). Covariates for the models were constructed from the answers to the questionnaire as each item of every question was translated into a binary variable via one-hot encoding (e.g. in Supplementary Data S1, Question 7, available at Rheumatology Advances in Practice online; item a becomes the covariate that equals 1 if the patient answered ‘yes’ to having psoriasis and 0 if the answer was ‘no’). After this covariate construction, the set of obtained covariates was augmented with the patient’s age (integer covariate) and sex (binary covariate).

PyCaret was used to split the data and train, tune and evaluate the models. The dataset was first split into two a training set (80%) and a test set (20%), by random stratified splitting. This method preserves similar fractions of positives (i.e. patients with non-inflammatory rheumatic diseases) and negatives (i.e. patients with inflammatory rheumatic diseases) in training set and test set. Before model training, covariates with low variation in the training set as well as (multi)collinear covariates in the training set were deleted. A covariate was considered to have a low variation if the same value was observed in more than 90% of the samples in the training set. Collinearity is the phenomenon in which one covariate is highly linearly correlated with another one (i.e. if the Pearson correlation coefficient is greater than 0.9), which increases the variance of the model coefficients, thus making these unstable and noisy for linear models [5].

Performance of a model was quantified in terms of accuracy, recall (also called sensitivity) and precision, which are all obtained from the confusion matrix [6]. For model training, PyCaret’s random grid search (RGS) method was used for hyperparameter tuning of parametrized models to find the parameters with the best model performance. RGS selects a predefined number of hyperparameter combinations, according to probability distribution, for which the model performance is evaluated. By default, we chose to evaluate 1000 hyperparameter combinations. As for each hyperparameter configuration, model coefficients were fitted independently; comparing hyperparameter combinations on the training data was done by 3-fold cross-validation to reduce the effect of randomness. Therefore, training data were split into three equally large subsets, called folds. Two folds (i.e. two-thirds of the 80% training data) were used for fitting model coefficients, given a selected hyperparameter combination, and the remaining fold, called the validation fold, was used for evaluation of this hyperparameter combination. As all three folds were used once as validation fold, it is implied that the validation scores of the models are the averages over the three different validation folds. Similarly, the training scores of the models are the averages over the corresponding training folds (i.e. the three different combinations of two folds leading to two-thirds of the 80% training data).

The best hyperparameter combination, leading to the so-called tuned model, was selected based on the validation score (the average score over the three validation folds). The tuned model coefficients were then fit on the whole training set. Finally, the performance of all tuned models was evaluated and compared on the test set (20% of the complete dataset).

Logistical impact of classification of inflammatory vs. non-inflammatory disease

To assess the impact of questionnaire-based classification on patient journeys, we quantified the logistical (dis)advantage of its use. We focused on the first month of the patient journey of each patient in our test set, and we assumed that (part of) the diagnostic test(s) performed during this period could have been done right before the first consultation with the rheumatologist. For each patient who was classified as having a non-inflammatory disease, we simulated various sets of diagnostic test(s) prior to the first consultation. After discussion with experts regarding the timing of diagnostic tests and consultations, we made the following assumptions to simulate a realistic reduction in a number of

• Diagnostic tests that are feasible to organise before first consultation blood tests, radiology examinations (plain radiograph or CT scan) and ultrasounds. More complex examinations (such as an MRI or capillary microscopy) can only be done by appointment.
• A blood test can only be done prior to the first consultation if, in the patient journey data, it occurred within 14 days (2 weeks) before or after the first consultation.
• If double testing happens within 14 days (2 weeks), the last test could be done simultaneously with the first one and therefore is redundant.
• If two consultations, irrespective of whether they are in-person or teleconsultation, happen within 14 days (2 weeks) after a diagnostic test, the last consultation is considered redundant and could be avoided.

To illustrate these assumptions and how we simulate reduction in a number of consultations, observe Table 1, which shows the first month of several patient journeys. The blood tests of Patients 1 and 2 could be done on the same day, prior to the first consultation, and thereby one consultation could be avoided for Patient 1. The blood test of Patient 3 is not eligible for this (by Assumption 2), which means that for Patient 3, no consultation could be avoided. For Patient 4, it holds true that if a blood test prior to the first consultation is done, one consultation could be avoided. Its second consultation could not be avoided because of the required MRI, which cannot be done prior to the first consultation. The patient journeys for Patient 5 illustrate Assumption 3, as in its original patient journey two radiology examinations succeed each other, without consultation(s) in between, but within 14 days. If a blood test is performed before the first consultation of Patient 6, Assumption 3 leads to one avoided consultation. Because the radiology examination still needs to be done, this moves to Step 2. If both a blood test and radiology examination are performed prior to the first consultation, both consultations from Step 2 and Step 3 are avoided. The effect of our proposed methodology for Patient 7 is similar to Patient 6; however, Patient 7 has double consultation instead of double diagnostic testing. Note that we assume both consultations relate to the immediate preceding diagnostic test (in this case, the ultrasound), so that if this test is not performed prior to the first consultation (as in the first adapted patient journey), we do not make a decision on whether one of the two consultations could be avoided.

As Table 1 shows, the logistical (dis)advantages of our proposed methodology for a rheumatology outpatient clinic depend on the set of diagnostic tests performed prior to the first consultation of non-inflammatory classified patients. A blood test, a radiological examination, an ultrasound and all combinations of these tests are eligible to do prior to the first consultation. Planning diagnostic tests for all non-inflammatory classified patients inevitably leads to overdiagnostics, so the final step is to check whether the advantage of reduction in the number of consultations outweighs the disadvantage of performing overdiagnostics. To this order, we assume fictional revenue for avoiding consultations and fictional costs for overconsumption of diagnostic we earn 1 (unit) for avoiding an in-person consultation or teleconsultation and we incur a cost of 0.1 for an unnecessary blood test, 0.2 per unnecessary radiology examination and 0.4 for unnecessary ultrasound. These fictional costs are based on the expertise of the rheumatologists in ZGT and estimated proportional to the cost of a consultation with a rheumatologist. In other words, experts valued a blood test cost 10 times less than a consultation with a rheumatologist, a radiology examination five times less and an ultrasound 2.5 times less. In Supplementary Table S2 (available at Rheumatology Advances in Practice online), we consider the patient journey of Patient 7 to illustrate overdiagnostics and fictional costs. For each patient in our test set, we created a comparable table. For each classification algorithm, for each of the patients classified by this algorithm as non-inflammatory, and for each set of diagnostic tests, we checked the (fictional) profit of carrying out this set of diagnostic tests before the first consultation. For each set of diagnostic tests, we averaged the profits obtained over the number of patients classified by the algorithm as non-inflammatory. Then we identified which combination of diagnostic tests and classification algorithm yields the highest average profit.

Results

In Table 2, we compare the group of patients who responded to the questionnaire (521 patients in the period February 2020–July 2021) to the historical cohort of all newly referred patients from January 2017 till February 2020 in the first two columns, and the patient group with inflammatory diagnoses (n = 2758, 39.1%) to the patient group with non-inflammatory diagnoses (n = 4290, 60.9%) in the last two columns. We observe that the cohort with a completed questionnaire is quite similar to the historical cohort in terms of the metrics and that patients with inflammatory diagnoses are more likely to be male and on average older compared with patients with non-inflammatory diagnoses. Also, their patient journey includes more consultations. Therefore, we initiated our methodology to reduce the length of non-inflammatory patient journeys by, as mentioned in the Methods section, simulating pre-first visit diagnostic tests for this group. We did not simulate these tests for inflammatory patients, as the statistics in Table 2 suggest they will visit the clinic after diagnosis anyway, so from the perspective of reducing consultations, it is less promising.

The questionnaire leads to 37 items (see also Supplementary Table S1, available at Rheumatology Advances in Practice online). We applied one-hot encoding to obtain binary covariates, removing the (multi)collinear ones and the ones for which more than 90% of the patients in the training set gave identical answers to the corresponding question. This resulted in 34 covariates, augmented with age and sex.

Table 3 shows the performance of the classification algorithms in terms of accuracy, recall (sensitivity) and precision after hyperparameter tuning. Recall that the non-inflammatory class has label 1, so recall refers to the ability to correctly identify non-inflammatory patients, and specificity refers to the ability to correctly identify the inflammatory patients. We highlight the performance of the Extra Trees Classifier (et) as it scores the highest on accuracy within the test set. Remark that, due to computation time, PyCaret tacitly lowered the number of evaluated hyperparameter combinations during model tuning for some models. Finally, remember that both training and validation scores in Table 3 are averages over corresponding folds (see Methods section).

Next, we want to find out which combination of classification algorithm and set of diagnostic tests prior to the first consultation for all non-inflammatory classified patients performs best from the clinic’s point of view. In Table 4, we compare all combinations based on the average profit per patient classified as having non-inflammatory rheumatic disease. The best-performing combination in terms of logistical costs is the K-nearest Neighbours Classifier (knn) together with a blood test and radiology examination before the first consultation. Note that knn was not the best-performing classifier in terms of accuracy, recall and precision, as this was et. This suggests that, from a logistical point of view, it is sometimes beneficial to misclassify a patient as having non-inflammatory rheumatic disease. In that case, such a patient undergoes diagnostic tests (which in this best-case scenario are a blood test and radiology examination) prior to its first consultation, which on average costs the clinic less compared to first consult the patient and then decide on diagnostics. This is due to avoiding consultations and/or teleconsultations.

This result relies on the cost of each diagnostic test. We performed a sensitivity analysis in which we evaluated 27 cost-realistic configurations (blood test cost 0.05, 0.1, 0.15; radiology 0.1, 0.2, 0.3; ultrasound 0.2, 0.4, 0.6) to study the impact on the preferable set of pre-first visit diagnostic tests. The results are depicted in Fig. 1, where we observe that for all cost configurations, it is optimal to offer all non-inflammatory predicted patients a blood test and radiology examination prior to the first consultation, and the second best option is always to offer the combination of all three diagnostic tests. For the combination of blood test and radiology examination, the knn classifier performs best for all cost configurations, followed by nb with a very small difference, whereas for the combination of all three diagnostic tests, this is vice versa (similar to the results in Table 4). For all other profitable sets of diagnostic tests, knn and nb are also the top-2 classifiers.

Finally, we provide more insight into the potential savings in relation to overdiagnostics. As knn outperforms nb for the optimal classifier–diagnostics combination, we study knn’s performance for all possible sets of diagnostic tests prior to the first consultation. As can be observed from Table 5, if all patients who are classified to be non-inflammatory receive a blood test prior to the first consultation, on average 0.1 in-person consultations are avoided per patient and 0.26 teleconsultations. The trade-off is that 44% of these patients had an unnecessary blood test (i.e. overdiagnostics). In the best scenario, where all non-inflammatory classified patients receive a blood test and radiology examination, on average 0.21 in-person consultations and 0.49 teleconsultations are saved per patient. In this case, 6% of all patients receive only an unnecessary blood test, 27% an unnecessary radiology examination, and 38% receive both diagnostic tests as overdiagnostics. In other words, for every 100 patients classified as having a non-inflammatory disease, the clinic would avoid 21 in-person consultations and 49 teleconsultations, but at the cost of doing 44 unnecessary blood tests and 65 unnecessary radiology examinations.

Discussion

The results confirm our first aim, that a pre-first visit digital questionnaire can accurately classify patients into non-inflammatory vs. inflammatory rheumatic diseases. Among the various machine learning-based algorithms, there is a slight preference for the et classifier with an accuracy of 0.771, sensitivity (recall) of 0.809 and precision of 0.833. This result is highly comparable to the study with the FRYQ Questionnaire, also used for triage of newly referred patients to a rheumatology department in the Netherlands [3]. In that study, differentiation between the two diagnostic classes could be done with an AUC-ROC (area under the curve–receiver operating characteristic curve) score of 0.70, using elastic net regularization of 87 covariates constructed from the questionnaire. In comparison, we use 34 covariates (constructed from our pre-first visit questionnaire). The promise of screening for inflammatory rheumatic disease among newly referred patients was also demonstrated in a study with a tool called ‘Rheumatic?’ [7]. This web-based screening tool identifies individuals with early signs of or at high risk of a subset of chronic inflammatory rheumatic diseases. The pilot study, conducted in three countries, reported patients with an immune-mediated arthritic disease or not, with AUC-ROC = 79.0. When they put a threshold on the score above which people should come to the rheumatologist, this led in the Dutch dataset to a high specificity of 0.91 but low sensitivity (also called recall) of 0.14 [7].

For the second aim of our study, we simulate an intervention to the diagnostic patient journey and study its efficiency by weighing the advantage of reduction in the number of consultations against the disadvantage of the number of (over)diagnostics, using fictional costs. This intervention simulates in each patient from our test set the planning of one or more diagnostic tests before a first consultation when the questionnaire indicates a non-inflammatory rheumatic diagnosis. We compare every combination of classification algorithm and set of diagnostic tests by calculating its average (fictional) profit. The best-performing combination is the knn together with a blood test and radiology examination prior to the first consultation. In this case, for every 100 patients classified as having a non-inflammatory disease, the clinic would avoid 21 in-person consultations and 49 teleconsultations, but at the cost of doing 44 unnecessary blood tests and 65 unnecessary radiology examinations. A large asset of our method is that the revenue and cost can be tailored to the clinic. However, performing more diagnostic tests prior to the first consultation will generally lead to more avoided consultations and a higher amount of overdiagnostics. In 2009, Agrawal conducted a one-stop clinic-based ultrasonography study [8]. In 33.3% of new patients with clinical suggestions of inflammatory arthritis, musculoskeletal ultrasound did not reveal any evidence of inflammation in or around the joints. They postulated that this can have a considerable impact on subsequent evaluations (with its economic implications), treatment considerations and counselling of patients [8]. In another study aiming at improving logistics in a rheumatology department, Väre et al. showed that a one-stop shop is feasible [9].

Our study has several limitations. First, it was conducted in a single clinic. Second, we conducted the study before and during the COVID-19 pandemic, which may have led to an underestimation of the number of avoided in-person consultations since rheumatologists actively reduced the number of physical clinic visits. However, it may also have led to an overestimation of the number of avoided teleconsultations, as those replaced in-person consultations. Third, we collected fewer questionnaires than expected, due to slow acceptance of the digital questionnaires, while we were unable to prolong the collection because of the introduction of a new health record system. Fourth, rheumatologists were free to choose the course of how to care for their patients. No local protocol or guideline was in use to diagnose patients with (non-)inflammatory rheumatic disease, which limits the generalizability of our numerical results to other clinics. This also applies to the assumptions that are made about the reducibility of consultations in a patient with regard to the practices of the rheumatologists in our clinic, these have been validated by experts, but for another clinic, other or additional assumptions may be more plausible and even necessary. Fifth, the relative costs of diagnostic procedures and rheumatological consultations are determined by rheumatologists, who are not necessarily experts in estimation of such costs. Although the use of relative costs avoids the influence of differences due to, e.g. cost of electricity, overhead and doctor’s fees, these costs may vary widely between clinics in various countries and may vary with treatment demand. To prove that our approach can lead to savings, a prospective study is needed. A first step in this could be that the clinic chooses a classifier that is used to inform rheumatologists in advance of the new patient’s diagnostic class, after which the rheumatologist could decide upon pre-first visit diagnostic tests. With the collected data from this prospective research, we then may demonstrate if it leads to financial savings. The next step then could be to offer all non-inflammatory classified patients a pre-specified set of diagnostic tests prior to the first consultation. Of course, future research should first reveal feasibility of questionnaire-based classification in daily practice.

Supplementary Material