Test‐retest reliability of cardiopulmonary exercise test‐derived metrics in individuals with COPD versus healthy controls

Authors: Milan Mohammad, Rie S. Thomsen, Iben E. Rasmussen, Amalie B. Andersen, Jacob P. Hartmann, Ronan M. G. Berg

Abstract

Background

Cardiopulmonary exercise testing (CPET) is usually considered the gold standard for assessing maximal oxygen consumption (V̇O2max), a health and performance marker in patients with chronic obstructive pulmonary disease (COPD). Despite the widespread application of CPET, the absolute and relative test‐retest reliability of CPET‐derived metrics remains unexamined.

Objective

To examine and compare test‐retest reliability of CPET derived metrics in individuals with COPD and healthy matched controls.

Methods

12 individuals with COPD and 12 healthy age‐ and sex‐matched controls were included in this case‐control study. Each participant completed two CPET on a bicycle ergometer on two different days. Absolute reliability was reported as smallest real difference (SRD) and relative reliability as coefficient of variance (CV) and intraclass correlation coefficients (ICC).

Main Results

SRD for peak oxygen uptake was 451.6 (267.4;1006.4) mL/min and CV was 7.8 (4.7;11.0)% in patients with COPD, whereas SRD was 244.2 (151.4;491.5) mL/min and CV was 3.0 (1.8;4.2)% in healthy controls but with no significant between group difference for SRD. CV values for all CPET derived metrics were found to be below 10%. Apart from peak workload achieved and peak minute ventilation, SRD and CV were significantly higher in COPD than in controls for all other CPET‐derived metrics.

Conclusion

This study provides test‐retest reliability estimates of the most widely used CPET derived metrics in individuals with COPD and healthy matched controls. Test‐retest reliability for most metrics derived from CPET were found to be lower in individuals with COPD when compared to healthy controls.

INTRODUCTION

Exercise intolerance is common in individuals with chronic obstructive pulmonary disease (COPD) and is of multifactorial origin (Jaitovich & Barreiro, 2018; Marillier et al., 2020). The assessment of maximal oxygen consumption (V̇O2max) through cardiopulmonary exercise testing (CPET) is often considered the reference method for evaluating cardiorespiratory fitness and identifying mechanisms underlying exercise intolerance (Herdy et al., 2016; Stickland et al., 2012). As such, many individuals with COPD undergo CPET both in clinical settings and as part of research studies focusing on exercise interventions (Peters et al., 2023). Several studies have demonstrated that a reduced exercise capacity measured by peak oxygen uptake (V̇O2peak) serves as a strong independent predictor of mortality in individuals with COPD regardless of forced expiratory volume in 1 s (FEV1) and age (Da Luz Goulart et al., 2022; Ewert et al., 2022; Oga et al., 2003; Yoshimura et al., 2014). In addition, CPET derived metrics reflective of pulmonary gas exchange efficiency such as minute ventilation (V̇E), oxygen uptake (V̇O2), carbon dioxide production (V̇CO2) and the ventilatory equivalent for carbon dioxide (V̇E/V̇CO2‐slope) provide both diagnostic and prognostic information in COPD (Behnia & Sietsema, 2023; Da Luz Goulart et al., 2022; Neder et al., 2016).

Despite the wide use and clinical value of CPET in COPD, the absolute and relative test‐retest reliability of various CPET‐derived metrics has not previously been investigated in COPD nor compared with age‐ and sex matched healthy controls. Thus, while previous studies have reported that various CPET metrics are reliable, both in healthy young individuals (Rivera‐Brown et al., 1995; Skinner et al., 1999) and in various clinical populations such as Parkinson's disease (Jensen et al., 2023), multiple sclerosis (Langeskov‐Christensen et al., 2014), and coronary artery disease (Coeckelberghs et al., 2016), it is unknown whether this applies to individuals with COPD during a standardised CPET. Earlier studies on COPD patients have only assessed test‐retest reliability using symptom limited exercise tests with no studies reporting both absolute and relative reliability for more metrics determined from a standardised CPET (Barron et al., 2014; Covey et al., 1999). Ventilatory restraints, including dynamic hyperinflation and impaired pulmonary gas exchange, together with lower limb exertion are factors possibly resulting in unfavorably test‐retest reliability in individuals with COPD. This contributes to insufficiently performed tests with higher between‐day variability than in various other healthy and patient populations (Behnia & Sietsema, 2023).

Therefore, the objective of this present study was to (1) provide between‐day test‐retest reliability estimates of the most widely used CPET‐derived metrics in individuals with COPD, and (2) to compare these estimates to those of age‐, and sex‐matched healthy controls.

MATERIALS AND METHODS

Ethical approval

The study was approved by the Regional Ethical Committee of the Capital Region of Denmark (file no. H‐23075064) and performed according to the most recent guidelines of the Declaration of Helsinki. All participants provided oral and written informed consent before enrolment. The study was registered on ClinicalTrials. gov (ID: NCT06257381).

Study design and setting

This case‐control study is reported according to the “Strengthening the Reporting of Observational Studies in Epidemiology” (STROBE) Statement: Guidelines for Reporting Observational Studies (von Elm et al., 2008). From 15th of March 2024 to 19th of September 2024, a total of 24 individuals were recruited to participate in the study at the Centre for Physical Activity Research, Rigshospitalet, Copenhagen, Denmark. We included 12 individuals with COPD and 12 healthy controls. The study consisted of two study days denoted as visit 1 and visit 2 (Figure 1). During visit 1, participants underwent preliminary testing including medical health interview and full pulmonary function testing followed by a CPET. On visit 2 the participants repeated the CPET identical with the test on visit 1. Visit 1 and 2 were repeated within 2‐10 days with exactly the same experimental set‐up including the same test investigator. Moreover, all participants were instructed to refrain from caffeine consumption, alcohol ingestion, vaping, and nicotine use for at least 24 h before each visit and to refrain from vigorous exercise 48 h before the visits.

Participants

The inclusion criteria for individuals with COPD were (1) age from 45 to 80 years, (2) FEV1 to forced vital capacity (FVC) ratio (FEV1/FVC‐ratio) below 0.7, (3) a Modified Medical Research Council score of 0–3 (Bestall et al., 1999), and (4) a resting arterial oxygenation > 90% without oxygen supplementation. The COPD diagnosis was confirmed by a medical team of licensed physicians after clinical history and lung function tests were assessed. The inclusion criteria for healthy controls were (1) age from 45 to 80 years, (2) normal values of FEV1, FVC and FEV1/FVC‐ratio, (3) a Modified Medical Research Council score of 0–3, (4) a resting arterial oxygenation > 90% without oxygen supplementation and (5) a normal single‐breath diffusing capacity for carbon monoxide corrected for haemoglobin (DL,COc).

Exclusion criteria for both COPD and healthy controls were known heart failure, ischaemic heart disease, cardiac arrhythmic disease, claudication, renal or liver dysfunction, malignant disease, pregnancy, symptoms of any disease within 2 weeks before the study and participation in pulmonary rehabilitation within 6 months. For healthy controls, also any kind of pulmonary disease was an exclusion criterion.

The cases (individuals with COPD) were matched on sex and age (+/– 3 years) with controls (healthy individuals).

Measurements

Medical examination

During initial visit 1, each study participant underwent a medical examination. This included auscultation of the heart and lungs, a concise medical history review, electrocardiogram, blood pressure, heart rate and review of all current drug use to confirm the absence of any exclusion criteria conflicts. Height (m) and weight (kg) were measured, and body mass index (BMI, kg/m^2^) was calculated as weight/height^2^.

Lung function testing

All lung function tests were performed by trained personnel following standardised protocols in line with international guidelines (Bhakta et al., 2023; Graham et al., 2019) as a part of the initial assessment. All tests were performed with Jaeger MasterScreen PFT pro system (CareFusion, Höchberg, Germany), and included dynamic spirometry, whole‐body plethysmography, and single‐breath uptake of carbon monoxide (CO). The following data were FEV1, FVC, FEV1/FVC‐ratio, total lung capacity (TLC), residual volume (RV), alveolar volume (VA), and DL,COc, both as absolute values, and as percent of predicted values according to standard reference equations (Stanojevic et al., 2022). Before measurements, participants' standing height, weight, and haemoglobin (Hb) levels were obtained. Hb measurements were performed using capillary blood samples and analysed with the HemoCue device (Hb 201 + ; HemoCue AB, Ängelholm, Sweden).

Cardiopulmonary exercise test

The CPET was completed on a bicycle ergometer (LC4, Monark Exercise AB, Vansbro, Sweden). The test commenced with 5 min baseline measurement with the participant placed in a resting sitting position on the bike, followed by a 5‐min warm‐up. This was ensued by an incremental increase in workload every minute until exhaustion. The test was terminated either due to medical conditions, reaching less than 60 rpm for 20 s., saturation level dropping below 80%, or voluntarily cessation by the participant. Workload and increments were calculated for the individual participant in accordance with physical activity level, age, sex, waist measurement and resting heart rate (Nes et al., 2011). Oxygen consumptions were measured breath‐by‐breath by using the Quark gas analyzer (Cosmed Quark CPET System, Cosmed, Srl, Rome Italy). If two out of three of the following CPET criteria were fulfilled, V̇O2max was considered achieved; (1) respiratory exchange ratio (RER) > 1.1, (2) Borg scale > 17, and (3) plateau in V̇O2 (determined by visual inspection of the test results) (Bassett & Howley, 2000; Radtke et al., 2019). If this was not the case, V̇O2 were reported as V̇O2peak. Furthermore, a subjective assessment of exhaustion was evaluated by the operator, and reasons for early test termination were noted. Rate of perceived exhaustion was measured using the Borg scale (6–20) at test termination rating both overall exhaustion, lung, and leg exhaustion. Heart rate was measured with a Garmin HR chest band, which was compatible with COSMED (Garmin International Inc., Olathe, KS, US). Saturation was measured by pulse oximetry (Nonin, Xpod). Adjustment of handlebars, saddle, and operator were ensured to be similar at both visits.

Outcome measures

Absolute between‐day reliability, estimated by the smallest real difference (SRD), as well as the corresponding relative reliability estimates, assessed by the coefficient of variance (CV) and intraclass correlation coefficient (ICC) were assessed for the following CPET‐derived V̇O2peak, peak workload (WL,peak), peak heart rate (HRpeak), peak oxygen pulse (V̇O2/HRpeak), peak minute ventilation (V̇Epeak), V̇E/V̇CO2‐slope, and peak RER (RERpeak).

Potential sources of bias

The likelihood of misclassification of COPD was minimal. The COPD diagnosis was rigorously validated through comprehensive lung function tests, adhering to widely accepted consensus guidelines (Stanojevic et al., 2022) and evaluated by a medical team of licensed physicians. All measurements were standardised and conducted consistently by the same team with the same equipment. However, the predominantly Caucasian composition of our study cohort limits the applicability of our results across diverse ethnic backgrounds.

Sample size

As this is a pilot trial, there is no prior information to base a sample size on. For such pilot studies the recommendation is a sample size of 12 per group, based on rationale about feasibility, precision about the mean and variance (Julious, 2005).

Statistical methods

All statistical analyses were conducted using the R statistical software version 4.3.3 (R Foundation for Statistical Computing, Vienna, Austria) within RStudio (version 1.4.1717). Normality of the data were assessed with visual inspection by histograms and Quantile‐Quantile (QQ) plots.

Normally distributed variables are reported as mean (standard deviation (SD)) and mean difference (95% confidence interval (95% CI): lower limit (LL); upper limit (UL)); otherwise, non‐normal distributed data are reported as median [25th percentile–75th percentile]. A Student's t‐test was used to detect differences in baseline characteristics between groups and a Mann‐Whitney U test was used when assumption of normality failed.

Repeated measurements were analyzed using linear mixed effects regression (lme4 package; version 1.1–35.3; Bates et al., 2015). Model terms were CPET metric ~ visit*group + age + sex + height + (1 | ID), with random intercepts for unique participant ID to account for repeated measures. Model assumptions were assessed via visual inspection of fitted values versus residual plots and gave no reason of concern. Post hoc pairwise testing of estimated marginal means was performed if significant results were conducted to ensure internal consistency; test‐retest reliability was only assessed for internally consistent variables.

Both absolute and relative test‐retest reliability was assessed (Hartmann et al., 2023). For absolute reliability, these involved calculations of SRD, which estimates the maximum difference between any two measurements on 95% of occasions when no real underlying difference is present, using a one‐way analysis of variance. Moreover, one‐way analysis of variance was used to determine the standard deviation within participants (SDw). SRD was calculated using the following formula (Vaz et al., 2013) SRD=(T−quantile with appropriate degrees of freedom×2*SDw2where T represents the t‐value from the Student's t‐distribution.

Relative reliability was assessed by CV, which expresses the proportion of variance (%) caused by measurement error (Vaz et al., 2013): CV=SDwMean of measurements×100

Based on the distribution of mean estimates and residual variance from a linear mixed model, we simulated the distribution of the CV to obtain 95% confidence intervals for the CV, following the method previously described (Liu, 2012). Moreover, ICC was also used as an additional measure of relative reliability. The ICC was calculated using a two‐way mixed‐effects model with the absolute agreement and multiple measurements ICC (Lee et al., 2012; Koo & Li, 2016).

The R‐package “clintools” was used to calculate SRD, CV, and ICC with 95% CI and p‐values for between group comparisons by bootstrapping using 1,000 iterations in the comparerel‐function from the clintools‐package (Hartmann et al., 2023; Olsen et al., 2023). Statistical significance was accepted at α < 0.05 (two‐sided).

RESULTS

Participant characteristics

Participant characteristics are outlined in Table 1. The two groups differed only in terms of lung function metrics as individuals with COPD had lower both raw values and % of predicted for FEV1, FEV1/FVC ratio, RV and DL,COc. Moreover, individuals with COPD had more smoking pack years.

Cardiopulmonary exercise test

All participants achieved complete exhaustion during each CPET assessed by the operator. In individuals with COPD 75% (18/24) of the tests fulfilled two out of three criteria for V̇O2max to be considered achieved. In the healthy control group this corresponded to 96% (23/24). A plateau in V̇O2 was reached in 67% (16/24) of the tests for the individuals with COPD, and 88% (21/24) for the healthy controls. RER > 1.1 was reached in 54% (13/24) of the individuals with COPD and 92% (22/24) of the healthy controls. Borg > 17 was reached in 79% (19/24) of tests for the individuals with COPD, whereas 96% (23/24) of tests for the healthy controls fulfilled this criterium. Since two out of three V̇O2max criteria were not met across all tests, the term V̇O2peak will be used to describe the highest oxygen uptake achieved during the single tests covering both if maximal oxygen uptake was achieved or not.

The individuals with COPD had lower values of all metrics derived from the CPET when compared to the healthy controls except for V̇E/V̇CO2‐slope for which individuals with COPD had higher values (Table 2).

Between‐day test‐retest reliability

All metrics from the CPET were internally consistent between study days, as no systematic between‐visit differences in any metric were detected for either COPD or the healthy control group with no main effect of visit (Table 2). SRD, CV, and ICC for all metrics in the two groups are provided in Table 3.

SRD for V̇O2peak was 451.6 (267.4;1006.4) mL/min and CV was 7.8 (4.7;11.0)% in COPD patients, whereas SRD was 244.2 (151.4;491.5) mL/min and CV was 3.0 (1.8;4.2)% in healthy matched controls. SRD for V̇O2peak (mL/min) tended to be higher in COPD than in controls with a 207.37 mL/min, but without reaching statistical significance (p = 0.135), while CV was significantly higher in the former (p = 0.024). Apart from WL,peak, V̇Epeak and RERpeak, SRD and CV were significantly higher in COPD than in controls for all other CPET‐derived metrics. ICC did not differ between groups for any metric.

DISCUSSION

This study found CV values of all CPET derived metrics in both groups to be below 10%. Additionally, it showed a significant difference in the test‐retest reliability of V̇O2peak between groups, with lower relative reliability according to the CV estimates observed in individuals with COPD. Additionally, both absolute and relative reliability itself were lower in individuals with COPD when considering for HRpeak, V̇O2/HRpeak and V̇E/V̇CO2‐slope. Together, these findings indicate that CPET‐derived metrics exhibit a high day‐to‐day variability in individuals with COPD.

The reliability of V̇O2peak has previously been investigated, with considerable variability in the statistical approaches. A study (Noseda et al., 1989) reported a CV of 9% for V̇O2peak in individuals with severe COPD, which is slightly higher than the 7.8% observed in our study. This discrepancy may be attributed to the greater severity of disease in their cohort, no restrictions before each test day, and the fact that participants underwent breathing training between tests. Another study on 24 individuals with moderate COPD conducting symptom limited exercise tests on a cycle ergometer found a CV of 11% for V̇O2peak. This is slightly higher than our results possibly ascribed to disease severity, time between test and differences in testing protocol. The time between tests were from 2 h to testing on a separate day, and the test protocol from test one to test two was different, as a standardised testing protocol was used on the first test and a personalised protocol was used on the second test (Barron et al., 2014). Only one previous study has evaluated both a COPD group and a healthy control group, reporting an ICC of 0.97 for individuals with COPD and 0.99 for healthy controls as the only measure of reliability for V̇O2peak (Covey et al., 1999). However, although widely used, ICC has limited value compared to SRD and CV, as it is sensitive to both within‐ and between‐group variations. This means that if ICC is reported on a highly heterogenous population, a high within‐group standard deviation could be the reason for high ICC values, regardless of the method's flaws (Hartmann et al., 2023).

With a CV value of 7.8% in the COPD group and of 3.0% in the healthy control, the latter group had a higher between‐day test‐retest reliability of V̇O2peak. This may reflect fluctuations in disease symptoms and limitations of multiple steps of the oxygen transport cascade, which can exacerbate dyspnea and leg fatigue during exercise, both of which are recognized as primary exercise‐limiting factors in individuals with COPD (Pinto‐Plata et al., 2007). Due to disease‐related limitations, achieving V̇O2max is often not possible for individuals with COPD (Covey et al., 1999; Elbehairy et al., 1985). As evidenced in this study, only 75% of tests performed by individuals with COPD met at least two of the three V̇O2max criteria compared to 96% in the healthy controls, indicating superior test validity in the control group. Inconsistency in exercise capacity between days in individuals with COPD might be a result of difference in ventilatory limitations on the given day. This may challenge the detection of changes in V̇O2peak in exercise intervention studies and could also affect the accuracy of prescribed exercise intensities when using acute exercise bouts determined from CPET.

The SRD for V̇O2peak in individuals with COPD and healthy controls was found to be 451.6 mL/min (24%) and 244.3 mL/min (9%), respectively. SRD indicates how much a measure has to increase to surpass day‐to‐day variability and measurement error, and thus the threshold for detecting a significant and meaningful physiological change in individuals with COPD is higher. For comparison, previous training intervention studies in individuals with COPD have reported increases in V̇O2peak ranging from 5% to 15% from baseline, depending on the intensity, duration, and modality of the exercise intervention (Arnardóttir et al., 2007; Coppoolse et al., 1999; Varga et al., 2007). Thus, the SRD in individuals with COPD derived from this study may somehow exceed the physiologically expected increase in V̇O2peak following a training intervention. This necessitates consideration when determining whether observed improvements in V̇O2peak reflect actual physiological adaptations rather than day‐to‐day variability in test measurements. Thus, a change smaller than SRD for the individual participant might be accepted if more physiological results point in the same direction. Despite these challenges, a consensus exists that a CV below 10% is typically regarded as an acceptable threshold for measurement methods (Aronhime et al., 2014; Liu, 2012; Shechtman, 2013). In the present study, all CPET metrics demonstrated CV values below 10% in both groups, indicating acceptable test‐retest reliability across all metrics, even though individuals with COPD exhibited high day‐to‐day variability. However, it must be kept in mind that any predefined threshold for when a given reliability estimate is ‘acceptable’ is arbitrary, as this depends on entirely its intended application, including the context of measurement, the availability of other measurement methods, the hypothesized effect size, and the relevance to the study objectives.

In addition to V̇O2peak, significant higher SRD and CV values for HRpeak, V̇O2/HRpeak, V̇E/V̇CO2‐slope were found in patients with COPD when compared to healthy controls, suggesting lower reliability in the COPD population. However, with all CV values to be below 10%. A previous study (Barron et al., 2014) on individuals with moderate COPD reported a CV of 8% for HRpeak and 13% for V̇E/V̇CO2‐slope, which is slightly higher than the values observed in our study and thus suggesting lower reliability. For V̇O2/HRpeak the study by (Barron et al., 2014) found a CV of 8%, similar to our results. In terms of SRD and CV for WL,peak, individuals with COPD were represented with 21.4 W and 3.9%, whereas healthy controls were represented with 19.0 W and 2.4% showing similar values between groups in our study, with the lowest CV value among all investigated CPET metrics, making WL,peak the most reliable test estimate. Notably, our CV values for WL, peak for both individuals with COPD and healthy controls were lower than those reported in previous studies finding a CV of 9.7% in individuals with severe COPD (Noseda et al., 1989) and 13% in individuals with moderate COPD (Barron et al., 2014). These discrepancies between studies may be attributed to differences in variations in the patient populations studied, time intervals between tests, testing protocols and the degree of control in testing environments.

While we matched the individuals with COPD and healthy controls for age and sex, the relatively small sample size (12 participants per group) may have limited the statistical power, increasing the risk of Type II error. The small cohort also precluded subgroup analysis based on the severity of COPD, which may have provided additional insights into how disease progression affects test‐retest reliability. Although we made efforts to conduct the CPET at the same time of day to control for circadian variations in physiological responses, logistical constraints sometimes prevented this. As a result, variability in test conditions across different time points may have introduced uncontrolled variability in CPET outcomes. Furthermore, a subjective approach was used to determine whether a plateau was achieved or not in the CPET to exhaustion limiting our findings to this approach. On the other hand, the fact that the same operator tested the same individual on both study days means that the reported test‐retest reliability estimates do not necessarily apply to situations where different operators perform the testing, in which case reliability may become lower. Finally, differences in medication use between the individuals with COPD and healthy controls were not controlled for. Given that individuals with COPD frequently use bronchodilators on as‐needed daily basis, this may have influenced their CPET performance on each given test day, potentially contributing to the increased between‐day variability.

CONCLUSION

In the present study, we have provided test‐retest reliability estimates for the most widely used CPET‐derived metrics during ergometer‐based exercise to exhaustion, both in COPD patients and age‐ and sex‐matched healthy controls. While both groups showed CV values below 10% for all CPET‐derived metrics, individuals with COPD did show lower between‐day test‐retest reliability for most metrics, likely due to day‐to‐day variations in disease manifestations, as well as questionable validity because many individuals with COPD are unable to achieve the current CPET test criteria.

AUTHOR CONTRIBUTION

Milan Mohammad: Design, data collection, data analysis, data interpretation, figures, first draft, revisions. Rie S. Thomsen: Data collection, data analysis, data interpretation, figures, first draft, revisions. Iben E. Rasmussen: Data collection, data interpretation, revisions. Amalie B. Andersen: Data collection, data interpretation, revisions. Jacob P. Hartmann: Design, data collection, data analysis, data interpretation, revisions. Ronan M. G. Berg: conception, design, data interpretation, revisions, supervision. All authors approved the final version of the manuscript and agreed to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved. Ronan Martin Griffin Berg is guarantor of this work and accepts full responsibility for the work and the conduct of the study, had access to the data, and controlled the decision to publish.

CONFLICT OF INTEREST STATEMENT

The authors declare no conflicts of interest.