Implementations of modulation complexity scores and their correlations with treatment plan quality in stereotactic radiation therapy

Authors: Mathieu Gaudreault, Phu Hoang Nguyen, Catherine Lawford, Rick Franich, Nicholas Hardcastle

Abstract

Background

During the delivery of contemporary stereotactic radiation therapy treatment, the radiation dose is dynamically shaped by the multileaf collimator (MLC). The modulation complexity score (MCS) is a metric that quantifies MLC apertures. However, inconsistent definitions of the MCS have been introduced in the literature. Furthermore, investigations of correlations between complexity metrics and dosimetric plan quality remain scarce.

Purpose

We aim to highlight differences between MCS definitions and assess their correlation with treatment plan quality in curated datasets of stereotactic radiation therapy treatment plans.

Methods

Volumetric modulated arc therapy treatment plans from planning challenges of multi‐metastasis stereotactic radiosurgery (SRS), pancreas stereotactic ablative body radiotherapy (SABR), and vertebral SABR were considered. According to the challenge guidelines, the quality of each plan was scored from 0 to 150. To quantify complexity, the two most used interpretations of the MCS were computed. In the first interpretation (beamMCS), the area aperture variability (AAV) was normalized by a virtual area constructed with the most open position of each leaf over all control points of the arc. In the second interpretation (cpMCS), the AAV was normalized by the virtual maximal leaf opening in each control point. Each quantity ranged between 0 (complex plan) and 1 (not complex plan). The Spearman correlation coefficient (rs) and its associated p‐value were calculated between MCS and plan score. The process was repeated by stratifying the data per site, treatment planning system (TPS), and MLC type (conventional versus high definition).

Results

The plans of 366 treatments were considered in the SRS (n = 107), pancreas (n = 137), and vertebral (n = 122) planning challenge. The plan score ranged from 86.2 to 148.3 (median = 135). All plans considered, the complexity was higher with beamMCS than cpMCS (median ± interquartile range IQR) = 0.13 ± 0.11/0.19 ± 0.11 with beamMCS/cpMCS, p‐value < 0.001). The beamMCS was weakly correlated with plan score (rs = 0.14, p‐value < 0.01) whilst the correlation was not statistically significant with cpMCS (p‐value > 0.17). SRS plans were the more complex whilst vertebral plans were the less complex in both interpretations. The beamMCS and the score were positively correlated in 2/5 TPS and with the conventional MLC. The cpMCS and the score were negatively correlated in the three challenges and 1/5 TPS. All other correlations were not statistically significant.

Conclusions

The two MCS interpretations yielded conflicting correlations with plan scores. The cpMCS was superior in assessing plan quality in this set of SRS and SABR plans. As complexity metrics may be useful tools in treatment planning optimization, standardization in their numerical implementation would be preferable.

INTRODUCTION

In modulated radiation therapy treatment, the radiation field intensity is modulated to optimize the dose to the tumor whilst limiting the dose received by adjacent healthy organs. Intensity modulation is achieved with a device shaping the field called the multileaf collimator (MLC) which is made of narrow high atomic number material (usually tungsten) leaves. ^1^ The leaves move continuously to predefined positions defined in control points (CP) with fixed gantry position in intensity modulated radiation therapy (IMRT) treatment or with moving gantry in volumetric modulated arc therapy (VMAT) treatment. The leaf positions are determined at the planning stage before treatment and are the result of a mathematical calculation optimizing the radiation dose to meet the prescription. ^2^ Accurate modeling of the MLC in the treatment planning system (TPS) is therefore essential to deliver the radiation that was planned and approved before treatment.

To quantify the modulation, “complexity” metrics have been introduced based on leaf positions and the amount of dose delivered. ^3^ , ^4^ Such metrics may be useful to predict the need for patient‐specific quality assurance or to guide treatment planning. ^5^ , ^6^ One of these metrics is the modulation complexity score (MCS), which is the product of three terms quantifying respectively the field shape and area, and the output weight. ^7^ In this context for the same output weights, a pattern with small variation in adjacent leaf positions and field area will have a higher MCS value and be considered less complex than a pattern with large variation in adjacent leaf positions and field area. Inconsistent definitions of the MCS have been introduced in the literature. ^7^ , ^8^ , ^9^ , ^10^ , ^11^ The differences between these definitions have not yet been addressed, nor their effect on the complexity.

A certain level of complexity is needed in high‐quality treatment planning to generate highly conformal dose to targets whilst keeping the dose to adjacent healthy organs to a minimal level. ^12^ However, as complexity is expected to increase with plan quality in challenging treatment plan geometry, dosimetric inaccuracies between planned and delivered doses are also expected to increase due to small beam apertures, large variations in dose rate or gantry speed, or low output intensity. ^10^ Therefore, increasing quality through complexity may lead to clinically undeliverable plans. Whilst several studies investigated correlations between complexity and deliverability, ^7^ , ^11^ , ^13^ , ^14^ , ^15^ , ^16^ reports about correlations between complexity and quality are scarce, probably due to the challenge involved in defining “quality” in the context of treatment planning.

The purpose of this study is two‐fold. We first aim to highlight mathematical differences between the two most used definitions of the MCS and establish their distinctness from each other. We further aim to investigate their correlations with treatment plan quality in curated datasets of stereotactic radiation therapy treatment plans.

METHODS

Treatment plans

The complexity analysis was performed on treatment plans selected from three treatment planning challenges hosted by the Trans‐Tasman Radiation Oncology Group (TROG). ^17^ The planning challenges covered multi‐metastasis stereotactic radiosurgery (SRS), ^18^ pancreas stereotactic ablative body radiotherapy (SABR), and vertebral SABR. ^19^ In each challenge, participants were asked to export a computed tomography (CT) image and its associated structure set, plan a treatment meeting the challenge protocol, and import the plan and its corresponding dose distribution to the challenge platform. The scoring system consisted of a list of dose‐volume metrics and conformity indices. The metrics were chosen based on clinical trials (NIVORAD [ACTRN12616000352404], MASTERPLAN [ACTRN12619000409178], and Local‐Hero [NCT02898727]) and were agreed upon by a committee consisting of the trial radiation oncologists, medical physicists, and radiation therapists. The scoring system was designed to award a minimum level of points for meeting clinical trial constraints, and increasingly higher points as the plan was improved from this base level. As such, each metric was associated with a minimum requirement, an ideal value, and a maximal score. If the metric's value was below the minimum requirement, the score of this metric was zero. If the metric's value was equal to or higher/lower than the ideal value for targets/organs at risk, the assessed score was the maximal score of this metric. If the value was between the minimum requirement and the ideal value, the score was a linear interpolation between zero and the maximal score for this metric. The maximal plan score was 150 in all three challenges. The scoring system used for each planning challenge is shown in Tables S1– S3. In this current study, only VMAT plans were considered, which represented around 75% of all plans. The analysis was performed by stratifying the dataset by planning challenge (SRS, pancreas, and vertebral), TPS (Eclipse [Varian Medical Systems, Palo Alto, USA], Monaco [Elekta, Stockholm, Sweden], RayStation [RaySearch Laboratories AB, Stockholm, Sweden], Pinnacle3 [Philips Radiation Oncology Systems, Fitchburg, WI, USA], and iPlan [Brainlab AG, Munich, Germany]), and MLC type (conventional MLC, high‐definition (HD)‐MLC).

MCS

The MCS is a composite index defined per arc that considers the aperture shape and area, and the output weights. ^7^ The aperture shape and area are respectively measured with the leaf sequence variability (LSV) and the AAV (Figure 1). MCS, LSV, and AAV range between 0 to 1, with complexity increasing with decreased metric values. The LSV quantifies the difference between adjacent leaf positions in one bank normalized by the difference between the maximal and minimal leaf positions inside the bank. In the reference frame chosen for this study, the value of a leaf position increases from bank B to bank A (see the Figure S1). In these terms, the LSV is defined per control point as the product of the contribution of each bank, LSV[cp] = LSVA[cp] × LSVB[cp], where A and B specifies the bank A and B respectively, and where (1)LSVZ[cp]=1−1l−1∑i=1l−1posiZ−posi+1Zcpmax(posZ)cp−min(posZ)cp,where l is the total number of leaves in the Z bank, where Z=[A,B], and where max(posZ)cp and min(posZ)cp are respectively the maximal and minimal leaf position in the Z bank in the control point cp (Figure 2). If max(posZ)cp−min(posZ)cp = 0, then posi=posi+1 for all leaves and therefore LSVZ[cp]=1. In this case, all leaves have the same positions.

Inconsistent definitions of the AAV have been introduced in the literature by the way the aperture area per control point was normalized. Originally, the AAV was defined as the variability in the aperture area over all control points of a given beam. ^7^ , ^20^ In this work, we named this interpretation beamAAV to differentiate it from the other interpretation. In this picture, beamAAV per control point is calculated as(2)beamAAV[cp]=∑i=1LPposiA−posiBcp∑i=1LPmax(posiA)beam−min(posiB)beam,where LP is the number of leaf pairs in the control point cp, where max(posiA)beam is the maximal position over all control points of the beam of the ith leaf in bank A, and where min(posiB)beam is the minimal position over all control points of the beam of the ith leaf in bank B (Figure 3 [top]). The aperture area of a given control point is therefore normalized by the virtual maximal aperture area constructed with the maximal position over all control points of each leaf in bank A and the minimal position over all control points of each leaf in bank B. This virtual maximal aperture area is expected to be larger than any true aperture area realization in a control point. In other words, the algorithm finds the maximal position over all control points of the beam of the bank A leaf that belongs to the ith leaf pair, repeats this process to find the minimal position of the bank B leaf, takes the difference between the two positions, repeats for all leaf pairs, and performs the sum over all leaf pairs. If max(posiA)beam−min(posiB)beam=0 for all ith leaf pairs, then leaves have the same positions and beamAAV is set to 1.

In the other interpretation, ^8^ , ^9^ , ^10^ the normalization factor is calculated per control point and not over all control points of the beam. We named this interpretation cpAAV in this work. In this interpretation, the numerator of the AAV can be understood as the average leaf pair opening (ALPO) whilst the denominator is a virtual leaf opening generated with the maximal leaf position in bank A and the minimal leaf position in bank B in the control point,(3)cpAAV[cp]=ALPO[cp]max(posA)cp−min(posB)cp=1LP∑i=1LPposiA−posiBcpmax(posA)cp−min(posB)cp.The leaves involved in the calculation of Equation (3) are shown in Figure 3 (bottom). If max(posA)cp−min(posB)cp=0, then cpAAV is set to 1. Equation (3) differs from Equation (2) as in the former, the numerator and denominator are of the order of the aperture area and therefore includes the contribution of one or more leaf pairs whereas in the latter, the numerator and denominator are of the order of a single leaf pair opening. Note that in Equation (3), the normalization represents an opening which may be virtual if the two extremal positions do not belong to the same leaf pair. In those terms, the beamMCS of a given arc is defined as(4)beamMCSarc=1MUarc∑cp=1nLSVA[cp]×LSVB[cp]×beamAAV[cp]×MU[cp],where MU[cp] are the monitor units of one control point and MUarc are the monitor units of the arc. The cpMCS of a given arc is defined as(5)cpMCSarc=1MUarc∑cp=1nLSVA[cp]×LSVB[cp]×cpAAV[cp]×MU[cp].Note that weighted average over the monitor unis may be solely applied to LSV and AAV such that (6)Xarc=1MUarc∑cp=1nX[cp]×MU[cp],where X∈{LSV=LSVA×LSVB,beamAAV,cpAAV}.

The notation used in Equations (4)–(6) has been introduced initially for IMRT treatment where the MU per CP are well defined. However, in VMAT treatment, there are n−1 non‐zero meterset weights as opposed to n control points. To assess a MU for each CP, a method has been introduced in the calculation of another metric, ^21^ , ^22^

(7)MU[cp]=12ΔMWcpcp+1+ΔMWcp−1cpMUarc,where ΔMWcpcp+1=cMWcp+1−cMWcp is the difference between successive cumulative meterset weights (cMW) with ΔMWcp−1cp=0 as cp=1 and ΔMWcpcp+1=0 as cp=n, where n is the number of control points in the reference beam, and MUarc is the total monitor units of the arc. By expanding terms in the sum, Equation (6) is equivalent to (8)Xarc=∑cp=1n−1X¯cp+1cp(cMWcp+1−cMWcp),where(9)X¯cp+1cp=12X[cp]+X[cp+1],namely the average of X between two successive control points is used. In other words, the value obtained with Equation (6), for which the difference between cumulative meterset weights is averaged, is identical to the value obtained with Equation (8), for which the metric is averaged. Note that Equation (8) differs from a previous interpretation ^11^ where the product of the average of LSV and AAV was used to define the so‐called vMCS. In this previous interpretation, terms proportional to LSV[cp]AAV[cp+1] and LSV[cp+1]AAV[cp] would be introduced and Equations (6) and (8) would not be equivalent. However, Equations (6) and (8) would be identical by dropping the cross terms and using (10)X¯cp+1cp=LSV[cp]AAV[cp]+LSV[cp+1]AAV[cp+1]2.Lastly, each quantity for the plan was defined as weighted by the number of monitor units of each arc,(11)Xplan=1MUplan∑arcXarcMUarc,where X∈{beamMCS,cpMCS,LSV,beamAAV,cpAAV}.

In our implementation, ^23^ all leaves contributing to the open field were considered. A leaf pair was considered closed if the spacing between opposite leaves was smaller than a minimal opening determined in each plan. If a jaw was positioned within a leaf, the fraction of the leaf contributing to the open field was considered in the calculation (except in the denominator of cpAAV). In the calculation of beamAAV, the leaf width was considered in both the numerator and the denominator. By definition, the leaf width was not considered in the calculation of cpAAV. The difference between the two MCS interpretations was investigated by calculating the correlation between each complexity metric and plan score. As MCS combines two metrics to deliver a composite result, individual component metrics (LSV and AAV) were also considered.

Statistical analysis

To assess correlations, the Spearman correlation coefficient (rs) and its associated p‐value were calculated between plan score and MCS, LSV, and AAV. Assuming a linear relationship, the correlation between the two MCS and AAV interpretations was calculated with the Pearson correlation coefficient (rp) and its associated p‐value. The difference in medians between distributions was assessed with the Wilcoxon‐signed rank test if the number of samples was the same or with the rank sum test otherwise. In all cases, a p‐value smaller than 0.05 indicated that the difference was statistically significant.

RESULTS

A total of 366 treatment plans were considered for which the number of plans per challenge, per TPS, and per leaf width at isocenter is shown in Table 1. Considering all plans, plan score ranged between 86.2 and 148.3 (median ± interquartile range [IQR] = 135.18 ± 16.89). A summary of the correlations evaluated in this work is shown in Table 2.

The plan score as a function of the results obtained with beamMCS and cpMCS is shown in Figure 4 (left). All plans considered, beamMCS was very weakly correlated with the score (rs = 0.14, p‐value < 0.01, n = 366) whilst the correlation between cpMCS and the score was not statistically significant (p‐value > 0.17, n = 366). The two MCS interpretations are compared in Figure 4 (right). The complexity was higher as calculated with beamMCS as compared with cpMCS (median ± IQR = 0.13 ± 0.11/0.19 ± 0.11 with beamMCS/cpMCS, p‐value < 0.001). There was a moderate correlation between the two MCS interpretations (rp = 0.67, p‐value < 0.001).

Stratifying by planning challenge, the correlation between beamMCS and the score was not statistically significant in the three challenges (p‐values > 0.39) (Figure 5). However, cpMCS was weakly anti‐correlated with the score in the three planning challenges (rs = ‐0.23/‐0.19/‐0.37 in the SRS/pancreas/vertebral planning challenges, n = 107/137/122, p‐values < 0.03). In the three challenges, plans were more complex as calculated with beamMCS than with cpMCS (median ± IQR = 0.04 ± 0.05 / 0.12 ± 0.11 in the SRS challenge, 0.19 ± 0.09 / 0.24 ± 0.07 in the pancreas challenge, and 0.12 ± 0.05 / 0.17 ± 0.07 in the vertebral challenge as measured with beamMCS/cpMCS, p‐values < 0.001 in the three comparisons). The SRS plans were more complex whilst vertebral plans were less complex, an observation that holds independently of the MCS interpretation used.

The MCS values obtained by TPS are shown in Table 3. Differences between beamMCS and cpMCS were statistically significant in all cases (p‐values < 0.001), except for the Brainlab's TPS. Probably due to the small number of plans in each stratum, the correlation with the score was only statistically significant with beamMCS with the Eclipse TPS in the vertebral SABR challenge (rs = ‐0.24, n = 72, p‐value = 0.04) and with the iPlan TPS in the SRS challenge (rs = 0.90, n = 5, p‐value = 0.04).

Interestingly, the two interpretations yielded conflicting results as the data was stratified by MLC type (see Figure S5). With beamMCS, treatments planned with the conventional MLC were more complex than treatments planned with the HD‐MLC (beamMCS median ± IQR = 0.12 ± 0.10/0.15 ± 0.11, with MLC/HD‐MLC, n = 218/148, p‐value < 0.001). However, with cpMCS, treatments planned with the HD‐MLC were more complex than treatments planned with the conventional MLC (cpMCS median ± IQR = 0.19 ± 0.11/0.17 ± 0.12, with MLC/HD‐MLC, n = 218/148, p‐value < 0.01). The beamMCS and plan score were weakly correlated in plans using the conventional MLC (rs = 0.22, p‐value < 0.01).

The LSV ranged between 0.12 and 0.83 (median ± IQR = 0.61 ± 0.18) and its correlation with the score was not statistically significant (p‐value = 0.6). Considering LSV only, the difference in complexity between vertebral and SRS plans was not statistically significant (p‐value = 0.4). However, both cases were more complex than pancreas plans (LSV median ± IQR = 0.66 ± 0.10 in pancreas plans, n = 137, versus 0.51 ± 0.28 in SRS plans, n = 107, and 0.57 ± 0.25 in vertebral plans, n = 122, p‐values < 0.001 in the two comparisons). By using LSV only, treatments planned with the conventional MLC were more complex than treatments planned with the HD‐MLC (LSV median ± IQR = 0.57 ± 0.16/0.68 ± 0.09, with MLC/HD‐MLC, n = 218/148, p‐value < 0.001). Furthermore, LSV and plan score were weakly correlated if the HD‐MLC was used (rs = ‐0.29, p‐value < 0.001) whilst the correlation was not statistically significant with the conventional MLC (p‐value = 0.54).

The plan score is shown as a function of both interpretations of AAV in Figure 6 (left). The beamAAV was weakly correlated with plan score (rs = 0.12, p‐value = 0.03) whilst cpAAV was weakly anti‐correlated with plan score (rs = ‐0.14, p‐value < 0.01). The beamAAV was smaller than cpAAV (median ± IQR = 0.23 ± 0.15/0.33 ± 0.17 for beamAAV/cpAAV, n = 366, p‐value < 0.001) as shown in Figure 6 (right). The same observation holds when the results were stratified by planning challenge, the largest difference between beamAAV and cpAAV being in SRS plans (median ± IQR = 0.06 ± 0.14/0.26 ± 0.35 for beamAAV/cpAAV, p‐value < 0.001) (Figure 7). By stratifying with MLC type, the two interpretations of AAV also provided conflicting results (see Figure S6). Treatment plans were more complex with the conventional MLC by using beamAAV whilst treatment plans were more complex by using the HD‐MLC by using cpAAV.

DISCUSSION

In this work, we highlighted differences between two inconsistent definitions of the MCS introduced in the literature. When correlations were statistically significant, cpMCS was mostly negatively correlated with plan score whilst beamMCS was mostly positively correlated with plan score. The cpMCS therefore yielded results that agreed with the expectation that complex plans were associated with high scoring. This observation was also obtained with other complexity metrics by using the same dataset. ^12^ One may argue that the latter may be expected from SRS and SABR plans, achieving high treatment plan quality sometimes comes with the cost of high complexity. However, beamMCS yielded results for which less complex plans were associated with high scoring, which is counter intuitive in this context. These contradicting results illustrate the effect of the two MCS interpretations. Therefore based on our findings, cpMCS was superior in assessing plan quality in this set of SRS and SABR plans.

The fundamental difference between cpMCS and beamMCS lies in the way the aperture area variability is measured. The former measures the variability per control point, whilst the latter measures the variability per beam. To illustrate the difference between the two metrics, consider the following examples. On the one hand, consider two arcs with constant area along the beam path. Suppose that one arc consists of an open square field, whilst the other arc is made of this square field but fractionated in two unequal parts randomly distributed. In this case, beamMCS = cpMCS = 1 in the arc made of the constant square field but beamMCS = 1 and cpMCS < 1 in the beam with fractionated smaller fields. This example illustrates that cpMCS may be more adequate for detecting small area aperture within control points if the total area remains more or less the same along the rotation. On the other hand, consider two control points where a rectangular area in the first control point is reduced to a smaller rectangular area in the second control point. In this case, beamMCS < 1 whereas cpMCS = 1. Another example would consist of a sliding window covering a rectangular area. This case would also lead to beamMCS < 1 but cpMCS = 1. Consequently, these examples illustrate that beamMCS may be more adequate for detecting small area apertures over all control points.

Our results also pointed out that plans with the conventional MLC were more complex than plans with the HD‐MLC as measured with beamAAV whereas plans with the HD‐MLC were more complex than plans with the conventional MLC as measured with cpMCS. This contradictory observation may be explained as follows. The conventional MLC has the potential to generate a larger maximal aperture area over all control points (the denominator of Equation 2) due to its larger leaf widths as compared with the HD‐MLC. This larger maximal aperture area will lead to a smaller beamAAV, which would result in plans being more complex with the conventional MLC as compared with the HD‐MLC. However, the HD‐MLC has the potential to produce higher leaf opening variability at a control point due to the increased number of contributing leaves as compared with the conventional MLC. This increased variability will lead to a smaller cpAAV, which would result in plans being more complex with the HD‐MLC as compared with the conventional MLC.

The cpMCS was previously used to compute its correlation with other complexity metrics on a multi‐centre planning audit of prostate and head and neck (HN) plans. ^10^ The cpMCS was negatively correlated with the edge metric (rs = ‐0.68/‐0.63 for prostate/HN plans) and plan irregularity (rs = ‐0.60/‐0.62 for prostate/HN plans). These negative correlations were expected as complexity increases with decreased MCS and with increased edge metric and plan irregularity. As the edge metric ^24^ and plan irregularity ^25^ are defined by control point, the implementation of cpMCS would be facilitated in an algorithm as compared with beamMCS. Furthermore, since these metrics may share the same numerical array defining the area per control point, strong correlations between each metric may be expected.

To our knowledge, no study reported a correlation between cpMCS and plan deliverability. However, several reports assessed the correlation of beamMCS with gamma passing rate (γPR) between planned and measured dose distributions to predict potential patient‐specific quality assurance failure. In particular, weak to moderate correlations were observed by using either the Pearson or Spearman correlation coefficient (rp = 0.46 with γPR(3%, 0 mm), n = 30; ^7^

rp = 0.5 with γPR(3%, 3 mm), n = 142; ^11^

rs = 0.21 with γPR(3%, 3 mm), n = 240; ^13^

rs = 0.37 with γPR(2%, 1 mm), n = 40; ^14^

rp = 0.91 with γPR(2%, 2 mm), n = 612; ^15^

rs = 0.42 with γPR(2%, 2 mm), n = 38 ^16^ ). A study comparing beamMCS and cpMCS in the context of plan deliverability would be interesting.

As shown in Table 2, cpAAV reported more significant statistical results than cpMCS. In clinical settings, reporting a single number as plan complexity is attractive due to its simplicity. However, the composite nature of the MCS often obscures the physical explanation for the plan's complexity. As such, independent evaluation of AAV and LSV is recommended, especially when plan quality is questioned. A further evaluation of LSVA and LSVB may also be useful depending on the clinical context. It is worth noting that other metrics specialized in detecting small area apertures, such as the small aperture score (SAS) ^26^ or the percentage of control points with segment area lower than 5 × 5 cm2 (%SA < 5 × 5 cm2), ^27^ may provide further insights.

In this work we modified the original interpretation of the MCS adapted for VMAT (the so‐called vMCS) ^11^ with Equations (6) and (8) as the original interpretation introduced terms proportional to LSV[cp]AAV[cp+1] and LSV[cp+1]AAV[cp]. The physical interpretation of these two terms is questionable. However, the maximum relative difference between MCS calculated with Equation (6) or (8) and the original interpretation was smaller than 10% in all treatment plans considered in this study. In a future effort to standardize complexity metrics, we do recommend reporting LSV and AAV separately as calculated with either Equation (6) or (8) as opposed to reporting their product.

We focused on the two most common interpretations of the MCS to highlight their differences. Other interpretations or modifications have been introduced in the literature, such as normalizing AAV with the maximum area over all control points, ^23^ restricting the calculation to specific leaves spanning an organ, ^9^ or the integration of angular dependency in CyberKnife plans. ^20^ The reference frame used in this study (shown in Fig. S1) was chosen to facilitate an implementation of the MCS. Other frames might be useful for validation, such as the one introduced in sec. 2 of the Supporting Information. Complexity metrics may be valuable tools in the optimization of treatment plan quality. As such, standardization is necessary. However, integration of complexity metrics in commercial TPS is currently lacking. ^5^

The minimum leaf gap determines the threshold for which a leaf pair is opened or closed. This parameter may differ between institutions and TPS, and its value is not stored in the DICOM format. In this work, we used the smallest non‐zero leaf pair opening as the minimum leaf gap. This choice may be wrong in some plans and using other values would have changed the results. Regarding the potential standardization of complexity metrics, the inclusion of the minimum leaf gap opening in the DICOM format would be useful.

Finally, evaluating plan quality is challenging and no gold standard currently exists. ^6^ Therefore the scoring system used in each challenge was subjective. Furthermore, the number of plans stratified by challenge and TPS was low. As a result, correlations between plan score and complexity were mostly not statistically significant with this stratification. A study designed to investigate correlations between plan quality and complexity by TPS would be interesting. Moreover, plan complexity was not included in the scoring system of each planning challenge considered in this study. Therefore, high quality may have been achieved at the expense of complexity and some plans may have been unrealistic in clinical practice. It is therefore expected that the level of complexity achieved in this cohort would be higher than in a clinical cohort.

CONCLUSION

The treatment plan quality score was correlated with plan complexity in a dataset composed of three planning challenges for SRS and SABR treatment. However, inconsistent definitions of the MCS led to different results, even leading to opposite relations between quality and complexity. The cpMCS demonstrated superior complexity assessment than beamMCS in the set of treatment plans considered. As complexity metrics may be valuable tools to optimize quality in treatment planning, standardization is necessary.

CONFLICT OF INTEREST STATEMENT

N.H. receives research funding from Varian Medical Systems for unrelated work. N.H. is a paid consultant of SeeTreat Medical.

Supporting information