Terry and colleagues [1] provided much needed information on the performance of four breast cancer risk prediction models in a high-risk population, the Breast Cancer Prospective Family Study Cohort (ProF-SC). Women in ProF-SC were recruited from high-risk clinics or were relatives of women with breast cancer in registries [2–4]. Terry and colleagues studied a subset of 15,732 women who were aged 20 to 70 years old and without previous breast cancer. They were followed for a median of 11.1 years to estimate 10-year risk of developing breast cancer. Eighty-two percent had at least one first-degree relative with breast cancer and 6.83% carried a BRCA1 or BRCA2 mutation (hereafter called BRCA mutation). For comparison, no more than 15% of women in the U.S. have an affected relative [5], and the prevalence of BRCA mutation carriers in the general population is only about 0.32% [6]. I estimated [7] that the annual breast cancer incidence rate in ProF-SC was 508/10⁵, which is 3.04 times greater than the U.S. National Cancer Institute’s Surveillance, Epidemiology and End Results Program (SEER) rate for white women (from 2011 to 2015), adjusted to the ProF-SC age distribution. Thus, ProF-SC is a high-risk cohort, ideally suited to assess the performance of risk models like BRCAPRO [8], BOADICEA [6] and IBIS [9], which include an autosomal dominant component of genetic risk and are widely used in genetic counselling. ProF-SC provides much more information on the performance of these models than all previous studies.

Terry and colleagues also evaluated the Breast Cancer Risk Assessment Tool (BCRAT) [10,11] that was designed for the general U.S. population. BCRAT warns that “This tool cannot be accurately used for women carrying a breast-cancer-producing mutation in BRCA1 or BRCA2” (https://bcrisktool.cancer.gov/). In fact, BCRAT recommends BOADICEA for such women.

There are important differences among these models. BCRAT is empirical and does not invoke a genetic model. The only information required is age, age at menarche, age at first live birth, number of previous breast biopsies (0, 1, ≥2), presence of atypical hyperplasia on any biopsy, number of affected first-degree relatives (0, 1, ≥2), and race/ethnicity. The three other models use extensive pedigree data and, importantly, incorporate data on BRCA1/2 status if available. BRCAPRO assumes that breast cancer risk is strictly autosomal dominant. BOADICEA allows in addition for residual familial correlation by including a polygenic risk component. IBIS also allows for residual familial correlation by including a low-penetrance common autosomal dominant gene in addition to a rare high-penetrance autosomal dominant gene like BRCA. In addition, IBIS includes non-genetic risk factors, unlike BRCAPRO and BOADICEA. For further details and recent developments, the reader can consult https://projects.iq.harvard.edu/bayesmendel/publications?page=1 for BRCAPRO, https://ccge.medschl.cam.ac.uk/boadicea/boadicea-web-application/ for BOADICEA, and http://www.ems-trials.org/riskevaluator/ for IBIS.

A key criterion for evaluating risk models is calibration, namely how well the observed number of breast cancers, O, agrees with the expected numbers of breast cancers predicted by the model, E. A ratio E/O > 1 indicates that the model overestimates risk, and E/O < 1 indicates underestimated risk. Unless the model is well calibrated, the predicted risks can mislead doctors and patients. Terry and colleagues examined the calibration of these models overall and in various subgroups. Most values for E/O (Table 1) are taken from table 3 in [1], but additional results for women age ≥50 were provided by MB Terry [12].

BRCAPRO consistently underestimated the observed numbers of breast cancers; E/O ranged from 0.45 to 0.65 (Table 1). BOADICEA and IBIS were well calibrated in the entire population and in BRCA-negative women, except that among all women aged ≥50 years, E/O = 1.11(95% CI 0.99–1.25) for BOADICEA and 1.08 (0.96–1.21) for IBIS, a statistically non-significant indication of slight overprediction.

BCRAT underestimated risk in the entire ProF-SC population (E/O = 0.79) and in women under age 50 (E/O = 0.55), but surprisingly, BCRAT was well calibrated (E/O = 1.06(0.95–1.19)) in women aged ≥50 years. To evaluate BCRAT, one should restrict attention to BRCA-negative women. In this group, E/O = 0.97(0.89–1.06), indicating excellent overall calibration. Among BRCA-negative women under age 50, E/O = 0.78(0.68–0.89) and among BRCA-negative women aged ≥50 years, E/O = 1.13(1.00–1.27). Thus, although BCRAT was not designed for high-risk populations, it was well calibrated in BRCA-negative ProF-SC women overall, though not as well as BOADICEA and IBIS within subgroups of women aged <50 and ≥50 years.

The 14,657 “BRCA-negative” women in [1] included 5395 (36.8%) women who were not tested for BRCA mutations and whose affected relatives were not tested. I call these women “non-tested.” The breast cancer incidence rate was higher in the non-tested women than in the tested women, probably because some of the non-tested women were BRCA-positive. I assumed that those who died, developed breast cancer or ended follow-up within ten years had 5 years follow-up on average and calculated breast cancer incidence rates per year as 0.197 for BRCA-positive women, 0.0392 for women who tested BRCA-negative, and 0.0467 for non-tested “BRCA-negative” women. If the non-tested group had had the same incidence rate as the tested BRCA-negative group, the incidence rate in the entire “BRCA-negative” group would have been reduced by a factor of (0.368×0.0392+0.632×0.0392)/(0.368×0.0467+0.632×0.0392)=0.899, with a corresponding reduction in breast cancers observed. Thus, the E/O ratios in Table 1 for all BRCA-negative women would be increased by a factor of 1/0.899 = 1.112, as shown in the row labeled “All adjusted” in Table 1. Similar adjustments are given separately for BRCA-negative women <50 and ≥50 years old with respective divisors 0.967 and 0.939 (Table 1). These adjustments suggest that BOADICEA, IBIS and BCRAT may overestimate risk slightly in women aged ≥50 years who tested negative or whose affected relatives tested negative.

TABLE 1

Table 1. Expected-to-observed ratios and concordance statistics.

I estimated what prevalence of BRCA mutations in the non-tested “BRACA-negative” women would account for the increased risk in that group. If the non-tested group is otherwise similar to the tested group, apart from BRCA mutations, the proportion p of mutation carriers in the non-tested group would satisfy 0.194p + 0.0392(1 − p) = 0.0467. The solution p = 0.0484 suggests that about 5% of the non-tested “BRCA-negative” women were, in fact, BRCA mutation carriers.

A second criterion for evaluating risk models is discriminatory accuracy, often measured by the area under the receiver operating characteristic curve, AUC, which is the probability that the projected risk from a randomly selected woman who developed breast cancer (a case) exceeds the projected risk from a randomly selected non-case (see section 6.4 in [13]). Terry and colleagues used the concordance statistic, which is usually close to the AUC [14]. Higher values of concordance indicate greater discriminatory accuracy. The concordance values for BRCAPRO, BOADICEA and IBIS were near 0.7 for the entire ProF-SC population (Table 1), with values near 0.75 for women <50 years old and near 0.60 for women ≥50 years old. The concordance for BCRAT was about 0.6 in these groups. In the BRCA-negative group, the concordance statistic for BCRAT was close to the concordances for BRCAPRO, BOADICEA and IBIS (Table 1).

It is desirable to simplify risk projection models to facilitate their clinical use. Terry and colleagues explored whether eliminating some pedigree information reduced performance. Among BRCA-negative women, there was little change in performance of BOADICEA, IBIS and BRCAPRO from dropping information on third-degree relatives and unaffected second-degree relatives; even using only first-degree relatives had little effect on calibration or concordance (supplemental table 2 in [1]). In the subset of 7737 BRCA-negative women with at least one affected second-degree relative, BOADICEA and IBIS underestimated risk if only data on first-degree relatives were used.

There is reason to believe that high concordances of BRCAPRO, BOADICEA and IBIS in ProF-SC reflect their incorporation of BRCA mutation data, rather than their extensive use of pedigree data. Their concordances were higher in younger women (Table 1), which is consistent with the higher relative risks from BRCA mutations in younger women [6]. In the BRCA-negative group, their concordances were little different from that of BCRAT, which uses only number of affected first-degree relatives. The prevalence of BRCA mutations in ProF-SC, 6.83%, is extraordinarily high. I calculated the concordance from only knowing BRCA mutation status in ProF-SC. Age-specific relative risks for BRCA1 and BRCA2 in [6] can be combined using the proportion of BRCA1 mutations in ProF-SC, 0.543, to produce relative risks r for BRCA, namely 34.0, 21.0, 15.8, 9.65, and 8.67 for ages 20–29, …, 60–69 years. Let h be the 10-year age-specific breast cancer risks in ProF-SC, which I estimated from table 2 in [1] as 0.0195, 0.0430, 0.0569, 0.0608, and 0.0670. Using these values of r and h, I calculated the age-specific probability that a case was a carrier (g₁) or not (g₀) and the probability that a non-case was a carrier (k₁) or not (k₀) and obtained the concordance from g₁k₀ + 0.5(g₀k₀ + g₁k₁). The age-weighted average of these age-specific concordances was 0.746. This calculation shows that just knowing BRCA status in ProF-SC yields a high concordance. For the general population with BRCA prevalence 0.32% [6] and h from SEER, this calculation yields a concordance of only 0.524. Thus, much of the discriminatory ability of BRCAPRO, BOADICEA and IBIS in ProF-SC derives from their inclusion of BRCA mutation data.

ProF-SC offers guidance for practice. BOADICEA and IBIS are well calibrated and take advantage of information on mutation status to improve discriminatory ability in high-risk populations like ProF-SC. BRCAPRO underestimated risk substantially in this population. Surprisingly, the simpler BRCAT performed reasonably well in BRCA-negative women.

Although well calibrated risk models provide useful information for clinical decisions and some public health applications, such as designing chemoprevention trials [15], higher concordances (well above 0.8) are needed for deciding who should not be screened or for achieving an appreciable population benefit from preventive interventions with adverse effects [16]. Previous theoretical calculations [16,17] indicated that adding mammographic density and single nucleotide polymorphism (SNP) scores might achieve a concordance near 0.7. IBIS recently introduced these factors [15], as did BOADICEA [18]. Assuming that risk had a lognormal distribution, I calculated from figures 1c and 2c in [18] that the modified BOADICEA had a concordance near 0.7 in the general population without BRCA measurements. Models like these combined with safer preventive interventions could reduce population risk. To achieve these benefits, safer interventions are needed, and the calibration of new risk models needs to be assessed with prospective studies such as that by Terry and colleagues.

CONFLICTS OF INTEREST

The author declares he has no conflicts. However, he helped develop the BCRAT models.

ACKNOWLEDGEMENTS

The author thanks Professor Mary Beth Terry and all the investigators in the Breast Cancer Prospective Family Study Cohort (ProF-SC). Professor Terry, on behalf of all the investigators in ProF-SC, provided data on women ≥50 years old in Table 1.

FUNDING

This work was supported by the Intramural Research Program of the Division of Cancer Epidemiology and Genetics, National Cancer Institute, National Institutes of Health, USA.

How to Cite This Article

Gail MH. Commentary on Terry et al., 10-Year Performance of Four Models of Breast Cancer Risk: A Validation Study. Lancet Oncol. 2019;20(4):504-17. Med One. 2019;4:e190017. https://doi.org/10.20900/mo.20190017