Discussion

This study describes the first nationwide proficiency testing for EGFR and KRAS mutation analysis, EGFR gene amplification analysis and EGFR expression. For EGFR expression analysis with IHC a consensus score of only 23% of the cases was reached, denoting a high variation in IHC results for most of the cases. Therefore, without further standardisation, EGFR expression analysis by IHC was considered not suitable for use in daily practice. In contrast, ISH and mutation analysis for EGFR and KRAS testing leads to a mean success rate of ≥97%, a sensitivity of ≥96% and a specificity of ≥95%.

To generate a norm for adequate molecular testing we propose that two cut-off levels should be chosen: one for the success rate (the percentage of correct answers is calculated as 100% minus the percentage of cases not leading to a test result and error cases combined), and one for the error level in the test outcome (the sum of false-positive and false-negative cases per test). In this study, two laboratories with success rate of 24–40% for ISH and KRAS mutation testing, respectively, clearly performed below a reasonable threshold. For the other laboratories taken together, the success rate was over 97%. To set a norm for success rate in a ring study with a limited number of test samples (<14) our study suggests that a maximum allowed error of one sample per test is reasonable. With this norm, two outlier laboratories were identified: one with two false-negative results and one with three false-negative results.

In theory, a 90% Bayesian confidence interval (CI) may be calculated for a number of samples in proficiency testing, see table 7. For instance, in a laboratory with eight out of 10 correct answers in a proficiency testing study, the chance is 90% that between 53% and 92% of the routine analyses will be correct. The chance is 5% that in daily practice this laboratory will generate the correct answer for more than 92% of cases. As it seems reasonable to set a norm in proficiency testing at the smallest number of correct cases where the upper limit of the 90% CI is above the 95%, theoretically in a setting of 10 cases at least nine cases should be correct.

For a specific set of samples in proficiency testing (n=10, 14, 20 or 30) the 90% CI (region between the 5th percentile and 95th percentile of the posterior probability distribution) is shown for the success rate (fraction of correct answers×100%). The 90% CI is constructed with Bayesian statistics,21 assuming uniform prior probability for the success rate on the interval between 0 and 1.

Our data suggest that 97% sensitivity and 97% specificity for mutation detection is feasible. Unfortunately, to determine whether every individual laboratory reaches this level precisely, an unacceptably large amount of samples needs to be tested in a ring study. However, the main aim of proficiency testing, providing external feedback about test performance, was reached with a limited number of samples, as in this study laboratories performing within the proposed error norm and outlier laboratories for a certain test were clearly established. Moreover, the necessity of participating in quality control ring studies of molecular diagnostic pathology is demonstrated.

Recently, for KRAS mutation analysis in colorectal cancer, arbitrary thresholds for success rates of DNA extraction and correct KRAS mutation identification were set at 95% and 97%, respectively.13 For lung cancer, a norm has not been recommended in the guidelines for EGFR and KRAS mutation testing.10 Importantly, our study supports that such mutation and gene amplification analyses can be performed with an average 95% success rate and 97% accuracy. Nevertheless, reporting the accuracy level, and also the sensitivity and specificity data, provides additional information.

In The Netherlands, participation in proficiency testing is part of the guideline, but a norm for proficiency testing is lacking. In the USA, in the ‘Clinical Laboratory Improvement Act’ of 1988, ‘unsatisfactory’ performance in proficiency testing is defined as failure to attain the minimum satisfactory score for an analyte for a single testing event. ‘Unsuccessful’ performance is defined as failure to attain the minimum satisfactory score for either two consecutive or two out of three consecutive events. Failure to return proficiency testing results to the proficiency testing programme within the time frame specified by the programme, or failure to participate, is also an unsatisfactory performance. In the USA (for clinical chemistry and microbiology) when testing a larger number of cases, failure to attain an overall testing event score of at least 80% is considered unsatisfactory. As to the consequences of an unsatisfactory performance the responsibility for improvement belongs to the laboratories performing the molecular test. Essential in this respect appears to be an internal quality control and validation programme with external samples. In Europe, the consequences for unsuccessful performance in molecular diagnostics have not yet been established. One option is that the laboratory itself directly withdraws the test from its diagnostic procedures. Alternatively, either the professional organisation or the government should establish consequences.

For the detection of EGFR amplification/polysomy in 2008, one laboratory reported three false-negative results, whereas in 2009 results were optimal. Although a high correlation between EGFR FISH and CISH analyses has been shown,11 12 the success rate of FISH was 100% in both years (in total 150 results), but for CISH the success rate was 67% (34 /51) in 2008 and 95% (37/39) in 2009 (see table 6). The low success rate for ISH in 2008 was largely due to a single laboratory (with a success rate of 24%) that had just begun to use EGFR ISH. It can be argued that a laboratory should not participate in a ring test as long as the test is not internally validated successfully. The scoring system to detect EGFR copy number changes in NSCLC used in this study differed from the more recently published guidelines using the Colorado criteria.22 23 Interestingly, that scoring system is significantly different from the HER2 scoring system, mainly because of the more frequent occurrence of polysomy in lung cancer compared with breast cancer. Therefore, the presence of cases with polysomy could affect the consensus rate in proficiency testing when using scoring systems other than the Colorado criteria.

A relation between EGFR mutation and gene amplification has been described: EGFR gene amplification often accompanies EGFR mutation with preferential amplification of the mutant allele.27 In our study, one out of two amplification-positive cases tested for EGFR mutations showed a mutation. For clinical relevance, the molecular diagnostic methods appear to be more important than IHC for the identification of lung adenocarcinoma patients who may benefit from EGFR inhibitor treatments.28

In the ring studies of 2008 and 2009 most laboratories did not use standard nomenclature. The EGFR mutations were interpreted as correct as long as the correct numbers of nucleotides that were deleted or inserted were mentioned. However, for future reference the use of standard nomenclature is strongly advised. Guidelines for standard nomenclature of mutations can be found on the website of the Human Genome Variation Society (http://www.hgvs.org/rec.html).

Most discordant sequencing data were found in samples containing less than 50% tumour cells; therefore, it is essential to mention the percentage of tumour cells of the material used for DNA isolation and the sensitivity of the assay and/or reliability of the specific analysis in the pathology reports. Remarkably, the variation in estimation of tumour cell percentage varies tremendously between laboratories for tissue cores that, from the first to the last section (4 μm each), were cut within a 400 μm paraffin block thickness (figure 1). This may be due to variation in interpretation between pathologists; for example, confusion between the percentage of nuclei that are in the tumour (the correct measurement) and the percentage area of the sample that is tumour, which is not the correct estimate but is the percentage that pathologists often measure; this will be subject of further research. Of utmost importance is the relation between the fraction of tumour cells in a clinical specimen and the analytical sensitivity of the method used. The latter denotes the threshold of the technique for mutation detection, which can be expressed as number of copies necessary to detect a signal or the minimum number of mutated cells required to detect a signal in a background of wild-type cells. For PCR and sequencing, a minimum amount of 20–30% tumour cells is required,29–32 while for other techniques, such as PCR heteroduplex analysis with denaturing high-performance liquid chromatography, this is 1%,33 for scorpion ARMS methodology it is around or below 1%, for PCR high-resolution melting sequencing it is around 2.5%,16 34 for peptide nucleic acid clamped PCR it is 1–10%,35 and for mutation-specific PCR it is 2.5%.35 Theoretically not only the fraction of tumour cells is relevant, but also the amount of mutated heterozygous or homozygous gene copies per tumour cell, possibly increased by either polysomy24 36–38 or (mutant allele specific) amplification.39 40 If this biological phenomenon occurs, the fraction of tumour cells may in theory be slightly lower, but still lead to mutation detection. Therefore, clinical samples harbouring a tumour cell fraction at the detection limit of the technique may be used for mutation detection, but in this situation only a mutation-positive test result will be regarded as informative.

Generally, a deletion mutation is easier to detect than a point mutation. The present series contained only one point mutation sample (DNA08). Indeed, the Thr790Met mutation was detected by only three of the four laboratories that examined exon 20. It is possible that with mutation sequencing analysis a difference in analytical sensitivity may be present between deletion and point mutations, especially in cases where the tumour cell fraction is close to the detection limit.

For proficiency testing in molecular diagnostics (DNA-based ISH/mutation testing) in The Netherlands, a 75% consensus has been reached since 1998 for >95% of the samples (Schuuring E, Thunnissen E unpublished data). In order to achieve a high consensus (for example, US standard 80%), high-quality test samples are required. An interlaboratory survey of HER2 IHC in breast cancer revealed that 20% of the ‘challenges’ (samples) showed too much variation in staining to ‘be considered valid proficiency testing challenges’. A similar sample number of 25% with low consensus (<80%) was reported in another study for HER2,41 42 and a study for CD117.43 In fact these data underscore the higher variation present in IHC compared with ISH and mutation analyses.

IHC detection of EGFR expression did not lead to a consensus score in 13 of the 17 cases. This may be explained by: (a) the use of six different antibodies (see table 2), (b) different epitope retrieval approaches,44 (c) variations in signal enhancement, (d) use of automatic or manual staining, (e) a tissue handling procedure not exactly similar to the one used in house, or (f) variation in interpretation of stained slides. The first three explanations probably have the largest impact. For different antibodies the range of variation is large, with maximum difference of around 30–50% in the number of percentage positive cases.45–47 Although the epitope-binding site may differ between antibodies, the affinity of binding may be more important, as laboratories E and F had the highest values for IHC scores, while they were targeting different domains, and also using different antigen retrieval methods. Basically, IHC analysis is not quantitative, but can at best be considered a semiquantitative method with certain limits. In a model system on quantitative IHC it was shown that a difference in epitope concentration by a factor of only four was the range between minimum staining and maximum staining intensity.48 The linear part of the dynamic range of IHC staining is close to a factor of two to three. Moreover after binding of the primary antibody, signal enhancement systems lead to an amplification of four to nine times, which is the difference between a maximum and a negative IHC outcome. Alternative explanations are less likely, such as: (a) the scoring system—scoring was standardised and performed according to a method described previously,15 which is similar to the HER2/neu IHC scoring for breast cancer in the Netherlands49; (b) the tissue sample—it was fixed for all laboratories in the same manner; (c) the paraffin TMA block—it was cut at the same time, avoiding possible influences over time50; (d) depth levels—these vary only minimally: <400 μm for both ring studies, leaving minimal space for heterogeneity; (e) difference between biopsy and resection—this was not an issue in this study.47 The lack of consensus for scoring EGFR IHC has also been reported in head and neck squamous cell carcinoma and colorectal carcinoma,51 and consequently not used for the selection of patients that might benefit from EGFR-targeted therapy. In this type of ring study, it is not necessary to standardise methodology beforehand, as it may well be that the techniques used are different, but that they still lead to the same outcome, as was previously shown for a ring study on tissue identification14 and also for the outcome of amplification and mutation analysis of this study. Nevertheless, this study clearly shows that further standardisation is required for EGFR IHC and that it should not be used to select lung cancer patients for EGFR TKI therapy at present. It is not excluded that future IHC studies with EGFR mutation-specific antibodies may yield more reproducible results.52 53 Currently, EGFR IHC plays no role in selection for clinical treatment.