Tag: false positives

Tackling false positives in biotech is a crucial strategy for enhancing the accuracy of genetic testing and analysis

Performing target validation well

Performing target validation well

Comments 0 Comment

Summary

This blogpost describes issues encountered in target validation and how to safeguard against poor reproducibility in RNAi experiments.

The importance of target validation

More than half of all clinical trials fail from a lack of drug efficacy. One of the major reasons for this is inadequate target validation.

Target validation involves verifying whether a target (protein/nucleic acid) merits the development of a drug (small molecule/biologic) for therapeutic application.

Failing to adequately validate a target can burden a pharma with roughly 800 million to 1.4 billion in drug development costs. Impact is not only monetary as large site closures  often result as companies struggle to save costs and a reduced production effort deprives patients of new medicines.

Performing target validation well

Special attention should therefore be given to performing target validation techniques well.

target validation techniques
Overview of target validation techniques (Lindsay, Nat Review Drug Discovery, 2013)

Many of these techniques involve inhibiting target expression to establish its relevance in a cellular or animal disease model. This can be performed with chemical probes, RNA interference (RNAi), genetic knock-outs, and even targeted protein degradation.

The reproducibility of these techniques however has been an issue of concern for drug developers. Less than half of all findings from peer-reviewed scientific publications was reported to be successfully reproduced.

Dismal rates of reproducibility from several pharma-led cancer-focused studies ranged from 11% (Amgen) to 25% (Bayer). A review by William Kaelin Jr sums up the common pitfalls of preclinical cancer target validation. One of his key points:

Cellular phenotypes caused by a chemical or genetic perturbant should be considered to be off-target until proved otherwise, especially when the phenotypes were detected in a down assay and therefore could reflect a nonspecific loss of cellular fitness. It is only by performing rescue experiments that one can formally address whether the effects of a perturbant are on-target.

The comment highlights the issue of reagent non-specificity as a notable contribution towards poor reproducibility.

Certainly, for RNAi the wide-spread off-target effects of siRNAs has been observed in numerous publications. The mechanism being well-established to be based on microRNA-like seed-based recognition of non-target genes. The effect dominates over on-target effects in many large RNAi screens, illustrating the depth of the problem.

Reagent non-specificity is not restricted to RNAi. There have been multiple reports of non-specificity for gene editing technique, CRISPR, which can be read about in detail here, here and here. Recent publications continue to shed more light on its potential off-targets as we learn more about this relatively new technique.

Even chemical probes may have multiple targets. It is hence imperative that more than one target validation technique be used to avoid confirmation bias.

Target validation – a story from Pharma

Back in 2013, when siTOOLs was just starting out, a pharma approached us with a target validation problem.

They were obtaining different results with 3 different siRNAs in a cellular proliferation assay. Despite all 3 siRNAs potently downregulating the target gene, they produced different effects on cell viability.

Which siRNA tool to trust?

target validation siRNA vs siPOOL pharma story
Three different siRNAs against the same target were tested in a cell proliferation assay. Despite all 3 siRNAs showing potent target gene silencing, effect on cell proliferation differed greatly.

A whole-transcriptome expression analysis performed for the 3 siRNAs and a siPOOL designed against the same target revealed the reason for the large variability.

target validation expression analysis siRNA vs siPOOL pharma story
How many genes can you affect with an siRNA? Whole transcriptome analysis by microarray was performed and number and % of up and down-regulated genes are shown over total number of genes assayed (18567).

Despite all siRNA tools affecting the same target, the difference in extent of gene deregulation was astounding. With the greatest number of off-target effects, it was not surprising that siRNA 3 showed an impact on cell proliferation.

In contrast, siPOOLs had 5 to 25X less differentially expressed genes compared to the 3 commercial siRNAs against the same target. An expression analysis carried out for another gene target showed similar results i.e. siPOOLs having far less off-targets.

The target was dropped from development. A great example where failing early is a good thing, though it was not without costs from validating the multiple siRNAs.

The recommended target validation tool

Functioning like a pack of wolves, siPOOLs increase the chances of capturing large and difficult prey, while making full use of group diversity to compensate for individual weakness.

siPOOLs efficiently counter RNAi off-target effects by high complexity pooling of sequence-defined siRNAs. This enables individual siRNAs to be administered at much lower concentrations, below the threshold for stimulating significant off-target gene deregulation. Due to having multiple siRNAs against the same target gene, target gene knock-down is maintained and in fact becomes more efficient.

siRNA vs siPOOL rtqPCR knock-down efficiency
siPOOLs increase targeting efficiency, avoiding knock-down variability. Figure shows rtqPCR quantification of target RNA levels when two siPOOLs vs two siRNAs against 36 genes were tested.

We still recommend using multiple target validation techniques. As a first evaluation however, siPOOLs are quick, easy and most of all, reliable.

Rescue experiments can also be performed with siPOOL-resistant rescue constructs.

Find out more

Little correlation between Dharmacon siGENOME and ON-TARGETplus reagents

Little correlation between Dharmacon siGENOME and ON-TARGETplus reagents

Comments 2 comments

The most common way to validate hits from Dharmacon siGENOME screens is to test the individual siRNAs from candidate pool hits (siGENOME reagents are low-complexity pools of 4 siRNAs).  In this deconvolution round, we normally see that the individual siRNAs for genes behave very differently and seed effects dominate (discussed here and here).

One could argue that deconvolution is not the correct way to validate candidate hits (even though it’s the method recommended by Dharmacon),  as testing the siRNAs individually will result in seed effects that are suppressed when the siRNAs are pooled.  One problem with this argument is that low-complexity pooling does not get rid of off-target effects (e.g. Fig 5 in this paper), something that is better done via high-complexity pooling.  But assuming it were true, validating with a second Dharmacon pool would be better.

Tejedor et al. (2015) performed a genome-wide Dharmacon siGENOME screen for regulators of Fas/CD95 alternative splicing.  ~1500 genes were identified by a deep-sequencing approach.  ~400 of those were confirmed by high-throughput capillary electrophoresis (HTCE, LabChip).  They then retested those ~400 genes (again by HTCE) using Dharmacon ON-TARGETplus pools.

The following plot shows the values for the siGENOME and ON-TARGETplus pools for the same genes (i.e. each point corresponds to 1 gene).

What’s measured is the percent of splice variants that include exon 6 following siRNA treatment.  That was compared to the values for a plate negative control (untransfected wells) and converted to a robust Z-score.  This is the main readout from the paper.

The Pearson correlation improves if the strong outlier at -150 for siGENOME is removed (R = 0.25), while the Spearman correlation is unchanged.

We see that a fairly small number of genes are giving reproducibly strong phenotypes (e.g. 13 of 400 have robust Z-scores less than -15 for both siGENOME and ON-TARGETplus reagents).

If we remove those 13 strong hit genes, the correlation approaches zero:

Even if the strong outlier for siGENOME is removed, the correlation is still near zero:

Although using a second Dharmacon pool removes some of the arbitrariness of defining validated hits (e.g. saying that 3 of 4 siRNAs must exceed a Z-score cut-off of X, or 2 of 4 siRNAs must exceed a Z-score cut-off of Y), the end result is similar:  A few strong  genes show reproducible phenotypes, while many of the strongest screening hits show inconsistent results.  The main problem, off-target effects in the main screen, is not fixed.

postscript

Tejedor et al. say that 200 genes were confirmed by ON-TARGETplus validation.  They consider a gene confirmed if the absolute value of the robust Z-score is greater than 2.  The Z-score is calculated using the median for untransfected plate controls.  I suspect that a significant proportion of randomly selected genes would also have passed this cut-off.

In table S3 (which has the ON-TARGETplus validation results), there are actually only 177 genes (including 2 controls) that meet this cutoff.  The supplementary methods state: Genes for which Z was >2 or <-2 were considered as positive, and a total number of 200 genes were finally selected as high confidence hits.

Which suggests that genes outside the cut-off were chosen to bring the number up to 200.

But if we look at the Excel sheet with the ‘200 hit genes’, it has 200 rows, but only 199 genes.  The header was included in the count.

This type of off-by-one error is probably not that uncommon.  In a case like this, it does not matter so much.

One case where it did matter was in the Duke/Potti scandal.  The forensic bioinformatics work of the heroes of the Duke scandal found that, when trying to reproduce the results from published software, one of the input files caused problems because of an off-by-one error created by a column header.  That was one of many difficulties in reproducing the Potti paper’s results which eventually led to its exposure.

CRISPR/Cas9 Screening – The “Copy-Number Effect”

CRISPR/Cas9 Screening – The “Copy-Number Effect”

Comments 1 comment

Several CRISPR/Cas9 screens identifying essential genes in cancer cell lines have been performed to date (Shalem et al., 2014, Hart et al., 2015, Kiessling et al., 2016). These typically take the form of pooled screens where sgRNA libraries targeting all genes or subsets of genes are introduced in parallel into Cas9-expressing cells, at a single sgRNA per cell. The sgRNAs exert a negative or positive selection pressure on cells based on their impact on cell viability and proliferation. The most depleted or enriched sgRNA sequences are determined by next-generation sequencing, revealing relevant gene ‘hits’. Very similar to how pooled shRNA screens are performed.

From these screens, several groups have observed a worrying phenomenon: CRISPR gRNAs targeting genomic regions of high copy number amplification showed a striking reduction in cell proliferation/survival. Dr William Hahn’s group at the Dana Farber Institute was one of the first to characterize this in a publication last year involving a CRISPR/Cas9 screen on 33 cancer cell lines looking for essential genes. In total, 123411 unique sgRNAs were used targeting 19050 genes (6 sgRNAs/gene), 1864 miRNAs and 1000 non-targeting negative control sgRNAs.

What they discovered is a little worrying to say the least.

The figure shows two genomic regions in two different cell lines (SU86.86 and HT29). At genomic coordinates highlighted by the red box, 3 tracks are shown. Top, copy number from the Cancer Cell Line Encyclopaedia (CCLE) SNP arrays, red indicating above average ploidy and blue showing below; middle, CRISPR/Cas9 guide scores with purple trend line indicating the mean CRISPR guide score for each CN segment defined from the above track; bottom, RNAi gene-dependency scores. AKT2 and MYC, known driver oncogenes at these loci, respectively, are highlighted in orange. For RNAi data, shRNAs targeting AKT2 used in Project Achilles were not effective in suppressing AKT2 (hence the negative result).

 

Key findings:

  • A striking enrichment of negative CRISPR guide scores (i.e. sgRNAs that reduced cell proliferation/survival) for genes that reside in genomic regions of high copy-number amplification.

 

  • Genes identified in CRISPR that reduced survival, did not have the same effect when disrupted by RNAi in the same cell lines (this RNAi screen was done by the same group but published 2 years before).

 

  • This enrichment was seen also for unexpressed genes, i.e. genes not transcribed. Meaning the reduced survival was not due to loss-of-function of the targeted gene.

 

  • Even for regions with low absolute copy numbers, a significant reduction in survival was observed compared to non-targeting control sgRNAs. Furthermore, the effect was dose-dependent with greater copy number amplifications producing larger negative CRISPR guide scores.

Notably, the correlation between copy number and genes that were scored high on essentiality was also observed when looking at data from other studies (Hart et al., 2015). The “copy number effect” would therefore produce a high number of false positives in CRISPR screens for essential genes in cancer cell lines. The graph above shows just how big an effect this is. Comparing genes identified as essential in a CRISPR screen vs RNAi screen, increasingly essential CRISPR-identified genes were more likely to reside on copy number amplifications (defined as having average sample ploidy > 2). This effect was notably absent for RNAi-derived essential genes.

Aside from false positives, the increased noise due to “copy number effects” also increases false negatives. MET, a gene identified by shRNA screens, for example, failed to be picked out by CRISPR screens as it is located on a chromosome 7 amplicon (7q31) in MKN45 cells (gastric cancer cell line) where all other gRNAs within that amplicon also scored as essential.

The authors go on to explore mechanisms behind the “copy number effect”. They found it was attributed to a DNA damage response stimulated by excessive cutting by Cas9. This response appeared p53-dependent and induced cell cycle arrest at the G2 phase, explaining the anti-proliferative effect. A similar response was seen for promiscuous sgRNAs that cut at multiple sites, with effects being more pronounced when cuts were spread over several chromosomes as opposed to a single chromosome.

How to manage this?

So far, most simply avoid analysing hits where sgRNAs lie at amplified regions or target multiple sites (Wang et al., 2017). However, these regions of copy number amplifications have been implicated in cancer and may contain relevant hits. Several computational methods have therefore recently been developed to correct for “the copy number effect”. Hahn’s group developed a computational algorithm called CERES based on data obtained from CRISPR sgRNA screens in 342 cancer cell lines representing 27 cell lineages.

Novartis also developed a Local Drop Out (LDO) algorithm that corrects obtained data based on examining gRNAs scores at direct genomic neighbours. When multiple neighbouring genes show similar drop out scores, effects are assumed to be due to “copy number effects”. This method has the advantage of not requiring prior knowledge of copy number, however it does require a sufficient density of gRNAs to accurately capture “copy number effects”.  They also had an alternative method, Generalized Additive Model (GAM) where copy number was taken into account.

 

How the CERES Model Works

The Results – copy number dependency is reduced while preserving essentiality of cancer-specific genes such as KRAS

 

A step towards the right direction but the penetrance of this effect still raises some concerns:

  • Although false positives are reduced with these computational methods, it is difficult to recapture false negatives. This is dependent on the gRNA having a stronger phenotype compared to neighbouring gRNAs on the amplicon which is not always the case. The LDO method for example still failed to recapture MET.

 

  • Guide scores can vary with cell line, sgRNA and experimental conditions, making it difficult to apply the same counter-measures to every experiment.

 

  • Given multiple cut sites trigger the same effect, how do we ensure multiple sgRNAs when introduced into a cell are not inducing a similar response? This is difficult to control in pooled screens, and poses a limitation in multiplex screens. Synthetic lethality screens for example with sgRNAs targeting multiple genes, might be subject to a higher false positive rate.

 

  • With even diploid genes (copy number = 2) having statistically significant growth reduction compared to haploid gene loci, the challenge still remains to delineate a true loss-of-function over a non-specific cellular response.

 

  • Negative sgRNA controls have to be carefully selected. From the study, non-targeting controls had little impact on viability compared to most other sgRNAs. Controls targeting non-expressed genes or non-essential loci have been recommended as better controls.

 

  • Lastly, although this effect seems to apply mostly to cancer cell lines that undergo a high rate of gene amplifications, similar effects may extend to polyploid tissues such as the liver.

Hence as always gene function should be determined by a variety of methods. Using RNAi for example to affirm a CRISPR-knockout phenotype would add greater confidence to a hit. To avoid those RNAi-related false positives however, its probably best to use siPOOLs.

 

Source of figures:

Aguirre, A. J., Meyers, R. M., Weir, B. A., Vazquez, F., Zhang, C.-Z., Ben-David, U., … Hahn, W. C. (2016). Genomic Copy Number Dictates a Gene-Independent Cell Response to CRISPR/Cas9 Targeting. Cancer Discovery, 6(8), 914 LP-929.

Meyers, R. M., Bryan, J. G., McFarland, J. M., Weir, B. A., Sizemore, A. E., Xu, H., … Tsherniak, A. (2017). Computational correction of copy-number effect improves specificity of CRISPR-Cas9 essentiality screens in cancer cells. bioRxiv. Retrieved from https://biorxiv.org/content/early/2017/07/10/160861.abstract

Other relevant sources:

Munoz, D. M., Cassiani, P. J., Li, L., Billy, E., Korn, J. M., Jones, M. D., … Schlabach, M. R. (2016). CRISPR Screens Provide a Comprehensive Assessment of Cancer Vulnerabilities but Generate False-Positive Hits for Highly Amplified Genomic Regions. Cancer Discovery, 6(8), 900 LP-913. Retrieved from https://cancerdiscovery.aacrjournals.org/content/6/8/900.abstract

de Weck, A., Golji, J., Jones, M. D., Korn, J. M., Billy, E., McDonald, E. R., … Kauffmann, A. (2017). Correction of copy number induced false positives in CRISPR screens. bioRxiv. Retrieved from https://biorxiv.org/content/early/2017/06/23/151985.abstract

 

Want to receive regular blog updates? Sign up for our siTOOLs Newsletter:

error

Like what you see? Mouse over icons to Follow / Share