Serology Review Memo, April 16, 2012 - MenHibrix
Date: April 16, 2012
To: File, STN 125363/21
From: Mustafa Akkoyunlu, MD. Ph.D., Committee Member
Through: Willie F. Vann, Ph.D., Chief,
Subject: Serology Review Memo for BLA 125363/21(MenHibrix)
Sponsor: GlaxoSmithKline (GSK)
This review pertains the sponsor’s response to the second CR letter issued on September 21, 2011 by CBER. The review will address the questions 1 to 3 which are about the human serum bactericidal assay (hSBA). My reviews on sponsor’s response are written after each question.
1. In response to Item 1a, you indicated that study 005 sera were not handled according to the SOP. You indicated that the study 005 ------------(b)(4)----------- was ---------------------------------------------(b)(4)----------------------------------------------------------------, which is significantly more than the validated --------------(b)(4)-----------. You hypothesized that excessive ------(b)(4)-------- may have led to ----(b)(4)---- hSBA titers in the Men Y retest. You then tested this hypothesis by retesting immune sera subjected to -----------------------(b)(4)------------------ in the hSBA. The results of your ------------(b)(4)-------- cycles experiment showed that the assay is robust for up to ----(b)(4)-- and there may be a small impact at ---(b)(4)---. Thus, these data do not support your hypothesis that the observed ---(b)(4)--- in MenY titers is attributable to ---------(b)(4)-------------- cycles.
In addition, you have also suggested that the (b)(4) of the study 005 sera may have played a role in the ---(b)(4)--- of hSBA titers, but you have not tested this hypothesis and have not established the storage time-point at which hSBA titers begin their ---(b)(4)--. We conclude that the reasons for the ---(b)(4)--- in study 005 hSBA titers remain unknown. Please provide any additional information you may have that would explain the ---(b)(4)--- in study 005 hSBA results.
The sponsor did not provide new explanations for the unusually –b(4)-- hSBA titers measured in the retesting of study 005 sera. The evaluations of other retest analysis and the QC data analysis suggest that overall the assay stability is within the pre-defined conditions although the repeat tests and the QC sera measurements clearly demonstrate that the hSBA titers ---(b)(4)--- in time.
2. RESPONSES TO ITEMS 2 AND 3
2.1. Item 2a
In response to Item 3a, you provided data relevant to the reliability of the hSBA for Men Y. Specifically, you presented a table of hSBA values from the Y assay for the sentinel samples included in study HIB-MENCY-TT-013 (Table 5). We note that seven out of 50 samples show a greater than four-fold discrepancy between the highest and lowest reported values. Four samples show results both above and below a titer of (b)(4), including one sample with a titer in the ---(b)(4)--------------------------------. The samples with only one replicate provided are not included in the totals. In addition, a substantial amount of data is missing from the table which precludes a complete assessment of assay stability.
Together, the ---(b)(4)---- titers seen in the repeat analyses for samples from Study Hib-MenCY-005 (refer to item 1 above), in conjunction with the ---(b)(4)--- titers and the discrepancies in the data submitted in response to Items 1b and 3a (refer to item 2 above), raise concerns with regard to the ability of the hSBA assay for the Y strain to produce reliable and consistent data over time. While it is acknowledged that sample ---(b)(4)--- may have been one factor leading to ---(b)(4)--- hSBA titers in the Men Y retest, adequate control of the assay during the sample analysis of the pivotal studies is critical. In this regard, we request the following additional information:
To evaluate whether small changes in the assay over time would have affected all groups from a given study equally, please provide the blinding and randomization scheme for analysis of the samples from the pivotal studies.
The blinding appears to be sufficient.
2.2. Item 2b
2.2.1. Item 2b i
Please provide data that demonstrate that the -----(b)(4)----- algorithm maintains consistent assay performance across changes in control and complement lots. Please provide a trending analysis for the -----(b)(4)----- values that demonstrates consistent assay performance within control and complement lots. Please show that the -----(b)(4)----- algorithm is independent of sample titer, i.e., that the variance of the -----(b)(4)----- ratio is constant relative to titer.
Clarification: CBER clarified on October 13th 2011 (e-mail from Kirk Prutzman) that they would like to see evidence that the impact of the -----(b)(4)----- factor on the variance of the samples would be independent of samples’ titers.
Provide data that demonstrate the -----(b)(4)----- algorithm maintains consistent assay performance across changes in control and hC’ lots.
The fact that -----(b)(4)----- factor—b(4)--- in time (Figure 3 and supplement 9) indicates that the actual titers of control sera were less than the expected titers, further indicating that hSBA retesting of samples (in this case controls) yields-b(4)- titers. However, as shown in figure 4, the aggregate ratio of retest titer to original titer (individual ratio) follows a trend over a range of study samples. This figure suggests that if adjustment factor was not used Individual Ratio for tests performed between 26-Jan-09 to 25-Jun-09 would have been significantly lower because the adjustment factor during this time was –(b)(4)-. Thus, it appears like adjustment factor has significantly increased the low titer measurements of the retested samples. In theory -----(b)(4)----- factor appears to have corrected the low test serum titers. However, since we do not know the reason(s) for progressive –b(4)--- in control sera the decrease in -----(b)(4)----- factor becomes a concern.
After consulting with Ms. Freyja Lynn/Consumer Safety Officer who is also a serology assay reviewer on this file, I am also convinced that low adjustment factor may be shifting low titer samples upwards thereby effectively creating a gap between negative samples and those above the protective titer of (b)(4). Table 5 shows the 44 samples out of b(4) samples that had an initial result within the -----(b)(4)------- from the Hib-MenCY-TT-009/010 study. ----(b)(4)--- samples are those samples falling in the range of ----(b)(4)---. The percentage of samples falling in the ---(b)(4)--- in study 009/010 is low. This low percentage could be a consequence of adjustment factor or it could be due to the immunogenicity of vaccines. Because the adjustment factor may be effectively creating a gap between negative samples and those that are above the protective titer of (b)(4), Dr. Barbara Krasnicka who is a biostatistics reviewer on this file has recalculated the efficacy outcome using (b)(4) and (b)(4) as cutoff levels. The results showed that the efficacy results do not change significantly even when more conservative cutoff levels are used.
The effect of -----(b)(4)---- titers versus the -----(b)(4)----- titers on the clinical samples against MenY strains is best seen in Tables 6, 7 and 8. These tables clearly show that -----(b)(4)----- increased the GMTs about two-fold. However, it is assuring to see that both the ----(b)(4)--- and the -----(b)(4)----- titers meet the predetermined criteria for this study.
2.2.2. Item 2b ii
Please present the analysis that demonstrates that the four-parameter model can be appropriately fitted to the bacterial count data generated in the assay. Please describe how the a and d parameters for each sample are determined and controlled. Please comment on whether the curve fitting is constrained, and if so, please explain how it is constrained. Please provide the basis for the criterion that each sample has an R2 greater than –(b)(4)-.
Please present the analysis that demonstrates that the four-parameter model can be appropriately fitted to the bacterial count data generated in the assay.
The literature suggests that in principal the four-parameter model yields more accurate titer calculation than the traditional methods. Determination of accurate a and d parameters depends on well behaving titration of samples yielding sigmoid curves. Such curves are commonly achieved in assays such as ---(b)(4)---. However, variations inherent to OPA results with less accurate titration rendering poorly defined a and d parameters. The statistical merits of four-parameter model to calculate titers can be best addressed by statisticians.
2.2.3. Item 2b iii
You presented quality control charts for positive controls in the Men Y hSBA assay (QC1 and QC2) for the testing period from July 2009 to June 2010 to demonstrate assay stability. We notice that in the QC chart for Control 1 (-------(b)(4)--------) for the period from July 2009 to January 2010 (Section 4.3.8, Figure 2, page 30), many data points are below the lower limit. For the period from February 2010 to June 2010 (Section 4.3.8, Figure 3, page 30), the target value for Control 1 (-------(b)(4)--------) is changed to a higher level. Although all data points are within the control limits, the range between the lower and upper control limits becomes much wider. In light of these observations, please explain why you conclude that the hSBA Men Y assay is stable.
Comment on the following:
- Data points for -------(b)(4)-------- are below the lower limit
- Range for new control is wider
- Explain why you conclude that the assay is stable
The sponsor indicates that the range of each control is initially set provisionally based on limited runs. When the control is further tested with the clinical samples the ranges are calculated on data obtained from more runs. Therefore a given control can have a provisional range and a final range. Freyja Lynn has raised an important concern regarding the use of provisional range for controls during clinical samples since provisional ranges appear to be wider than the final ranges. The sponsor argues that since the CVs of control titers are well below the pre-accepted CV of 45% the controls performance is acceptable. Ideally, samples need to be re-evaluated after the control ranges are finalized.
2.2.4. Item 2b iv
The time period covered by these QC charts (July 2009 to June 2010) began several months after the testing of samples from studies 009 and 010 (Jan 2009 to February 2009) was completed. Thus, these QC charts do not provide information regarding the assay stability at the time the testing of samples from the clinical studies supporting this BLA was performed. Please provide data that support the stability of the assay covering the actual testing period from study -005 to study -010. Data that would be supportive include all QC charts form controls with trending analyses, reagent qualification data for any new controls or complement introduced during the analysis of samples from a given study, and all sentinel data. A detailed and continuous time line depicting the changes in controls and complement lots during the entire testing period should also be included.
Table 10 lists all reagent changes occurred between 2006 and 2009. The criteria used to qualify new reagents (Human Complement and bacterial ---(b)(4)---) were:
- The geometric mean ratio of positive samples had to be within the range of –(b)(4)--.
- The agreement between results with the two reagents had to be –(b)(4)-
According to the list, all reagents met the qualification criteria. However, the list also shows that -------(b)(4)------ method was used in the qualification of reagents starting from 08/13/2007. Therefore the issues outlined for -------(b)(4)------ are expected to effect the qualification of reagents also.
Supplement 7 Figure 4 shows the QC charts from the testing period of Phase III trial Hib-MenCY-TT-009/010. This chart shows that QC performance was reasonably good and the fact that there were no reagent changes occurred during the testing of Phase III samples suggests that possible biases that could be introduced during the qualification of new reagents has been avoided .
We are concerned that missing data for the samples from Study -013 added as sentinel samples in routine hSBA testing of samples from Studies -009 and -010 may have biased the results of the Men Y assay stability evaluation, especially for week 1. Out of the b(4) samples tested in week 1, only 20 samples have valid titer results. Eight samples have a missing value code “TC”, meaning that they were supposed to be retested at the lower dilution because less than 2 dilution points of the curve have –b(4)---------. Since these missing TCs are not missing at random, excluding these samples could make the GMR at week 1, relative to the initial reference, higher than the true ratio had those TC samples been re-tested (based on their titers at weeks 2- 4). Overall, the GMRs during the four weeks clearly suggest that a reduction in MenY titers from the initial reference values is also present for these sentinel samples from study -013. Also, the concordance analysis may not be useful for evaluating this unidirectional (--(b)(4)--) assay stability issue and its potential impact on the clinical studies results, because there are many samples with titers (b)(4) initially and few samples near the cutoff point. Please comment.
Please comment on:
- Missing data for sentinel 013 samples may bias the MenY assay results
- GMRs suggest reduction in titers is also present for sentinel samples
Missing data for sentinel 013 samples may bias the MenY assay results and GMRs suggest reduction in titers is also present for sentinel samples
The sponsor emphasized that the hSBA of the pivotal studies Hib-MenCY-TT-009 and -010 were conducted over four consecutive weeks with a single (b)(4) and ----(b)(4)------- lot. Again, it is important that the pivotal study was conducted without changing (b)(4) and (b)(4) --- since the qualification of (b)(4) and - (b)(4)- may have introduced high degree of fluctuation in the assay performance.
The sponsor indicates that 8 samples tested during the first week resulted with less than –b(4)----------. Historical hSBA titers for these samples had shown that these samples did result with -b(4)---. In subsequent weeks (weeks 2, 3, and 4) these (b)(4) samples did yield greater than –b(4)---- therefore they were included in the calculations other than week one. The sponsor indicates that the differences in hSBA titers for these (b)(4) samples are due to the use of different -- (b)(4)-- . According to the sponsor, the initial testing was done with lot b(4)and the Phase III testing was done with lot - (b)(4)- . According to the sponsor ---(b)(4)---- activity was higher than lot (b)(4). As a result, the ---(b)(4)--- factor for lot (b)(4) was (b)(4) while it was (b)(4) for the lot (b)(4). These events further underscore the inadequacy of the method used to qualify -(b)(4)-. The sponsor states that initial dilutions used for the (b)(4) samples in week one did not factor the ---(b)(4)--- factor for the ---(b)(4)---, while the initial dilutions for these (b)(4) samples in subsequent weeks were adjusted according to the –(b)(4)- derived ---(b)(4)--- factor. A re-calculation of the week 1 GMR by including the titers of the (b)(4) samples resulted with a value of 0.61 (instead of 0.81) and the GMR of 0.61 for week 1 is inline with the GMR results of weeks 2 (0.63), 3 (0.67), and 4 (0.60). However these results also suggest that GMR hSBA values consistently –(b)(4)-- in time and an explanation for the cause of this –(b)(4)-- is not provided.
The second CR letter questions further identified deficiencies about the hSBA methods used by the sponsor. An important deficiency is linked to the use of ---(b)(4)--- method. Because the acceptance range is very wide (--(b)(4)--), the ----(b)(4)---- factor appears to be introducing a significant bias to final titer calculations. This bias is also effecting the qualification of new reagents. The evaluations of each --(b)(4)-- lot qualification runs by sponsor showed that majority of the values fall within the range of --(b)(4)-- around the mid-point, further suggesting that the accepted range is too liberal. Nevertheless, it appears like a bias due to qualification of new reagents was not introduced to the testing of Phase III trial samples because reagents were not changed during the testing of Pivotal Phase III trials 009/010.
The effect of titer ----(b)(4)---- can be especially significant for the samples with titers at the lower end (close to cut-off level of (b)(4)). Titers around the cut-off level may have been moved upwards as a result of ----(b)(4)---- . Thus, samples that would normally be negative may be shifted above cut-off value of (b)(4). The number of such samples may affect the vaccine efficacy calculations. Because of this possibility CBER has recalculated Phase III trial data by using more conservative cut-off levels (----(b)(4)---- ). The results of the recalculation showed that the efficacy conclusions were not different as compared to the original results obtained using a cut-off titer of (b)(4) . Thus, since no new reagents were introduced during the Phase III sample testing and efficacy calculations based on more conservative cut-off levels support the original conclusions the hSBA used in the evaluation of Menhibrix is appears to be appropriate for its intended use.
Although data suggest that the assay was appropriate for the evaluation of Menhibrix file, the identified and the unidentified deficiencies of the assay remains as concern for use of this hSBA in future studies. The identified deficiencies are discussed above and they can be listed as:
- Calculation of ----(b)(4)---- factor.
- Qualification of reagent lots.
- The use of four parameter logistic test in titer calculation.
However, our initial concern regarding the ---(b)(4)--- in hSBA titers of retested samples remains unresolved. It is still not clear why hSBA titers are ---(b)(4)--- when samples are retested after relatively long term storage. This phenomenon appears to be independent of the above listed deficiency of the hSBA used by the sponsor.