Performance assessment of four HIV self-test devices in South Africa: A cross-sectional study

Volume 117| Number 1/2 January/February 2021 Research Article https://doi.org/10.17159/sajs.2021/7738 © 2021. The Author(s). Published under a Creative Commons Attribution Licence. Performance assessment of four HIV self-test devices in South Africa: A cross-sectional study AUTHORS: Mohammed Majam1 Alex E. Fischer1: Naleni Rhagnath1 Vanessa Msolomba1 Willem D.F. Venter1 Laura Mazzola2 Samanta T. Lalla-Edward1


Introduction
The UNAIDS and the World Health Organization (WHO) 90-90-90 strategy released in 2015 has been adopted globally. 1 Despite significant progress made towards improving HIV testing rates in South Africa using the conventional, facility-based approach, it was still insufficient to reach the goal of testing 90% by 2020. 2 Inclusion of HIV self-testing (HIVST) in the South African strategy was considered to complement (by promoting use in populations who do not usually exhibit facility-based health seeking behaviour) and supplement (by providing a different option for HIV testing) existing methods while possibly improving HIV testing uptake, thereby facilitating target attainment. 3,4 HIVST involves self-sampling of the user's oral fluid or blood specimen (dependent on the kit requirement), performing the HIV rapid diagnostic test (RDT), and then interpreting the result. The HIVST kits are intended to be used in a private setting, by a general population that encompasses a broad range of ages, education and literacy levels and nationalities. The benefit of HIVST includes immediate and confidential test results, and may encourage testing by groups who may otherwise avoid testing due to stigma, or the time and effort required for a clinic visit. HIVST can also promote more frequent testing, enable earlier diagnosis of HIV, may modify risk behaviours and may empower people to become more proactive and engaged in their health-care decisions. [5][6][7] The first HIVST RDT approved for home use by the US Food and Drug Administration (FDA) was OraQuick ADVANCE Rapid HIV-1/2 Antibody Test in 2012 8 and since then, studies have continued to show the benefits of HIVST across several populations 5,[9][10][11] . Based on this growing body of evidence, the WHO released guidelines for HIVST use in 2016, and strongly recommends HIVST as a way to supplement existing HIV testing services. 12 These guidelines recommend that only validated, WHO pre-qualified products should be used in public health programmes, and this position has also been adopted by the South African National Department of Health in their National HIV Self Screening Guidelines 2018. 13 In order to validate products, the WHO Prequalification of In Vitro Diagnostics coordinated through the Department of Essential Medicines and Health Products has begun a prequalification process for HIVST to identify products which follow the best practices and standards set by international groups, including the International Medical Devices Regulatory Forum, the Global Harmonization Task Force, the US FDA and the European Regulatory Authorities. 14 In December 2017, the WHO released its Technical Specification Series for the prequalification process for HIV self-test devices. The WHO prequalification process includes a review of the device packaging, instructions for use, analytical and clinical performance data, as well as a manufacturing site inspection. Device manufacturers must also demonstrate that self-testing is supported by evidence from studies that explore usability and clinical performance, among a broad population of untrained intended users. 14 The HIV Self-Testing Assessments and Research (HSTAR) programme at the Wits Reproductive Health and HIV Institute (Wits RHI) is a Bill and Melinda Gates Foundation funded programme to support HIVST developers looking to submit their device for prequalification and those seeking to enter the South African market, by independently providing data on HIVST usability (HSTAR001) and usability, performance and accuracy (HSTAR003) in the hands of untrained users.
The usability testing of seven prospective HIVST devices was recently completed with contrived results, as part of the HSTAR001 trial in Johannesburg, South Africa, and the usability index for each device was high, ranging from 84.2% to 97.6%. 15 Following a similar methodology, this study (HSTAR003) aimed to build on those results, and inform the WHO prequalification process by evaluating the usability of four HIVST candidates in clinical practice, with real-time results, instead of contrived ones. Additionally, the clinical performance and accuracy of these HIVSTs was investigated using sensitivity and specificity, by comparing results with the laboratory fourth-generation ELISA as the gold standard.

Study design
This cross-sectional study was implemented from March 2017 until November 2018, using the WHO prequalification published guidance. The HIVST devices were evaluated independently of the manufacturers and in series, to ensure no cross-contamination of assessments. To prevent participants from enrolling for more than one device, a fingerprint scanning Biometric Enrolment System was used.

HIVSTs
Four HIVST devices were assessed: three fingerstick whole blood devices and one oral fluid device. The three fingerstick devices were respectively produced by Biosure Ltd (United Kingdom), Biolytical Laboratories (INSTI) (Canada) and Chembio Diagnostic Systems (USA), while the oral fluid test was produced by Orasure Technologies (USA). Each HIVST device included the manufacturer's instructions for use (IFU) and other kit components, which were presented as intended for sale or distribution in South Africa. No additional job aids, demonstration or assistance were provided other than the manufacturer packaged materials.

Study participants
Convenience sampling was used to recruit adult participants from Wits RHI clinical trial sites in the inner city of Johannesburg. Included volunteers had to be at least 18 years old, had to be able to read English and to be first-time HIV self-testers with a self-reported unknown HIV status. Individuals were excluded if they had any prior experience with HIV self-testing or were health workers or lay counsellors who had performed HIV testing. Also excluded were participants who had received an experimental HIV vaccine or were taking HIV pre-exposure prophylaxis, persons known to be HIV positive or to have any extenuating condition (such as intoxication or acute sickness) which would interfere with the process. 15 Using the WHO Prequalification Technical Specification Series document for guidance, a blended sample size of 900 participants was required for the usability assessment of each device. This sampling intended to blend high-risk and low-risk populations, and during training recruiters were made cognisant of recruiting equal gender participation, diverse age groupings and diverse education levels. 14

Field procedures
All study procedures were conducted by a team of Good Clinical Practice trained researchers, and the self-testing followed the same procedures as HSTAR001 15 in that participants were handed a sealed test kit and they were provided with no further information about the device or test procedure. They were then requested to perform the test while being silently observed. The observer documented the process using a product-specific questionnaire. This was followed by a posttest interview.
Instead of being handed a contrived result to interpret, the participants' real self-test result was noted by the participant, then independently read and confirmed by a research nurse. In order to evaluate the performance and accuracy of the HIVST results, a 5-mL blood sample was drawn at the conclusion of each self-test, and a fourth-generation laboratory ELISA test (ABBOTT Laboratories, Chicago, USA) was performed within 24 h at the Wits Clinical Laboratory Services (a South African National Accreditation System (SANAS) approved, Good Clinical Laboratory Practice compliant facility). The ELISA laboratory test was used as the gold standard for the calculation of clinical sensitivity and specificity for each HIVST device.
HIV status was subsequently determined on site for all participants, irrespective of HIV status on the HIVST, using nurse-administered professional tests following the South African National Confirmatory Testing Algorithm. 13 Fingerstick samples were obtained using the Advanced Quality™ Rapid Anti-HIV 1&2 Test (RDT1) and the Abon™ HIV 1/2/O Tri-Line Human Immunodeficiency Virus Rapid Test Device (RDT2). If both the HIVST and RDT1 indicated a non-reactive/negative result, the participant was diagnosed as HIV negative. If one or both tests were reactive/positive, then the RDT2 test was performed. If both professional tests (RDT1, RDT2) were negative, then the participant was diagnosed as HIV negative. If both professional tests (RDT1, RDT2) were positive, then the participant was diagnosed as HIV positive and provided with a medical referral. In cases of discordant professional test results, the ELISA test was used for final diagnosis, and the participant was referred to a clinical site for the test results and follow-up.

Data collection
For the recently completed HSTAR001 usability assessment, the WHO prequalification literature was used to design, pilot test and implement a product-specific semi-structured questionnaire for data collection 15 which was also used in the current HSTAR003 study. The usability questionnaire comprised a HIVST process checklist guided by IFU steps, used to calculate usability index and a post-test interview that investigated the participants' competency, experiences and recommendations. For performance and accuracy evaluations, the ELISA laboratory test results were provided back to the research staff as an electronic copy within 24 h via email, and a hard copy was hand delivered within 7 days.

Data analysis
After data collection, field workers transcribed the questionnaire results into an MS Excel database. Quantitative data were analysed with descriptive statistics. Each batch of test kits went through a quality control check and 10% of all data entries were also checked by administrators for quality control.
Sensitivity and specificity were analysed to measure the performance and accuracy of each HIVST. Sensitivity refers to the ability of the HIVSTs to accurately detect truly positive tests, while specificity refers to the ability of the HIVSTs to correctly filter out truly negative test results. Both outcomes improve as they approach 100%, and their calculations are presented in Figure 1. The data supporting the results of this study are available upon request to the corresponding author.

Ethical considerations
The study was approved by the Human Research Ethics Committee of the University of the Witwatersrand (No. 161110). All participants signed an informed consent form and participants received a reimbursement for their participation. The manufacturer played no part in the study design, procedures or analysis of findings.

Usability assessment
The four HIVSTs had an average usability index of 97.1% (95.9-98.8%) on their product-specific usability assessment ( Table 2). The full usability indexes for each HIVST are available in Supplementary table 1. Despite the high usability, there were several spoiled tests (233/3600; 6.5%), in which critical errors prevented the test from producing a valid result. The majority of spoiled tests came from specimen collection errors (101/3600; 2.8%) or process errors (160/3600; 4.4%). A small number of spoiled tests were due to participants asking for assistance (7/3600 (0.2%) or quitting (12/3600; 0.3%). Four (0.1%) participant results were also deemed invalid due to defective kits, as they did not present a positive internal control line, even though the participants correctly completed all steps.
The process and collection errors that limited usability were specific to each device. Common errors across most fingerstick devices were due to incorrect lancing technique or lancet placement, resulting in insufficient blood available, failure to transfer the blood specimen to the device or buffer, or failure to apply the correct volume of buffer. For the oral fluid test, the most common errors were incorrect sampling technique during swabbing of the gum, and not transferring the device into the buffer solution.
Biosure and Chembio had the most spoiled tests. The Chembio and Biosure products use identical kit components and follow the same principle of testing; however, the kit components are packaged differently and have a different IFU design to align with Chembio and Biosure branding. The most common error seen across both products was related to the step: 'Push hard through the foil cap until fully seated in the buffer cap.' Those that made errors with this step had not pushed hard through the foil cap, and only inserted the tip of the device into the buffer which resulted in an inactive test and invalid result (no lines on test strip).

Performance assessment
Only participants who successfully achieved a self-test result on their own (3367/3600 (93.5%); range: 816/900 (90.7%) to 877/900 (98.2%)) were included in the performance calculation for clinical sensitivity and specificity; any incomplete tests or quits were not used to calculate the device performance. In total, there were 498 (14.  Table 2.

Discussion
While previous studies have evaluated the usability of HIVSTs with contrived results, this report is the first South African report on the clinical performance of multiple devices with real-time results interpretation. The results of this study add to the growing body of evidence that supports the use of HIVSTs as a user-friendly and accurate testing approach to reach populations that may not have access to traditional clinic-based testing. A 2018 systematic review assessed the reliability of HIVSTs from 20 reports across 16 studies conducted between 1995 and 2016. In this review, 16 (80%) had a specificity greater than 98%, and although sensitivity varied substantially, 18 (90%) of the reports had a sensitivity greater than 80%. 16 Furthermore, an Orasure study from Singapore in 2012 (n=994) achieved a similar sensitivity of 97.4% and a specificity of 99.9%. 17 Another recent study of INSTI in Kenya (n=354) also revealed comparable results to our study with a sensitivity of 98.99% and a specificity of 98.15%. 18 A total of 330 (94.29%) participants found the device was easy to use, and the 15.1% of participants who tested positive in this study was slightly higher than the national prevalence of 13.1%. 19 While corroborating previous results [15][16][17][18] , this South African study demon strates the sensitivity and specificity values of four HIVSTs to be higher than those attained during performance measurement for FDA approval 17 , with a substantial sample size as outlined in the requirement for WHO prequalification. The National Department of Health in South Africa requires that any HIVST it procures or that is used on their sites must be approved by the South African Health Products Regulatory Authority (SAHPRA) or be prequalified by the WHO.     The high sensitivity and specificity of each HIVST evaluated in this study suggests that they should all be considered for approval, as they also meet all of the other standards outlined by the WHO prequalification documents. Each batch of devices was manufactured under ISO 14385 standards required for the design and manufacture of medical devices and each HIVST included IFUs with minimal language and simple pictorial instructions. At the time of this publication submission, two of the four devices in this assessment, OraSure and INSTI, had been prequalified by the WHO using data generated in this study. 20 Subsequent to this study Chembio also received prequalification. Data from these studies have been separately shared with SAHPRA, the South African National Department of Health and the South African National Institute for Communicable Diseases in order to facilitate the approval and usage of the products in implementation programmes such as the Self-Test Africa (STAR) project.
Despite the high levels of sensitivity and specificity, there were a number of user errors (notably with Biosure and Chembio), highlighting areas for improvement. Refining and tailoring the IFU to target markets (an action consequently implemented by Biosure and Chembio) and simplifying the device design could increase the overall usability of the device, thus further minimising errors. Whilst errors are expected in the hands of untrained users, it is imperative that users are able to recognise that an error has been made, and that the test invalidates itself, i.e. no control line/dot appears when a critical error is made. Tests which do not have specimen control lines, and produce control lines in the absence of any human specimen, can prove to be detrimental to HIVST as it could lead to an increase in false negative results. In order to build from these results and create a more robust body of evidence, future testing should be conducted with, and opinions elicited from, more diverse groups that include wider demographics and participants who are recruited independently of a clinical setting.

Limitations
This study has several limitations. A selection bias may have been created with convenience sampling, and while the evaluation of the devices in series ensured no cross-contamination, the general population may have become more aware of HIVST by the time the last device was tested, due to limited but expanding media coverage. The readability and comprehension of test instructions (we used only English IFUs for this evaluation) may be context and population specific, which limits the generalisation of these findings. Furthermore, an observation bias may be present, as the study was conducted under observation in a clinical setting, instead of alone in their homes.
Similar to the limitations of the HSTAR001 usability study, there is no validated or standardised usability test for HIVSTs, so the productspecific semi-structured questionnaire from HSTAR001 was used to quantify usability. 15 No direct comparisons could be made because of the different device components and non-standardised IFUs across kits. The sensitivity and specificity of each test also do not allow for direct comparisons, as these results were independently benchmarked against a gold standard, and not each other.
A fifth HIVST, Atomo, withdrew from the study halfway through data collection, so these results were not included in the aggregated data, or explored in the discussion, but the manufacturer did independently receive WHO prequalification for the device after withdrawing from the study. 21

Conclusions
The four devices that were fully evaluated in this study and performed well, are among a growing number of HIVSTs intended to enter the South African market; OraSure, Chembio and INSTI have already received their WHO prequalification 22 and Biosure also received approval for use in South Africa. The results of this HSTAR003 performance evaluation methodology may also be used to guide similar evaluations among different populations. In the coming years, various HIVSTs will gain approval and enter the marketplace, which means that policies and distribution channels must be appropriately developed to accommodate this influx.