Is it possible to conduct clinical trials during a pandemic? The example of a trial of hydroxychloroquine

Introduction

Hydroxychloroquine (HCQ), a drug normally used to treat malaria and certain autoimmune disorders, was reported to have antiviral activity in vitro against coronaviruses – and specifically SARS-CoV-2 – by researchers of the Wuhan Institute of Virology, China, at the beginning of February, early on in the COVID-19 pandemic.¹ 
The proposed mechanism of action is that it blocks virus infection by increasing endosomal pH required for virus cell fusion and glycosylation of viral surface proteins and it also has immune-modulating activity, synergistically enhancing its antiviral effect in vivo.^1,2 Based on these observations, in the complete absence of drugs with proven efficacy against COVID-19, hydroxychloroquine has been used off label as potential anti-COVID-19 drug by many physicians worldwide.³ Attention to hydroxychloroquine intensified in mid-March 2020, when the results of a small study were published, showing that the combination of hydroxychloroquine and azithromycin had quickly reduced the amount of the virus in COVID-19 patients.⁴
Hydroxychloroquine definitely entered the public consciousness when US President Donald Trump called it a potential ‘game-changer’ in a press conference and strongly encouraged its use. From that moment, the debate began taking on political overtones.

The Food and Drug Administration (FDA) issued an emergency use authorization (EUA) on 28^th March to allow hydroxychloroquine use in hospitalized patients with COVID-19 when a clinical trial was not available. On 22^nd May a large retrospective analysis which reported an increased risk of serious heart disorders in patients treated with hydroxichloroquine was published on Lancet journal.⁵ The Lancet paper was then retracted on 13^th June in a major scandal over the authenticity of the unreleased patient database.⁶ However, in the meantime, FDA had revoked the EUA,⁷ the World Health Organization (WHO) halted its trial of hydroxychloroquine in hospitalized COVID-19 patients,8 and the European Medicine Agency (EMA) as well as national authorities, including the Italian Drug Agency (AIFA), stopped the off-label use of the drug recommending to use hydroxychloroquine only in properly designed clinical trials under close supervision.^9,10     
A lot of publications have been made so far on the use of hydroxychloroquine for treatment or prevention of COVID-19, but only few coming from high-ranked published studies^11-15 and there is still need for further evidence on benefits and risks of hydroxychloroquine in different settings.

Several randomized controlled studies for treatment and chemoprophylaxis with hydroxychloroquine have been designed (see in https://www.clinicaltrials.gov) and hopefully they will help to gain definitive results on efficacy and safety of this drug. However, in our experience, it has become extremely difficult to carry on a clinical trial of hydroxichloroquine. We wonder if it will still be possible to draw any conclusions on its efficacy and safety based on a “normal” scientific basis. In the following, we will argue that scientific assessment of empirical evidence cannot be undergone without considering a post-normal context.¹⁶

Objectives

The objective of the present study is to analyse the factors that had an impact on the conduction of a clinical trial of hydroxychloroquine.

Methods

The PROTECT Trial

The PROTECT trial is a controlled, open label, cluster-randomised, superiority trial with parallel group design aimed at assessing the efficacy of hydroxychloroquine in:

• prevention of COVID-19 or related symptoms in SARS-CoV-2-exposed subjects (household members and close contacts of COVID-19 patients);
• treatment of early phase asymptomatic or paucisymptomatic COVID-19 patients.

According to the study design, 2,000 SARS-CoV-2-exposed subjects (contacts) and 300 SARS-CoV-2 infected subjects (positive PCR test on a rhinopharyngeal or oropharyngeal swab for SARS-CoV-2), asymptomatic or paucisymptomatic not in treatment with any anti-COVID-19 medication (Index cases) will be randomized (2:1) either to receive Hydroxychloroquine or to Observation (the comparator in this trial is observation, since at the time of study start, neither treatment was provided to asymptomatic or paucisymptomatic subjects nor prophylaxis was available for contacts).
The primary endpoints are:

• for contacts, the proportion of subjects who become symptomatic and/or swab positive in each arm within 1 month from randomization;
• for index cases: the proportion of subjects who become swab negative in each arm within 14 days from randomization.

The study is conducted at Cancer Institute of Romagna (IRST) IRCCS (Emilia-Romagna Region, Northern Italy) in collaboration with the Departments of Public Health of Emilia-Romagna Region and of other North-Central Italy regions which agreed to collaborate.17 Potentially-eligible subjects are identified by Public Health Departments through contact tracing activities. Given the impossibility for subjects diagnosed positive to SARS-CoV-2 and for their close contacts to move from their home, most of the study procedures are carried out through phone calls and web-based applications. Subjects who agree to participate are randomized to Hydroxychloroquine or Observation. Hydroxychloroquine is shipped by the IRST pharmacy to subjects’ home within 24 hours from randomization. Telephonic interviews and electronic patient reported outcomes (ePRO) are then used to collect data in an electronic case report form (eCRF), during treatment and up to 6 months after study entry.
The PROTECT trial has received approval by AIFA and by the Ethic Committee of the National Institute for Infective diseases “L. Spallanzani” IRCCS (Rome) – which is the national ethic committee competent for all the trials on COVID-19 in Italy – in April and has started on 9^th May. 

Statistical analysis

Descriptive statistics on PROTECT trial during the first five months of the study are calculated. In detail, number and percentage of reported, eligible, enrolled cases and case-contacts by day for the period 14^th May-30^th September 2020. 
Time trends of proportion of acceptance to enter in the trial are reported, as scatter plot of daily proportions and as a predicted probability by a logistic regression model with restricted cubic splines on calendar time (4 equispaced knots) with 95% confidence intervals.¹⁸ The analysis was conducted on the positive cases and repeated on the case-contacts.
Other model specifications were tested by log-likelihood ratios between nested models – linear trends, segmented regressions, cubic splines with lesser number of knots.

The statistical strategy was to flexibly describe the temporal pattern of probability of acceptance by positive cases and case-contacts. After fitting a wiggly non-linear model – cubic spline with 4 knots – as reference, we checked if more parsimonious models still provide a good fit comparatively to the more complex reference model.¹⁹

Covariates

At ecological level, we recorded the time trend of the epidemics, seasonality, and media coverage of HCQ criticism as potentially explicative variables. We did not consider complex multiple regression analysis, because of the potential ecological fallacy, the multicollinearity among predictors, the relatively small study size. We consider a qualitative assessment of the role of different predictors a safer and wiser approach than a potentially misleading quantification relying on unverifiable model assumptions.²⁰

Press review

Italian newspapers (national and local) from 1^st March to 30^th September 2020 have been searched for the keywords ‘hydroxychloroquine’, ‘chloroquine’, ‘plaquenil’. 
The search was carried out by querying the database of the daily press review. The service was provided by an external agency: Telpress Italia Spa. The database contains articles from the main and local (Romagna area) Italian non-scientific newspapers in print and on-line.

Results

The PROTECT trial was designed between March and April 2020, when the COVID-19 pandemic was at its peak, Italy was in total lockdown, and there were no approved drugs for cure or prevention of COVID-19. At that time, AIFA allowed off-label use of HCQ for treating COVID-19 patients.  
In planning the trial, our assumptions regarding enrolment in the study were:

• an incidence of at least 250 new cases/week in the regions of interest during the phase two of the pandemic (post-lockdown);
• for each case, one or two close contacts being identifiable;
• 70%-80% of the subject being properly traced and notified by each public health department to the coordinating centre;
• 70%-80% of the subject being eligible (asymptomatic/paucisymptomatic) and having no contraindications to assume HCQ;
• 80% of the subject agreeing to participate.

Based on these assumptions, we expected to achieve the target enrolment size in about 3 months from study starting.  
Over more than 4 months of active enrolment (from 9^th May to 30^th September), 1,020 subjects have been reported to the trial staff by the health departments, of which 633 contacts and 387 COVID-19 positive cases (the proportion of cases and contacts was in line with our assumptions, but the absolute number was half of the number assumed of 2,300 subjects, due to the evolution of the COVID-19 epidemics); 706 adult subjects were potentially eligible (706/1,020; 69%, in line with our assumptions).

After being contacted, 579 (579/706; 82%) subjects did not enter the study, due to lack of compliance with inclusion/exclusion criteria or explicit refuse at telephonic interview, and only 127 (18%) were enrolled, of which 59 positive cases and 68 contacts (in strong contrast with our assumptions) (table 1). 
The percentage of acceptance of positive cases was higher (22%; 59 out of 274 subjects) than that for contacts (16%; 68 out of 432 subjects).
The time trend of probability of acceptance was markedly different. The observed percentage of acceptance for positive cases was well above 20% up to August 2020, then declined. Among contacts, we observed a continuous decline from June 2020 onward.

The regional epidemic curve (figure 1) between 25^th February and 30^th September 2020 shows a bimodal pattern, with a number of cases in September equal to the number registered in early May and a minimum in mid-June.
The season was relatively warm (average temperature in Emilia-Romagna Region: May 22°C, June 27°C, July 30°C, August 29°C, September 25°C). The pattern was opposite in shape to the epidemic curve.
The debate over efficacy and safety of HCQ was very present in the media, with a peak in the month of June. A synthesis of the events regarding HCQ is reported in table 2. The lay press – national and regional, i.e., of Emilia-Romagna Region – published the first news against HCQ on the last few days of May and for the end of June most media declared that evidence of HCQ ineffectiveness was reported in the scientific literature.

At the end of September 2020, leading national newspaper²¹ summarized the changed status of knowledge from May (HCQ recommended) to September (HCQ abandoned) (figure 2).

Figure 3 reports the frequency of articles in the national and regional media on HCQ in the period March-September 2020. Based on the frequency of articles, it is clear that after the end of May, following the first criticisms on the use of HCQ, the media decided that HCQ was no more an attracting news for the public. Based on a simple qualitative content analysis, articles reporting criticism to the use of HCQ were published only during the month following the publication of the controversial studies based on the Surgisphere data (approximately, 1^st June-1^st July 2020, red lines in figure 3). For those interested, see comments on retraction.^22-24

Fitting flexible non-linear models on daily data provided a more detailed picture, as shown in figure 4 (positive cases) and in figure 5 (case-contacts). Acceptance proportions peaked on mid-June 2020 and then started to decline. For contacts, we observed almost a linear decreasing trend.

The trend was clearly not linear for positive cases: likelihood ratio (LR) test for linearity: chi2 6.86 (2 df); p=0.032. For case-contacts there was no deviation from linearity, LR chi² was 0.40 (2 df); p=0.819.
For positive cases, there was a decrease in the probability of acceptance by the end of the period of coverage by the media on the criticism to the use of HCQ. For case-contacts, the probability of acceptance shows a decrease proportional to the decreasing frequency of articles on HCQ by the media.

Discussion

The PROTECT trial was designed following established rules and complying the new requirements regarding COVID-19 containment.

In all phases of the trial, urgency was a driving force. Great efforts were undertaken to set up a properly designed randomized trial in a very short time and with the many critical aspects to afford in the ‘COVID-19 era’ (how to contact and visit subjects, how to obtain informed consent, how to provide the study drug, how to monitor the safety, how to collect data). The set-up benefited from a positive exchange with AIFA and according to the Article 17 of the Law Decree No. 18 of 17^th March 2020, clinical trial protocols are preliminarily evaluated by the AIFA Technical Scientific Committee (CTS) and subsequently approved, after evaluation by the AIFA Competent Authority (Clinical Trial Office) and by the Ethics Committee of the National Institute for Infectious Diseases 'Lazzaro Spallanzani', as single national Ethics Committee.

We do the best as possible, but the trial was run in a difficult and rapidly-changing context. One could argue that the main reason for low accrual could be the drop down of infected people that has been observed after the lockdown period, probably also thanks to the season. However, in October 2020, the second wave of SARS-CoV-2 virus spread has occurred in Italy, but no changes in the accrual has been observed.

Indeed, in the time span covered in the present paper, we faced a series of changing scenarios. We ran the trial in a period of rapidly evolving epidemiological context. The number of subjects diagnosed with COVID-19 in Emilia-Romagna Region dropped down from more than 500/day of April, when the pandemic was at its peak, to less than 50 between end of May and mid of August, when the enrolment window was supposed to be open. Also the number of admission to intensive care units and the number of deaths related to COVID-19 decreased accordingly, leading to a more relaxed feeling and behaviour in the population with regard to the virus.

The national and regional health service underwent to a profound change in term of logistic and organization. Due to the COVID-19 containment measures patient-physician relationship was impaired and appeared to be limited to phone contact. As a consequence, almost all aspects of the on-going trial conduct were reorganized. Potentially eligible subjects were contacted only over the phone by the personnel of a cancer institute, not by their own general practitioner; the drug was shipped at home by a courier. This may have generated some concern and a feeling of uncertainty in the potentially eligible subjects. 
The pandemics dictated also several changes of Regulatory Authorities’ guidelines. The AIFA, as well as other European Agencies, firstly allowed the off-label use of HCQ for COVID-19, then suddenly retired it and recently revised its use again. Although AIFA recommended the use within clinical trials under close supervision, this contributed to a general distrust to the drug and, by consequence, on the clinical investigations aimed to test HCQ.

Last, but not least, the trial suffered by a changing attitude of the media regarding HCQ. The prevailing mood in the lay press passed from a positive feeling where HCQ was presented as a recommended drug to a negative feeling where HCQ was presented as a drug to be abandoned due to high toxicity and poor efficacy.
Up to now, the PROTECT trial lesson must be framed in a broader context. the characteristics of the COVID-19 pandemic, its rapid spreading, the collapse of the health system, the profound uncertainties on the biology of the disease and the therapeutics, the urgency to set up a reaction to pandemic on one side, and on the other side the characteristics of modern societies – the so-called knowledge-based societies – which lead to a massive public discussion of scientific arguments. Scientific results, usually published in scientific journals, appeared as pre-print in open repositories. Lack of peer review and public discussion in lay media are some of the new features of the current situation. Policies regarding the pandemic mitigation or contrast were strongly influenced by this kind of scientific literature – there are many examples, among others see Onge et al.²⁵
The difficulties to scrutinize the quality of scientific results was not limited to those published in repositories not subjected to peer-review, but in a context of urgency affected also scientific peer-reviewed journals (see, for example, the Lancet retraction of the paper on HCQ).⁷

The traditional conduct of the scientific research, as exemplified by a randomized clinical trial, controlling for external sources of variation, was going to be impossible to do. The consequence is that new questions arise. The epidemic changed the patient-physician relationship, which was mainly cast on chronic diseases or long lasting infectious diseases; e.g., tuberculosis. The question then is: How to maintain trust in face of urgency and uncertainty?
The epidemics evolves rapidly on time and the number of eligible patients changes rapidly following the epidemic waves. The trial logistic is usually planned to enrol on a pretty uniform time schedule. Then, how to maintain quality and comparability when the enrolment rate has marked variation on a relatively short time span?
The urgency dictated by the epidemic and the high stakes involved lead to a strong politicization of the debate on mitigation or therapeutics policies – a debate mostly outside the traditional scientific academia. How to maintain the agnostic position of the researcher in conducting a trial?

The criteria to evaluate the scientific evidence in the authorization procedures, adopted by the Central Agencies, were relaxed due to the urgency created by the pandemic and to the complete lack of knowledge of the epidemiology of a novel emerging pathogen.²⁶ Sometimes, this urgency conflicted with the precautionary principle, too. This situation speeded up the process opened by the mounting wave of real-world evidence. Studies that in pre-COVID-19 era could have been considered as ‘proof-of-principle’ and motivate the conduct of a randomized trial, become currently the only usable empirical evidence in an authorization procedure. The time required by a randomized trial to provide an answer is strongly inadequate in face of a pandemic. Under normal circumstances, trial implementation can take years, whereas the pandemics placed pressure on studies able to quickly generate evidence. The academia is divided on the criteria to appraise the empirical evidence from not randomized trials.^26,27 

The above-mentioned state of things has profound consequence on the subjects who were proposed to enter a randomized study. They cannot rely on the traditional patient-physician relationship to establish a trusted relationship within the trial. They were exposed to the open and evolving debate on the treatment under evaluation. These difficulties are evident in the conduction of the PROTECT trial. The adhesion rate of positive SARS-CoV-2 cases, which was high at the beginning of the trial due to a positive expectation of care, dropped in the months of the highly-politicized debate on the treatment. Contacts of a positive case, lacking of an explicit motivation to adhere, tended to decline acceptance to enter the trial very quickly, following the debate on the lay media.

In both of them, there is a crisis of trust. This is the most striking consequence of the current situation. The lesson to be learned is to rely on an ‘extended peer community’^16,28,29 of which the eligible subjects and the broader public are essential components. It is not possible to conduct a randomized trial without planning the construction of an extended community, face the determinants of adherence, openly discuss the scientific rationale, define the outcome with the participants, identify with the participants and the broader public the implications of the trial results in advance – by presenting before starting the study all the possible study outcomes. (for an example in environmental epidemiology see De Marchi et al.).³⁰
The challenge is: Is it possible to restore trust? Is it possible to obtain trusted results? Is it possible to do these on time?
We do not know how; we believe the traditional approaches do not fit to these tasks. The urgency is the more demanding requirement. Probably, we must plan participatory studies with interim analysis or design a sort of mixed study in which observational approaches complement the more stringent experimental conditions.
Indeed, as reported on 25^th March 2020 by Waltner-Toews et al:³¹ “This pandemic offers society an occasion to open a fresh discussion on whether we now need to learn how to do science in a different way”.

Conclusions

In conclusion, at least in the current pandemic situation, it is not possible to rely on ‘normal’ applied science, i.e., classic randomized trials. At the end, randomized trials would not be able to enrol the expected number of subjects, would probably be emended or suspended, would be ended prematurely. In all these cases, they would not be able to provide the expected strong empirical evidence.
The extended peer community should be set up since the design phase. Study protocols should include specific investigations on motivations to agree to be enrolled and on determinants of study compliance. Study protocols should implement innovative approaches of participants’ engagement – better if in a two-stage design. In the first stage, the study protocol should describe how to set up a participatory research group for designing the trial and developing subject-/population-specific engagement approaches. In the second stage, the eligible subjects are engaged for the trial according to the approaches agreed in the first stage. This is similar to well-known strategies in tools development in industrial settings – the so-called user-centred design. See Walden et al.³² for example in a clinical context.
The trial should be always sequential or adaptive; in other words, the possibility of an early stop should be planned in all trials.
Lastly, hybrid designs should be encouraged. It should be remembered in choice of the study design that in some context the unique feasible design could be an observational study. Careful control of sources of confounding should be planned a priori whenever possible and instrumental variable approaches should be considered.

Conflicts of interest: none declared. Authors are involved in several aspects of the PROTECT Trial.

Funding disclosure: IRST IRCCS is funding this trial and will have full oversight of the design of the study, collection, analysis, and interpretation of data and in writing the manuscript.

References

1. Wang M, Cao R, Zhang L, et al. Remdesivir and chloroquine effectively inhibit the recently emerged novel coronavirus (2019-nCoV) in vitro. Cell Res 2020;30(3):269-71.
2. Liu J, Cao R, Xu M, et al. Hydroxychloroquine, a less toxic derivative of chloroquine, is effective in inhibiting SARS-CoV-2 infection in vitro. Cell Discov 2020;6:16.
3. Gautret P, Lagier JC, Parola P, et al. Hydroxychloroquine and azithromycin as a treatment of COVID-19: results of an open-label non-randomized clinical trial. Int J Antimicrob Agents 2020;56(1):105949.
4. Sanders JM, Monogue ML, Jodlowski TZ, Cutrell JB. Pharmacologic treatments for Coronavirus disease 2019 (COVID-19): a review. JAMA 2020;323(18):1824-36.
5. Mehra MR, Desai SS, Ruschitzka F, Patel AN. RETRACTED: Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet 2020;S0140-6736(20)31180-6.
6. Mehra MR, Ruschitzka F, Patel AN. Retraction – Hydroxychloroquine or chloroquine with or without a macrolide for treatment of COVID-19: a multinational registry analysis. Lancet 2020;395(10240):1820.
7. Food and Drug Administration. Coronavirus (COVID-19) Update: FDA Revokes Emergency Use Authorization for Chloroquine and Hydroxychloroquine. FDA 15.06.2020. Available from: https://www.fda.gov/news-events/press-announcements/coronavirus-covid-19-update-fda-revokes-emergency-use-authorization-chloroquine-and
8. World Health Organization. Coronavirus disease (COVID-19): Hydroxychloroquine. WHO 19.06.2020. Available from: https://www.who.int/news-room/q-a-detail/coronavirus-disease-covid-19-hydroxychloroquine
9. European Medicines Agency. COVID-19: reminder of the risks of chloroquine and hydroxychloroquine. EMA 29.05.2020. Available from: https://www.ema.europa.eu/en/news/covid-19-reminder-risks-chloroquine-hydroxychloroquine
10. Agenzia Italiana del Farmaco. AIFA sospende l’autorizzazione all’utilizzo di idrossiclorochina per il trattamento del COVID-19 al di fuori degli studi clinici. AIFA 26.05.2020. Available from: https://www.aifa.gov.it/-/aifa-sospende-l-autorizzazione-all-utilizzo-di-idrossiclorochina-per-il-trattamento-del-covid-19-al-di-fuori-degli-studi-clinici
11. Randomised Evaluation of COVID-19 Therapy. No clinical benefit from use of hydroxychloroquine in hospitalized patients with COVID-19. RECOVERY 05.06.2020. Available from: https://www.recoverytrial.net/news/statement-from-the-chief-investigators-of-the-randomised-evaluation-of-covid-19-therapy-recovery-trial-on-hydroxychloroquine-5-june-2020-no-clinical-benefit-from-use-of-hydroxychloroquine-in-hospitalised-patients-with-covid-19
12. Boulware DR, Pullen MF, Bangdiwala AS, et al. A Randomized Trial of Hydroxychloroquine as Postexposure Prophylaxis for Covid-19. N Engl J Med 2020;383(6):517-25.
13. Mitjà O, Corbacho-Monné M, Ubals M, et al. Hydroxychloroquine for Early Treatment of Adults with Mild Covid-19: A Randomized-Controlled Trial. Clin Infect Dis 2020:ciaa1009.
14. Cavalcanti AB, Zampieri FG, Rosa RG, et al. Hydroxychloroquine with or without Azithromycin in Mild-to-Moderate Covid-19. N Engl J Med 2020;383(21):2041-52.
15. Abella BS, Jolkovsky EL, Biney BT, et al. Efficacy and Safety of Hydroxychloroquine vs Placebo for Pre-exposure SARS-CoV-2 Prophylaxis Among Health Care Workers: A Randomized Clinical Trial. JAMA Intern Med 2021;181(2):195-202.
16. Funtowicz SO, Ravetz JR. Science for the post-normal age. Futures 1993;25(7):739-55.
17. Nanni O, Viale P, Vertogen B, et al. PROTECT Trial: A cluster-randomized study with hydroxychloroquine versus observational support for prevention or early-phase treatment of Coronavirus disease (COVID-19): A structured summary of a study protocol for a randomized controlled trial. Trials 2020;21(1):689.
18. Harrell FE JR. Binary Logistic Regression. In: Harrell FE JR. Regression Modeling Strategies With Applications to Linear Models, Logistic and Ordinal Regression, and Survival Analysis. Cham, Springer, 2015.
19. Witte JS, Greenland S. A nested approach to evaluating dose-response and trend. Ann Epidemiol 1997;7(3):188-93.
20. Saltelli A, Funtowicz SO. When All Models Are Wrong. Issues in Science and Technology 2013;30(2):74-85.
21. Dusi E. I farmaci bocciati e quelli promossi. Le armi anti virus. La Repubblica, 29.09.2020.
22. Offord C. WHO Halts Hydroxychloroquine Testing Over Safety Concerns. The Scientist 27.05.2020. Available from: https://www.the-scientist.com/news-opinion/who-halts-hydroxychloroquine-testing-after-safety-warning-67573
23. Lancet, NEJM retract controversial COVID-19 studies based on Surgisphere data. Retraction Watch 04.06.2020. Available from: https://retractionwatch.com/2020/06/04/lancet-retracts-controversial-hydroxychloroquine-study/
24. A month after Surgisphere paper retraction, Lancet retracts, replaces hydroxychloroquine editorial. Rectration Watch 10.07.2020. Available from: https://retractionwatch.com/2020/07/10/a-month-after-surgisphere-paper-retraction-lancet-retracts-replaces-hydroxychloroquine-editorial/
25. Onge PS, Campan G. The Flawed COVID-19 Model That Locked Down Canada. MEI 04.06.2020. Available from: https://www.iedm.org/the-flawed-covid-19-model-that-locked-down-canada
26. Oldenburg CE, Doan T. Rigorous Randomized Controlled Trial Implementation in the Era of COVID-19. Am J Trop Med Hyg 2020;102(6):1154-55.
27. Adam D. Who Believes in Hydroxychloroquine? Issues in Science and Technology 16.09.2020. Available from: https://issues.org/hydroxychloroquine-hcq-controversy/
28. Funtowicz SO, Ravetz JR. Peer Review and Quality Control. International Encyclopedia of the Social & Behavioral Sciences 2015. 
29. Wynne B. Misunderstood misunderstanding: social identities and public uptake of science. Public Understanding of Science 1992;1:281-304.
30. De Marchi B, Biggeri A, Cervino M, et al. A Participatory Project In Environmental Epidemiology: Lessons From The Manfredonia Case Study (Italy 2015-2016). Public Health Panorama 2017;3(2):321-27.
31. Waltner-Toews D, Biggeri A, De Marchi B, et al. Post-Normal Pandemics: why CoViD-19 requires a new approach to science. Recenti Prog Med 2020;111(4):202-04.
32. Walden A, Garvin L, Smerek M, Johnson C. User-centered design principles in the development of clinical research tools. Clin Trials 2020;17(6):703-11.