Int J Pharm Pharm Sci, Vol 13, Issue 2, 50-55Original Article

DEVELOPMENT OF QUALITY INDICATORS FOR PHARMACEUTICAL CARE PERFORMANCE ASSESSMENT IN NIGERIAN COMMUNITY PHARMACIES: A DELPHI PANEL

OKPALANMA NNEOMA*, EKWUNIFE OBINNA

Department of Clinical Pharmacy and Pharmacy Management, Faculty of Pharmaceutical Sciences, Nnamdi Azikiwe University, Awka, Nigeria
*Email: nneomaokoli@yahoo.co.uk

Received: 22 Jul 2020, Revised and Accepted: 14 Dec 2020

ABSTRACT

Objective: This study aimed to develop quality indicators for assessing pharmaceutical care performance in the Nigerian community pharmacies.

Methods: We searched for existing quality indicators through an extensive literature review. The identified quality indicators consisted of thirty-four items in 10 core components. The Delphi method was used to arrive at a consensus on quality indicators for assessing pharmaceutical care performance in the Nigerian community pharmacies by surveying a panel of experts. There were 3 rounds of the Delphi panel conducted by consulting a panel of 10 experts in pharmaceutical care practice. A mean score>3.5, median>3.5, an absolute value between the median and mode<1.00 was used to establish consensus on the quality indicators for assessing pharmaceutical care performance in the Nigerian community pharmacies.

Results: Ten studies met the inclusion criteria. A list of 33 quality indicators comprising of 10 core components emerged from the extensive literature review. They included quality management, documentation of care, communication with the patient, management of clinical risk in pharmacy, compounding, dispensing and patient care, follow-up of pharmacotherapy counselling, drug inventory and stocking, training of pharmaceutical staff. At the end of the round three votings, 24 statements of the quality indicator reached consensus in nine core components: quality management (5), continuity of care (1), communication with patients (3), clinical risk management (5), dispensing (1), follow-up of pharmacotherapy (3), counselling (1), logistics (3), training of pharmacy staff (2).

Conclusion: This study developed a set of 24 quality indicators for assessing pharmaceutical care performance in community pharmacies in Nigeria.

Keywords: Delphi panel, Quality indicator, Pharmaceutical care, Community pharmacy

© 2021 The Authors. Published by Innovare Academic Sciences Pvt Ltd. This is an open access article under the CC BY license (https://creativecommons.org/licenses/by/4.0/)
DOI: https://dx.doi.org/10.22159/ijpps.2021v13i2.39179. Journal homepage: https://innovareacademics.in/journals/index.php/ijpps.

INTRODUCTION

Pharmaceutical care is a philosophy of practice for pharmacists. The concept tasks pharmacists to assume responsibility with other health professionals to ensure patients positive health outcomes. Pharmaceutical care practice leads the pharmacist to access the therapy received by patients for appropriateness, identify drug-related problems and resolve the identified drug-related problem. Besides medication supervision, pharmaceutical care practice encourages pharmacists to undertake patients counselling [1, 2]. 

Several barriers to pharmaceutical care practice have been documented in the literature [3, 4]. Lack of standards for pharmacists to apply in their daily practice is the main obstacle to pharmaceutical care implementation [3]. Thus, it is important to establish practice standards to aid pharmaceutical care practice. Such practice standards should be environment-specific while following the general norms of pharmaceutical care practice. This is important given the peculiarities of community pharmacy practice in Nigeria. One of such being that community pharmacy co-exists with drug vendors who do not have formal pharmacy training and are authorized to sell only over the counter drugs.

The Delphi method was used to achieve consensus on quality indicators for assessing pharmaceutical care performance in the Nigerian community pharmacies by surveying a panel of experts. RAND Corporation developed Delphi technique in Santa Monica, California, in the 1950s during the Cold War when the US devised a Delphi project to forecast the impact of technology on military capabilities’ development [5]. Delphi technique is a research design that uses a series of questionnaires to gain consensus involving multiple iterations to get information and results from panel members between each round [5, 6]. To regard a procedure as Delphi, four features are necessary. These features include iteration, controlled feedback, anonymity and statistical analysis of responses [5, 7]. 

Quality indicators rate the quality of care provided by healthcare professionals. They deal with the characteristics of service providers, steps involved in providing care, and the effect of care on the health status [8]. Quality indicators are used to enhance quality improvement activities by either making comparison between institutions over time or by supporting consumers to choose their health care providers [9]. Absence of quality indicators in health care services is disastrous and highly unprofessional as they are needed to assess and improve the quality of medical practice [10].

To the best of our knowledge, there is no set of quality indicators available to assess pharmaceutical care performance in community pharmacies in Nigeria. Therefore, this study used a Delphi panel to develop Nigerian specific quality indicators for assessing pharmaceutical care performance in community pharmacies.

MATERIALS AND METHODS

Methods

We searched for existing quality indicators developed for assessing pharmaceutical care performance in community pharmacies through an extensive literature review. Studies that were selected for inclusion included: (1) studies that developed quality indicators for pharmaceutical care practice; (2) studies that reported the impact of the quality indicators on community pharmacy practice; and (3) studies published in the English language. We excluded studies that reported only abstracts or literature review of the research work and studies that assessed only one aspect of pharmaceutical care practice.

After reviewing the selected studies, we selected 33 quality indicators comprising ten components [11-19]. The ten components were quality management, documentation of care, communication with the patient, management of clinical risk in pharmacy, compounding, dispensing and patient care, and follow-up of pharmacotherapy, counselling, drug inventory and stocking, training of pharmacy staff.

Selection and description of participants

The researchers identified the criteria required in choosing the panel of experts before the Delphi panel. Pharmacists with knowledge and experience of pharmaceutical care practice in a community pharmacy were selected as experts. Some administrators of pharmacy practice were also included as members of the panel of experts. Specifically, the Anambra State Pharmacist Council of Nigeria (PCN) officer, the Director of Pharmaceutical Services (DPS), the Deputy Director of Pharmaceutical Services (DDPS) and community pharmacists who were members of the pharmacy inspection and regulation team represented the panel of experts. In total, ten panels of experts participated in this study. A study stated that ten to fifteen panels of experts are needed to yield satisfactory results for a study using the Delphi method [20]. Another study suggested that 5 to 10 experts are adequate for content validation [21].

Technical information

There were three rounds of the Delphi panel for this study. In the first round, the participants received a questionnaire containing the list of indicators with clear and specific instruction for each participant. Table 1 shows the list of indicators distributed in the first round of the Delphi panel. The participants voted by either ticking agree or disagree beside each statement.

The experts were also given the opportunity to offer comments and suggested more items necessary in developing the questionnaire. The response frequencies were anonymously calculated. A statement needs 80% of the panel of experts to agree to it (that is 8 out of 10 experts) to be accepted at each round. This was based on a study which suggested that at least 80% of experts (in a case where 10 experts participated in a consensus study) must agree on an item to meet consensus [21].

According to the feedback received from the panelists, the statements that did not meet the 80% consensus got rephrased and the instrument summarized for the second round of the Delphi panel. In round 2 of the Delphi panel, the 10 panels of experts received the summarized questionnaire from round 1. The experts used the same voting methods described in round 1 but with the knowledge of the expert’s earlier scores and comments. The participants could change their minds on their votes while reflecting on the group’s result and yet their responses were recorded anonymously. Analysis of the responses was as in round 1 of the Delphi panel. The researcher deleted all statements that did not meet the experts’ agreement from the questionnaire. Some statements got adjusted based on the feedback from the experts and the instrument summarized for the third round. In the third and last round of the Delphi panel, the participants received the questionnaire in a Likert scale format with instruction to vote the degree to which they agree with the indicators using a 5-point scale where 1 meant the idea was highly irrelevant and 5 meant the idea was highly relevant. The goal of the third round was to gain consensus. At the end of the third round, the Delphi procedure was considered completed [22]. 

Ethics

The Ethics Committee of Chukwuemeka Odumegwu Ojukwu University Teaching Hospital Awka, Anambra State, Nigeria, approved the study (COOUTH/CMAC/ETH. C/VOl.1/0068). Participants gave their informed consent before the commencement of the study.

Statistical analysis

In the third round of the Delphi panel, indicators considered valid were those that met the following criteria: (1) 80% of the participants rated three or higher on a 5-point scale; (2) the mean score on the Likert scale was>3.5; (3) the median on the Likert scale was>3.5; and (4) the absolute value of the difference between the median and the mode was<1.00 [5, 6, 7, 23].

RESULTS

Ten studies met the inclusion criteria and were used to generate a set of potential quality indicators [11-19]. A list of 33 quality indicators comprising ten components emerged from the summary findings. The ten components were: quality management, documentation of care, communication with the patient, management of clinical risk in pharmacy, compounding, dispensing and patient care, follow-up of pharmacotherapy counselling, drug inventory and stocking, training of pharmacy staff At the end of round 1 voting and after summarizing the comments made by the experts, a consensus was reached for twenty-five out of the thirty-three quality indicators (i.e. ≥ 8 panel members voted “agree” for each of the 25 statements). These 25 quality indicators were from nine components. These quality indicators got approved for the next round. Consensus was not reached for eight of the quality indicators after round 1 (i.e. ≥ 8 panel members voted “disagree” on the statement). The eight quality indicators included: documented errors that did reach the patients; documented errors reported to the physician or primary care provider; patients offered instructions on inhalation medication and use with first dispensing; availability of a standard operating procedure for the release of compounded medication before dispensing of the drug to the patient; medication compounded for each patient that followed standardized procedure; dosage of the compounded medication for children (0 to 6 y) cross-checked by a pharmacist; expired medication dispensed to the patient and thus discovered and reported by the patient (each month) and participation in a national program for the patient-reported side effect of drugs in the National Pharmacovigilance Center.

In round 2, the panel of experts received the 25 statements of the quality indicator that reached consensus together with the eight statements that did not reach consensus in round 1 with the accompanying comments. After round 2 voting, the experts reached consensus on 24 statements (i.e. ≥ 8-panel members voted “agree” on the 24 statements). The panel also reached consensus to omit nine statements from the questionnaire (i.e. ≥ 8-panel members voted “disagree” on the nine statements). Consensus was not reached for a total of nine statements after round 2. Table 2 gave details of the second round of the Delphi panel.

The aim of round 3 was to seek clarification for statements that reached consensus and generate more statements that will help in developing the quality indicators. In round three, a consensus was reached for the 24 statements of the indicators after voting (i.e. ≥8 of the participants’ votes rated 3 or higher on a 5-point scale; the mean score on the Likert scale was>3.5; the median on the Likert scale was>3.5; and the absolute value of the difference between the median and the mode was<1.00).

The final list of quality indicators consisted of 24 statements distributed in nine components. Five statements related to quality management, one statement related to continuity of care, three statements related to communication with the patient, five statements related to clinical risk management, one statement related to dispensing, three statements related to follow-up of pharmacotherapy, one statement related to counselling, three statements related to logistics, and two statements related to training of pharmacy staff. Table 3 shows the result of the third round of the Delphi panel.

Table 1: Quality indicator obtained after the first round of delphi technique

Quality management

  1. The number of years of practice as a pharmacist.

  2. The number of patient-reported adverse effects that were reported to the national pharmacovigilance center.

Documentation of care

  1. Pharmacist ability to obtain information on patients’ actual use of the drug before dispensing and to document all the necessary information obtained in the patients’ record.

  2. A standard means of documenting medication errors (e. g. wrong dosage, wrong substance, wrong compounding) that occurred during the work process in the pharmacy and was realized before the drug reached the patient.

  3. The number of medication errors discovered during dispensing of medications that were effectively documented.

  4. Complaints made by patients that were documented by the pharmacist.

  5. The number of documented errors that reached the patient.

  6. Formal documentation of allergic reactions.

Communication with patient

  1. The number of patients who were given instructions on the use of inhalation medication with first dispensing.

  2. An educational program and plan are mapped out for each patient, especially patients with chronic diseases.

  3. Drug information adequately provided by the pharmacist to patients chronically taking 3 classes of drugs concomitantly.

  4. The number of patients adequately educated on drug interactions (drug-drug, drug-food interactions).

Management of clinical risk in pharmacy

  1. Stipulated guidelines were implemented in managing clinical risks in the pharmacy.

  2. There is formal documentation of all action taken when a drug interaction occurs.

  3. A protocol is available for informing patients on contraindication of drugs especially new patients.

  4. Management and formal documentation of patients drug allergic reactions, most especially new patients.

  5. Referral of patients to other healthcare providers and effective documentation.

Compounding

  1. Compounded medications were produced after undergoing standard operating procedure before being dispensed to patients.

  2. The number of drugs compounded for patients that followed standard procedure.

  3. Dosage of compounded mediation for children (0 to 6 y) is checked by the pharmacist.

Dispensing and patient care

  1. The number of dispensed medication with the patient given information on the drug.

  2. Dispensed medication cross-checked by the pharmacist (percentage).

Follow up of pharmacotherapy

  1. Percentage of patients using NSAIDs, especially elderly patients (>70 y), also received gastroprotection.

  2. The pharmacist ensures that gastroprotection was added to drugs of patients on NSAIDs in 80% of cases, especially patients>70 y where gastroprotection was lacking.

  3. The number of patients followed up on pharmacotherapy and documented.

Counselling

  1. A professional guideline was employed while counseling patients on medication.

Drug inventory and stocking

  1. A professional guideline was employed in assessing suppliers of drugs.

  2. There is a standard system for checking expired drugs.

  3. The number of drugs that were discovered to have expired internally before being dispensed each month.

Training of pharmaceutical staff

  1. Participation in mandatory professional development courses, seminars and continuing education for pharmacists.

  2. Percentage of pharmaceutical staff with adequate training in pharmacy operations.

  3. The side effect of drugs reported by patients was entered into a national program in the national pharmacovigilance centre.

Table 2: Quality indicator obtained after the second round of Delphi technique

Quality management

  1. The number of years of practice as a pharmacist.

  2. The number of patient-reported adverse effects that were reported to the national pharmacovigilance center.

Documentation of care

  1. Pharmacist ability to obtain information on patients’ actual use of a drug before dispensing and to document all the necessary information obtained in the patients’ record.

  2. A standard means of documenting medication errors (e. g. wrong dosage, wrong substance, wrong compounding) that occurred during the work process in the pharmacy and was realized before the drug reached the patient.

  3. The number of medication errors discovered during dispensing of medications that were effectively documented.

  4. Complaints made by patients that were documented by a pharmacist.

  5. The number of documented errors that reached the patient.

  6. Formal documentation of allergic reactions.

Communication with patient

  1. The number of patients who were given instructions on the use of inhalation medication with first dispensing.

  2. An educational program and plan are mapped out for each patient, especially patients with chronic diseases.

  3. Drug information adequately provided by the pharmacist to patients chronically taking 3 classes of drugs concomitantly.

  4. The number of patients adequately educated on drug interactions (drug-drug, drug-food interactions).

Management of clinical risk in pharmacy

  1. Stipulated guidelines were implemented in managing clinical risks in the pharmacy.

  2. There is formal documentation of all action taken when a drug interaction occurs.

  3. A protocol is available for informing patients on contraindication of drugs especially new patients.

  4. Management and formal documentation of patients drug allergic reactions, most especially new patients.

  5. Referral of patients to other healthcare providers and effective documentation.

Compounding

  1. Compounded medications were produced after undergoing standard operating procedure before being dispensed to patients.

  2. The number of drugs compounded for patients that followed standard procedure.

  3. Dosage of compounded mediation for children (0 to 6 y) is checked by the pharmacist.

Dispensing and patient care

  1. Number of dispensed medication with the patient given information on the drug

  2. Dispensed medication cross-checked by the pharmacist (percentage)

Follow up of pharmacotherapy

  1. Percentage of patients using NSAIDs, especially elderly patients (>70 y) also received gastroprotection.

  2. The pharmacist ensures that gastroprotection was added to drugs of patients on NSAIDs in 80% of cases, especially patients>70 y where gastroprotection was lacking.

  3. The number of patients followed up on pharmacotherapy and documented.

Counselling

  1. A professional guideline was employed while counselling patients on medication.

Drug inventory and stocking

  1. A professional guideline was employed in assessing suppliers of drugs.

  2. There is a standard system for checking expired drugs.

  3. The number of drugs that were discovered to have expired internally before being dispensed each month.

Training of pharmaceutical staff

  1. Participation in mandatory professional development courses, seminars and continuing education for pharmacists.

  2. Percentage of pharmaceutical staff with adequate training in pharmacy operations.

  3. The side effect of drugs reported by patients was entered into a national program in the national pharmacovigilance centre.

Table 3: Result of rating indicator in the third round

Components and indicators Consensus
Mean Median (Me-Mo) Status
Quality management
Number of years of practice as a community pharmacist 3.7 4 0.0 valid
Number of patient-reported to have adverse effects and was reported to the national pharmacovigilance center. 4.3 4.5 0.5 valid
Documentation of care
The ability of the pharmacist to obtain information on patients’ actual drug use before dispensing and to document all information in the patients’ record. 4.7 5 0.0 valid.
A standard means of documenting medication errors (e. g. wrong dosage, substance, wrong compounding) that occurred during the work process in the pharmacy and was realized before the drug reached the patient. 4.4 5 0.0 valid
Number of medication errors discovered during dispensing of medications that were effectively documented. 4.3 4.5 0.5 valid
Number of documented complaints made by patients and documentation of a patient allergic reaction. 4.2 4 0.0 valid
Communication with patient
An educational program and plan is mapped out for each patient, especially patients with chronic disease. 4.1 4.5 0.5 valid
Drug information is adequately provided to patients currently in chronic consumption of 3 different classes of drugs. 4.4 4 0.0 valid
Number of patients adequately educated on drug interactions (drug-drug, drug-food interactions). 4.3 4.5 0.5 valid
Management of clinical risk in pharmacy
Stipulated guidelines were employed in clinical risks management in pharmacy 4.5 4 0.0 valid
In case of a drug interaction, actions taken are formally documented. 4.4 5 0.0 valid

There is a protocol available to inform patients on drug contraindications

most especially for new patients.

 3.9 4 0.0 valid
Allergic reactions are managed effectively for all patients, especially for new Patients. 4.3 4 0.0 valid
Referral of patients to other healthcare providers and effective documentation. 4.2 4 0.0 valid
Dispensing and patient care
Number of dispensed medication with the patient given Information on the drug. 3.6 4 0.0 valid
Follow up of pharmacotherapy
The number of NSAID users whom gastro protection was administered, especially patients>70 y old. 4.1 4 0.0 valid
The pharmacist provided gastro protection especially to patients>70 y who are using NSAIDs on a daily basis and who it was not dispensed. 4.2 4 0.0 valid
Number of patients followed up on pharmacotherapy and documented. 4.2 4 0.0 valid
Counselling
A professional guideline was employed while counselling patients on medications 4.4 4 0.0 valid
Drug inventory and stocking
Suppliers of drugs assessed according to the professional guideline 4.1 4 0.0 valid
There is a standard system for checking expired drugs 4.4 5 0.0 valid

The number of drugs discovered to have expired and reported before they were

dispensed each month

 3.8 4 0.0 valid
Training of pharmaceutical personnel
Participation in mandatory professional development courses, seminars and continuing education for pharmacists. 4.7 5 0.0 valid
Percentage of pharmaceutical staff with adequate training on pharmacy operations 4.0 4 0.0 valid

Valid= Accepted; ×= Rejected; (Me-Mo) = Median-Mode

DISCUSSION

The study developed a set of quality indicators for assessing the pharmaceutical care performance of community pharmacies in Nigeria. The set of quality indicators covers a great range of community pharmacy practice. The experts in community pharmacy practice and major stakeholders in community pharmacy practice reached a consensus on the quality indicators for assessing pharmaceutical care performance in the Nigerian community pharmacies.

It has been opined that quality indicators for assessing the performance of a healthcare delivery system should be selected based on inputs of health care providers with adequate knowledge and experience [14]. Since the late 1970s, Delphi technique has been employed for such exercise e. g. selecting quality indicators [34].

There are quality indicators developed for specific disease states [24-33]. Since the late 1970s, quality indicators in health care have been arrived at with the use of the Delphi technique [34]. For instance, a study revealed the effectiveness of the Delphi panel in establishing consensus in developing prescribing indicators [25], developing indicators for chronic diseases [35], developing performance indicators for emergence medicine [36] and developing indicators for laboratory performance [33]. In our study, we used the Delphi technique to develop a tool for assessing the performance of community pharmacists concerning their pharmaceutical care delivery. A panel of ten experts in the pharmaceutical sector conducted a three-round Delphi panel. The panel of experts reached consensus on twenty-four quality indicators in nine core components, including quality management (2 indicators), documentation of care (4 indicators), communication with patients (3 indicators), management of clinical risk in pharmacy (5 indicators), dispensing and patient care (1 indicator), follow-up of pharmacotherapy (3 indicators), counselling (1 indicator), drug inventory and patient care (3 indicators), training of pharmaceutical staff (2 indicators). The number of quality indicators that resulted in our study is fewer to those obtained from the study by [14] in the Netherlands. The Netherlands study developed 42 quality indicators for community pharmacy care. The indicators included the following: patient counselling (6 indicators), clinical risk management (10 indicators), compounding (7 indicators), dispensing (3 indicators), monitoring of medication use (11 indicators), quality management (5 indicators). Another study [12] developed 66 quality indicators across 10 categories. The difference in the number of quality indicators of these studies compared to ours could be explained in part by the lesser standards of care typically obtainable in developing settings. Nevertheless, introducing these 24 indicators is an important step towards improving pharmaceutical care practice in community pharmacies in Nigeria. The major stakeholders in pharmacy practice participated in validating the questionnaire and this will increase its validity and possibly its acceptance for use in the Nigerian context. Furthermore, the acceptance and popularity of the quality indicators will depend on its proper use, feedback from the community pharmacists and the use of the indicator by the pharmacy inspection team during their routine inspection.

This study had some possible limitations which could have emanated from the prominent disadvantages of the Delphi technique, such as ‘‘bandwagon effect,’’ ‘‘noise,’’ and ‘‘peer pressure for conformity’’. We tried to avoid these limitations by successfully protecting each expert's identity with strict confidentiality and by avoiding personal contacts among experts. Additionally, ‘‘respondent’s fatigue’’ was minimized by limiting the Delphi rounds to only three.

CONCLUSION

This study developed a set of 24 quality indicators for assessing pharmaceutical care performance in community pharmacies in Nigeria. If used for assessment, these quality indicators should impact positively on the quality of pharmaceutical care practiced in community pharmacies in Nigeria.

ACKNOWLEDGEMENT

We express our sincere gratitude to all the pharmacists that participated in this study.

FUNDING

Nil

AUTHORS CONTRIBUTIONS

NO designed the study, conducted the literature search, collected and processed data from the pharmacists and patients, analyzed and interpreted the data, wrote the first draft of the manuscript. OIE developed the concept of the study, guided in designing the study, supervised the study and corrected the first draft of the manuscript. All the authors approved the final draft of the manuscript

CONFLICT OF INTERESTS

The authors declared none.

REFERENCES

  1. Dauti M, Idrizi EA, Malaj L. Pharmaceutical care in community pharmacy in the republic of macedonia. A compared study with EU countries. ESJ 2014;3:313-8.

  2. Foppe Van Mil JW, Schulz M, Tromp TH, FJ Dick. Pharmaceutical care, Europian developments in concepts, implementation, teaching, and research: a review. Pharm World Sci 2004;26:303-11.

  3. Petkova V, Georgiev ST, Dinitrov ZI, Radivoeva M. Implementation of pharmaceutical care knowledge in Bulgarian community pharmacy. Pharm Education 2006;6:107-10.

  4. Penna RP. Pharmaceutical care: Pharmacy’s mission for the 1990s. Am J Hosp Pharm 1889;47:545–7.

  5. Vantamay N. Using the Delphi technique to develop effectiveness indicators for soci marketing communication to reduce health-risk behavior among youth. Southeast Asian J Trop Med Public Health 2015;46:949-57.

  6. Barzekar G, Aziz A, Mariapan M, Ismail MH, Hosseni SM. Delphi technique for generat­ing criteria and indicators in monitoring ecotourism sustainability in northern forests of Iran: case study on Dohezar and Sehezar Watersheds. Folia Forest Pol 2011;53:130-41.

  7. Rowe G, Wright G. The delphi technique as a forecasting tool: issues and analysis. Int J Forecast 1999;15:353-75.

  8. Chartrand M, Guenette L, Brouillete D, Cote S, Huot R, Landry J, et al. Development of quality indicators to assess oral anticoagulant management in community pharmacy for patients with atrial fibrillation. J Manag Care Spec Pharm 2018;24:357-65.

  9. Fujita K, Moles RJ, Timothy F Chen. Quality indicators for the responsible use of medicines: a systematic review. Br Med J 2018;8:e020437.

  10. Sagarananda G, Parbati D, Gulam MK. Assessment of patient-care, facility care and complementary drug use indicators among elderly patients in a tertiary care hospital of Western Nepal. Asian J Pharm Clin Res 2020;13:83-6.

  11. Arkaravichien W, Wongpratat A, Lertsinerdom S. Quality indicators to compare accredited independent pharmacies and accredited chain pharmacies in Thailand. Int J Clin Pharm 2016;38:899-909.

  12. Teichet M, Schoenmakers T, Kylstra N, Mosk B, Bouvy ML, van de Vaart F, et al. Quality Indicators for pharmaceutical care: A comprehensive set with national scores for dutch community pharmacies. Int J Clin Pharm 2016;38:870–9.

  13. Grey E, Harris M, Rodham K, Wenss MC. Characteristics of good quality pharmaceutical services common to community pharmacies and dispensing general practices. Int J Pharm Pract 2016;24:811-8.

  14. Bie DJ, Kijlstra NB, Daemen BJG, Bouvy ML. The development of quality indicators for community pharmacy care. Br Med J 2015;20:666-71.

  15. Schoenmakers TW, Teichert M, Braspenning J, Vanderink L, De Smet PA, Wensing M. Evaluation of quality indicators for dutch community pharmacies using a comprehensive assessment framework. J Manag Care Spec Pharm 2015;21:144-52.

  16. Halsall D, Noyce RR, Ashcroft DM. Characterizing healthcare quality in the community pharmacy setting: insights from a focus group study. Res Soc Adm Pharm 2012;3:360-70.

  17. Winslade N, Taylor L, Shi S, Schuwairth L, Vander Vieuten C, Tomblem R. Monitoring community pharmacists quality of care: a feasibility study of using pharmacy claims data to assess performance. BMC Health Serv Res 2011;11:12.

  18. Trap B, Hansen EH, Trap R, Kahsay A, Simoyi T, Oteba MO, et al. A new indicator-based tool for assessing and reporting on good pharmacy practice. Southern Med Rev 2011;3:4-11.

  19. Benrimoj SI, Werner JB, Raffaek C, Roberts AS, Costa FA. Monitoring quality standards in the provision of non–prescription medicines from Australian community pharmacies: results of a national progamme. Qual Saf Health care 2007;16:354-8.

  20. Suwaratchai P, Sithisarankul P, Sriratanban J, Chenvidhya D, Phonburee W. Utilize the modified delphi technique to develop trauma care indicators. J Med Assoc Thai 2008;91:99-103.

  21. Eubank BH, Mohtadi NG, Lafve MR, Wiley JP, Bois AJ, Borman RS, et al. Using modified clinical consensus for the diagnosis and treatment of patients with rotator cuff pathology. BMC Med Res Methodol 2016;16:56.

  22. Murry JW, Hammors JO. Versatile methodology for conducting qualitative research. Rev High Educ 1995;18:423-36.

  23. Dalkey NC, Helmer O. An experimental ap­plication of the delphi method to the use of experts. Manag Sci 1963;9:458-67.

  24. Hermanides HS, Hulscher MEJL, Schouten JA, Prins JM, Geerlings SE. Development of quality indicators for the antibiotic treatment of complicated urinary tract infections: a first step to measure and improve care. Clin Infect Dis 2008;46:703–11.

  25. Campbell S, Cantrill J, Roberts D. Prescribing indicators for UK general practice: delphi consultation study. Br Med J 2000;321:425–8.

  26. Morris C, Cantrill J, Hepler C, Noyce P. Preventing drug-related morbidity–determining valid indicators. Int J Qual Health Care 2002;14:183–98.

  27. Morris C, Cantrill J. Preventing drug-related morbidity–the development of quality indicators. J Clin Pharm Ther 2003;28:295–305.

  28. To T, Guttmann A, Lougheed M, Gershon A, Dell S, Stanbrook M, et al. Evidence-based performance indicators of primary care for asthma: a modified RAND appropriateness method. Int J Qual Health Care 2010;22:476–85.

  29. Uphoff E, Wennekes L, Punt C, Grol R, Wollersheim H, Hermens R, et al. Development of generic quality indicators for patient-centered cancer care by using a RAND modified delphi method. Cancer Nurs 2012;35:29–37.

  30. Caughey GE, Ellett LMK, Wong TY. Development of evidence-based Australian medication-related indicators of potentially preventable hospitalizations: a modified RAND appropriateness method. Br Med J 2014;4:e004625.

  31. Martirosyan L, Braspenning J, Denig P, de Grauw W, Bouma M, Storms F, et al. Prescribing quality indicators of type 2 diabetes mellitus ambulatory care. Qual Saf Health Care 2008;17:318–23.

  32. Yoon D, Park I, Schuemie M, Park M, Kim J, Park R. A quantitative method for assessment of prescribing patterns using electronic health records. PLoS One 2013;10:e75214.

  33. Zinn J, Zalokowski A. The use of delphi panel for consensus development on an indicator of lab performance. CLM 1999;13:425-8.

  34. Boulkedid R, Abdoul H, Loustan M, Sibony O, Alberti Care. Using and reporting the delphi method for selecting healthcare quality indicators: a systematic review. PLOS ONE 2011;6:e20476.

  35. Roland M, Holden J, Campbell S. Quality assessment for general practice supporting clinical governance in primary care groups. Manchester National primary care research and development centre; 1998.

  36. Bettie E, Mackway Jones K. Delphi study to to identify performance indicators for emergency medicine. Emerg Med J 2004;21:47-50.