General survey planning:
- How do I decide whether to conduct a national or state survey?
- Is the sample size of medicine outlets large enough?
- Is clustering the sample to the main urban centre biasing the sample to urban areas?
- Why was the sample size of medicine outlets expanded?
- My country has multiple administrative divisions; which ones should I choose as survey areas?
- Should I include survey areas that do not contain medicine outlets for a given sector? For example, should I include a survey area where there are private sector medicine outlets but no public sector medicine outlets?
- What should I do if there is no public hospital in a given survey area?
- How can I identify private medicine outlets for sampling if a list is not available centrally?
- What should I do if my private sector contains both licensed retail pharmacies and licensed drug stores with substantially different prices?
- What should I do if my country has more than two “other” sectors that are supplying medicines to a substantial number of patients?
- Is it possible to survey more than 30 medicine outlets per sector?
- Is the number of targeted medicines large enough?
- Why has the core list of 30 medicines been divided into a global core list and regional core lists?
- How were core medicines selected?
- Why are most sold generic equivalents no longer surveyed?
- What if core medicines are registered in my country but are not widely available?
- What if some of my supplementary medicines do not have an international reference price?
- Why does each survey medicine have a specific dosage form and strength when multiple forms and strengths are available?
- Is it possible to survey more than 50 medicines?
- If my public sector has both a national Essential Medicines List (EML) and individual state or provincial EMLs, which one should I use in the survey?
- What should I do if the manager at a medicine outlet does not give permission to collect the data?
- Why is there a recommended pack size when data is collected on unit price?
- If the lowest-priced generic is available in the recommended pack size but also in a larger pack size that has a lower unit price, which price should I record?
- Why are back-up facilities visited?
-What is meant by the median and interquartile range?
- Why do we report median prices rather than average prices?
- What does a median price ratio (MPR) represent?
- What is a matched pairs analysis?
- What do WHO/HAI surveys tell us about medicine availability?
- What is considered an affordable treatment?
- How do my country's survey results compare to other countries' results?
GENERAL SURVEY PLANNING
How do I decide whether to conduct a national or state survey?
In most countries, the methodology can be applied as a national survey. However, if your country is large in geographical size, has a high population and/or has a large number of medicine outlets, it may be more appropriate to apply the methodology as a state or provincial survey. This decision must be made on an individual country basis; contact HAI or WHO for advice.
Is the sample size of medicine outlets large enough?
The WHO/HAI Medicine Prices project recommends collecting prices on targeted medicines at 30 outlets per sector. This recommendation was based on decades of experience in small sample research methods by various WHO programmes and other organisations, and is further supported by data from completed WHO/HAI Medicine Prices surveys. In general, 30 outlets will be sufficient. However, if availability of medicines is very low or if price differences are known to be large, a survey manager may wish to increase the number of outlets sampled to obtain enough data points to get reliable estimates. Also, median prices for medicines on the list that are found only rarely will be less reliable than median prices for more commonly found medicines. Survey managers should consider each medicine on the list before the survey and if they are not commonly used in the country, replace the medicine with another that is more widely available.
In 44 WHO/HAI Medicine Prices surveys up to October 2007, we constructed 95% confidence intervals for median medicine prices using a simple nonparametric method (see http://www.math.unb.ca/~knight/utility/MedInt95.htm). When 20 to 29 price observations were collected, the confidence interval typically extended 3% above and below the median price of an originator brand product in the private sector. In other words, if the median price (or standardized MPR) was found to be 200, then the 95% confidence interval would be from 194 to 206. When 10 to 19 prices were observed, the confidence interval was about +/- 5% (or 190 to 110, in the example). Median prices of lowest priced generics are somewhat less reliable – about +/- 12% when there are 20 to 29 prices collected and +/- 15% when 10 to 19 prices are collected. We also found that the interquartile range of outlet prices (or IQR, which is defined by the 25th and 75th percentile prices) is very close to the nonparametric 95% confidence interval in most cases. Thus, survey managers can use the IQR as an informal substitute for the 95% confidence interval.
Is clustering the sample to the main urban centre biasing the sample to urban areas?
A validation study conducted in Peru showed that the sample is not biased to urban areas. In the study, the number of survey areas in the standard methodology was expanded both in number and in distance from the urban centre. At the time of the validation study the methodology included four survey areas: the main urban centre and three other administrative areas within one days’ travel from the main urban area. The validation study showed that increasing the number of survey areas, or increasing their distance from the main urban centre did not significantly affect the results obtained.
Why was the sample size of medicine outlets expanded?
Although the original sample size (20 medicine outlets per sector) was found to be nationally representative in the Peru validation study, the low availability of medicines observed in many surveys meant that there was less price data available for analysis than anticipated. As such, the sample size was increased (30 medicine outlets per sector) to ensure sufficient price data for robust analysis.
My country has multiple administrative divisions; which ones should I choose as survey areas?
General parameters for the selection of survey areas are as follows:
- Survey areas should cover a population of approximately 100,000 to 250,000
- All six survey areas should be reachable within one day's travel from the main urban centre
- A 4 hours’ drive should cover a reasonable portion of the survey area
- Survey areas should be as nationally representative as possible, bearing in mind the practicalities of conducting the survey
Should I include survey areas that do not contain medicine outlets for a given sector? For example, should I include a survey area where there are private sector medicine outlets but no public sector medicine outlets?
Yes, you should still include the survey area but the sample size in other areas should be increased accordingly. In the example above, the sample of public sector medicine outlets would be increased in other surveys areas (one extra outlet per survey area) to maintain a total sample size of 30 (6 public outlets/survey area x 5 survey areas, instead of 5 public outlets/survey area x 6 survey areas).
What should I do if there is no public hospital in a given survey area?
You should first check to make sure the administrative divisions you have chosen are not too small; survey areas should be large enough to contain at least one district or regional hospital. If you still have a survey area that does not contain a hospital, you will need to select an alternate health facility to survey as the “anchor” around which the sample is clustered. Choose the facility with the highest level of care available in the survey area; if there are multiple facilities that offer the same level of care, select one at random. When in doubt contact WHO or HAI for advice.
How can I identify private medicine outlets for sampling if a list is not available centrally?
Area supervisors in the survey areas should consult with local officials in order to identify private medicine outlets. For this reason it is important that area supervisors are selected that are familiar with the local area and have local contacts capable of providing the necessary information.
What should I do if my private sector contains both licensed retail pharmacies and licensed drug stores with substantially different prices?
You should survey licensed retail pharmacies as the private sector and survey licensed drug stores as an “other” sector. Another option would be to survey both licensed retail pharmacies and licensed drug stores as part of the private sector, and then conduct a sub-group analysis of each (i.e. analyse each separately). However, this approach is discouraged as your sample size for each group will be smaller than recommended, thereby weakening your results.
What should I do if my country has more than two “other” sectors that are supplying medicines to a substantial number of patients?
You may choose to 1) limit your study to two “other” sectors, or 2) survey more than two “other” sectors. The disadvantage of Option 1 is that you will be missing price and availability results from a sector that is potentially important for patients’ access to medicines. The disadvantage of Option 2 is the added time and resources required to survey additional sectors. Further, the workbook can only accommodate two “other” sectors, so data from additional sectors will have to be entered into a second workbook, and any comparisons across sectors in different workbooks will have to be conducted manually. The decision to include multiple other sectors in the survey depends on the individual survey objectives and on the importance of each other sector in patients’ access to medicines. When in doubt, contact WHO or HAI for advice.
Is it possible to survey more than 30 medicine outlets per sector?
Yes, the survey should include at least 30 medicine outlets per sector to ensure a representative sample. However, you are free to increase the sample size, bearing in mind the additional time and resources it will require to visit additional outlets. The survey workbook can accommodate up to 120 medicine outlets per sector. Note that in increasing the sample size, it is recommended that the number of survey areas be increased to provide a more representative sample, rather than increasing the number of medicine outlets surveyed in each survey area.
Is the number of targeted medicines large enough?
From WHO/HAI’s experience in dozens of surveys to date, 30 to 50 medicines has been found more than adequate to illustrate pharmaceutical pricing problems that may be present in a country (such as unaffordable prices, excessive total mark-ups over manufacturing price, low availability) as well as positive features and opportunities (such as relatively affordable generics or sound procurement practices). A Medicine Prices survey tends to show both a diverse set of findings within a country at the same time as a coherent overall summary.
One phenomenon that has been noted, however, is that MPRs and availability may differ quite a lot from one medicine to another. Individual medicines have their own quirky stories in terms of their history on the market, production requirements, distribution channels, and patterns of use (different demographic characteristics, treated diseases, seasons, etc.). This natural variety and quirkiness means that even a very stable statistic such as the median (we calculate the median availability and median MPR across medicines) has some limits in applicability. The median across the basket of targeted medicines in a survey should be considered a rough approximation of the true picture of all similar medicines in a country. (Similar means they could have been selected using the same criteria that was used to select the targeted medicines.)
WHO/HAI conducted a study in which non-parametric 95% confidence intervals were constructed around these medians. The confidence intervals were fairly wide, and, conveniently, are closely approximated by the IQRs. In general, the upper and lower bounds of the confidence interval around the median MPR were at a relative distance of 50-100% from the median, while the distance from the median to each bound of the CI around median availability was about 20%. Fairly large confidence intervals means that results summarized across surveyed medicines should always be reported as findings for a set of widely used essential medicines, and not as a precise figure describing an entire country. When medians across medicines are compared within a survey (for example, between sectors, or between originators and generics) or between country surveys, it is important to use only matched lists of medicines. Also, the IQR around each median should be reported alongside the median, in order to provide a sense of the natural variation in the data.
Why has the core list of 30 medicines been divided into a global core list and regional core lists?
Core survey medicines have been divided into a global list and region-specific lists to account for the wide variations in the general usage of individual medicines. Regional core medicines lists have been developed as a complement to the global core list to address regional differences in the usage of medicines.
How were core medicines selected?
An analysis was conducted on the availability of the original set of 30 core medicines in the over 40 surveys conducted prior to December 2006. The results of this analysis, together with data from additional sources such as IMS Health national databases, were used to select a global core list of 14 medicines that are widely used in many countries/regions and meet the following criteria:
Used to treat common acute and chronic conditions that cause significant morbidity and mortality
Recommended, usually as first-line courses of treatment, in global, regional and national treatment guidelines.
Available in standard formulations
Why are most sold generic equivalents no longer surveyed?
The difficulty in identifying most sold generics (MSGs), and the variation in methods of identifying MSGs across surveys, made the analysis of MSG data difficult to interpret and of limited value. It was therefore decided to exclude MSGs from the survey.
What if core medicines are registered in my country but are not widely available?
In the surveys conducted to date, perceived non-availability of certain medicines has often proved to be wrong. Before removing a medicine from the global or regional list, verify its availability and, when in doubt, include it in the survey to avoid a missed opportunity. If, after conducting the survey, you find that a medicine is not generally available in the country, you can exclude this medicine/product from the analysis by “turning off” the corresponding row in the workbook. On the Field Data Consolidation pages, “Column D: Include in analysis?” allows you to include/exclude data from individual medicine products. Changing the ‘1’s in Row 10 to ‘0’s will exclude medicine products from calculations (excluded rows are shaded grey).
What if some of my supplementary medicines do not have an international reference price?
It is strongly recommended that all medicines have an international reference price. However in some cases, such as surveys of therapeutic groups, some medicines may not have an international reference price. In such cases the analysis of the medicines needs to be two-fold: 1) using Median Price Ratios (see Manual section Chapter 8, Box 8.1) for those medicines with an international price; and 2) using the median price in local currency. The median price in local currency is automatically calculated and presented in the last column of the individual Field Data Consolidation pages, in data entry view with ratios turned on to display summary data for individual medicines (see manual section 7.4.1) , but is not presented on the “sector summary” or “product summary” pages.
Additional guidance is provided in the document Using the WHO/HAI Medicine Price Methodology to study a therapeutic group of medicines, also available on the manual’s CD ROM. Note that the analysis of availability and affordability of medicines does not rely on the international reference price and is therefore unaffected by the existence or not of an international reference price.
Why does each survey medicine have a specific dosage form and strength when multiple forms and strengths are available?
Each survey medicine must have a specific dosage form and strength so that the data collected at individual medicine outlets is comparable and the analysis is comparing “like with like”. This has to be borne in mind when analyzing the availability data; a medicine might not have been available in the dosage form/strength collected in the survey, but may have been available in an alternate dosage form/strength. To address this issue, the manual recommends that if a core medicine is available in a different dosage form/strength, the alternate should be included as a supplementary medicine.
Is it possible to survey more than 50 medicines?
Each survey workbook is capable of holding data for a maximum of 50 medicines. Should you wish to survey more than 50 medicines, you can do so by using a second workbook. Since the Medicine Price Data Collection form is generated from the workbook, you will need to create one form that contains all survey medicines. This can be done by copying the data collection form from one workbook into a Word document, and adding the medicines from the second workbook manually. Also note that summary data that includes medicines from two different workbooks (e.g. average availability or median MPR for all survey medicines) will need to be computed manually.
If my public sector has both a national Essential Medicines List (EML) and individual state or provincial EMLs, which one should I use in the survey?
If you are conducting a national survey, use the national EML. If you are conducting a state or provincial survey, you may wish to use either the national EML or the state/provincial EML, depending on your survey objectives. For example, if your state/provincial survey is part of a series of surveys aimed at examining prices and availability across the country, you will probably still want to use the national EML. Alternatively, if you are conducting the survey on behalf of a state or provincial health authority, you will probably want to use the state/provincial EML.
What should I do if the manager at a medicine outlet does not give permission to collect the data?
If upon contact with a sample medicine outlet the manager does not give permission to conduct the survey, the back-up facility should be used. In this case the back-up facility becomes the sample facility, and it will be necessary to identify another back-up in the event that fewer than 50% of medicines are available.
Why is there a recommended pack size when data is collected on unit price?
Medicine pack sizes often affect the unit price (e.g. price per tablet); generally a lower unit price is offered when a larger pack size is purchased (economy of scale). Recommended pack sizes are therefore used in the survey to avoid differences in unit price caused by variations in pack size.
If the lowest-priced generic is available in the recommended pack size but also in a larger pack size that has a lower unit price, which price should I record?
You should record the unit price that corresponds to the recommended pack size, even if this price is higher than that of another available pack size. In this way you will increase the comparability of results across medicine outlets and avoid price variations caused by economies of scale.
Why are back-up facilities visited?
Where fewer than 50% of medicines are available in a given medicine outlet, a second back-up facility is used to collect an additional set of data. This is to ensure that enough price data is collected to allow for robust analysis. Note that the data from the original facility is kept and analysed in order not to skew the results for medicine availability.
What is meant by the median and interquartile range?
After prices of a medicine have been collected from a group of outlets, the median is a statistical result meaning the mid-point of all the prices. One can arrange all the prices from smallest to largest and the median is the price in the middle (if there are an odd number of prices) or the mid-point between the two in the middle (if there are an even number). Essentially, half of the prices are below the median and half are above. The median allows you to summarize all the prices with a single numerical result.
The interquartile range is defined by the 25th and 75th percentile prices, after ordering by price. The 25th percentile price is the midpoint between the minimum price and the median price. One quarter of all the prices are smaller than 25th percentile and three quarters are larger. Likewise, the 75th percentile price is half-way between the median and the maximum, with three quarters below and one quarter higher. Thus, the interquartile range (or IQR), which is all the prices between the 25th and 75th percentile, is the center-most half of all prices.
The WHO/HAI workbook also calculates medians and IQRs of MPRs and availability results across medicines. An MPR (median price ratio) is calculated for each medicine surveyed, then these are arranged in order by size, and the median is defined as the midpoint of MPRs. Similarly, each medicine has an “availability”, the percentage of surveyed outlets in which it was found. Median availability is the midpoint of all the availabilities when they are in order by size.
Why do we report median prices rather than average prices?
The median is generally considered to be a better summary of price data than the average. (The average, or mean, is the sum of all prices, divided by the number of prices). Prices tend not to be clustered in a neat group, like, for example, the heights of different men. Rather, there tend to be many prices on the lower end (but none below zero), fewer higher prices, and an occasional few very high prices. The few high prices have an excessive influence on the average result, such that the average may not be representative of most prices. The average is likely to be higher than most prices. The median statistic will remain representative even when there are some very unusual prices or even price errors.
What does a median price ratio (MPR) represent?
The MPR is an expression of how much greater or less the local medicine price is than the international reference price, e.g. an MPR of 2 would mean that the local medicine price is twice that of the international reference price.
What is a matched pairs analysis?
In the survey, matched pairs analysis is used to compare summary data (median MPR) for two different categories of medicines (e.g. originator brands vs. lowest priced generics, or public sector vs. private sector). For median price ratios to be calculated for patient prices, medicines need to be found in at least 4 outlets. As a result of variations in the availability of individual medicines, summary data for originator brands will likely not contain the same group or basket of medicines as summary data for lowest-priced generics. Similarly baskets of medicines found in the public sector will also vary to those found in the private sector. Since these baskets contain different medicines with different prices, they are not comparable. In the matched pairs analysis, only those medicines found in both baskets are included in the calculation of summary data, thereby generating comparable results.
What do WHO/HAI surveys tell us about medicine availability?
Availability results are useful but should be interpreted with some caution and, ideally, an understanding of the context, particularly when examining results from another country. Several factors can affect availability results – for example, the timing of the survey, or the decisions of the survey managers about which medicines to survey. Medicines on a global or regional list may not be in widespread use locally, or local prescribers and consumers may prefer another strength or form. Therefore, the median availability across medicines in a survey may not be a reliable representation of a country’s medicine situation. Nevertheless, extreme median availabilities (for example, 0%, or greater than 80%) probably tell us something meaningful.
Availability results offer many potential insights. For example, when large price differences exist between the public and private sectors, differences in availability should also be noted. Likewise, in the private sector, the relative availability between originator and generic forms will have an impact on what patients are able to obtain. Public sector availability should usually be considered in the context of a national essential medicines list. In other words, there is an important difference between intending and failing to have a medicine in stock, versus never intending to stock an item. Relative availability between two medicines may suggest something about the demand or perceived importance of the medicines. (In the private sector, widely-stocked medicines may be considered good sellers. In the public sector, a stock-out could mean insufficient supply to meet high demand, or alternatively, a low priority for stocking when procurement budgets are limited.) In general, the more outlets in the survey sample that have the data, the more confident we can be that the median price (and Median Price Ratio) calculated across outlets is trustworthy.
What is considered an affordable treatment?
A treatment which costs less than 1 days’s wage for the lowest-paid government worker is generally considered affordable. However in many countries, a substantial proportion of the population earn less than the lowest government wage. If this is the case in your country you may wish to change the daily wage in Cell J6 of the Treatment Affordability page of the workbook and see how affordable medicines would be for people on a lower income. However, remember to change to Cell J6 back to the daily wage of the lowest-paid government worker before sending your workbook to HAI or WHO.
How do my country's survey results compare to other countries' results?
Because each survey has its own moment in time and a team of investigators shaping the application of the WHO/HAI methods, comparisons within a single survey are usually more reliable and valid than cross-country comparisons. Nevertheless, an international comparison can be an eye-opening exercise – one which this methodology is designed to facilitate. A cross-country comparison will be more convincing and relevant if it is within a year or two in time, and if the two countries are in the same region and have comparable cultural or socioeconomic characteristics. Their pharmacy sectors need not be similar, and in fact differences in survey results may point to the consequences of different structures or specific policies. To the extent possible, compare results for the same specific medicines, and the more matched pairs that can be identified, the better.