6 Pilot Study and Pre-test

Nisha Jolly Nelson

Objective

The objective of this module is to introduce two important preliminary steps in social research, namely pilot study and pre-test. It describes the importance, procedure, and limitations of these techniques as well.

Introduction

This module explains the importance of Pilot study and Pre-test in social research. We would take up the task of explaining the meaning, advantages, and limitations of pilot study first. A pilot study is one of the crucial stages in a research project. It is necessary and useful in providing the groundwork in a research project. The term ‘pilot study’ refers to mini versions of a full-scale study also called feasibility studies, as well as the specific pre-testing of a particular research instrument such as a questionnaire or interview schedule. Pilot studies are important element of a good study design. Conducting a pilot study does not guarantee success in the main study, but it does increase the likelihood. Pilot studies fulfil a range of important functions and can provide valuable insights for other researchers.

Pre-tests are preliminary tests of the measures used on a small sample of the population to be studied. It is the final stage in the tool development process. Pre-testing is the use of questionnaire in a small pilot study to ascertain how well the questionnaire works. Thus, pilot study is a smaller version of a larger study that is conducted to prepare for that study. Pilot studies are used as feasibility studies, to ensure that the ideas or methods behind a research idea are sound and to understand the study protocol before launching a larger study. But pre-testing is the testing of research instruments, including questionnaires or interview schedules used to carry out the research. Now let us discuss these methodological tools in detail.

Learning Outcome

After reading this module you will be able to explain what pilot study is and pre-test in social research. You will be able to describe the importance, procedure, and limitations of pilot study. It will also enable you to get an idea of pretesting and its importance in social research

Pilot Study

In social science research, the term pilot study is used in two different ways. It can refer to so-called feasibility studies which are “small scale version[s], or trial run[s], done in preparation for the major study” (Polit et. al. 2001: 467). A pilot study can also be the pre-testing or ‘trying out’ of a particular research instrument (Baker 1994: 182-3). Thus, a pilot study is a small-scale implementation of a larger study or a part of a larger study. Pilot studies last for shorter period of time and usually involve a smaller number of participants, sites or organisations. Pilot studies can be used in any methodological setting, especially when attempting to collect data in a new format or location or to simply examine potential roadblocks before full implementation. A pilot study may also be viewed as a feasibility study. A feasibility study is completed to determine if the full study can be accomplished. Feasibility studies are practical when there is concern that a full-scale study may not be possible due to concerns about cost, procedures, personnel and other issues. Pilot studies are not simply exploratory in nature. They are designed with a clear purpose of developing some conclusions and pushing an area of research (Schreiber 2008). It resembles the main study in many respects, including an assessment of the primary outcome. In some cases, this will be the first phase of the substantive study and data from pilot phase may contribute to the final analysis. This can be referred to as an internal pilot. Or at the end of the pilot study, the data may be analysed and set aside, a so called external pilot (http://www.nets.nihr.ac.uk/glossary). Pilot studies range from relatively informal trying out procedures on a handful of participants, to efficacy studies, or to small scale clinical trial interventions (Hertzog 2008).

It has been said that pilot studies are likely to be “under discussed, underused and underreported” (Prescott and Soeken 1989: 60). Full reports of pilot studies are rare in the research literature (Lindquist 1991; Muoio et al. 1995, van Teijlingen et al. 2001). When reported, they often only justify the research methods or particular research tool used. Too often research papers only refer to one element of the pilot study, for example, to the ‘pre-testing’ or ‘pilot testing’ of a questionnaire (de Vaus 1993). Such papers simply state: “the questionnaire was tested for validity and reliability.” When pilot studies are mentioned in more detail in academic papers and reports, researchers regularly comment that they “had learned from the pilot study” and made the necessary changes, without offering the reader details about what exactly was learnt. Some of these processes and outcomes from both successful and failed pilot studies might be very useful to others embarking on projects using similar methods and instruments. This is particularly important because pilot studies can be “time-consuming, frustrating, and fraught with unanticipated problems, but it is better to deal with them before investing a great deal of time, money, and effort in the full study” (Mason and Zuercher 1995). It has also been argued that the current research climate demands accountability from researchers, which means that there is a need to ensure the best possible use of research results (Crosswaite and Curtice 1994). Well-designed and well-conducted pilot studies can inform us about the best research process and occasionally about likely outcomes. Therefore, investigators should be encouraged to report their pilot studies, and in particular to report in more detail the actual improvements made to the study design and the research process.

4.1. Doing a Pilot Study: Why is it Essential?

Conducting a pilot study is important in social research. It serves many purposes:

It often provides the researcher with ideas, approaches, clues the researcher may not have foreseen before conducting the study. Such ideas and clues increase the chances of getting clearer findings in the main study.
It permits a thorough check of the planned statistical and analytical procedures, giving the researcher a chance to evaluate their usefulness to the data. The researcher may then be able to make needed alterations in the data collecting methods and therefore analyse data in the main study more efficiently.
It can greatly reduce the number of unanticipated problems because the researcher has all opportunity to redesign parts of his/her study to overcome difficulties that the pilot study reveals.
It may save lot of time and money. The pilot study almost always provides enough data for the researcher to decide whether to go ahead with the main study.
In the pilot study, the researcher may try out a number of alternative measures and then select those that produce the clearest result for the main study.
Pilot study is especially beneficial for students. The less research experience the student has, the more s/he likely to benefit from a pilot study. Because of that possibility the student should attempt a pilot study whenever possible.
A pilot study also helps in answering methodological questions.

4.2. Steps in Conducting a Pilot study

Pilot studies can be based on quantitative and/or qualitative methods and large-scale studies might employ a number of pilot studies before the main survey is conducted. The researchers may start with “qualitative data collection and analysis on a relatively unexplored topic, using the results to design a subsequent quantitative phase of the study” (Tashakkori & Teddlie 1998: 47). The first phase of a pilot might involve using in-depth interviews or focus groups to establish the issues to be addressed in a large-scale questionnaire survey. Next the questionnaire, e.g., the wording and the order of the questions, or the range of answers/options on multiple-choice questions, might be piloted. A final pilot could be conducted to test the research process, e.g., the different ways of distributing and collecting the questionnaires.

4.3. Pilot Study Procedures to Improve the Internal Validity of a Tool (Questionnaire/Interview Schedule)

We may note down the following advantages of a pilot study for preparing a questionnaire or interview schedule (Peat et al. 2002: 123):

Administer the questionnaire to pilot subjects in exactly the same way as it will be administered in the main study
Ask the subjects for feedback to identify ambiguities and difficult questions
Record the time taken to complete the questionnaire and decide whether it is reasonable
Discard all unnecessary, difficult or ambiguous questions
Assess whether each question gives an adequate range of responses
Establish that replies can be interpreted in terms of the information that is required
Check that all questions are answered
Re-word or re-scale any questions that are not answered as expected
Shorten, revise and, if possible, pilot again.

4.4. Reasons for Conducting a Pilot Study

Pilot studies are conducted for a range of different reasons. The most important reasons are as follows:

Developing and testing adequacy of research instrument
Assessing the feasibility of a (full-scale) study/survey
Designing a research protocol
Assessing whether the research protocol is realistic and workable
Establishing whether the sampling frame and technique are effective
Assessing the likely success of proposed recruitment approaches
Identifying logistical problems which might occur using proposed methods
Estimating variability in outcomes to help determining sample size
Collecting preliminary data
Determining what resources (finance, staff) are needed for a planned study
Assessing the proposed data analysis techniques to uncover potential problems
Developing a research question and research plan
Training a researcher in as many elements of the research process as possible
Convincing funding bodies that the research team is competent and knowledgeable
Convincing funding bodies that the main study is feasible and worth funding
Convincing other stakeholders that the main study is worth supporting

Self Check Exercise – 1

What is a pilot study?

A pilot study is a small-scale implementation of a larger study or a part of a larger study. Pilot studies last for shorter amounts of time and usually involve a smaller number of participants, sites or organisations. Though traditionally associated with quantitative experimental design pilot studies can be used in any methodological setting, especially when attempting to collect data in a new format or location or to simply examine potential roadblocks before full implementation. A pilot study may also be viewed as a feasibility study. A feasibility study is completed to determine if the full study can be accomplished.

Why are pilot studies important?

Pilot studies are important because it provides the researcher with ideas, approaches and clues the researcher may not have foreseen. It permits a thorough check of the planned statistical and analytical procedures, giving the researcher a chance to evaluate their usefulness. The researcher may then be able to make needed alterations in the data collecting methods and therefore analyse data in the main study more efficiently. It can greatly reduce the number of unanticipated problems because the researcher has all opportunity to redesign parts of his/her study to overcome difficulties that the pilot study reveals. It may save lot of time and money. The pilot study almost always provides enough data for the researcher to decide whether to go ahead with the main study. In the pilot study the researcher may try out a number of alternative measures and then select those that produce the clearest result for the main study. Pilot study is especially beneficial for students. The less research experience the student has, the more s/he likely to benefit from a pilot study.

4.5 Why are Pilot Studies not reported?

Pilot studies are not reported due to publication bias. Most of the journals accept only papers that have statistically significant results and not to report non-significant effects (Mahoney 1977; Chann 1982; Dickersin 1990). Papers reporting methodological issues, such as those identified during the pilot phase of a study, are less attractive to publishers. However, it is equally important to ensure that lessons learned with respect to the research method are shared, otherwise respondents may be subjected to poorly developed tools or money may be wasted because methods of recruitment failed. A consistent selection bias favouring reports of primary research over papers on research methods, theoretical thinking, or secondary analysis, can lead to many of researchers re-inventing the wheel without having had the opportunity to learn from other people’s experience.

4.6. Limitations of Pilot Studies

Pilot studies also have a number of limitations. These include the following:
Possibility of making inaccurate predictions or assumptions on the basis of pilot data
Problems arising from contamination and
Problems related to funding (Van Teijlingen and Hundley 2002).
Let us discuss these issues in detail.

a) Inaccurate Predictions and assumptions:

Completing a pilot study successfully is not a guarantee of the success of the full-scale survey. Although pilot study findings may offer some indication of the likely size of the response rate in the main survey, they cannot guarantee this because they do not have a statistical foundation and are nearly always based on small numbers. Furthermore, other problems or headaches may not become obvious until the larger scale study is conducted.

b) Contamination:

A further concern is that of contamination. This may arise in two ways:

a) Where data from the pilot study are included in the main results.
b) Where pilot participants are included in the main study, but new data are collected from these people.

Social scientists engaged in predominantly quantitative research are likely to argue that: “an essential feature of a pilot study is that the data are not used to test a hypothesis or included with data from the actual study when the results are reported” (Peat et al. 2002: 57). The obvious concern is that if there were problems with the research tool and modifications had to be made in the light of the findings from the pilot study, data could be inconsistent or inaccurate. However, where an established and validated tool is being used and the pilot study is determining other methodological aspects, such as recruitment rates, it could be argued that such data may be of value.

A more common problem is deciding whether to include pilot study participants or site(s) in the main study. Here the concern is that they have already been exposed to an intervention and, therefore, may respond differently from those who have not previously experienced it. This may be positive, for example, the participants may become more adept at using a new tool or procedure. However, it may also be negative with participants showing a decline in following a protocol because it is no longer novel. Indeed, changes in behaviour have long been recognised and a ‘run in’ period, where an intervention is introduced prior to a study, is often used for these reasons. The concern about including participants from the pilot study in the main study arises because only those involved in the pilot, and not the whole group, will have had the experience. In some cases, however, it is simply not possible to exclude these pilot-study participants because to do so would result in too small a sample in the main study. This problem arises in particular where the samples are clusters, for example, schools, prisons or hospitals. In such cases, one can conduct a sensitivity analysis (or subgroup analysis) to assess to what extent the process of piloting influences the size of the intervention effect.

Contamination is less of a concern in qualitative research, where researchers often use some or all of their pilot data as part of the main study. Qualitative data collection and analysis is often progressive, in that a second or subsequent interview in a series should be ‘better’ than the previous one as the interviewer may have gained insights from previous interviews which are used to improve interview schedules and specific questions. Some have therefore argued that in qualitative approaches separate pilot studies are not necessary (e.g. Holloway 1997: 121). For example, a qualitative interviewer conducting 15 focus group interviews will listen to the recordings or read through the transcripts of the first three or four in order to improve the questions, the way of introducing the issues into the group interview or even to add new topics. Thus, although there is no specific pilot study, analysis of the earlier focus groups may help improve the later ones. However, Frankland and Bloor (1999: 154) argue that piloting provides the qualitative researcher with a “clear definition of the focus of the study” which in turn helps the researcher to concentrate data collection on a narrow spectrum of projected analytical topics. Piloting of qualitative approaches can also be carried out if “the researcher lacks confidence or is a novice, particularly when using the interview technique” (Holloway 1997: 121).

c) Funding

Problems may also arise where a pilot study requires a significant investment of resources, making it difficult for the study team to call a halt to the research after an unsuccessful pilot study. Researchers might be tempted to make considerable changes in the main study, rather than deciding that the proposed study is not possible with the available resources, time, population, etc. In contrast, funding bodies may be reluctant to fund a further study if the pilot has been substantial as they may view the research as no longer original, especially if results from the pilot study are published.

Self Check Exercise -2

Why are pilot studies not reported?

Pilot studies are not reported due to publication bias. Papers reporting methodological issues, such as those identified during the pilot phase of a study, are less attractive to publishers.

What are the problems with pilot studies?

Possibility of making inaccurate predictions or assumptions on the basis of pilot data
Problems arising from contamination and
Problems related to funding

Pre-test

The objective of this section is to understand the importance of pre-testing. Pre-test is an important part in social research. Despite their widely recognized importance in survey research, pre-tests have received little methodological attention (Hunt et.al.1982). Whether a study is to employ questionnaires, interviews, observation or available data, a pre-test is vital. As has been mentioned earlier, pre-tests are preliminary tests of the measures used on a small sample of the population to be studied. No matter how carefully one designs a measure for questionnaire, interviews, or observations it is still advisable to give it an actual try. A pre-test of a questionnaire may demonstrate that some of the questions are unintelligible to respondents. In a questionnaire with open–ended questions, the researcher might find that the respondents are giving inadequate answers, suggesting a need to reword the question. An interviewer undertaking a pre-test might find that the respondents do not feel that that the interview is legitimate and may refuse to be questioned, signalling the need to reword the introductory remarks. Very often pre-testing of questionnaire is done in a hurried, non systematic fashion. Lehmann (1979) has pointed out that the pre-testing stage in the research process is one “most likely to be squeezed out due to cost and time pressures”. Similarly reports on pre-testing are almost absent in the literature on social sciences.

5.1 Fundamental Issues in Pre-testing

Let us examine some of the fundamental issues pertaining to pre-testing process. According to Hunt et. al. (1982), there are five fundamental issues with respect to pre-testing. These are:

What specific item should be pre-tested?
What method should be used to conduct pre-test?
Who should do the pre-testing?
Who should be the subjects in the pre-test?
How large a sample is needed for the pre-test? Let us examine each issue in detail.

a) Items that should be pre-tested:

This can be separated into three categories:

Items about the questionnaire itself
Items about specific questions
Items about data analysis.

The items about questionnaire which has to be pre-tested include: length, layout, the format for the questions used, and the number of lines to leave for replies and the sequencing of questions. The pre-test can also be used as a device to estimate response rate for the questionnaire.

Second, we need to pre-test individual questions. The interviewer who is doing the pre-test should carefully observe the respondent as s/he is filling out the questionnaire/Interview schedule. If a respondent hesitates at a particular question, the question may be ambiguous or confusing, or may contain terminology unfamiliar to the respondent. The interviewer should probe the respondent after each question and /or at the end of the questionnaire to ascertain how the respondent interpreted each question and whether the respondent had problems with any questions.

Third, the pre-testing process can and should be used to pre-test data analysis procedures and the complete research design. Coding and tabulating the procedures should be pre-tested with dummy tables prepared to facilitate this process. The results of the open-ended questions on the preliminary questionnaire may be used to suggest new research hypothesis that could be tested. In short, the pre-testing of the questionnaire should be considered a “dry run” of the entire research project. The ultimate question is will the instrument provide data of sufficient quality and quantity to satisfy the objectives of the research project?

b) Method by which the pre-test is administered

A second issue in pre-testing is the method by which the pre-test should be administered. Three methods often discussed are personal interviews, telephone interviews, and mail self-reports.

Most writers suggest that the first series of pre-test should be conducted by personal interview even if the questionnaire ultimately will be administered by telephone or through mail (Boyd, Westfall and Stasch 1977). Personal interviews enable the interviewer to notice reactions, hesitations and other cues by the respondent that could not be obtained via telephone or mail. Subsequent pre-tests should be conducted by means of the administration method to be used in the ultimate research.

Personal interview pre-tests may be conducted by the debriefing method or by the protocol method. In the debriefing method, the respondent is asked to fill out the questionnaire completely while the interviewer makes careful observations. After the questionnaire is completed, the interviewer probes the respondent for any potential problems with the format of the questionnaire and with individual questions. In the protocol method, the subject is asked to “think loud” as he or she is filling out the questionnaire. The interviewer records the concurrent verbalizations or makes careful notes of them.

c) Selection of Interviewers

A third issue in pre-testing is the selection of interviewers. It is recommended (Boyd, Westfall, and Stasch 1977) that the interviewers carrying out the task must be competent to do it as s/he must be able to perceive uneasiness, confusion and resistance among respondents.

d) Nature of the respondents.

There is little consensus pertaining the issue of the nature of respondents. To Hull and Hawkins (1976), for pre-testing we must use respondents who are similar as possible to the target respondents. Similarly Zaltman and Burger (1975) suggest that the pre-test subjects should be “typical” or “representative”. A different position is taken by Galtung (1969). To him for the purpose of pre-testing it is necessary to have a statistically sophisticated probability sample or a purposive quota sample, but a heterogeneous sample with extreme cases is indispensable. Brown and Beik (1975) recommend that “the trial include adverse and favourable as well as typical respondents. If a questionnaire is administered to the extremes of the sample (intellectual, emotional and attitudinal) it should be more than satisfactory for the “typical” respondents also.

e) Size of the pre-test sample

The final issue pertaining to pre-testing is the size of the pre-test sample. There appears to be substantial consistency with regard to this issue among various authors. Some authors simply recommend that the sample be small (Zaltman and Burger 1975). Others give specific sizes. For example, Ferber and Verdoorn (1962) suggest a sample of 12 is satisfactory. Boyd, Westfall and Stasch (1977) recommend 20, whereas Backstrom and Hursh (1963) indicate that the sample of 30 is adequate. The unanimity is that, there are grounds of believing that the size of the pre-test sample is not fixed, but it should be a function of the instrument and the target population. For example, long, complex instruments would seem to require larger pre-test samples than short, simple instruments. Likewise, if the ultimate questionnaire is to be used with very sophisticated target populations, it would require a larger pre-test sample than would one intended for sophisticated audiences.

5.2 Pre-testing Interview Guidelines

Respondents should be at ease and the interviews are to be taken in an undisturbed place. Interviewer should motivate respondents to participate and they have to initially explain the purpose of the interview. Convince the respondents as “experts” and his/her opinion is valuable. The interviewer should also avoid certain things. They should not tell the people that they are wrong. They are not supposed to argue or contradict with the opinion of the respondents. Also the interviewer is not allowed to teach or advice the respondents.

Conclusion

Pilot studies play an important role in the development of scientific social research. It is used to determine the feasibility of conducting a large scale study. It will help the researcher to understand the weakness or strength of the proposed study. It can also be used to pre-test the instrument used to collect the information. While pilot study is a smaller version of a larger study that is conducted to prepare for that study, pre-test is the testing of the research instruments such as the questionnaire or the interview schedule. Like pilot study, pre-testing the instrument is equally important to ensure that the questions are understood by the respondents and there are no problems with the wording or measurement. Pre-testing involves the use of a small number of respondents to test the appropriateness of the questions and their comprehension. The different aspects to be evaluated during the pre-test are the acceptability of the wording of the questions in the local cultural context, willingness of the respondents to co-operate in the research, determining whether the questionnaire will meet the objectives of the research. Thus pre-testing may help in putting questions in proper sequence, using acceptable wording, question spacing, structuring of answers and all other instructions that will help the interviewers in the interview process.

Self Check Exercise-3

What is pre-test?

A very important part of the questionnaire construction process is known as pre-testing. Pre-test is the final step in the questionnaire/schedule (tools for collecting data) development process. Pre-testing a draft questionnaire/schedule in a small pilot study allows the researcher to ascertain how well the questionnaire works. This involves testing research instrument in conditions as similar as possible to the research, but not in order to report results but rather to check for problems in wording of questions, lack of clarity of instructions and in fact, anything that could pose problem to the instrument’s ability to collect data in an economical and systematic fashion. Pre-tests should be conducted systematically, with potential respondents and using the same method of administration. The temptation to hurry over them, using just a convenience sample, should be avoided. It is also beneficial to pre-test the questionnaire with specialists in question construction, who may be able to pick up potential difficulties which might not be revealed in a pre-test with respondents. If there are a variety of respondent types, all should be included in the pre-test, and if the questionnaire is to be in several languages, it should be tested in each language.

What are the fundamental issues in Pre-testing?

There are five fundamental issues with respect to pre-testing. These are:

What specific item should be pre-tested?
What method should be used to conduct pre-test?
Who should do the pre-testing?
Who should be the subjects in the pre-test?
How large a sample is needed for the pre-test?

Summary

Pilot study and pre-test are important components of survey research, affording researchers a valuable opportunity for reflection and revision of their research. Pre-testing is the method of validating the survey instrument and its measurement while pilot testing is the dress rehearsal of survey administration and procedures.

you can view video on Pilot Study and Pre-test

Web Sites

www.ncbi.nlm.nih.gov/pubmed/12216297
sru.soc.surrey.ac.uk/SRU35.html
http://www.jstor.org/stable/3151627
srmo.sagepub.com/view/the-sage-encyclopaedia-of…research../n715.xml
uni-mysore.ac.in/courseworklisc/ppt/Manjunatha%20sr.ppt
www.slideshare.net/yogeshnamdeo/pilotstudy-34346040
uir.unisa.ac.za/bitstream/handle/10500/164806chapter5.pdf
http://www.nets.nihr.ac.uk/glossary

References

Backstrom, Charles H and G.D. Hursh. Survey Research. Evanston, IL: Northern University Press, 1963.
Baker, T.L. Doing Social Research. New York: McGraw-Hill Inc, 1994.
Beck, Michael S Lewis, Bryman, Alan, Liao, Tim Futing. The Sage Encyclopedia of Social Science Research Methods. Thousand Oaks: Sage Publications, 2000.
Bowling, A. Research Methods in Health: Investigating Health & Health services. Buckingham: Open University Press. 1997.
Boyd, Harper W. Jr., Ralph Westfall, and Stanley F. Stasch. Marketing Research-Text and Cases, Homewood IL: Richard D Irwin, Inc, 1977.
Brown, Lyndon O, and Leland L Beik. Marketing Research-Methodological Foundations. Hinsdale IL: The Dryden Press, 1969.
Burns, N. and Grove, S.K. Understanding Nursing Research. Philadelphia: W.B. Saunders Company.
1999.
Chann, S.S. “The epidemiology of unpublished randomised controlled trials”. Clinical Research 30 (1982): 234A.
Crombie, I. K. and Davies, H.T.O. Research in Health Care: Design, Conduct and Interpretations of
Health Services Research. London: Wiley, 1997.
Crosswaite, C. and Curtice, L. “Disseminating research results-the challenge of bridging the gap between health research and health action”. Health Promotion International 9 (1994): 289.
de Vaus, D.A. Surveys in Social Research. London: UCL Press, 1993.
Dickersin, K. “The existence of publication bias and risk factors for its occurrence”. Journal of the American Medical Association 263 (1990): 1385-1389.
Ferber, Robert and P.J. Verdoorn. Research Methods in Economics and Business. New York: The Macmillan company, 1962.
Frankland, J. and Bloor, M. Some issues arising in the systematic analysis of focus group material. In Developing Focus Group Research: Politics, Theory & Practice, edited by Barbour, R. and Kitzinger, J. London: Sage, 1999.
Galtung, Johan. Theory and Methods of social research. New York: Columbia university press, 1969.
Holloway, I. Basic Concepts for Qualitative Research, Oxford: Blackwell Science, 1997.
Hundley, V, Milne, J, Leighton-Beck, L. et al. “Raising research awareness among midwives and nurses: does it work?” Journal of Advanced Nursing 31, 1 (2000): 78-88.
Lindquist, R. “Don’t forget the pilot work!” Heart Lung: 20, 1 (1991): 91-92.
Mahoney, M.J. “Publication prejudices: an experimental study of confirmatory bias in the peer review
system”. Cognitive Therapy Research 1 (1977): 161-175.
Mason, D.J. and Zuercher, S.L. “Pilot studies in clinical nursing research”. Journal of the New York State Nursing Association, 26 (1995): 11.
Misakian, A.L. and Bero, L.A. “Publication bias and research on passive smoking”. Journal of the American Medical Association 280, 3 (1998): 250-253.
Muoio, R., Wolcott, L., and Siegel, H. “A win-win situation: The pilot program”. Journal of
Continuing Education in Nursing 26 (1995): 230-233.
Oxman, A.D. Cook, D.J. Guyatt, G.H. “Users’ guides to the medical literature. VI. How to use an overview. Evidence-Based Medicine Working Group”. Journal of the American Medical Association 272 (1994): 1367-1371
Peat, J., Mellis, C., Williams, K. and Xuan W. Health Science Research: A Handbook of Quantitative
Methods, London: Sage, 2002.
Peters, D. and Ceci, S. “Peer review practices of psychological journals. The fate of published articles, submitted again”. Behavioural & Brain Sciences 5 (1982): 187-255
Polit, D.F., Beck, C.T. and Hungler, B.P. (ed). Essentials of Nursing Research: Methods, Appraisal
and Utilization. Philadelphia: Lippincott Williams & Wilkins, 2001.
Prescott, P.A. and Soeken, K.L. “The potential uses of pilot work”. Nursing Research 38 (1989): 60-62.
Rosenberg, K.M. and Daly, H.B. Foundations of Behavioural Research: A Basic Question Approach,
Fort Worth/London: Harcourt Brace College Publishers, 1993.
Schreiber, B James. The sage Encyclopaedia of qualitative research. New Delhi: Sage Publications, 2008.
Shelby D. Hunt, Richard D Sparkman, Jr and James B. Wilcox. “The pre-test in survey research: Issues and Preliminary Findings”. Journal of Marketing Research Vol 19, No 2 (1982): 269-273.
Smith, R. and Roberts, I. “An amnesty for unpublished trials”, British Medical Journal 315 (1997):
622.
Tashakkori, A and Teddlie, C. Mixed Methodology: Combining Qualitative & Quantitative
Approaches, Thousand Oaks: Sage, 1998.
Teijlingen van, E., Rennie, A.M., Hundley, V., Graham, W. “The importance of conducting and reporting pilot studies: the example of the Scottish Births Survey”, Journal of Advanced Nursing 34 (2001): 289-295.
Van Teijlingen, Edwin and Hundley, Vanora. Social Research Update. Issue 35, University of Surrey, 2001.
Zaltman, Gerald and Philip C. Burger. Marketing Research: Fundamental and Dynamics, Illinois: The Dryden Press, 1975.