Setting: Comparisons in performance of questionnaires are of interest in international comparisons of prevalences and risk factors, after translation and development of new questionnaires. Factors such as sex, age, educational level and smoking habits may influence the performance of questionnaires.
Objective: 1) To discuss questionnaire comparisons in general, using as an example a Norwegian respiratory symptoms questionnaire compared with a translation of the British Medical Research Council questionnaire on chronic bronchitis; and 2) to examine whether reliability differs in subgroups.
Design: A population sample of 935 residents of Hordaland County, Norway, completed two questionnaires in a short interval of time. Agreement and Cohen's kappa were calculated.
Results: Prevalences were significantly different between the two questionnaires for most symptoms. Agreement decreased from non-smokers through ex-smokers to smokers, whereas the kappa statistic increased. Agreement increased as the educational level increased, whereas the kappa statistic declined. No significant age and sex effects on agreement and kappa were observed.
Conclusion: Small changes in the phrasing of a question can have considerable effect on prevalence estimates. Reliability statistics are prevalence dependent. Reliability may differ according to smoking habit and educational level. Questionnaires should be tested in the population that they are to be used in.