Thought I’d put up a couple of initial thoughts/feed-back on the testing:
First off it’s becoming clear that some of the questions are ambiguously worded. This is especially obvious in the results for questions 12 and 14.
Question 12, looking at categorizing sources, is worded in such a way that it is not clear whether the question refers to the story extract itself or the sources used in the story extract. This means that respondents incorrectly label the source as “Primary” (correct for the sources used for the story extract) and “Tertiary” (correct for the story extract itself and therefore the correct answer for the question).
The other one that people are obviously getting wrong because of the wording (including myself) is question 14. This question asks what element of a study design is not a strength of the study.
This implies that you are to critique the design as it is actually presented, not how it could have been. Thus people are choosing the option that is “least wrong”. A bit of a change in this wording to make it clear what design could have been used but wasn’t or that could have made the study better or even restricting the answer options to just the study elements present in the background information would probably bring the score for this question up.
Interestingly there are a few questions nobody has gotten wrong, indicating they may be a little too easy (but perhaps the sample size is still too small yet, 45 responders so far).
The first question “Which of the following is a valid scientific argument?” has a 100% responder correctness score. As does Question 16 on the proportions of house building materials and question 20 on the rat population. Question 27 “Which of the following actions is a valid scientific course of action?” also has a 100% score.
So, great stuff so far, as I mentioned there’s been about 45 people taking part so far and things already are shaping up nicely. So spread the word and lets see how many people we can get. If possible it would be nice to get constructive criticism on the question wording like I have done above that can be feed back to the original test designers.
Thanks for the interest so far and keep it up!