Yesterday’s post about the Science Literacy test has gotten some good responses.
Thought I’d put up a couple of initial thoughts/feed-back on the testing:
First off it’s becoming clear that some of the questions are ambiguously worded. This is especially obvious in the results for questions 12 and 14.
Question 12, looking at categorizing sources, is worded in such a way that it is not clear whether the question refers to the story extract itself or the sources used in the story extract. This means that respondents incorrectly label the source as “Primary” (correct for the sources used for the story extract) and “Tertiary” (correct for the story extract itself and therefore the correct answer for the question).
The other one that people are obviously getting wrong because of the wording (including myself) is question 14. This question asks what element of a study design is not a strength of the study.
This implies that you are to critique the design as it is actually presented, not how it could have been. Thus people are choosing the option that is “least wrong”. A bit of a change in this wording to make it clear what design could have been used but wasn’t or that could have made the study better or even restricting the answer options to just the study elements present in the background information would probably bring the score for this question up.
Interestingly there are a few questions nobody has gotten wrong, indicating they may be a little too easy (but perhaps the sample size is still too small yet, 45 responders so far).
The first question “Which of the following is a valid scientific argument?” has a 100% responder correctness score. As does Question 16 on the proportions of house building materials and question 20 on the rat population. Question 27 “Which of the following actions is a valid scientific course of action?” also has a 100% score.
So, great stuff so far, as I mentioned there’s been about 45 people taking part so far and things already are shaping up nicely. So spread the word and lets see how many people we can get. If possible it would be nice to get constructive criticism on the question wording like I have done above that can be feed back to the original test designers.
Thanks for the interest so far and keep it up!
Scepticon 
This means that respondents incorrectly label the source as “Primary” (correct for the sources used for the story extract)
While the research falls within my definition of primary source, it doesn’t come within the definition you give of primary source. There is nothing to indicate that the research had been submitted to a peer-reviewed journal, which the definition you provide requires.
While technically true I think the question and info provided are sufficiently worded to make that implication clear.
It left me wondering, as a non-scientist, whether this was a scientific term of art with which I was unfamiliar. The information is sufficiently clear for the purposes of the question, but I may now have an inaccurate understanding of what a primary source of information is. Does the definition of primary source, when used in a scientific sense, require submission to a peer-reviewed journal? I wouldn’t have thought it did, but now I don’t know.
Yeah, I think it really does require that. Though I’m not sure if there exists any definition that makes that “official” (there could be I just don’t know). But it is the scientific “Literature” (ie the peer reviewed journals) that is primary, not the studies that happen to be in people’s file draws.
I’ll look into that though, thanks.
Okay – that was just something I didn’t know. As a non-scientist, I would have thought that the results of an experiment could be a source. Perhaps not a good source, but one, nonetheless.
If that’s not how science does things, that’s okay. This is science literacy, after all!
I’m a bit flummoxed by the evaluation of my response to question 26, to wit:
“For Q26 your answer was
It is not a credible source of scientific research because only studies
showing the effectiveness of the company?s drugs were included in the
journal.
and the correct answer was:
It is not a credible source of scientific research because only studies
showing the effectiveness of the company’s drugs were included in the
journal.”
Please attempt to unflummox yourself. If you will direct your attention to the question mark where an apostrophe should be in “company’s” you will see the reason for the erroneous incorrect answer.
Some browsers seem to display some of the symbols incorrectly and the script that checks the answers does a simple comparison on the text. If it deviates in any way then it’s wrong. But we know the truth and you can mentally increase your score by one.
I thought I’d fixed all of those issues but apparently not. Thanks.