Last week an interesting paper was published outlining the development of a test designed to evaluate science literacy. The basic idea of the test is to examine a number of different skills that are involved in evaluating scientific claims and facts.
I liked this idea so much I decided to put together an online version of the quiz that people could take and get instant feedback for. This could also be adapted for use in an educational environment as intended in the original paper.
To that end I need beta testers, people who would like to take the test and give me feed-back on how to improve it. A couple of things I need to mention at this point:
First, the test will ask for your email address in order to send a summary of how well you did on the test. Your email address will not be used for any other purpose and indeed I’ve set up the back-end so that the address will be deleted as soon as your email is sent – your private information will remain that way.
With this in mind – If you have feed-back about your performance, there is no way for me to tell which entry is yours unless you give me the time you submitted the form.
Secondly, I have used some custom html code in creating the form which doesn’t integrate seamlessly with the google drive document that runs most of the functionality. This means if you miss a required question (all of them are required FYI) than you will be re-directed to the original form of the test, this will look a bit different and will not contain the pictures (but will have urls you can use to get to them).
You can either continue with this version of the form or hit your browser’s Back button and fill out the questions you missed there.
Finally, If you have and comments or suggestions leave them here and I’ll see what I can do. Also I am actually fairly mediocre at html coding and such like so if there is anyone who wants to volunteer their services to upgrade the quiz I’m happy to share the load :).
Ok, here’s the link to get you started, have fun.
[Edit: Here’s a link to some initial analysis of the results]
Scepticon 
Interesting. Have passed on to daughter doing GCSE exams at school in U.K. Suggested she shows to Science Teachers and see if something of interest for them to use. Managed 25 out of 28 which was quite pleased with for a non-scientist. Guess at least some of my skeptical reading has sunk in, along with a smattering of school learning many years ago. Would be great if it spread further. Has it been touted at JREF? Maybe send to them?
Yeah, I got 25/28 too.
Showing a science background isn’t necessarily beneficial either :).
Would be great if more teachers got to see this so thanks for that.
JREF huh? good suggestion. I don’t have much to do with those folks directly (though I think they’re great) so I’ll have to find an “in”. You you’ve already got one feel free to send details..
Thanks a lot.
Interesting idea and good to see some effort in this area. Some feedback regarding the test:
– My score was 24/28 and I am a working scientist
– IMO, the wording of some questions is confusing (e.g. Q4, Q7, Q14, Q21), perhaps due to questions phrased in the negative, e.g. “which of the following is *not* a strength of the trial” rather than “which of the following are weaknesses of the trial”.
– In Q14, none of the options seem to be strengths of the trial (2500 isn’t that large, there’s no indication that participants were chosen randomly, and participants were not randomly assigned to groups). However, “All of the above” was marked as incorrect in my test. It would be good to clarify this question.
– In Q15, it’s debatable whether graph C or D is better. No information is given about how stress was measured in the study. One would hope real studies are not so crude as to characterize a complex issue like stress as simply “high” or “low”. I think a good study should be designed to produce a graph like option C (this answer is marked as incorrect)
– The quiz is oriented strongly towards biological sciences. It would be nice to include questions from the physical sciences.
– The quiz would be quite challenging for members of the general public
Hope this feedback is useful.
RE ambiguous wording. Agreed, see today’s post (https://scepticon.wordpress.com/2013/01/23/science-literacy-test-initial-follow-up/)
I think diversifying the question topics would be a great idea. I suspect the bio focus is a consequence of the specialities of the initial authors, got any suggestions?
Spread the quiz to more of the public and maybe we’ll find out.
Great feedback, thanks.
Awesome quiz – What are you planning to do with the results? I would be curious to see a split of the results between different demographics, particularly people with a science vs non-science background, so maybe you could add some fields to collect that data also?
I’m thinking so, initially this was just supposed to be a de-bugging run and to see what people thought of the questions. But it’s taken off a bit so I am thinking of adding demographic data. Trouble is now that it’s getting new entries every couple of minutes it’s kind of hard to update :).
I’ll see if I can update an offline version an then quickly swap out bits all at once…
Fun quiz. Do you think that Q22 should stipulate that the outside researchers are neutral?
Good question. I think the question works as it is (ie even if the scientists weren’t neutral they might still be more reliable than the other sources) but it is something that is assumed by “outside” rather than explicitly stated for complete clarity.
So thanks for that, I was just mentioning on another site that I wanted insights exactly like this one.
VERY BLOODY ANGRY. Took the test then asked me to log into google and got rid of my answers.
I’m sorry you had problems. No one else has mentioned any similar issues and you shouldn’t need to log into google to participate. At what point did you have problems? On submitting or pressing the continue button at the end of the first page. If you didn’t not complete every question then this would have caused problems.
Did the test but couldn’t get my answers 😦 … Maybe continue button is broken? I am using google chrome.
hmm, yeah something is off. I’ll see what I can do.
I liked the test and I did it. BUT when I hit the continue button nothing happened. Neither on my PC with Firefox nor on the Ipad with Safari.
May be you could post a list of answers.
Yes, sorry. Looks like google changed something about thier drive scripts rendering this one inactive. And I’m not expert enough to fix it.
I’ll post the answer so that scores can be calculated manually – good idea.
the same thing happened to me: nothing happens (Safari 7.0.6); sad, because i wanted to make this (really nice!) test with my students these days…
Here are the answers, sorry again it isn’t working. I’m willing to share the back end with anyone who can help 🙂
1 A strain of mice was genetically engineered to lack a certain gene, and the mice were unable to reproduce. Introduction of the gene back into the mutant mice restored their ability to reproduce. These facts indicate that the gene is essential for mouse reproduction.
2 Graph C
3 Conduct a statistical analysis to determine if females live significantly longer than males.
4 To evaluate the effect of a new diet program, researchers compare weight loss between participants randomly assigned to treatment (diet) and control (no diet) groups, while controlling for average daily exercise and pre-diet weight.
5 Several research studies have found a new drug to be effective for treating the symptoms of autism; however, a government agency refuses to approve the drug until long term effects are known.
6 The Malathion killed the beetles, causing fewer tadpoles to be eaten.
7 Type “A” mice with Lymphoma were more common than type “A” mice with no tumors.
8 Compared to users of standard dumbbells, users of the Shake Weight were able to lift weights that were significantly heavier at the end of an 8-week trial.
9 Scientists are selected to help conduct a government-sponsored research study on global climate change based on their political beliefs.
10 The resource may not be accurate, because the purpose of the site is to advertise a product.
11 Results from an experimental study demonstrated that individuals randomly assigned to consume one diet soda each day were twice as likely to have a stroke compared to those assigned to drink one regular soda each day.
12 Tertiary (Media reports, encyclopedia entries or documents published by government agencies.)
13 All of the above
14 Randomly assigning participants to control and experimental groups.
15 Figure D
16 36%
17 the article was evaluated by unbiased third-party experts
18 The largest increase in meat consumption has occurred in the past 20 years.
19 The uncertainty in the estimate of the actual mean caffeine content will be larger in study 1 than in study 2.
20 48%
21 The decreasing trend in violent crime rates may be caused by something other than violent video games
22 a research study conducted by outside researchers
23 500
24 Researchers are making inferences about a population using estimates from a smaller sample.
25 yearly screening data on autism symptoms for immunized and non-immunized children from birth to age 12
26 It is not a credible source of scientific research because only studies showing the effectiveness of the company’s drugs were included in the journal.
27 The scientific journal, Science, retracts a published article after discovering that the researcher misrepresented the data.
28 River shrimp are more abundant in lower elevations because pools at these sites tend to be deeper.
The idea that a scientist is somehow less biased than any other member of professional society is manifest NONSENSE. Science is a human driven activity. Scientists have agendas are just as baised as anyone else. Much misconduct, gatekeeping and other political shenanigans go on. Paul Davies notes all of this…
There is PLENTY of physics out there left UNDISCOVERED. All theories and laws are provisional and perfectible. Modifications are fully expected in the future. We do not know enalry enough . Major HUMILITY is needed by these stuffy science types.
Scientists are biased, very true, everyone is. But science is self correcting in that the individual biases tend to balance out and if there is a name to be made by refuting someone else’s work then that is what will happen.
Humility is required, and in the scientists I know, very much in evidence. Know any scientists? I do and the word “stuffy” does not come to mind.
Hmm… it doesn’t seem to Work anymore …
yeah, I don’t have the skills to debug. I guess that is the price I pay for using a platform that can change at any time.
I like this test and think it should be shared in social media. I have put together a simpler one written by Richard Carrier and posted on the secular web infedels.org http://infidels.org/library/modern/richard_carrier/SciLit.html#5
I looked at a couple web enabled testing sites and think quizbean is the most attractive and simple for this task. Please take it here:
quizbean.com/r/qb2NyBbpW2YV
I will work on moving this TOSLS test on this format to see what we can learn.
Cool, would love for this to be more widely known. I couldn’t seem to access the quizbean page without creating an account (which I haven’t done yet). is this right? Can quizes be taken by people without registering? Seems like that would be a significant barrier.