Prolific: Recruiting participants for online Psychology studies

I’ve just run a research study in which I paid to recruit participants for the first time. I used the online Prolific recruitment website and it went really well. So I thought I’d share my experience.

Using Prolific for recruiting participants to my PhD studies

Prolific is a website for recruiting human participants to research studies. People (participants) sign up to do studies, get notified when studies for which they are eligible are available, and get small amounts of money in return if they participate in a study.

According to Prolific’s About page, the company was set up in the last 2-3 years by a former PhD student, Katia (and her friend Phelim), who had struggled to recruit human participants for her research. So the focus is very much on recruitment for research studies with a personal appreciation of what’s involved in such activities. As I write this, the participant pool seems to include about 37,400 people and they have recently been upgrading their website so it seems to be going well.

How does it work?

Prolific does not host the studies themselves. You can set up your actual study using all kinds of software; for example, a survey on Qualtrics or Survey Monkey, or an experiment on Gorilla. Prolific just helps you recruit participants to it.

You can register for Prolific with a researcher account or a participant account (or both). As a researcher, you can set up studies and add credit to your account to run them. As a participant, you provide some personal information so that Prolific can offer you only studies for which you are eligible. If you participate in a study, you get paid for your time and you can receive that money yourself or donate it to charity.

How do you use Prolific as a researcher?

When you’ve set up your online study outside of Prolific, you create a new study in Prolific and give it details of your study, including the URL to the study itself:

I set up my study in Gorilla. Gorilla integrates nicely with Prolific in that it generates a URL that collects participants’ Prolific IDs automatically (so the participant doesn’t have to manually enter it). As a researcher, you can also enter the Prolific ‘completion URL’ into your Gorilla experiment so that participants don’t need to manually enter a completion code into Prolific before they can receive payment. This mostly worked seamlessly for me, though I messed up the set up at one point because I accidentally limited my Gorilla study to one fewer participant than on Prolific; don’t do that, it means that Prolific sends the last participant to Gorilla which then rejects them and Prolific sends another participant, and so on…

As you can see in the screenshot, Prolific calculates up-front the cost of your study. The cost depends on the number of participants you want to recruit and how much you want to pay them, plus Prolific’s commission. Prolific enforces a minimum payment to participants of the equivalent of £5 per hour (I paid participants £1.25 for a 15-minute study; some took longer than 15 minutes, most took much less but all received the same amount).

You then provide a textual description of your study so that potential participants can decide whether they’re interested in taking part. After that, you can optionally select to ‘prescreen’ participants according to their basic demographics or other more specific features of their lives:

I selected that participants should be UK residents which, Prolific helpfully informed me, restricted me to about 18,000 people in the participant pool. Aside from basic demographic details, most of the prescreener questions are optional for participants to complete (though not completing them restricts how many studies they’re eligible to participate in). The fewer prescreeners you include in your study, the more potential participants are eligible to do your study.

Prolific are quite firm that you must not include screening questions in your actual study (e.g. asking participants if they are a certain age and ending the study if they are not). Instead, you must use the prescreeners so that ineligible participants don’t even get offered your study. This is because it gets really annoying, as a participant, to be offered a study and then start it, only to then be told you’re not eligible.

Finally, you have to confirm that you’ve tested your study and various other things. I specified not to include the page in an iframe. When, as a participant, you start a study, Prolific displays a panel above the study that contains your Prolific details. For studies where the participant has to manually enter their Prolific ID etc for tracking their participation, that’s maybe useful. For my study, though, Gorilla handled all that automatically and an extra panel on the page just unnecessarily used up screen space.

You can now publish your study, as long as you’ve credited your account with enough money to cover the calculated cost (you can request a refund for any credit you don’t spend). At this point Prolific displays your study to eligible participants and email subsets of eligible participants to notify them that there’s a new study they can take part in. It’s quite good fun watching the live dashboard update as participants start your study:

Prolific keeps recruiting until it reaches your target recruitment number (21 in the screenshot above). You then have 21 days to ‘approve’ participants so that they get paid. Prolific has a few criteria you can legitimately use to approve or reject participants. I included some ‘attention questions’ in my study and only participants who got a certain number correct were paid (in practice, all of them were fine). I also did some other checks but ultimately accepted all the complete sets of data.

One participant, for some reason, was not presented with all the questions but otherwise completed the study. This appeared to have been a weird technical blip in the study itself so I approved the participant even though I couldn’t use their data because it wasn’t their fault. I also, separately, gave a bonus payment of 25p to one participant who had tried to take part but had been bounced out of my study because of my mistake in setting it up (see above) and they contacted me to let me know.

I ended up running the study in Prolific three times. The first time had participants whitelisted to just my participant ID (I’m registered with a participant account as well as a researcher account) so that I could test it (the recommended way to test that Prolific integrates properly with your study software). The second time, I collected 20 data from participants and then checked that everything was going okay. I then approved their payments but that automatically ‘completed’ the study so I couldn’t just add 21 more participants to the recruitment target. Instead, I had to create the study again by just duplicating it in Prolific (which retained all the same details to integrate with Gorilla) and then screening out anyone who had taken part previously. This worked fine but was a bit unnecessary and annoying. The workaround is to ‘pause’ your study before approving and adding additional participants, then unpausing the study to continue running it with the new recruitment target.

All in all, it’s all pretty easy to use, though it’s worth reading relevant parts of the Prolific documentation to understand how it works and what it can do for you, especially with prescreening and with integrating Prolific with your online study software. I was a bit slow setting up my first study but in future it will be quicker.

Isn’t the sample biased by recruiting through a website like Prolific?

All samples are biased unless they’re completely random and, even then, randomly-selected participants will drop out (or just refuse to take part) so you get some bias of self-selection. This happens in all research that involves human participants. What’s important is that you try to get as representative and suitable sample as possible for the population that you are studying.

My research is on people’s perceptions of household energy. Because experiences of household energy vary according to the country you live in (e.g. in the US, aircon is far more prevalent than in the UK), I decided to design my studies for people with experience of living in UK households. A large proportion of Prolific’s participant pool is UK-based, which suits my studies well.

The demographics of Prolific’s participant pool are biased towards Caucasian participants (though about representative of the UK) and towards younger and middle-aged people. I think it can be assumed that it is also implicitly biased towards people who are willing, comfortable, and able to use websites to participate. Interestingly, despite the large proportion of students in Prolific’s participant pool, the majority of participants in my study were not students (which was perfect for my studies). If you’re interested, Prolific have a collection of links to resources about online versus lab-based studies.

For my study, Prolific was great. I’m getting close to the end of my PhD and I just need to run some small, exploratory, online studies quickly on people who live in UK households. To check the findings of these studies with other sub-demographics (eg older people in the UK, greater ethnic diversity of people in the UK, people in the UK who are less comfortable with using websites and computers, people not in the UK), in future studies, I would need to find another method of recruitment to complement this one.

How is Prolific different from Amazon’s Mechanical Turk?

I did look into using Amazon’s Mechanical Turk (MTurk) after a friend’s positive experience of using it for her own PhD study. MTurk is a similar kind of service but for any type of work that can be done online (not just research studies), though researchers have taken to using it quite a lot. The problem for me was that MTurk’s participant pool is mostly in the US and India and I needed to recruit UK residents. Prolific describe that and other differences between them and MTurk. They also provide a link to an independent study that found Prolific was generally better than MTurk for research studies (at least along criteria that I cared about).

Isn’t there a danger of recruiting only professional study participants?

Prolific claims to avoid this problem (which has been observed on MTurk) by notifying different subsets of eligible participants so that it isn’t just the fastest people in the participant pool who get to participate in all the studies all the time.

Any problems?

I was initially uncertain about how well my study would go because I’d also registered as a participant to get a feel for the experience from that perspective (I recommend doing this) and experienced a couple of problems. These were partly technical problems caused by the site upgrades. I’d also wondered how reliably participants got offered the studies because, as a participant, I’d had to complete many, many prescreener questions before being offered a very small number of studies (though I think this was maybe because of Prolific’s policy of not encouraging the same few participants to do all the studies).

Ultimately, my study was fine and I recruited my initial 20 participants in about 18 minutes, which was amazing! And they seemed to be fairly representative of the participant pool with a greater range of ages than I’d expected.

The other main problem was that I discovered that, as a researcher, I could download a lot more information about my participants than I’d expected or was ethically cleared to obtain. As both a researcher and a participant, this made me uncomfortable. However, I emailed the team and they quickly investigated and addressed the problem, prioritising it to get fixed within a few days. Researchers now have access only to a limited set of non-identifying data about participants plus the participants’ responses to any prescreeners that were selected for the study.

The Prolific support team has been brilliant. You can contact them by email or there’s an in-site messaging system; if there’s no one available, they’ll email you later. They’ve responded helpfully to every contact I’ve made and they regularly update their help/FAQ system.

Is it worth using Prolific?

I will definitely use Prolific again for another study in the next few weeks so I, obviously, encourage you to sign up as a participant. 🙂 Based on my overall positive experience as a researcher, I recommend it to other researchers and students as an option to consider for their own studies. If you want to give it a go, it’d be great if you could use my Prolific referral link which gives me credit towards future studies I run.

Why I love parkrun

Parkrun is a weekly, timed, 5km run. It happens every Saturday at 9am in England and Wales (and many other countries) and at 9.30am in Scotland. It’s run by volunteers and is free to enter. You register once (at parkrun.org.uk) and print out your barcode. Then turn up at a parkrun with your barcode and run, jog, or walk the 5k course. When you cross the finishing line, you receive a second barcode and someone scans both barcodes. Then, later that day, you get a txt message and email telling you your time.

DSC_0229

There are currently 784 different weekly parkruns across the world. 376 are in the UK, where parkrun began, and the rest are spread across 10 other countries around the world. It all started in 2004 with just 13 runners and four volunteers at Bushy Park in London. As the name suggests, parkruns are never on roads, or crossing roads. And it’s a run, not a race; the emphasis is on doing your personal best (whether that’s finishing in a particular time or just taking part at all).

How I started parkrunning

I never wanted to run. I had no interest in running. I wasn’t even that keen on exercise, except for walking into town (or, for a brief time, cycling to work sometimes). Friends kept posting annoying photos on Facebook of their parkrun achievements on a Saturday morning while I was still sleeping.

Then I randomly heard an interview with Paul Sinton-Hewitt, founder of parkrun, on Radio 4 last summer. After hearing another parkrun feature on the radio around Christmas time, I found myself running three times round a cold, muddy field in Winchester with my friend, Gemma. That was the 17th January and that was my first parkrun. Two weeks later, I was staying with my runner friend, Wing, in London and we did the Greenwich parkrun. It was cold, muddy, and snowy that time. Then I did the Winchester parkrun every two weeks until I went on holiday to the US in April.

IMG_9582

In Raleigh, North Carolina, I dragged my friend Chris along to his first parkrun in Durham, NC (8am start there!). We thought it was amusing that a man from Stockport, UK, had driven through the night from a business trip in Atlanta to run the Durham parkrun before driving back again. That really seemed beyond commitment…

Then I ran every Winchester parkrun between my US holiday in April and my Sicily holiday in June. Then my parents drove 3 hours with me from where we were staying in Sicily to stay overnight in Palermo so that I could run the new Uditore parkrun on the Saturday morning.

At this point, I accepted that I was hooked. Not just on parkrun but on running in general. It turns out, I actually enjoy running.

What I love about parkrun

Stats! For every parkrun you finish, your personal stats page on the website gets updated with the event you were at, your time, your position, and various other things. Something like this:

stats

When I started parkrunning, my goal was to get to the end without stopping or walking. I had actually run 5k about half-a-dozen times a few years earlier and I knew that I could complete it without stopping as long as I went really, really slowly (basically jogging at a brisk walking pace). After my fourth parkrun, I discovered that I wasn’t quite so knackered at the finishing line and started to go a little bit faster. If you get a PB (personal best time) at a particular parkrun, that’s recorded on the website too.

While I enjoyed watching my PBs coming down, I was actually more motivated by the ‘collecting’ aspect of parkrun. That was what helped me get out of bed on a Saturday morning and get down there. Every parkrun you attend is recorded. When you have done 50 parkruns, you get a free t-shirt. Then another free t-shirt at 100, 250, and even 500 parkruns (just one, very dedicated, man has that one so far). By the end of 2015, I’d done 21 parkruns and volunteered at 11 (volunteering is recorded too and has its own free t-shirt to aim for).

Attempting (and not quite succeeding) to get a PB at Southsea parkrun in June.
Attempting (and not quite succeeding) to get a PB at Southsea parkrun in June. (Photo thanks to Southsea parkrun’s photographer on 13/06/15.)

After the first couple of times, getting up ‘early’ on a Saturday morning wasn’t so bad. My parkrunning friends, Tim and Andy, actually claim better times when a bit hungover or tired. And you can always (family permitting) go back to bed afterwards. According to Facebook, it’s not just me who often ends up napping on a Saturday afternoon after parkrun in the morning.

I also love the friendly and sociable atmosphere, especially at the smaller parkruns or if you volunteer. Runners are all ages; women; men; children running alongside; children in prams; dogs on leads. By my 8th or 9th parkrun, I’d caught up with a 60-year-old regular runner and, as I got faster, an 11-year-old.

What I learnt about myself

I actually enjoy running. That was a surprise! Not crazy marathon distances with all the training that involves but I quite like running 5-10k distances. In particular, I love sprinting. No matter how tired I am running the distance, I can almost always find the reserves to sprint the last 100m; kind of like my main course tummy being full but still having space in my dessert course tummy.

I discovered that my body can do amazing things that I didn’t believe back in January. In just 6 months, I dropped 10 minutes from my PB. I completed my first parkrun in January in 38:11 and couldn’t imagine being able to do it in under 30 mins. But I got there in July.

#ThisGirlCan run 5k in less than 30 mins! (In fact, we both did that week.)
#ThisGirlCan run 5k in less than 30 mins! (In fact, we both did that week.)

It turns out I’m quite determined! Not only in running distances in a given time but also in finding ways to get to local parkruns whilst on holiday, even if they’re not that nearby. Running, mostly at parkrun, improved my self-confidence.

The runner’s ‘high’ is definitely a thing; once I started enjoying running, I found that I usually felt great after a run, even if I was really tired. I also found running to be calming. And I’ve made new friends both at parkrun and through the (it turns out) masses of people at work who are into running.

It turns out, too, that I’m a sucker for gamification. Strava is both a blessing and a curse. It’s great for the sociable aspect of running and for challenging yourself to get better times. But getting ‘kudos’ from your Strava friends and badges from Strava can be a bit too encouraging at times. That, combined with loving the social aspect and the physical wellbeing I got from running, led me to overdoing it and getting shin splints (bruised feeling in my shins). A common running rookie error of doing too much too soon. In my case, I think it was sprinting downhill on tarmac during lunchtime runs, on top of running too often and too far (in addition to parkruns) over about 3-4 weeks in the summer.

What I learnt about running

The main thing is to build up gradually. I knew from the start that I could run 5k very slowly so I just gave it a go. As I got fitter, I started to get faster. By running, I could feel myself getting fitter, physically and mentally. So in June I started running more with friends, not just at parkruns.

I didn’t understand that, to do a lot of running, you need to be quite fit. The muscles that you need to be strong when running (eg ankles, calves, thighs, buttocks) don’t really get stronger by running alone (or not very quickly anyway). It’s important to have strong abdominal (‘core’) muscles and glutes (buttock muscles) in order to have better posture and control over your movement. I didn’t have very strong any-of-these muscles so when I suddenly increased the amount of running I was doing, my shins got overstressed.

I had been warned to build up gently but I didn’t really understand why. Moreover, I didn’t realise that any injury might not reveal itself until after I’d stopped running. I’d assumed it would hurt whilst running but, although I was tired on my last run, my shins (and knees) didn’t start hurting until later that day and over the next few days.

2015-08-15 09.06.47

Frustratingly, my shins still aren’t recovered enough to run any more than a gentle walk/jog yet. I now go to the gym three times a week to do both cardio exercises (like rowing machine) and weights, as well as doing some stretching exercises that the physiotherapist recommended. As a result, my knees are better (my thigh muscles are now less tight) and I’m generally stronger, fitter, and I have better balance (I can stand on one foot without wobbling loads!). But I still have some way to go.

So, moral of the story: don’t go mad! Other moral of the story: cross-training (doing non-running exercise too) is important for running. I think I would probably have been okay if I’d stuck to just doing the parkruns each week as that was a nice distance each week and gave me plenty of rest time in-between. I have no expertise in any of this, of course, and have just gleaned this information from my GP, the physiotherapist, gym instructors, and online. If you want to know what the experts say and debate, go take a look online, in particular, Runners World seems pretty good at summarising the latest research and controversies.

Volunteering at parkrun

Being injured and unable to run for a while encouraged me to get involved in the volunteering side of parkrun. All parkruns are organised and run by volunteers each week. The number of volunteers required each week and the roles they take depends partly on the size of the parkrun (that is, the number of runners).

_20150829_101513

The timing technology is neat and it’s all about enabling events to scale up to larger numbers of runners without getting overwhelmed. I’ve only volunteered at Winchester parkrun, which typically has between 150 and 250 runners. I’m told that the nearby Southampton parkrun, which often has around 600 runners, handles the barcode scanning etc at the finishing line with necessary military precision.

2015-08-22 08.50.51

I’ve now done a range of volunteer roles, including marshalling (pointing runners in the right direction on the course and cheering them on), giving out finishing tokens, time-keeping, scanning barcodes, and tailrunning (where you run last on purpose). There are other roles like photographer, run director (responsible for the run that week), and event director (responsible for that parkrun event every week). Each time you volunteer, you get another parkrun recorded on the website. After volunteering at 25 parkruns, you get a free t-shirt.

2015-11-21 08.45.22

How to start parkrunning

To give it a go, find your nearest parkrun (look on your country’s website if you’re not in the UK). Dig out your trainers.

Register on the parkrun website and print out your personal barcode.

Get out of bed on Saturday morning and go along to your local parkrun. You don’t have to run if you don’t feel up to it. You can walk, even if no one else is. Don’t worry about other people’s times; you’re only competing against your own goals, whatever they are.

If you really don’t feel up to it, try downloading a couch-to-5k app and doing that in your own time before you tackle a parkrun.

More than anything, though, enjoy it! And don’t forget your barcode! No barcode, no time.

_20151003_094802

#loveparkrun


This post is based on my OggCamp 2015 talk. The talk slides are on SlideShare.net but the speaker notes are badly presented on there so I decided to write it up properly.

Lord Nuffield and Tampon Club in WWII

Eagle-eyed AndySC was out and about visiting Nuffield Place and spotted this for Tampon Club:

tamponclub-wwII

“an early example of someone taking @tampon_club needs into account!”
— AndySC

Lord Nuffield is the Nuffield of Nuffield Hospitals and the William Morris of Morris Minors. Nuffield Place is his old home, which is now preserved by the National Trust for people to visit and find out about the things he did, including his contributions to the war effort during World War II.