Week 8 practical

Audio, trial data again, preloading stimuli, saving data to the server trial by trial

The plan for week 8 practical

This week we are going to look at code for a perceptual learning experiment based on the experiment described in Levi-Ari (2017) (and in fact using her stims, see below). There’s no new material to look at in the Online Experiments with jsPsych tutorial.

This experiment involves audio stimuli, which we haven’t worked with before, but jsPsych has plugins for audio so that is fairly straightforward. In our implementation we are also going to reuse some code from the word learning experiment to handle randomisation of button positions and using data to keep track of which buttons appeared where. The main new things we’ll introduce (in addition to audio stimuli, but jsPsych makes that very easy) are:

Remember, as usual the idea is that you can work through these practicals in the lab classes and, if necessary, in your own time - I recommend you use the lab classes as dedicated time to focus on the practicals, with on-tap support from the teaching team, but if you don’t get through everything in lab time you should do at least some work in your own time to keep up to speed.

Acknowledgments

Dr. Shiri Lev-Ari (who is the author of Lev-Ari, 2017) very kindly shared the participant instructions and audio stimuli for her experiment and gave me permission to make them public here, so the code uses the same audio files as the real experiment. Shiri wasn’t able to share the visual stims since she didn’t have sharing permission for all of them, but my research assistant Rachel Kindellan did some epic google image searching and found equivalent images which are licensed to be shared - so the images are not identical to the ones used by Dr. Lev-Ari, but should be close enough to give you a feel for the experiment.

A perceptual learning experiment

First: you build it!

As with last week, we’d like to give you an opportunity to try to build (parts of) this experiment yourself, and we’ll provide you with a template that includes some extra stuff (the social networks questionnaire, instructions, preloading of audio stimuli) that we pre-built for you so you can focus on the more interesting parts of the experiment.

As per last week, you need a bunch of files for this experiment - the html and javascript templates for your attempt (which we are calling my_perceptual_learning.html and my_perceptual_learning.js), the html and js file for our implementation (perceptual_learning.html and perceptual_learning.js), a php file (for saving data), and then several folders containing images and sound files. Again, rather than downloading them individually, download the following zip file and then uncompress it into your usual practicals folder:

Extract these files to a folder called something like perceptual_learning, alongside your grammaticality_judgments, self_paced_reading and word_learning folders. As per last week: our implementation saves data on the server, but it won’t save any data for you unless you are running it on the jspsychlearning server, so I suggest uploading it there - upload the whole perceptual_learning folder to your public_html folder on the jspsychlearning server and play with it there (if your directory structure is as I have been using so far, where all the exercises are in a folder called online_experiments_practicals, then the url for your implementation will be https://jspsychlearning.ppls.ed.ac.uk/~UUN/online_experiments_practicals/perceptual_learning/my_perceptual_learning.html and the URL for the final implementation will be https://jspsychlearning.ppls.ed.ac.uk/~UUN/online_experiments_practicals/perceptual_learning/perceptual_learning.html

If you run through our implementation of the experiment you’ll see that, in addition to the usual instructions, the experiment consists of three stages:

Remember when you are running through our implementation, if you look at the data display at the end that will include spoilers on how to write the code yourself, so depending on how much you want to challenge yourself you might want to avoid clicking through to the data! You can also check that our implementation saves data for you in your server_data folder, in a file called perceptuallearning_data.csv (if that file isn’t there after you run our code, try clicking the “Refresh” button in cyberduck to refresh the file list) - but again, beware spoilers!

To implement your own experiment, you need to figure out what plugin(s) you need for picture selection and phoneme categorization trials (think about the stimulus and the response), then look at the documentation for that plugin to see how to use it and/or edit previous bits of code you have written that do something similar. I would recommend starting with the phoneme categorization trials, since they are simpler than the picture selection trials - the picture selection trials involve buttons with pictures rather than text, which we haven’t shown you yet, so unless you want to dig deep in the jsPsych documentation you should just try to implement a simple text-only buttons version of the experiment and then look at our implementation to see how to get buttons with images instead of text.

In our implementation, the sound files and images you encounter (in random order) in the picture selection phase are:

And in the phoneme categorization trial the sounds you hear (in random order) are:

Remember, it’s OK to implement this in a very simple way, with a long manually-constructed trial list - but if you feel more adventurous and want to use some of the ideas that were introduced in the last couple of weeks (nested timelines, functions to build trials, saving data, randomisation of buttons) that’s great too. Give it a go, see how you get on, and ask for help if/when you get stuck!

Walk-through of our implementation

Our implementation of the perceptual learning experiment in perceptual_learning.html and perceptual_learning.js uses the audio-button-response plugin to play audio and get button-click responses - we haven’t used it before but it works in basically the same way as other button-response plugins you have used.

Social network questionnaire

The full set of questions is available in the supporting information of Lev-Ari (2017). As you can see, there are various text-box questions which are easy enough to implement in jsPsych using the survey-html-form plugin (look again at the end of self-paced reading practical). In the code I have only implemented the first three questions, which all require numeric answers:

var social_network_questionnaire = {
  type: jsPsychSurveyHtmlForm,
  preamble:
    "<p style='text-align:left'> <b>Social network questionnaire</b></p>\
              <p style='text-align:left'> In this questionnaire we would like to \
              gather information about your linguistic interactions. We realize \
              that some of the estimates are difficult to make. Please do your \
              best and be as accurate as possible.</p> \
              <p style='text-align:left'> Important: When providing estimates for \
              your exposure in a week, keep in mind that your habits may vary \
              considerably depending on the day of the week (e.g., weekday vs. weekend). \
              Please be as accurate as possible and do not simply multiply your \
              estimate for one day by 7.</p>",
  html: "<p style='text-align:left'>How old are you? <br> \
              <input required name='age' type='number'></p> \
         <p style='text-align:left'>With how many people do you converse orally \
         in a typical week? (Please only include people with whom you regularly \
           talk for longer than 5 minutes)<br> \
              <input required name='n_speak_to' type='number'></p> \
           <p style='text-align:left'>How many hours do you usually spend on \
           conversing orally with people in a typical week?<br>\
              <input required name='hours_speak_to' type='number'></p>",
};

Picture selection trials

Remember that on each picture selection trial the participant hears a description (“the fresh dill” etc) and the clicks on one of two images. We can do this fairly straightforwardly using the audio-button-response plugin (here is the documentation): we specify a stimulus which is the audio file we want to play, and then we can make the buttons be images rather than text; you have seen button response trials several times already, but never with images on the buttons, so that involves something a bit new.

You will notice that the directory for this experiment contains folders called picture_selection_sounds and picture_selection_images. Those contains all the sound files and images we should need. My implementation just uses a few of these, but you can look at a file called perceptual_learning_stims.csv, included in the zip file you downloaded, to see the full list of what images and sounds are available and how they fit together. The important things at this point are:

So what we want to do on each picture selection trial is play the sound file and show two buttons with the pair of images. We’ll start off doing this without images and with text buttons instead, and gradually build up to the implementation that’s in the code.

Forgetting about images for a moment, we could build a simple selection task like this:

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: ["fresh_dill", "dry_dill"],
};

That would play the sound file picture_selection_sounds/fresh_dill.mp3 (note that we have to tell the code that this sound file can be found in the folder picture_selection_sounds), and give the participant two text-button options, “fresh_dill” and “dry_dill” (so no images yet). The order of those choices is fixed though - the correct response is on the left - and unless we want to manually randomise when constructing our trial list it might be wise to borrow some of the tricks from the word learning experiment and randomise the order of the buttons on every trial. We can do that in exactly the same way as in the word learning experiment: set the choices to an empty list initially, then fill it out by shuffling ["fresh_dill", "dry_dill"] in the trial’s on_start:

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: [],
  on_start: function (trial) {
    var shuffled_choices = jsPsych.randomization.shuffle(["fresh_dill", "dry_dill"]);
    trial.choices = shuffled_choices;
  }
};

That will be fine, but as with just about anything in programming, there are several ways to do it! In the on_start of a trial you can access the trial choices directly, so another way to achieve the same outcome is to set the trial choices to a default order initially, then access, shuffle and over-write those choices in on_start, like this:

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: ["fresh_dill", "dry_dill"], //start with default order
  on_start: function (trial) {
    //can access the current trial choices with trial.choices, and shuffle them
    var shuffled_choices = jsPsych.randomization.shuffle(trial.choices);
    //then over-write with the shuffled version
    trial.choices = shuffled_choices;
  }
};

These both have the same end result, so which one you use (start with empty choices then complete in on_start vs start with choices in a default order then shuffle and over-write in on_start) depends on which you find more intuitive, and whether one of them makes life easier elsewhere - we’ll go with the second option here.

As usual, the jsPsych plugin won’t record the order of the shuffled choices for us, so we will have to use the data object again to keep track of what order the buttons are in and which one the participant clicked on. Again we can borrow some code and tricks from the word learning experiment to do this:

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: ["fresh_dill", "dry_dill"],
  on_start: function (trial) {
    var shuffled_choices = jsPsych.randomization.shuffle(trial.choices);
    trial.choices = shuffled_choices;
    trial.data = { button_choices: shuffled_choices };
  },

  //at the end, use data.response to figure out
  //which label they selected, and add that to data
  on_finish: function (data) {
    var button_number = data.response;
    data.button_selected = data.button_choices[button_number];
  },
};

So we make a note in data of that randomised order, under the heading button_choices. Then in on_finish (i.e. after the participant has pressed a button) we use the response information recorded automatically to work out which choice the participant selected, and store that in data as button_selected. By this point you might be getting a bit puzzled about the various parameters of trial.data associated with the participant’s response, so to try to help you keep those straight:

At this point we are getting close to what we want - play a sound file, click on a button, with the button position randomised, and record what button is selected - but of course our buttons just have the text “fresh_dill” and “dry_dill” in them, and actually we want the images from the corresponding image files. One way to do that is to replace our simple text button choices with <img ...> tags, which tell the browser to show an image rather than some text. For instance, this would work:

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: [
    "<img src=picture_selection_images/fresh_dill.jpg width=250px>",
    "<img src=picture_selection_images/dry_dill.jpg width=250px>",
  ],
  on_start: function (trial) {
    var shuffled_choices = jsPsych.randomization.shuffle(trial.choices);
  ...

(with … meaning everything else the same as the earlier example). We use the img tag to make the browser include an image, and then use src to tell it where to get the image from (note that have to include the directory name, picture_selection_images/, and the image type, .jpg). We also tell it how big we want the image to be - in this case, telling it to display the image at 250 pixels width. Then everything else will work as before, the two buttons will be shuffled, and we’ll end up with clickable image buttons. The only downside is that this is quite redundant - for both buttons we specify exactly the same information about the file path, the width etc, which seems kind of inefficient and potentially error prone. Also, all that button formatting stuff will be saved in data (under data.button_choices and data.button_selected) and will eventually be added to our experiment data, which is a bit messy and will make our data hard to look at.

There is actually a way around this, which is to use the button_html parameter of the plugin, which allows us to specify how all the buttons should be shown. We can use this to specify that the choices should be used as part of the name for image files when building the buttons. Integrating this into the code (with … meaning everything else the same as the earlier example):

var dill_trial = {
  type: jsPsychAudioButtonResponse,
  stimulus: "picture_selection_sounds/fresh_dill.mp3",
  choices: ["fresh_dill", "dry_dill"],
  on_start: function (trial) {
    var shuffled_choices = jsPsych.randomization.shuffle(trial.choices);
    trial.choices = shuffled_choices;
    trial.data = { button_choices: shuffled_choices };
  },
  button_html:
    '<button class="jspsych-btn"> <img src="picture_selection_images/%choice%.jpg" width=250px></button>',
    ...

What that extra bit of code in button_html does is set up a template for the buttons that the information in choices is slotted into (the marker %choice% tells the code where to drop in the information from choices, i.e. once we have shuffled and filled out the choices, this will be “fresh_dill” or “dry_dill”) - so it will use “fresh_dill” as (part of) the file name for one button, “dry_dill” for the other. Then I only specify all the button formatting once, and also in my data I am just going to end up recording the important varying information (the image name, rather than the full path and all the button format stuff), which makes my experiment data less awful to look at. Whenever you need buttons with images on them you can use this same trick - it takes a while to work out the first time, but once you’ve done it once and know the code works it’s easy to re-use!

We are nearly done, but of course this is just the specification for one trial, and we are going to need to specify many trials - they will all have the same format and just differ in stimulus and choices, so I really don’t want to write them all out in long format when building my trial list. There are several ways around this - I could use timeline variables, I could use nested timelines, but I am going to stick with the approach we have been using in the last few weeks and write a function that builds a trial for us. I am also going to exploit the fact that I can give a slightly different sound file name for manipulated or non-manipulated sound files (manipulated audio files have “_man.mp3” at the end, non-manipulated audio just has “.mp3” at the end). My function allows me to specify what I want the trial to look like at a high level (what is the sound file, is it the manipulated sound file or not, and what are the two buttons going to show?) and it will build a trial featuring the correctstimulus and choices for me. Here’s my function:

function make_picture_selection_trial(sound, target_image, foil_image) {
  var sound_file = "picture_selection_sounds/" + sound + ".mp3";
  var trial = {
    type: jsPsychAudioButtonResponse,
    stimulus: sound_file, //use the sound_file name we created above
    choices: [target_image,foil_image], //these will be shuffled on_start
    button_html:
      '<button class="jspsych-btn"> <img src="picture_selection_images/%choice%.jpg" width=250px></button>',
    post_trial_gap: 500, //a little pause between trials

    //at the start of the trial, randomise the left-right order of the images
    //and note that randomisation in data as button_choices
    on_start: function (trial) {
      var shuffled_choices = jsPsych.randomization.shuffle(trial.choices);
      trial.choices = shuffled_choices; //set trial choices now
      trial.data = {
        block: "picture_selection",
        button_choices: shuffled_choices,
      };
    },
    on_finish: function (data) {
      var button_number = data.response;
      data.button_selected = data.button_choices[button_number];
      save_perceptual_learning_data_line(data); //save the trial data
    },
  };
  return trial;
};

My make_picture_selection_trial function takes three arguments: sound, target_image, and foil_image.

The function doesn’t do anything fancy, and mainly just bundles up the information we provide into a single audio-button-response trial. There are a couple of things to note:

Now I can use my make_picture_selection_trial to build some picture selection trials and then shuffle them so they appear in random order.

var selection_trials_unshuffled = [
  make_picture_selection_trial("fresh_dill_man", "fresh_dill", "dry_dill"),
  make_picture_selection_trial("orange_telephone","orange_telephone","black_telephone"),
  make_picture_selection_trial("angel_wing", "angel_wing", "airplane_wing"),
  make_picture_selection_trial("animal_ear", "animal_ear", "animal_nose"),
];

var selection_trials = jsPsych.randomization.shuffle(
  selection_trials_unshuffled
);

I like this because it’s nice and clear to me what the sound and images are going to be. Note in this case I have manipulated audio for the /d/ trial (in “dill”) but not for the /t/ trial (in “telephone”) and not for the filler trials (NB: there are no manipulated audio files for the fillers, so if you try to set manipulated to true for those it will complain that it can’t find those audio files).

That’s it for the picture selection phase of this experiment, which is the most complex part.

Phoneme categorization trials

This will be relatively simple because we can basically re-use the code from the picture selection trials, but simplify it a bit. Remember that in each phoneme categorization trial the participant hears a voice (either the same speaker as in the picture selection phase or a new male speaker) and indicates by button press if that word is “dean” or “teen” (i.e. whether it started with a /d/ or a /t/); we can do all this using the audio-button-response plugin. We can use essentially exactly the same approach as above, except that we always present the same two choices on each trial (dean vs teen), and don’t bother randomising the left-right order of those on-screen (because Lev-Ari didn’t). Here’s the code:

function make_categorization_trial(sound) {
  //add the path and file extension
  var sound_file = "phoneme_categorization_sounds/" + sound + ".mp3";
  var trial = {
    type: jsPsychAudioButtonResponse,
    stimulus: sound_file,
    choices: ["dean", "teen"],
    post_trial_gap: 500,
    on_start: function (trial) {
      trial.data = {
        block: "phoneme_categorization",
        button_choices: trial.choices,
      };
    },
    on_finish: function (data) {
      var button_number = data.response;
      data.button_selected = data.button_choices[button_number];
      save_perceptual_learning_data_line(data);
    },
  };
  return trial;
}
  

var categorization_trials_unshuffled = [
  make_categorization_trial("samespeaker_VOT5"),
  make_categorization_trial("samespeaker_VOT10"),
  make_categorization_trial("samespeaker_VOT15"),
  make_categorization_trial("samespeaker_VOT20"),
  make_categorization_trial("samespeaker_VOT25"),
  make_categorization_trial("samespeaker_VOT30"),
  make_categorization_trial("samespeaker_VOT50"),
  make_categorization_trial("samespeaker_VOT80"),
];

var categorization_trials = jsPsych.randomization.shuffle(
  categorization_trials_unshuffled
);

A few things to note:

Saving data trial by trial

The rest of the code sets up the timeline in the usual way (creating the instruction screens, using concat to concatenate the various lists of trials, then running the timeline). The only interesting new thing is the function save_perceptual_learning_data_line which we call each time a critical trial (picture selection or phoneme categorization) finishes. We pass the trial’s data to this function, and it digs out the data we want, uses join to stick together that list of data into a comma-separated string, and then uses the save_data function that we introduced last week to write that data to a file called perceptuallearning_data.csv. Since save_data appends to the end of the file, each new line of data will be added, rather than over-writing the previous contents of the file. So if you look in your server_data folder as you progress through the experiment, remembering to refresh the cyberduck view, you should see the file getting larger as a new row of data is added after each trial. Watching the data files grow in this way as you run an online experiment is one of life’s guilty pleasures (at least for me).

function save_perceptual_learning_data_line(data) {
  // choose the data we want to save - this will also determine the order of the columns
  var data_to_save = [
    data.block,
    data.trial_index,
    data.time_elapsed,
    data.stimulus,
    data.button_choices,
    data.button_selected,
    data.response,
    data.rt,
  ];
  // join these with commas and add a newline
  var line = data_to_save.join(",") + "\n";
  save_data("perceptuallearning_data.csv", line);
}

One shortcoming of this method is that our csv file won’t have column names, which is not ideal. I have therefore used another plugin, the call-function plugin, to write the column names to the csv file before the first data-generating trial. The call-function plugin allows us to run some code behind the scenes - it has no components that are visible to the participant (i.e. no stimulus, no response), but it takes a func parameter which accepts a function that will do some work for us. In this case I am going to use that to run the save_data function, to write a single line of data (just the column names) to our data file. The code to do this looks like this:

var write_headers = {
  type: jsPsychCallFunction,
  func: function () {
    //write column headers to perceptuallearning_data.csv
    save_data(
      "perceptuallearning_data.csv",
      "block,trial_index,time_elapsed,stimulus,button_choice_1,button_choice_2,button_selected,response,rt\n"
    );
  },
};

Notice that the thing I am writing to the data file (using the save_data function) is a single comma-separated string with a new line at the end, so it’s in the correct format to display properly in my csv data file. Then I’ll slot this write_headers trial near the start of the experiment timeline, just after the consent screen, so that the headers get written before any real data.

Preloading

Downloading stimuli (audio or images) over the internet takes time, and the amount of time it takes depends on the speed of your participants’ connection, which is a problem in an experiment where we care about presenting stimuli for a set amount of time or generally presenting an experiment without lots of random lags and delays while stimuli are loaded. To avoid this, jsPsych provides a way of preloading stimuli, using the preload plugin - you drop a preloading trial into your timeline at the point where you want to load the stimuli, and when the timeline reaches that point it loads the items marked for preloading into the participant’s browser, ensuring they’ll be instantly available later when you need them; while the stimuli are preloading the participant sees a little progress bar so they know that something is happening. The code for creating this preload trial is very simple and looks like this:

var preload = {
  type: jsPsychPreload,
  auto_preload: true,
};

Then I slot this preload trial into my timeline just after the consent screen, to preload the stimuli at that point. You’ll notice that I have specified auto_preload: true in the preload trial. Each jsPsych plugin flags up some parameters for preloading - for instance in the audio-button-response plugin it’s the stimulus parameter that is marked up to be preloaded (similarly, in an image-button-response trial it’s the image stimulus that is marked for preloading). So by specifying auto_preload: true I am telling the preloading trial to go and find those stimuli in the timeline that are flagged in this way and preload them - it’ll do that for us, so all our audio stimuli will be preloaded and should play seamlessly when we get to the audio trials.

You might notice that when you run through the experiment that the images in the buttons take a moment to load at the start of each trial - this will be particularly pronounced if you are on a slow network. Why is that happening if we have a preload trial in our timeline?? That’s because audio-button-response doesn’t know that it needs to look out for images buried among the trial choices - audio-button-response just marks the stimulus parameter for preloading, but it doesn’t automatically preload our button images, because they don’t appear in a place it expects to have to preload (and actually knowing that they are images would involve some fairly complicated inference looking at the button_html parameter, so it’s not surprising it doesn’t try). It is actually possible to tell jsPsych to preload extra images or audio, which you can do by adding a manual preload list of images or audio to the preload trial - check out the the documentation for the preload trial, and look in particular at the images parameter of the plugin and the example titled “Manually preloading an image” to see how to add an images parameter to the preload trial.

Exercises with the perceptual learning experiment code

Attempt these problems. After you have attempted them you can look at my notes.

References

Lev-Ari, S. (2017). Talking to fewer people leads to having more malleable linguistic representations. PLoS ONE, 12, e0183593.

Re-use

All aspects of this work are licensed under a Creative Commons Attribution 4.0 International License.

Course main page

Project maintained by kennysmithed Hosted on GitHub Pages — Theme by mattgraham