Friday practical, participant-to-participant interaction

web sockets, python servers, incrementally building a timeline

The plan

Here we are going to look at code for a dyadic interaction task based on the Combined condition of the experiment described in Kanwal et al. (2017).

In terms of the trial types we need to present to participants, this experiment is actually very simple, and uses elements of the code we developed in the practicals on word learning and perceptual learning.

Participants go through an initial observation phase where they are exposed to two objects paired with short and long labels. This is basically identical to the observation phase of the word learning experiment in word_learning.js.
When communicating with their partner, participants sometimes act as what I’ll call the director - they are presented with an object and asked to select one of two labels to send to their partner. This is very similar to the production trial phase in the word learning experiment. The only difference is that we need to add a manipulation of production effort, making the longer labels more onerous to send. I explain how we do that below.
When communicating with their partner, participants sometimes act as the matcher - they are presented with a label and asked to select one of two objects which they think their partner is labelling. This is very similar to the picture selection trials in the perceptual learning experiment, except that we are just using a text label rather than an audio file.

However, there is one substantial complication: rather than participants completing this experiment individually, they play in pairs, sending messages back and forth with their partner. We therefore need some infrastructure to allow two participants, working on web browsers anywhere in the world, to interact via the restricted communication channel we provide. The code for this isn’t actually too complicated - I figured it out! - but I am going to hide most of the detail from you; the code is available and commented if you want to look at it or edit it, but you don’t have to. Instead I’ll try to explain to you how it works, at a conceptual level, and you can take the details on trust until you need to build a similar experiment yourself.

Remember, as usual the idea is that you do as much of this as you can on your own (might be none of it, might be all of it) and then come to the practical drop-in sessions or use the chat on Slack to get help with stuff you need help with.

A dyadic interaction experiment

Getting started

You need a bunch of files for this experiment - an html file, a few js files, some images, and then a folder containing some python code (this is where most of the magic happens). Download the following zip file and then uncompress it into your usual jspsych folder:

Download dyadic_interaction.zip

As usual, the code makes some assumptions about the directory structure it’s going to live in - in particular, it needs a copy of jspych. Regardless of whether you are putting this on your own computer or in your public_html folder on the server, these should sit in a folder called dyadic_interaction, alongside a copy of the jspsych-6.2.0 folder. This code should actually run OK on your local computer, but it won’t save your data if you run it locally - so to get the full experience, you need to run it on the server. Furthermore, there is one tweak you need to make before your data will save:

You need to edit save_data.php so that it points to your server_data folder. Open that file in an editor and change the path /home/USER/server_data/ to reflect your user name.

Once you have done that you can open dyadic_interaction.html in your web browser if you are running it locally, or go to http://jspsychks.ppls.ed.ac.uk/~USER/dyadic_interaction/dyadic_interaction.html if you want to run it on the server. And this time, rather than running it in one window, you will need to open the same code in two browser windows (both on your computer is fine, or you can play with a friend if you are running it on the server and if they can connect to the VPN) - it’s a dyadic game, it needs two players!

First, get the code and run through it so you can see what it does. Then read on. For this practical I am going to focus on the conceptual level of how the dyadic interaction happens, and avoid stepping through the javascript in too much detail - like I said, most of it is re-used from earlier experiments anyway.

Clients and servers for dyadic interaction

You are already with the notion of a server that hands communicates with some clients, because that’s how code you have put on the jspsychks server works - a client (your web browser) contacts the server and says, e.g., “hey, give me what you have at ~ksmith7/public_html/word_learning/word_learning.html” and the server sends says “sure, here’s the contents of that file” and sends back the html file; the client then says “hmm, looks like a need some javascript files too please” and the server sends those over too, then based on what’s in the html and javascript files your browser shows stuff on the screen, collects button clicks etc. All the mechanics of how this works is hidden from us, but the basic infrastructure is information flowing between your browser and the server - they send messages back and forth, and act on the messages they receive.

We can use exactly the same sort of information flow to build a dyadic interaction experiment: we’ll have some code sitting on a machine somewhere that receives requests from clients and sends information back to them; the clients then process that information and send further messages to the server. The code on the server handles the logic of how the dyadic communication game works - it keeps track of which clients (participants’ web browsers) are connected, pairs them up into dyads, tells them who should be director and who should be matcher, and so on. Then the code running on the clients handles the participant side of things - what the participants see, what options they have to click on - and sends their responses back to the server.

Our clients are written in javascript and jsPsych - they show stimuli, present buttons etc, just like in all the experiments we have looked at so far, the only difference being that the server tells them what kind of trial to run, and when they complete certain trials they send info back to the server telling them e.g. what label the participant selected.

Our server for this experiment happens to be written in python, rather than javascript - I could have written it in javascript, but to be honest I am more confident in python and that was easier! The server code receives and sends simple messages to the clients, keeps track of who is connected, and controls the progress of all the dyads that are currently running the experiment, sending them the right messages at the right time. It’s this relaying of messages back and forth via the python server that allows two participants in different locations to feel as if they are playing a communication game with each other - they are both connected to the python server, and the server hands out commands to one participant in a dyad based on what the other participant is doing. For instance, when one participant playing as director selects a label, they send their choice back to the python server and then the server sends it on to the other participant in that dyad to play a matcher trial.

That’s the basic idea. The flow of information between the clients and the server is a little bit intricate, because each trial in the experiment has several phases that the server needs to handle. In the diagram below I have tried to sketch out the kinds of information that pass back and forth between the server and two clients when you run the code. You read this from the top down, messages in blue are going from a client to the server, messages in orange are going from the server to the clients, white text boxes are actions that the clients or servers take based on the messages they receive. Participants complete the observation phase of the experiment individually, so this information flow only starts once participants are done with training on the lexicon and ready to interact with another participant, at which point they connect to the python server. This diagram covers the initial connection by 2 clients, formation of a pair, the pre-interaction instructions, and then a single communication trial with a director trial by participant ad30074fhd, a matcher trial by their partner 6apogh342, feedback to both, and then the start of the next trial where the roles flip.

flow of information between server and clients

One final note: I am running the python server for this experiment on a different server, blake4.ppls.ed.ac.uk - unfortunately access to the necessary ports on jspsychks is restricted to people on the Edinburgh network, they’re worried about the security of the server. This means that you’ll be restricted to playing with the client-side code, you can’t run your own version of the server on the jspsychks unfortunately. But you can still see the code and play around with the client, and if you know how you can actually run a local server on your own computer if you want to mess with that side of things.

Practicalities

How do we actually do this stuff in practice? The inner workings of the python server will have to remain a black box - the code is part of the zip file for this practical, if you know python you can have a look if you want, but you don’t have to (other than to make a couple of very simple edits detailed below to set up a private version or edit the trial list). But I do want to give you a flavour of how some of the jsPsych side of things works. In particular:

How the code is organised.
How we run the non-interactive observation phase.
How we send and receive messages to/from the python server.
How we build a timeline dynamically, based on messages from the python server.

Organisation of the code

Following the model for the audio recording practical, I have bundled up some of the technical stuff for the client-server communication in a separate file, dyadic_interaction_utilities.js, and then all the jsPsych stuff is in dyadic_interaction.js. For this experiment we also have an extra plugin that I created, called jspsych-image-repeatbutton-response.js, which sits alongside the other js files for the experiment - this is a minor modification to the standard jspsych-image-button-response.js plugin, I just copied that code and edited it a little bit to set up a trial type where the participant has to repeatedly click the button to complete the trial. JsPych doesn’t care who wrote the plugins, as long as they have the correct format it will use them no problem, so creating new plugins is pretty easy, particularly when they are based on existing ones.

The observation phase

When someone enters the experiment, the first thing they do is go through the observation phase. This is a solitary activity, so we handle it just like a normal jspsych experiment - we build some trials (image-keyboard-response trials) to show objects and labels, build a timeline of those trials (roughly lines 80-140 of dyadic_interaction.js), and then run through that timeline as normal. So far, so standard.

Sending and receiving messages from the server

The final two trials of the “normal” part of the experiment, after the observation phase, are called instruction_screen_enter_waiting_room and start_interaction_loop and look like this:

var instruction_screen_enter_waiting_room = {
  type: 'html-keyboard-response',
  stimulus: "<h3>Instructions before entering the waiting room</h3>\
  <p style='text-align:left'>Once the participant clicks through here they will connect to the server \
  and the code will try to pair them with another participant.</p>\
  <p>Press any key to begin</p>"
}

var start_interaction_loop = {type:'call-function',
                              func: interaction_loop}

instruction_screen_enter_waiting_room is a very boring html-keyboard-response trial, showing some dummy instructions. start_interaction_loop is another jsPsych trial, of a type we have not used before: call-function. A call-function trial just runs a javascript function, specified in the func parameter - in this case, we are asking it to run the function interaction_loop, which is going to do some important work for us. Unlike all the other plugins we have used so far, the call-function plugin is completely invisible for participants - it starts some code running, but nothing appears on the screen, no images are shown, no responses are collected.

So what does the interaction_loop function do? It appears with comments in the dyadic_interaction_utilities.js file, you can look if you are keen, but basically it does two main things:

It creates a connection between the client browser and the python server, using a communication protocol called WebSockets, which allow 2-way communication between the browser and the python server. This channel will stay open for the duration of the experiment, and allows the client and the server to communicate seamlessly by sending and receiving messages over the socket. The code for creating the socket is actually very simple, one line of code:
```
ws = new WebSocket("wss://blake4.ppls.ed.ac.uk/ws11/")
```
blake4 is my server for running this kind of experiment, and ws11 is one of the channels on that server that are set up to allow this kind of communication between the server and external clients.
Once the connection is created, interaction_loop then listens on that socket for messages from the python server. Whenever a message comes in, it does some basic parsing of the message and figures out what the client needs to do. For instance, if the server sends a message that looks like "{command_type:WaitingRoom}", this triggers the client to run a function called waiting_room(), which we’ll see in a moment, that creates a trial and adds it to the trial timeline - so the message from the python server eventually causes something to happen on the participant’s screen, i.e. a jsPsych trial is run. That’s how the whole experiment works - messages from the server trigger stuff in interaction_loop that cause trials to be created and added to the trial timeline. When I wrote the python server code I worked out what the minimal set of commands was that needed to come from the python server to the client to make this experiment work, then I wrote bits of code inside interaction_loop to process those commands.

The only other thing in dyadic_interaction_utilities.js is a function called send_to_server(message) - we call this function with a particular message we want to send back to the python server, and it sends it over the socket. For instance, when the participant finishes reading some instructions that the server sent over, we call:

send_to_server({response_type:"INTERACTION_INSTRUCTIONS_COMPLETE"})

which sends a message back to the python server to let it know that this participant has now finished reading the instructions; receiving this message triggers a response in the python server which allows the experiment to progress.

Building the timeline dynamically

OK, so what happens when the server sends over a message like "{command_type:WaitingRoom}", prompting the interaction_loop function processes to run the function waiting_room() - how does this make stuff happen on the participant’s screen?

Here’s the waiting_room() function, which is defined in the dyadic_interaction.ps code.

function waiting_room() {
  var waiting_room_trial = {type:'html-keyboard-response',
                            stimulus:"You are in the waiting room",
                            choices:[],
                            on_finish:function() {jsPsych.pauseExperiment()}}
  jsPsych.addNodeToEndOfTimeline(waiting_room_trial,jsPsych.resumeExperiment)
}

As you can see the guts of this is just a fairly boring html-keyboard-response trial, putting some text on-screen telling participants they are in a waiting room (in the real experiment we had some cat videos they could watch while they wait, but I am giving you the spartan version). But there are a couple of noteworthy things going on in the trial’s on_finish function and then after the trial is created which I’ll explain in a second once I have set the scene.

You should already be familiar with the idea that jsPsych experiments run through a timeline of trials - as each trial is completed you move to the next in the timeline, until you hit the end of the timeline at which point the experiment stops. In all the experiments we have seen so far, we define the timeline up-front, then the participant just runs through it. That poses a challenge for our dyadic interaction experiment, because we can’t define the timeline in advance - as soon as people start interacting, we need the python server to tell us what trials to run in what order.

The solution to this is to build the timeline as we go - every time the python server tells us what kind of trial to run we need to add that trial to the timeline and run it. Fortunately jsPsych provides a function for this kind of thing, called jsPsych.addNodeToEndOfTimeline(trial,continuation) - this will add trial to the very end of the timeline, then it will call continuation (which has to be a function, more on that in a minute). So we can use jsPsych.addNodeToEndOfTimeline to add new trials on the end of the timeline as they come in from the python server.

The second issue we have to deal with is that jsPsych is always moving forward - as soon as a trial is completed it will move to the next trial, and if there are no trials left in the experiment it will exit the experiment. And once it’s exited, it won’t re-start if you add stuff into the timeline - when it’s done it’s done. This is a problem when we are adding trials one at a time to the end of the timeline - we have to avoid running out of trials/track and having the experiment come to a crashing halt before we are actually finished with the participant.

Avoiding running out of trials

Fortunately, jsPsych provides functions to pause and resume the timeline - jsPsych.pauseExperiment() and jsPsych.resumeExperiment() - which we can use to make sure we never run out of trials: we pause the experiment when we are waiting for instructions from the python server (one of the first things interaction_loop does is pause the timeline to await instructions), then resume it when we have a trial to run, run the trial, and then pause it again as soon as that trial has finished, while we await further instructions from the server.

Now you are in a position to figure out what the waiting_room() function does. You’ll see in the on_finish parameter of the trial, we pause the timeline - that’s us pausing the timeline once to trial completes, to await further instructions from the server. And the final line of the function, after the waiting room trial has been created, is

jsPsych.addNodeToEndOfTimeline(waiting_room_trial,jsPsych.resumeExperiment)

That adds the trial we just created to the timeline, then once it’s been added allows the timeline to resume (only to be paused again when the trial finishes). All the functions that are called when the python server sends over a command have this structure - add the trial, resume the timeline, pause the timeline when the trial completes. The code includes a bunch of functions with this same basic structure, that add trials to the timeline based on prompts from the python server - they are called waiting_room(), waiting_for_partner(), show_interaction_instructions(), partner_dropout() (informs the participant that something has gone wrong with the experiment, which is usually the other player dropping out!), director_trial(target_object,partner_id) (adds a director trial to the timeline), matcher_trial(label,partner_id) (adds a matcher trial to the timeline), display_feedback(score) (tells the participant whether the last round of communication was a success or not), and end_experiment(). These are all commented up in dyadic_interaction.js if you want to take a look.

One other thing to note about the waiting_room_trial created by the waiting_room() function: it lasts forever! It’s an html-keyboard-response trial, and it doesn’t have a set trial_duration, so it needs keyboard input to end. But its choices are set to [], so it doesn’t accept any keyboard input. That’s a slightly weird trial type to create, but very handy when you want to give a participant a wait-message of uncertain duration. But at some point we will need to kick the timeline out of this trial, i.e. when another command comes in from the python server. We do that using the jsPsych.finishTrial(), which simply causes the current trial to end - so several of our functions that create new trials include a check to see if we are currently in one of these infinite-wait trials, and if so end that trial using jsPsych.finishTrial().

A half-hearted effort at manipulating production effort

Kanwal et al. (2017) use a click-and-hold method for increasing production effort: participants have to hold the mouse click for longer to send the longer label to their partner. I went for something slightly simpler to implement, a multiple-click trial type where you click multiple times to send the label, more clicks required for longer labels. This is wrapped up in the jspsych-image-repeatbutton-response.js plugin I wrote, which I mentioned above - this is a minor modification to the standard jspsych-image-button-response.js plugin. I think it actually should be possible to achieve the same effect with a normal jspsych-image-button-response.js trial which loops (jsPsych provides some built-in infrastructure to create looping trials), but I was having a hard time figuring it out so eventually I gave up and just made my own plugin! At some point I’ll have to replace this with a click-and-hold plugin to more closely match the Kanwal et al. method, I had a half-working version of that but again ran out of time to make it work more neatly, sorry.

Optional: Running your own private python server

The code defaults to connecting to a python server running on blake4, my research server in Edinburgh. That means that everyone who connects is going into the same waiting room and will be paired with the first available other player - so if you are testing the code at the same time as another student on the course, you might end up playing with them rather than yourself! That might be fun but it also might be irritating. If you want you can set up your own private server to connect to, so you are guaranteed to have the server to yourself - you can only run this server locally (jspsychks isn’t set up in a secure way so I wasn’t able to open ports for people not on the Edinburgh network). This means you will also have to run the client html code locally (i.e. by opening the html file on your computer), so you won’t be able to save data if you are using your own server. This also assumes that you know how to run python on your computer - if not, maybe skip this and just work with the shared server.

To do this you need to change one line in dyadic_interaction_utilities.js code so that the code uses your own server - this is around line 40 in that file, and you want to change it to:

ws = new WebSocket("ws://localhost:9011")

Once you have fixed that in the code, you have to start up your own private python server on your computer. This requires running python - again, if you don’t know how to do this, maybe skip this step, you won’t be missing out on much. But if you want to try it: open up a terminal window, a drab looking thing where you can enter text commands at a prompt, navigate to the server folder in your dyadic_interaction folder (probably using the cd command), and then at the prompt type

python dyadic_interaction_server.py

and hit return, you should see a little message saying something like “starting server”. Then if you open the dyadic_interaction.html file in a browser on your local computer, when clients connect you’ll get a stream of messages printed out reporting the progress of the experiment and the events that are happening from the python server’s perspective.

Exercises with the dyadic interaction experiment code

Run the experiment and look at the CSV data files it creates. Check that the contents of the data files make sense and how they relate to what you see as a participant.
How would you increase the number of training trials in the observation phase of the experiment?
In the Accuracy condition of Kanwal et al., participants interacted in dyads but there was no additional production effort associated with producing longer labels. How would you edit the jsPsych code to replicate the Accuracy condition? [Hint: you need to be looking at the director_trial function in dyadic_interaction.js, which creates the three sub-trials that make up a director trial].
How could you manipulate production effort in the other direction, making long labels even more effortful to produce?
[Only if you managed to set up a private server] It’s also possible to change the number of interaction trials, but this will involve editing the python server code and running your own private python server with your edited version. The relevant line is line 61 of dyadic_interaction_server.py, which looks like this:
```
target_list = ['object4','object4','object4','object5']*2
```
This creates a variable, target_list, which consists of 6 occurrences of object4 and 2 of object5. Even if you have never seen python before, hopefully you can guess how to edit this list to change the relative proportions of the two objects or the total number of trials.
[Only if you managed to set up a private server, optional] Can you create a version of the experiment where participants deal with more than 2 objects, e.g. 4 objects, organised in two pairs with a short ambiguous label shared across the two objects in each pair, or maybe 4 objects with unique labels and a single shared short label? This will involve changing the trial lists in the observation and interaction phases, but also some parts of the client code that make assumptions about which objects and labels participants will encounter during interaction.

References

Kanwal, J., Smith, K., Culbertson, J., & Kirby, S. (2017). Zipf’s Law of Abbreviation and the Principle of Least Effort: Language users optimise a miniature lexicon for efficient communication. Cognition, 165, 45-52.

Re-use

All aspects of this work are licensed under a Creative Commons Attribution 4.0 International License.

Course main page

Project maintained by kennysmithed Hosted on GitHub Pages — Theme by mattgraham