Week 9 practical

web sockets, python servers, incrementally building a timeline

The plan for week 9 practical

This week we are going to look at code for a dyadic interaction task based on the Combined condition of the experiment described in Kanwal et al. (2017).

Important note Unlike the rest of the code on this course, we had to use version 7 of jsPsych for this one - version 8 is brand new, they removed one of the functions that we need for this code to work (addNodeToEndOfTimeline) and while they are working on putting it back, it hasn’t come together quite in time. Version 7 is very similar to version 8 - there are some very small differences in how we add fields to the data generated by each trial, but you probably won’t notice much difference!

In terms of the trial types we need to present to participants, this experiment is actually very simple, and uses elements of the code we developed in the practicals on word learning, perceptual learning, and iterated learning.

So there is actually very little that’s new here in terms of basic trial types. However, there is one substantial complication: rather than participants completing this experiment individually, they play in pairs, sending messages back and forth with their partner. We therefore need some infrastructure to allow two participants, working on web browsers anywhere in the world, to interact via the restricted communication channel we provide. The code for this isn’t actually too complicated - I figured it out! - but as per last week, I am going to hide most of the detail from you; the code is available and commented if you want to look at it or edit it, but you don’t have to (apart from to carry out a couple of very simple edits detailed below). Instead I’ll try to explain to you how it works, at a conceptual level, and you can take the details on trust until you need to build a similar experiment yourself.

A dyadic interaction experiment

Getting started

You need a bunch of files for this experiment - an html file, a few js files, some images, and then a folder containing some python code (this is where most of the magic happens). Download the following zip file, uncompress it, and stick it alongside your other experiment folders on jspsychlearning.

If your directory structure is as I have been using so far, where all the exercises are in a folder called online_experiments_practicals, then the URL for the final implementation will be https://jspsychlearning.ppls.ed.ac.uk/~UUN/online_experiments_practicals/dyadic_interaction/dyadic_interaction.html. You can open the experiment in the usual way in your browser. But this time, rather than running it in one window, you will need to open the same code in two browser windows (both on your computer is fine, or you can play with a friend) - it’s a dyadic game, it needs two players!

You build (some of) it, if you want

You haven’t been shown the methods for connecting two participants for real-time communication yet, so I don’t expect you to be able to implement the full experiment from scratch yourself. But as I explain above, you have nearly all the tools to build something that looks like a real-time interaction experiment - you could think of it as a confederate priming experiment like we looked at last week, where you have one genuine participant interacting with a computer partner who produces descriptions in a fixed order. So if you want to have a go at coding up the observation trials, or thinking about how you can do some kind of director and matcher trials, go for it. But if you just want to see how we did it, and attempt the exercises at the end, that’s OK too.

Our implementation

For this week I am going to focus on the conceptual level of how the dyadic interaction happens, and avoid stepping through the javascript in too much detail - like I said, most of it is re-used from earlier experiments anyway, the new bit is the dyadic interaction.

Clients and servers for dyadic interaction

You are already familiar with the notion of a server that communicates with some clients, because that’s how code you have put on the jspsychlearning server works - a client (your web browser) contacts the server and says, e.g., “hey, give me what you have at ~ksmith7/public_html/online_experiments_practicals/word_learning/word_learning.html” and the server sends says “sure, here’s the contents of that file” and sends back the html file; the client then says “hmm, looks like I need some javascript files too please” and the server sends those over too, then based on what’s in the html and javascript files your browser shows stuff on the screen, collects button clicks etc. All the mechanics of how this works is hidden from us, but the basic infrastructure is information flowing between your browser and the server - they send messages back and forth, and act on the messages they receive.

We can use exactly the same sort of information flow to build a dyadic interaction experiment: we’ll have some code sitting on a machine somewhere that receives requests from clients and sends information back to them; the clients then process that information and send further messages to the server. The code on the server handles the logic of how the dyadic communication game works - it keeps track of which clients (participants’ web browsers) are connected, pairs them up into dyads, tells them who should be director and who should be matcher, and so on. Then the code running on the clients handles the participant side of things - what the participants see, what options they have to click on - and sends their responses back to the server.

Our clients are written in javascript and jsPsych - they show stimuli, present buttons etc, just like in all the experiments we have looked at so far, the only difference being that the server tells them what kind of trial to run, and when they complete certain trials they send info back to the server telling them e.g. what label the participant selected.

Our server for this experiment happens to be written in python, rather than javascript - I could have written it in javascript, but to be honest I am more confident in python and that was easier! The server code receives and sends simple messages to the clients, keeps track of who is connected, and controls the progress of all the dyads that are currently running the experiment, sending them the right messages at the right time. It’s this relaying of messages back and forth via the python server that allows two participants in different locations to feel as if they are playing a communication game with each other - they are both connected to the python server, and the server hands out commands to one participant in a dyad based on what the other participant is doing. For instance, when one participant playing as director selects a label, they send their choice back to the python server and then the server sends it on to the other participant in that dyad to play a matcher trial.

That’s the basic idea. The flow of information between the clients and the server is a little bit intricate, because each trial in the experiment has several phases that the server needs to handle. In the diagram below I have tried to sketch out the kinds of information that pass back and forth between the server and two clients when you run the code. You read this from the top down, messages in blue are going from a client to the server, messages in orange are going from the server to the clients, white text boxes are actions that the clients or servers take based on the messages they receive. Participants complete the observation phase of the experiment individually, so this information flow only starts once participants are done with training on the lexicon and ready to interact with another participant, at which point they connect to the python server. This diagram covers the initial connection by 2 clients, formation of a pair, the pre-interaction instructions, and then a single communication trial with a director trial by participant ad30074fhd, a matcher trial by their partner 6apogh342, feedback to both, and then the start of the next trial where the roles flip.

flow of information between server and clients

Practicalities

How do we actually do this stuff in practice? The inner workings of the python server will have to remain a black box - the code is part of the zip file for this week, if you know python you can have a look if you want, but you don’t have to (other than to make a couple of very simple edits detailed below to set up a private version or edit the trial list, if you want to try those). But I do want to give you a flavour of how some of the jsPsych side of things works. In particular:

Organisation of the code

Following the model of the last couple of weeks, I have bundled up some of the technical stuff for the client-server communication in a separate file, utilities.js, and then all the jsPsych stuff is in dyadic_interaction.js.

The observation phase

When someone enters the experiment, the first thing they do is go through the observation phase. This is a solitary activity, so we handle it just like a normal jspsych experiment - we build some trials (image-button-response trials) to show objects and labels, build a timeline of those trials (roughly lines 130-190 of dyadic_interaction.js), and then run through that timeline as normal. So far, so standard.

Sending and receiving messages from the server

The final two trials of the “normal” part of the experiment, after the observation phase, are called instruction_screen_enter_waiting_room and start_interaction_loop and look like this:

var instruction_screen_enter_waiting_room = {
  type: jsPsychHtmlButtonResponse,
  stimulus:
  "<h3>You are about to enter the waiting room for pairing!</h3>\
  <p style='text-align:left'>Once you proceed past this point we will attempt to pair \
  you with another participant. As soon as you are paired you will start to play the \
  communication game together. <b> Once you are paired, your partner will be waiting for you \
  and depends on you to finish the experiment</b>, so please progress through the experiment \
  in a timely fashion, and please if at all possible <b>don't abandon or reload the \
  experiment</b> since this will also end the experiment for your partner.</p>",
  choices: ["Continue"],
};

var start_interaction_loop = { type: jsPsychCallFunction, func: interaction_loop };

instruction_screen_enter_waiting_room is a very boring html-button-response trial, showing some instructions emphasising to participants that there is another participant depending on them. start_interaction_loop is another jsPsych trial, of a type we have used a couple of times: call-function. A call-function trial just runs a javascript function, specified in the func parameter - in this case, we are asking it to run the function interaction_loop, which is going to do some important work for us. The call-function plugin is completely invisible for participants - it starts some code running, but nothing appears on the screen, no images are shown, no responses are collected.

So what does the interaction_loop function do? It appears with comments in the utilities.js file, you can look if you are keen, but basically it does two main things:

ws = new WebSocket("wss://jspsychlearning.ppls.ed.ac.uk" + my_port_number)

where my_port_number is a string that is defined in the dyadic_interaction.js code, and is set by default to "/ws1/" (although see below on changing this to connect to your own private python server). Computers have many many ports they can connect to other computers over, this is simply one of the available ones.

The only other interesting thing in utilities.js is a function called send_to_server(message) - we call this function with a particular message we want to send back to the python server, and it sends it over the socket. For instance, when the participant finishes reading some instructions that the server sent over, we call:

send_to_server({response_type:"INTERACTION_INSTRUCTIONS_COMPLETE"})

which sends a formatted message back to the python server to let it know that this participant has now finished reading the instructions; receiving this message triggers a response in the python server which allows the experiment to progress.

Building the timeline dynamically

OK, so what happens when the server sends over a message like "{command_type:WaitingRoom}", prompting the interaction_loop function processes to run the function waiting_room() - how does this make stuff happen on the participant’s screen?

Here’s the waiting_room() function, which is defined in the dyadic_interaction.js code.

function waiting_room() {
  var waiting_room_trial = {
    type: jsPsychHtmlButtonResponse,
    stimulus: "You are in the waiting room",
    choices: [],
    on_finish: function () {
      jsPsych.pauseExperiment();
    },
  };
  jsPsych.addNodeToEndOfTimeline(waiting_room_trial);
  jsPsych.resumeExperiment();
}

As you can see the guts of this is just a fairly boring html-button-response trial, putting some text on-screen telling participants they are in a waiting room (in the real experiment we had some cat videos they could watch while they wait, but I am giving you the austere version). But there are a couple of noteworthy things going on in the trial’s on_finish function and then after the trial is created which I’ll explain in a second once I have set the scene.

You should already be familiar with the idea that jsPsych experiments run through a timeline of trials - as each trial is completed you move to the next in the timeline, until you hit the end of the timeline at which point the experiment stops. In all the experiments we have seen so far, we define the timeline up-front, then the participant just runs through it. That poses a challenge for our dyadic interaction experiment, because we can’t define the timeline in advance - as soon as people start interacting, we need the python server to tell us what trials to run in what order.

The solution I have used is to build the timeline as we go - every time the python server tells us what kind of trial to run we need to add that trial to the timeline and run it. Fortunately jsPsych provides a function for this kind of thing, called jsPsych.addNodeToEndOfTimeline(trial) - this will add trial to the very end of the timeline. So we can use jsPsych.addNodeToEndOfTimeline to add new trials on the end of the timeline as they come in from the python server. (NB this is the function that is currently disabled in version 8 of jsPsych!).

The second issue we have to deal with is that jsPsych is always moving forward - as soon as a trial is completed it will move to the next trial, and if there are no trials left in the experiment it will exit the experiment. And once it’s exited, it won’t re-start if you add stuff into the timeline - when it’s done it’s done. This is a problem when we are adding trials one at a time to the end of the timeline - we have to avoid running out of trials/track and having the experiment come to a crashing halt before we are actually finished with the participant.

Avoiding running out of trials

Fortunately, jsPsych provides functions to pause and resume the timeline - jsPsych.pauseExperiment() and jsPsych.resumeExperiment() - which we can make sure we never run out of trials: we pause the experiment when we are waiting for instructions from the python server (one of the first things interaction_loop does is pause the timeline to await instructions), then resume it when we have a trial to run, run the trial, and then pause it again as soon as that trial has finished, while we await further instructions from the server.

Now you are in a position to figure out what the waiting_room() function does. The final two lines of the function, after the waiting room trial has been created, are

jsPsych.addNodeToEndOfTimeline(waiting_room_trial);
jsPsych.resumeExperiment();

That adds the trial we just created to the timeline, then once it’s been added allows the timeline to resume, meaning that the participant will see the waiting room message. Then once the waiting room trial completes (on_finish) we pause the experiment again as we wait for fresh instructions from the server. All the functions that are called when the python server sends over a command have this structure - add the trial, resume the timeline, pause the timeline when the trial completes. The code includes a bunch of functions with this same basic structure, that add trials to the timeline based on prompts from the python server - they are:

One other thing to note about the waiting_room_trial created by the waiting_room() function: it lasts forever! It’s an html-button-response trial, and it doesn’t have a set trial_duration (because we don’t know how long the wait will be), so it needs keyboard input to end. But its choices are set to [], so it doesn’t put any buttons on-screen for the participant to click. That’s a slightly weird trial type to create, but very handy when you want to give a participant a wait-message of uncertain duration. But at some point we will need to kick the timeline out of this trial, i.e. when another command comes in from the python server. We do that using the jsPsych.finishTrial(), which simply causes the current trial to end - so several of our functions that create new trials include a check to see if we are currently in one of these infinite-wait trials, and if so end that trial using jsPsych.finishTrial().

A simple manipulation of production effort

Kanwal et al. (2017) use a click-and-hold method for increasing production effort: participants have to hold the mouse click for longer to send the longer label to their partner. I went for something slightly simpler to implement, a multiple-click trial type where you click multiple times to send the label, and more times for longer labels. I achieve this using a looping trial (as part of director_trial) - when the participant selects a label we measure its length and set this to the required number of clicks to exit the loop:

n_clicks_required = label_selected.length; //this determines how many times we click in the loop

then the looping trial runs until n_clicks_required have been provided.

Advanced: Running your own private python server

The code defaults to connecting to the python server on a port called “ws1” on the jspsychlearning server, which is a python server I have set running (ws1 is a secure port Alisdair has set up for this kind of thing). That means that everyone who connects is going into the same waiting room and will be paired with the first available other player - so if you are testing the code at the same time as another student on the course, you might end up playing with them rather than yourself! That might be fun but it also might be irritating, so if you want you can set up your own private server to connect to, so you are guaranteed to have the server to yourself.

To do this you need to change two things in the code:

Once you have fixed that in the code, you have to start up your own private python server. You can do this from cyberduck. In the “Go” menu on cyberduck there is an option “Open in Terminal” (this might be called “Open in Putty” if you are using Windows; you might also be asked to give permission for the terminal to run and/or asked for your password at this point, which is the password you use to log onto jspsychlearning server). Select the server folder in your dyadic_interaction folder in cyberduck then select this “Open in Terminal” option. This should pop up a terminal window, a drab looking thing where you can enter text commands at a prompt. At the prompt type

python dyadic_interaction_server.py

and hit return, you should see a little message saying something like “starting server”, and then when clients connect you’ll get a stream of messages printed out reporting the progress of the experiment and the events that are happening from the python server’s perspective. If you can’t get this to work, we’ll be happy to help you set it up - I suspect the process might differ a little bit across different versions of cyberduck.

Exercises with the dyadic interaction experiment code

As usual, try these yourself, and once you have had a go, you can look at our notes.

References

Kanwal, J., Smith, K., Culbertson, J., & Kirby, S. (2017). Zipf’s Law of Abbreviation and the Principle of Least Effort: Language users optimise a miniature lexicon for efficient communication. Cognition, 165, 45-52.

Re-use

All aspects of this work are licensed under a Creative Commons Attribution 4.0 International License.

Course main page

XXProject maintained by kennysmithed Hosted on GitHub Pages — Theme by mattgraham