Further Trinity Lancaster Corpus research: Examiner strategies

This month saw a further development in the corpus analyses: the examiners. Let me introduce myself, my name is Cathy Taylor and I’m responsible for examiner training at Trinity and was very pleased to be asked to do some corpus research into the strategies the examiners use when communicating with the test takers.

In the GESE exams the examiner and candidate co-construct the interaction throughout the exam. The examiner doesn’t work from a rigid interlocutor framework provided by Trinity but instead has a flexible test plan which allows them to choose from a variety of questioning and elicitation strategies. They can then respond more meaningfully to the candidate and cover the language requirements and communication skills appropriate for the level. The rationale behind this approach is to reflect as closely as possible what happens in conversations in real life. Another benefit of the flexible framework is that the examiner can use a variety of techniques to probe the extent of the candidate’s competence in English and allow them to demonstrate what they can do with the language. If you’re interested more information can be found in Trinity’s speaking and listening tests: Theoretical background and research.

After some deliberation and very useful tips from the corpus transcriber, Ruth Avon, I decided to concentrate my research on the opening gambit for the conversation task at Grade 6, B1 CEFR. There is a standard rubric the examiner says to introduce the subject area ‘Now we’re going to talk about something different, let’s talk about…learning a foreign language.’  Following this, the examiner uses their test plan to select the most appropriate opening strategy for each candidate. There’s a choice of six subject areas for the conversation task listed for each grade in the Exam information booklet.

Before beginning the conversation examiners have strategies to check that the candidate has understood and to give them thinking time. The approaches below are typical.

  1. E: ‘Let’s talk about learning a foreign language…’
    C: ‘yes’
    E:Do you think English is an easy language?’ 
  1. E: ‘Let ‘s talk about learning a foreign language’
    C: ‘It’s an interesting topic’
    E: ‘Yes uhu do you need a teacher?
  1. It’s very common for the examiner to use pausing strategies which gives thinking time:
    E: ‘Let ‘s talk about learning a foreign language erm why are you learning English?’
    C: ‘Er I ‘m learning English for work erm I ‘m a statistician.’

There are a range of opening strategies for the conversation task:

  • Personal questions: ‘Why are you learning English?’ ‘Why is English important to you?’
  • More general question: ‘How important is it to learn a foreign language these days?’
  • The examiner gives a personal statement to frame the question: ‘I want to learn Chinese (to a Chinese candidate)…what do I have to do to learn Chinese?’
  • The examiner may choose a more discursive statement to start the conversation: ‘Some people say that English is not going to be important in the future and we should learn Chinese (to a Chinese candidate).’
  • The candidate sometimes takes the lead:
  • Examiner: ‘Let’s talk about learning a foreign language’
  • Candidate: ‘Okay, okay I really want to learn a lo = er learn a lot of = foreign languages’

A salient feature of all the interactions is the amount of back channelling the examiners do e.g. ‘erm, mm’  etc. This indicates that the examiner is actively listening to the candidate and encouraging them to continue. For example:

E: ‘Let’s talk about learning a foreign language, if you want to improve your English what is the best way?
C: ‘Well I think that when you see programmes in English’
E: ‘mm
C: ‘without the subtitles’
E: ‘mm’
C: ‘it’s a good way or listening to music in other language
E: ‘mm
C: ‘it’s a good way and and this way I have learned too much

When the corpus was initially discussed it was clear that one of the aims should be to use the findings for our examiner professional development programme.  Using this very small dataset we can develop worksheets which prompt examiners to reflect on their exam techniques using real examples of examiner and candidate interaction.

My research is in its initial stages and the next step is to analyse different strategies and how these validate the exam construct. I’m also interested in examiner strategies at the same transition point at the higher levels, i.e. grade 7 and above, B2, C1 and C2 CEFR. Do the strategies change and if so, how?

It’s been fascinating working with the corpus data and I look forward to doing more in the future.

Continue reading

Birmingham ERP Boot Camp

Last week I attended a 5-day ERP Boot Camp at the University of Birmingham, and this was an incredible opportunity for me to learn from ERP experts and get specific advice for running my next ERP experiments. The workshop was led by two of the most renowned ERP researchers in the world, namely Professor Steven Luck and Dr Emily Kappenman. Luck and Kappenman are both part of the Centre for Mind and Brain at the University of California, Davis, which is one of the world’s leading centres for research into cognitive neuroscience. They are both among the set of researchers who set the publicationjen workshop blog 1 guidelines and recommendations for conducting EEG research (Keil et al. 2014), and Luck is also the developer of ERPLAB, which is a MATLAB Toolbox designed specifically for ERP data analysis. Moreover, Luck is the author of the authoritative book entitled An Introduction to the Event-Related Potential Technique. Before attending the ERP Boot Camp, most of the knowledge that I had about ERPs came from this book. Therefore, I am extremely grateful that I have had this opportunity to learn from the authorities in the field, especially since Luck and Kappenman bring the ERP Boot Camp to the University of Birmingham just once every three years.

There were two parts to the ERP Boot Camp: 2.5 days of lectures covering the theoretical aspects of ERP research (led by Steven Luck), and 2.5 days of practical workshops which involved demonstrations of the main data acquisition and analysis steps, followed by independent data analysis work using ERPLAB (led by Emily Kappenman). Day 1 of the Boot Camp provided an overview of different experimental paradigms and different ERP components, which are defined as voltage changes that reflect a particular neural or psychological process (e.g. the N400 component reflects the processing of meaning and the P600 component reflects the processing of structure). Most of the electrical activity in the brain that can be detected by scalp electrodes comes from the surface of the cortex but, in the lecture on ERP components, I was amazed to find out that there are some ERP components that actually reflect brain stem activity. These components are known as auditory brainstem responses. I also learnt about how individual differences between participants are typically the result of differences in cortical folding and differences in skull thickness, rather than reflecting any functional differences, and I learnt how ERP components from one domain such as language can be used to illuminate psychological processes in other domains such as memory. From this first day at the Boot Camp, I started to gain a much deeper conceptual understanding of the theoretical basis of ERP research, causing me to think of questions that hadn’t even occurred to me before.

Day 2 of the Boot Camp covered the principles of electricity and magnetism, the practical steps involved in processing an EEG dataset, and the most effective ways of circumventing and minimizing the problems that are inevitably faced by all ERP researchers. On this day I also learnt the importance of taking ERP measurements from difference waves rather than from the raw ERP waveforms. This is invaluable knowledge to have when analysing the data from my next experiments. In addition, I gained some concrete advice on stimulus presentation which I will take into account when editing my stimuli.

On day 3 of the Boot Camp, we were shown examples of ‘bad’ experimental designs and we were asked to identify the factors that made them problematic. Similarly, we discussed how to identify problematic results just by looking at the waveforms. These was really useful exercises in helping me to critically evaluate ERP studies, which will be useful both when reading published articles and when thinking about my own experimental design.

From the outset of the Boot Camp, we were encouraged to ask questions at any time, andJen workshop blog 2 this was particularly useful when it came to the practical sessions as we were able to use our own data and ask specific questions relating to our own experiments. I came prepared with questions that I had wanted to know the answers to for a long time, as well as additional questions that I had thought of throughout the Boot Camp, and I was given clear answers to every one of these questions.

Furthermore, as well as acquiring both theoretical and practical knowledge from the scheduled lectures and workshops, I also gained a lot from talking to the other ERP researchers who were attending the Boot Camp. A large proportion of attendees focused on language as their main research area, while others focused on clinical psychology or other areas of psychology such as memory or perception. I found it really interesting to hear the differences of opinion between those who were primarily linguists and those who were primarily psychologists. For instance, when discussing the word-by-word presentation of sentences in ERP experiments, the psychologists stated that each word should immediately replace the previous word, whereas the linguists concluded that it is best to present a blank white screen between each word. Conversations such as this made it very apparent that many of the aspects of ERP research are not standardised, and so it is up to the researcher to decide what is best for their experiment based on what is known about ERPs and what is conventional in their particular area of research.

Attending this ERP Boot Camp was a fantastic opportunity to learn from some of the best ERP researchers in the world. I now have a much more thorough understanding of the theoretical basis of ERP research, and I have an extensive list of practical suggestions that I can apply to my next experiments. I thoroughly enjoyed every aspect of the workshop and I am very grateful to CASS for funding the trip.

Spoken BNC2014 book announcement

We are excited to announce a forthcoming book which will be published as part of the BNC2014 logoRoutledge Advances in Corpus Linguistics series. “Corpus Approaches to Contemporary British Speech: Sociolinguistic Studies of the Spoken BNC2014” (edited by Vaclav Brezina, Robbie Love and Karin Aijmer) will feature a collection of research which is currently being undertaken by the recipients of the Spoken BNC2014 Early Access data grants.

With exclusive early access to approximately five million words of Spoken BNC2014 data, the book’s contributors will present a range of innovative studies which each analyse the corpus from a sociolinguistic perspective.

Following the public release of the complete Spoken BNC2014 (approximately ten million words) in late 2017, the book is anticipated to follow shortly thereafter. The agreement of the book with Routledge joins a previously announced special issue of the International Journal of Corpus Linguistics (IJCL), which will feature a range of work by other recipients of the Spoken BNC2014 Early Access data grants.

 

Participants needed for psycholinguistic experiment!

My PhD research combines methods from corpus linguistics and psychology in order to find out more about how language is processed in the brain. The method that I use from psychology is known as electroencephalography (EEG), and this involves placing electrodes across a participant’s scalp in order to detect some of the electrical activity of the brain. More specifically, I use the event-related potential (ERP) technique, which involves measuring the electrical activity of the brain in response to particular stimuli. When I carried out my pilot study earlier this year, this was the first time the EEG/ERP method had been used in the Department of Linguistics and Language, making it a really exciting project to get involved with.

Having completed my pilot study and obtained some really interesting results, I have refined my methods and hypotheses and I am now ready to recruit participants for my next two experiments. For one experiment which will take place in late August, I am looking for 16 native speakers of Mandarin Chinese; for another experiment which will take place in October, I am looking for 16 native speakers of English. I would really appreciate hearing from anyone who is interested in taking part! The whole procedure takes about 1 hour; it takes about 20-30 minutes for me to attach all of the electrodes, and the experiment itself takes an additional 20-30 minutes.

If you do decide to take part, you will wear a headcap containing 64 plastic electrode holders which the electrodes are clipped into, as well as 6 electrodes around your eyes and 2 electrodes behind your ears. The electrodes make contact with your skin via a conductive gel which enables some of the electrical signals in your brain to propagate to the electrode wires and into the AD-box, where the electrical signal is amplified and converted from analog to digital format. The amplified signals are then transmitted to the USB2 receiver via a fibre-optic cable, before being relayed onto the data acquisition computer where your brainwaves can be viewed as a continuous waveform. Before starting the experiment, I will ask you to blink, clench your teeth, and move your head from left to right so that you can see how these movements affect the observed waveform.

jen expermient

The experiment itself involves reading real language data that has been extracted from the British National Corpus. This consists of sentences which are presented word-by-word on a computer screen. After reading each sentence, you will be asked to respond to a true/false statement based on the sentence that you have just read.

Before conducting my pilot study, I carried out a number of test-runs on other postgraduate students and each one of them found it to be a really interesting experience. For instance, Gillian Smith, another PhD research student in CASS, agreed to take part in one of my test-runs and here she describes her experience as a participant:

“Getting to be involved in Jen’s experiment was a great opportunity! Having never participated in such a study before, I found the whole process (which Jen explained extremely well) very interesting. I particularly enjoyed being able to look at my brainwaves after, which is something I have never experienced. Likewise, having electrodes on my head was a lovely novelty.”

gill jen experiment


I would really like to hear from any native speakers of Mandarin Chinese or native speakers of English who would be interested in taking part in one of these experiments. Please email j.j.hughes(Replace this parenthesis with the @ sign)lancaster.ac.uk to express interest and to receive more information.

CASS goes to Weihai!

 

China 1

Between the 28th July and the 2nd August, Carmen Dayrell and I represented CASS at the 3rd Sino-UK Summer School of Corpus Linguistics. The summer school was organised by Beijing Foreign Studies University and was hosted at the Weihai campus of the University of Shandong, China. A research symposium followed the summer school on the 3rd August where we presented our research to representatives from both universities. The research symposium gave us a taste of how corpus linguistics is used in a different culture and we heard papers on a range of different topics, such as Alzheimer’s research, work on translations, Chinese medicine, and analyses of media discourse.

Our summer school sessions introduced students to corpus linguistics and gave them an overview of the discipline’s development within a UK context. We also discussed the range of projects ongoing at CASS and foregrounded the interdisciplinary focus of the Centre’s work. After the formal lectures, we ran hands-on sessions demonstrating how to use Graphcoll and CQPweb and conducted seminars using material from the Climate Change and Discourses of Distressed Communities projects to test the students’ frequency, keywords, and concordance analysis skills. The students really engaged with the sessions and were particularly taken with Graphcoll. They enjoyed doing the practical sessions, which they said were different to how they usually learned. Everyone in the classroom worked really hard and asked great questions that showed how interested they were in Lancaster’s tools.

China 2

Weihai is an absolutely beautiful place. The university sits with a sandy beach on one side and a mountain on the other. Because of this, Weihai campus is considered to have good Fung Shui. The place itself was described as a small city by those who live here, but ‘small’ is relative when compared to cities the size of Lancaster. Carmen and I enjoyed our time in China (despite a long journey involving flight cancellations and a trip to a Beijing hotel in the middle of the night) and loved seeing how well the students took to corpus linguistics and the materials that we prepared for them. The trip was a great success and we look forward to future collaborations between Lancaster and Beijing Foreign Studies University.

China 3

Dealing with Optical Character Recognition errors in Victorian newspapers

CASS PhD student, Amelia Joulain-Jay, has been researching to what extent OCR errors are a problem when researching historical texts, and whether these errors can be corrected. Amelia’s work has recently been featured in a very interesting blog post on the British Library’s website – you can read the full post here.

 

40th Anniversary of the Language and Computation Group

Mahmoud

Recently I was given the chance to attend the 40th anniversary of the Language and Computation (LAC) group at The University of Essex. As an Essex alumni I was invited to present my work with CASS on Financial Narrative Processing (FNP) part of the ESRC funded project . Slides are available online here.

The event celebrates 40 years of the Language and Computation (LAC) group: an interdisciplinary group created to foster interaction between researchers working on Computational Linguistics within the University of Essex.

There were 16 talks by Essex University alumnus and connections including Yorick Wilks, Patrick Hanks, Stephen Pulman and Anne de Roeck. http://lac.essex.ac.uk/2016-computationallinguistics40

The two day workshop started with Doug Arnold from the Department of Language and Linguistics at Essex. He started by presenting the history and the beginning of the LAC group which started with the arrival of Yorick Wilks in the late 70s and others from Language and Linguistics, this includes Stephen Pulman, Mike Bray, Ray Turner and Anne de Roeck. According to Doug the introduction of the cognitive studies center and the Eurotra project in the 80s led to the introduction of the Computational Linguistics MA paving the way towards the emergence of Language and Computation. Something I always wondered about.

The workshop referred to the beginning of some of the most influential conferences and associations in computational linguistics such as CoLing, EACL and ESSLLI. It also showed the influence of the world events around that period and the struggle researchers and academics had to go through, especially during the cold war and the many university crises around the UK during the 80s and the 90s. Having finished my PhD in 2012 it never crossed my mind how difficult it would have been for researchers and academics to progress under such intriguing situations during that time.

Doug went on to point out how the introduction of the World Wide Web in the mid 90s and the development of technology and computers helped to rapidly advance and reshape the field. This helped in closing the gap between Computation and Linguistics and the problem of field identity between Computational Linguists coming from a Computing or Linguistics background. We now live surrounded by rapid technologies and solid networks infrastructure which makes communications and data processing a problem no more. I was astonished when Stephen Pulman mentioned how they used to wait a few days for the only machine in the department to compile a few lines-of-code of LISP.

The presence of Big Data processing in 2010 and the rapid need for resourcing, crowd-sourcing and interpreting big data added more challenges but interesting opportunities to computational linguists. Something I very much agree with considering the vast amount of data available online these days.

Doug ended his talk by pointing out that in general Computational Linguistics is a difficult field; computational linguists are expected to be experts in many areas, concluding that training computational linguists is deemed to be a challenging and difficult task. As a computational linguist this rings a bell. For example, and as someone from a computing background, I find it difficult to understand how part of speech taggers work without being versed in the grammatical aspect of the language of study.

Doug’s talk was followed by compelling and very informative talks from Yorick Wilks, Mike Rosner and Patrick Hanks.

Yorick opened with “Linguistics is still an interesting topic” narrating his experience in moving from Linguistics towards Computing and the challenge imposed by the UK system compared to other countries such as France, Russia and Italy where Chomsky had little influence. This reminded me of Peter Norivg’s response to Chomsky’s criticism of empirical theory where he said and I quote: “I think Chomsky is wrong to push the needle so far towards theory over facts”.

In his talk, Yorick referred to Lancaster University and the remarkable work by Geoffrey Leech and the build up of the CLAWS tagger, which was one of the earliest statistical taggers to ever reach the USA.

“What is meaning?” was Patrick Hanks talk’s opening and went into discussing word ambiguity saying: “most words are hopelessly ambiguous!”.  Patrick briefly discussed the ‘double helix’ rule system or the Theory of Norms and Exploitations (TNE), which enables creative use of language when speakers and writers make new meanings, while at the same time relying on a core of shared conventions for mutual understanding. His work on pattern and phraseologies is of great interest in an attempt to answer the ”why this perfectly valid English sentence fits in a single pattern?” question.

This was followed by interesting talks from ‘Essexians’ working in different universities and firms across the globe. This included recent work on Computational Linguistics (CL), Natural Language Processing (NLP) and Machine Learning (ML). One of those was a collaboration work between Essex University and Signal– a startup company in London.

The event closed with more socialising, drinks and dinner at a Nepalese restaurant in Colchester, courtesy of the LAC group.

In general I found the event very interesting, well organised and rich in terms of historical evidences on the beginning of Language and Computation. It was also of great interest to know about current work and state-of-the-art in CL, NLP and ML presented by the event attendances.

I would very much like to thank The Language and Computation group at Essex Universities for the invitation and their time and effort organising this wonderful event.

Mahmoud El-Haj

Senior Research Associate

CASS, Lancaster University

@DocElhaj

http://www.lancaster.ac.uk/staff/elhaj

Tracking terrorists who leave a technological trail.

Dr Sheryl Prentice’s work on using technology to aid in the detection of terrorists has been gaining a lot of attention in the media this week! Sheryl’s discussion of the different ways in which technology can be used to tackle the issue of terrorism and how effective these methods are was originally published in The Conversation, and then republished by the ‘i’ newspaper on 23rd June 2016. You can read the original article here.

Corpus Data and Psycholinguistics Seminar

On the afternoon of Thursday 19th May 2016, CASS held its first ever psycholinguistics seminar which brought together researchers from both linguistics and psychology. The theme of the seminar was “Corpus Data and Psycholinguistics”, with a particular focus on experimental psycholinguistics.

The afternoon consisted of four 40-minute presentations which covered a range of different experimental methods including eye-tracking and EEG. Interestingly, the notion of collocation also emerged as a strong theme throughout the presentations. Different types of collocation were addressed, including bigrams, idioms, and compounds, and this prompted thought-provoking discussions about the nature of collocation and the relationship between psycholinguistic results and the different statistical measures of collocation strength.

The first presentation was delivered by Professor Padraic Monaghan from the Psychology Department at Lancaster University. In this presentation, Padraic provided an engaging introduction to computational modelling in psycholinguistics, focusing mainly on connectionist models where the input determines the structure of processing. This talk prompted a particularly interesting observation about the relationship between connectionist models and parts-of-speech tags in corpora.

In the second presentation, Dr Phil Durrant from the University of Exeter provided a critical perspective on his own earlier work into whether or not psycholinguistic priming is evident in collocations at different levels of frequency, and on the distinction between the related notions of collocation and psychological association. This presentation also provided a really interesting insight into the different ways in which corpus linguistics and psychological experimentation can be combined in psycholinguistic studies. This really helped to contextualise the studies reported in the other presentations within the field of psycholinguistics.

After a short break, I presented the results of the first of several studies which will make up my PhD thesis. This initial study pilots a procedure for using EEG to determine whether or not the brain is sensitive to the transition probabilities between words. This was an excellent opportunity for me to gain feedback on my work and I really appreciate the input and suggestions for further reading that I received from participants at this event.

The final presentation of the afternoon was delivered by Professor Michaela Mahlberg and Dr Gareth Carroll from the University of Birmingham. This presentation drew upon eye-tracking data from a study exploring literary reading in order to pinpoint the methodological issues associated with combining eye-tracking techniques with literary corpora, and with corpus data more generally.

With such an interesting series of talks sharing the theme of “Corpus Data and Psycholinguistics”, the CASS psycholinguistics seminar proved to be a very successful event. We would like to thank the presenters and all of the participants who attended the seminar for their contribution to the discussions, and we are really looking forward to hosting similar seminars in the near future.

TLC and innovation in language testing

One of the objectives of Trinity College London investing in the Trinity Lancaster Spoken Corpus has been to share findings with the language assessment community. The corpus allows us to develop an innovative approach to validating test constructs and offers a window into the exam room so we can see how test takers utilise their language skills in managing the series of test tasks.

Recent work by the CASS team in Lancaster has thrown up a variety of features that illustrate how test takers voice their identity in the test, how they manage interaction through a range of strategic competences and how they use epistemic markers to express their point of view and negotiate a relationship with the examiner (for more information see Gablasova et al. 2015). I have spent the last few months disseminating these findings at a range of language testing conferences and have found that the audiences have been fascinated by the findings.

We have presented findings at BAAL TEASIG in Reading, at EAQUALS in Lisbon  and at EALTA in Valencia. Audiences ranged from assessment experts to teacher educators and classroom practitioners and there was great interest both in how the test takers manage the exam as well as the manifestations of L2 language. Each presentation was tailored to the audience and the theme of the conference. In separate presentations, we covered how assessments can inform classroom practice, how the data could inform the type of feedback we give learners and how the data can be used to help validate aspects of the test construct. The feedback has been very positive, urging us to investigate further. Comments have praised the extent and quality of the corpus and range from the fact that the evidence “is something that we have long been waiting for” (Dr Parvaneh Tavakoli, University of Reading) to musings on what some of the data might mean both for how we assess spoken language and the implications for the classroom. It has certainly opened the door on the importance of strategic and pragmatic competences as well as validating Trinity’s aims to allow the test taker to bring themselves into the test.  The excitement spilled over into some great tweets. There is general recognition that the data offers something new – sometimes confirming what we suspected and sometimes – as with all corpora – refuting our beliefs!

We have always recognised that the data is constrained by the semi-formal context of the test but the fact that each test is structured but not scripted and has tasks which represent language pertinent to communicative events in the wider world allows the test taker to produce language which is more reflective of naturally occurring speech than many other oral tests. It has been enormously helpful to have feedback from the audiences who have fully engaged with the issues raised and highlighted aspects we can investigate in greater depth as well as raising features they would like to know more about. These features are precisely those that the research team wishes to explore in order to develop ‘a more fine-grained and comprehensive understanding of spoken pragmatic ability and communicative competence’ (Gablasova et al. 2015: 21)

One of the next steps is to show how this data can be used to develop and support performance descriptors. Trinity is confident that the features of communication which the test takers display are captured in its new Integrated Skills in English exam validating claims that Trinity assesses real world communication.