Marginal Gains and the Plateau Effect: The Benefits of Evidence Based Practice (EIP)

The above twitter dialogue is a snippet of a conversation I had with Gary Jones and other educationalists on social media about the role of research in education. After almost 15 years as a teacher, and 7 as a senior leader, it would appear that there is at last a strong focus on building a profession based partly on research, to support experience based practice. But to what extent should teachers and school leaders be influenced by the body of evidence presently circulating? Both Gary Jones and Dylan Wiliam offer these perspectives on evidence based practice/disciplined practice:

“Research can suggest to teachers what to look at, it cannot tell you whether it should be adopted, as that is the teacher and school’s decision...a far more relevant and useful question would be: can research help reduce teacher fallibility?”  

and

People often want to know “what works” in education, but the simple truth is that everything works somewhere, and nothing works everywhere. That’s why research can never tell teachers what to do—classrooms are far too complex for any prescription to be possible, and variations in context make what is an effective course of action in one situation disastrous in another. Nevertheless, research can highlight for teachers what kinds of avenues are worth exploring and which are likely to be dead ends, and this is why classroom formative assessment appears to be so promising.

However, personally I find myself nodding in agreement with Alex Quigley’s comments in his blog post: Just Don’t Call It Research about how teachers can best become evidence based practitioners:

I’m sure ‘disciplined inquiry‘ could be mangled to mean what we want it to mean, but I very much like Paul Hood’s helpful breakdown of the characteristics of disciplined inquiry: Meaningful topics are addressed Systematic, clearly described procedures are employed and described so that readers can follow the logic of the study and assess the validity of the study’s conclusion There is sensitivity to the errors that are associated with the methods employed and efforts are made to control the errors or consider how they influence the results Empirical verification and sound logic are valued: and Plausible alternative explanations are considered I like these characteristics because they encourage teachers to apply their practical wisdom – recommended by Wiliam – but in a more disciplined way that we can typically apply in our busy working week, by using  ‘logic‘ and considering ‘plausible alternatives‘.

I also agree with Alex about the need to avoid the cherry picking of research. Therefore, in my attempts to avoid confirmation bias, I regularly read the work of David Didau and Greg Ashman in support of traditionalists approaches to teaching.  Their writings (Didau’s in particular) help me to question some of my deep (progressive) rooted beliefs about teaching.  Didau’s sustained references to cognitive science, and his style of writing always make it difficult to entirely deny his assertions. Especially as he fills his writings with subtle reminders about the need to question one’s beliefs about teaching:

“If your beliefs don’t bear up under close critical evaluation, then maybe. just maybe, you believe in something silly.”

"But the irrefutable evidence that we are so often wrong should at least give us reason to question our intuition. If the cognitive scientists are right, we could make a profound difference to how well our students learn. If all their empirical evidence turns out to be wrong, no one’s died. It may not be worth betting your life on, but it outweighs the risk of going with a hunch.”

Despite Didau’s reminders, research can never tell the complete story.  Teaching that has resulted in outstanding outcomes for my students year on year is evidence that what I am doing works.   Yet there is always room for improvement; and perhaps (for this teacher) evidence informed practice can lead to aggregated marginal gains and help myself and others avoid the plateau effect:

“…There appear to be important gains in teaching quality in the first year of experience and smaller gains over the next few career years. However, there is little evidence that most teachers continue to improve after the first three years.”  Steven G. Rivkin, Eric A. Hanushek, and John F. Kain in 2005

The particular area of my practice that I have been attempting to improve as an evidence informed practitioner is feedback. Whilst feedback has the potential to have a significant positive impact on learning, as evidenced by Hattie’s effects table below:

http://www.learningandteaching.info/teaching/what_works.htm

It also has the potential to have a damaging impact on learning:

"In their review of feedback studies conducted between 1905 and 1995, Kluger and DeNisi (1996) found that in 38% of well-designed studies, feedback actually made performance worse—one of the most counterintuitive results in all of psychology."

I therefore politely request that as you read the remainder of this blog, please try and answer the following questions on my behalf:

1. Have my students been actively involved in the feedback process?
2. Do my students know what to do in order to improve their work?
3. John Hattie stated; “I don’t think it is right that kids come to our schools to watch us work.”  Is this what is happening in my classroom?
4. Has learning taken place?
5. Has evidence informed practice benefited my teaching?

Part 1  - The Learning Environment

Both Dylan and Didau suggests that positive relationships are essential for effective teaching:

"Trust is a better predictor of teacher success than expertise. A teacher may know their subject inside out, but if their pupils don't trust them they won't feel very motivated."  Didau (2015: 344)

and

"Without the relationships all the research in the world won't matter." Dylan Wiliam

So I began, as I always do when joining a new school, with building positive relationships with the students. My series of blogs on creating a classroom culture for learning, has on the whole, been a reflection of this process: We make the weather, Mistakes, Being Human 1, Being Human 2, Along with my class blog Asker International School Grade 7 Blog.

Part 2 - Feedback Strategies

The Task Grade 7 students were required to retell part of the original story of The Three Little Pigs.  Areas to consider: Audience First Person Perspective: Wolf or the Third Pig or alternative characters Contemporary version Speech Internal Monologue SPaG

In line with the International Baccalaureate's philosophy of backward planning, students were given the rubric at the beginning of the unit.  The language of the rubric was not dumbed down - I am of the opinion that students need to work within the language of the domain they are studying. Students must therefore become accustomed to using this language in the present context, and any future context.  Whilst I am a fan of the use of rubrics for backward planning, I am also conscious that they potentially have a number of pitfalls, pitfalls I am confident can be overcome providing certain step are taken. For example, in an article published by Faculty Focus Maryellen Weimer states:

“In the study, research reports written using the rubric were significantly better than those written not using a rubric. It is fair to ask whether use of the rubric improved their research report writing in general or only this one time on this one assignment.”

One must therefore be aware that whilst the use of a rubric appears to benefit short term performance, there is no suggestion from this research that this performance will lead to improved performance within a different context, or a future context that requires the same knowledge and/or skills. However, Daniel Willingham argues that:

“Knowledge tends to be inflexible when it is first learned. As you continue to work with the knowledge, you gain expertise; the knowledge is no longer organized around surface forms, but rather is organized around deep structure...For example, your knowledge of calculating the area of rectangles may have once been relatively inflexible; you knew a limited number of situations in which the formula was applicable, and your understanding of why the formula worked was not all that clear. But with increasing experience, you were able to apply this knowledge more flexibly and you better understood what lay behind it. Similarly, it is probably expecting too much to think that students should immediately grasp the deep structure beneath what we teach them. As students work with the knowledge we teach, their store of knowledge will become larger and increasingly flexible, although not immediately.”

Possible Solution: The use of spacing in conjunction with unit rubrics can ensure that the short term gains made by using rubrics can be sustained over time, and hopefully ensure that the information is stored in long term memory: “If nothing has changed in long-term memory, nothing has been learned.” Kirschner, Sweller & Clark

Teacher Feedback

My next step was to create opportunities for feedback to the students that matched my philosophy: that feedback is most effective in promoting learning if it involves the students in the process of deciding what the next steps should be, so that they are not passive recipients of the teacher’s judgments of their work. The steps provided below have attempted to stay true to this philosophy.

Firstly, I provided written feedback to students to consider based on the rubric, and the school literacy marking code.  The feedback was in response to students writing a first draft that only contained the first five sentences of their text.  I wanted the students to focus on producing quality rather than quantity to begin with.  Feedback (in all forms) would simply be focused on improving these initial first five sentences.

Secondly, in accordance with Andrew Quigley's advice, I also had a quick chat with the students to ensure they understood the comments/code provided in an attempt to avoid misunderstanding:

"Too often our elegant written commentaries are ignored or simply misunderstood.  We need to talk it through."

The need to talk to my students about the written feedback is particularly important considering that English is the second or third language for many of the students in my class.

Grade 7 Asker International School

Although, I am aware that Didau would suggest that this form of feedback simply implies mimicry rather than learning.

Students Responses To Teacher Feedback

Dylan Wiliam's argues that there is no point giving feedback unless you allow time to use the feedback, and that feedback should be more work for the student than it is for the teacher.  As such, students were given time during the following lesson to write their next draft.

Student A: first language Norwegian, second language English

Student B: first language Chechnyan, second language Norwegian, third language English (started learning 18 months ago)

Preparing Students for Peer Feedback

A recent study that focused on learning in the IB Primary Years had this to say about peer assessment:

“Teachers were more equivocal about peer assessment, expressing doubts about its value as a valid assessment tool and cautioning about the risks of using peer assessment or feedback in the absence of strong modelling for students.” IB Research Summary Assessment 2016

In contrast, Hattie's comments regarding peer assessment paint a much more positive picture::

“A lot of students gain a tremendous amount of learning from other students in their class...As you are learning something and you are starting to get a grasp of something, when your fellow peer or student says it correctly you are more likely to learn it than if another teacher says it or you read it again.”

I am of the opinion that as teachers it is our role to ensure that students are equipped with an understanding of how to conduct peers assessment, and that as Dylan Wiliam suggests ensure the feedback is cognitive rather than emotional, which is the benefit of having a rubric to work with.  As such, prior to the peer feedback session students wrote down what they should look for based on their knowledge of the assessment rubric. I also intentionally avoided students placing this information into grade bands due to the potential issue surrounding interpretation of the various verbs that are often used to distinguish one grade band from another:

"A rubric is not like an engineering standard, Sadler explained. In such a standard, we can specify to the millimetre something quite explicit. Instead, a rubric will contain vague words such as ‘coherence’ and ‘flow’ and will have a graduation in performance that will move from ‘demonstrating a good command of sentence structure’ to ‘demonstrating a sophisticated command of sentence structure’ and so on. The danger is obvious; different teachers will interpret these statements differently." Ashman (2016: 95)

Additionally, I did not wish the students to align their marking to a particular grade band, rather to simply offer advice that would help move a fellow student forward in their (apparent) learning. This was to avoid the issue of students simply focusing on/or trying to work out the grade the feedback comments might be aligned to:

"When teachers pair grades with comments, common sense would tell us that this is a richer form of feedback. But our work in schools has shown us that most students focus entirely on the grade and fail to read or process teacher comments. Anyone who has been a teacher knows how many hours of work it takes to provide meaningful comments. That most students virtually ignore that painstaking correction, advice, and praise is one of public education’s best-kept secrets"

An interesting development from this process was that some students brought new criteria in the form of descriptive writing techniques that they referred to as rich vocabulary.  What transpired next was a conversation about the ways in which the students could enrich the language they were using. I therefore had a decision to make: either direct the students to focus only on the original rubric requirements, or to allow the students to make comments on additional elements attached to the notion of good writing.  In this case my response to allow additional information beyond the requirements of the rubrics was influenced by both David Didau's and Greg Ashman's comments on the use of assessment rubrics.  Didau states the following relating to the restrictive nature of rubrics:

“Using mark schemes to assess students’ work narrows the validity of what constitutes good writing to what’s on the mark scheme. If it’s on the rubric, we credit it, if it’s not, we don’t.”

Greg Ashman also appears to be of the same opinion when he writes:

"Imagine the following scenario: we have used our expert concept of quality or a sample of papers to derive a set criteria that distinguish between essays of differing levels of quality. These criteria are only a sample of all the aspects that we could notice but these are what we write into our assessment rubric. Teachers then view this rubric as a target and specifically focus on teaching students to demonstrate evidence of meeting these specific criteria, perhaps at the expense of other aspects of the performance." (Ashman 2016: 97)

I am inclined to think, that I had not recently read the work of both of these two educationalists then my response would have been to request that the students remain to the confines of the original rubric (anchoring effect maybe?). Instead, I saw this as an opportunity to tap into student curiosity, which is the bedrock of the inquiry based classroom. This was followed up by a series of lessons on using the senses to invoke descriptive writing similar to the information in the clip below.  It was also a classic example of how as teachers we should ensure that we teach the students and not just the subject.

Peer Feedback

The students did not conduct peer feedback in their usual classroom.  Instead, they moved to a neighbouring classroom.  This was part of my drive this year to adhere to the following advice:

"If you’re after sustained improvement then you want to introduce as much variability into your teaching as possible: change rooms, change seating, change displays, remove the comforting and familiar background to lessons; mix up topics.  These desirable topics will slow down performance but will lead to increased long term retention and transfer of knowledge between contexts.” (Didau 2015: 127)

Grade 7 Asker International School

This clip is on the school blog and in the public domain

Checking Peer Feedback

When you read about research that suggests that 80% of the feedback students get on their work is from each other, and 80% of that is wrong, then alarm bells perhaps start ringing.  But only as a reminder to ensure certain strategies are employed. First I requested that students read out the feedback they had received from their peers:

 I also made certain that on my travels around the classroom I checked the written feedback for incorrect advice.  And whilst there were some mistakes, I agree with Alex Quiqley that these are opportunities for clearing up misunderstandings.  It would appear from my experience that the potential positive impact of peer assessment definitely outweighs the potential negatives providing teachers are aware of its possible pitfalls.  

Draft 3

Student were requested to write their next draft based on the feedback from the peer assessment session, and the lessons relating to the senses and descriptive writing.

                               

Self Assessment

Unlike peer feedback, self-assessment appears a favourable strategy in IB schools: “Self-assessment was highly valued for its ability to build reflective lifelong learners.” IB Research Summary Assessment 2016

In groups, students created the self assessment sheet (without access to the original rubric):

 

Students then completed the document individually with the addition of two individualised areas for improvement:

Final Version

Student B: first language Chechnyan, second language Norwegian, third language English (started learning 18 months ago)

Student C: first language Mandarin, second/third language Norwegian/English

Final versions can be found here

Reflection

1. Have my students been actively involved in the feedback process?
2. Do my students know what to do in order to improve their work?
3. John Hattie stated; “I don’t think it is right that kids come to our schools to watch us work.”  Is this what is happening in my classroom?
4. Has learning taken place?
5. Has evidence informed practice benefited my teaching?

What Next… Spacing: Redesigning the Grade 6,7 & 8 MYP Language and Literature Curriculum

Creating a Classroom Culture for Learning: Well-being (part 2 guilt)

There will always be more to do...

/

In part 1 I argued the need for teachers to fully engage in a positive mindset towards well-being, otherwise attempts to have a positive work life balance will be futile.  Although pressures placed on teachers by school leadership, and short term knee jerk reactions by government ministers often make it difficult for teachers to have a sense of well-being.  In my opinion, the biggest barrier to achieving such a balance is ourselves.  We often refer to our employment as a vocation rather than a job.  We talk about teaching as a calling.  We surround ourselves with emotive language to remind ourselves of the importance of our profession.  Phrases such as

"We are not just teachers we are managers of the worlds greatest resource: Children!"

"Every child matters"

"No child left behind"

"What sculpture is to a block of marble, education is to the human soul"

"Education is not filling of a pail but the lightening of a fire"

"Squeeze every last drop of learning out of every lesson"

are constant sources of inspiration but also place a heavy burden on our shoulders. And often this burden goes by the name of guilt.  The kind of guilt caused by thinking that you are not doing enough to help someone or some people.  Perhaps you have a student or students who is/are constantly struggling to understand a certain concept, but now you have other obligations during the weekend you must fulfil.  Or you receive an email form a parent at 9pm about how his/her daughter is frustrated and upset that she does not seem to understand the Order of Operations in Maths, yet her work shows that she has a grasp of the basics.  This is despite the fact that you have already given days and weeks of your free time considering this student when planning your lessons, but now you find you can't continue to do so.  Why? Perhaps your own children want to have you for a little while, or you life partner would like to hang out, or other students warrant your attention. Whatever the reason the guilt now starts to get to you, yet you continue to try to figure out ways to help the students despite the toll it's taking on you. Psychologists use the term 'compassion fatigue to explain this feeling of being burnout.

Remember it is your choice whether you wish to continue to make sacrifices needed to help the students, however also remember that acting out of guilt can only drain you further and make you a less effective teacher.  Your well-being is important to the long term success of your students.  If you can't maintain your health, how are you expected to control the climate in your classroom.

Creating a Classroom Culture for Learning: Being Human Part 2

In order to build positive relationships with students I have put forward the idea that it is not enough to just know your students but that they must also know you (Being Human part 1). Whilst allowing your students to see you as something more than a teacher might make some teachers feel uncomfortable, should it help foster positive relationships, then surely this is an avenue worth pursuing.  Hattie's diagram below is a gentle reminder about the potential impact positive teacher student relationships can have on student achievement, with at least three of the factors in Hattie’s diagram related to the teacher student dynamic: teacher-student relationships; not labelling students & expectations.

An effect-size of 1.0 is typically associated with:

• Advancing learners' achievement by one year, or improving the rate of learning by 50%
• A two grade leap in GCSE, e.g. from a C to an A grade
• An effect size of 1.0 is clearly enormous! (It is defined as an increase of one standard deviation)  1

With the above in mind, below are a few more strategies you may wish to consider in an effort to show the students you are human.

i) Personal Anecdotes

1. Anecdotes show that you are human, something more than a teacher.  
2. Anecdotes are a tool you can use successfully (sometimes) to make a point.
3. Anecdotes help you avoid being a hypocrite. I think it is important that we model expectations.  
4. Anecdotes allow you to model the idea of reflection.  I have rarely met a teacher who did not ascribe to the notion that schooling is about developing the whole child.  Yet whilst modelling positive behaviour, modelling expected outcomes, and modelling language are all generally accepted norms/expectations in many classrooms.  Modelling reflection (by a teacher) beyond the realm of school work is not common discourse in schools.
5. Anecdotes sometimes enable students to be comfortable in sharing their thoughts and feelings about the themes, topics, issues covered in class.  It helps build mutual trust, and helps you create open dialogue.

Example Anecdote

During my honeymoon my wife and I climbed Half Dome in California, We set off at 5 am and made it to the top at around 2 pm, we were back down by 7.15 pm.  It is one of my most important memories from my honeymoon road trip around California. Important because I got so much wrong.  I failed to pay attention to the experience of the journey, take in the scenery, talk to fellow walkers, and enjoy the company of my life companion. Instead I was only intent on achieving my goal of getting to the summit and back down again by nightfall.  This is an anecdote I have used to remind students (and myself as a school leader) that completing work is not always about finding the right answer to a problem, it’s also about the journey taken.  For example, students are sometimes so intent on finishing a piece of extended writing quickly they forget to consider all the ingredients necessary to produce a quality piece of work (slow writing). With regards to Maths, how often do students complete maths problems without showing their working out? And how often do we witness poor teamwork by students (and adults) in the pursuit of getting work completed quickly...

However, sharing personal anecdotes does come with a health warning.  Be careful that the classroom does not become about you. This is simply a tool to ensure lessons are about them.

ii) The Playground

When on break and/or lunch duty get involved.  Talk to students about what book they are reading, or if they have watched any movies lately, or what they think of Taylor Swift’s latest song, or squeeze.  I would even suggest involving yourself in some of the playground activities such as hopscotch, skipping, basketball, football, table tennis, or even sledging/sledding:

Mr Mark Asker International School January 2015

iii) The Corridor

Smile and say 'Hi' to students on the corridor.  Remember the journey to the staffroom or the classroom is just as important as reaching your final destination.

1 http://www.teacherstoolbox.co.uk/T_effect_sizes.html

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License

Good Reads: Didau and Tomsett

5/5 for both books: 

What if everything you knew about education was wrong? David Didau

This much I know about love over fear. John Tomsett

                                        

"Arguing with teachers who have a vested interest in being right about a particular ideological view of education is like presenting fundamentalist Christians with the incontrovertible evidence of evolution.  It just bounces off them and makes them even more convinced that they are right and you are wrong.  We desperately scramble to reduce dissonance by justifying beliefs in the face of disconfirming evidence and often dismiss it as the exception that proves the rule." (Didau 2015: 50)

For 14 years I have followed my faith, building a bank of skills and strategies in a constant effort to be a better servant.  I have been lucky to have helped convert many young people to the path of righteousness, confident that the tools at my disposable would stand the test of time.  However, my faith has been recently rocked by the release of another sacred text. One that has declared that much of what we believe is false.  The book: "What if you knew everything about education was wrong" by David Didau,  has challenged some of my strongest convictions, (or should I now say meta-beliefs) regarding education.  But rightly so!

Ok...perhaps I am exaggerating a little, but the book has certainly made me ask serious questions about my own pedagogical views. Particularly my lack of self scrutiny regarding assertions I have always assumed to be true.  Assertions I have actively passed confidently on to my colleagues.  Assertions that I have never attempted to prove, simply because they feel right.  How could I come to doubt some of my views on education so easily you may ask? It's only a book you say.  True, but it is a very intelligent book, especially the first 4 chapters, which are infuriatingly clever.  I almost guarantee that by the end of chapter 4 Didau will have lured you into the uncomfortable realisation that should you dismiss his ideas, then it is most likely due to one of many theories regarding bias you have just read, rather than as a result of any sound evidence.   With these theories anchored to your thinking, it becomes easy to accept that what Didau is preaching about education is potentially true.  In my case, the fear of cognitive dissonance, allowed me to be open to his line of thinking. Extremely annoying, but at the same time, extremely refreshing!

Whilst Didau's book plays with your mind, is calculated and clinical.  Tomsett's book is warm, fuzzy, and tugs at your heart.

What is refreshing about Tomsett's approach is that he presents himself as a fallible headteacher who has made mistakes in his personal and work life.  I hope most readers sees this as a strength rather than a weakness - as I do.  His comments regarding the need for headteachers to avoid losing their emotional intelligence - to always work in adult to adult mode, rather than adult to child mode when talking to staff is a fitting end to a book that provides a strong argument for servant leadership, one that recognises the importance of teachers in schools, rather than as a means to an end.

In writing this post about Tomsett's book I am sending a message to both my present and future self. The message to my future self  - between 8 and 10 years in the future to be exact is advice about parenting:

"Cars are a great places to talk about profound things.  If you have a teenage son who needs to tell you something, pop him in the passenger seat and drive - he'll open up in minutes because he can converse with you whilst you both stare straight ahead." (pg 167)

I have two boys, and judging by their character, I have a feeling I will be driving them both around a lot in the future.  With regards to the present, Tomsett’s book reminds me that my upbringing should not be remembered as a barrier that I had to overcome to be where I am today, but instead a ladder to success.  The qualities and values that my parents instilled in me, and the experiences that came from a working class upbringing is why I have succeeded in the world of education, not despite of it.

Creating a Classroom Culture for Learning: Well-being (Part 1)

Creating a Classroom Culture for Learning: Put your Oxygen Mask on First.

There are two reasons why airlines tell parents to put their oxygen masks on before their children when in an aeroplane. First, if you are unable to think clearly due to a drop in oxygen, you won't be able to be much help to yourself or your child. Second, if you become anxious, it will cause your child who looks to you to be calm in a crisis to become more anxious.  If you do not have children, substitute them for someone who would require your help in such a situation.

Now swap the aeroplane for your classroom; the teacher is the parent and the children are your students.  To most teachers it may seem like a contradiction to think that by looking after yourself first and foremost, you are looking after your students.  Teacher well-being is so important to ensuring that a culture of learning flourishes in the classroom.  I have mentioned previously about the importance of the teacher in controlling their own classroom climate, and the importance of teacher-student relationships.  How are these likely to be sustained if we let our own well-being deteriorate?

So I urge you to put on your own oxygen mask first when necessary. Your own well being is vital for student success. If common-sense  is not convincing enough then please consider the following:

"Researchers have long known about the infectious nature of stress. Pass-along strain runs rampant in relationships and work settings. Studies have shown that there is "crossover" stress from one spouse to the other, between coworkers, and "spill over" from the work domain to home. The stress contagion effect, as it's known, spreads anxiety like a virus. Our mirror neurons help suck us into the emotional eruptions of others."  1

The potential for your own emotional and mental state to impact on the classroom should be sufficient motivation to care for one’s well-being.  Keep the weather in your classroom fit for learning, don’t be the weather front that brings forth high winds that disrupt the learning environment.

If teachers are to be truly receptive to strategies supporting well-being, then they must embrace a well-being mindset. The first of The Five Pillars of Islam (Shahada) is the belief that there is no God but Allah, and Muhammed is his prophet (PBUH).  The other four are actions.  Actions that you would only complete should you first believe in the Shahada. One of the hardest things for a committed teacher to believe is that occasionally putting his/her own well being first is not selfish but selfless. Once you believe, then hopefully you will act accordingly, and look for opportunities to promote your own well-being.

For the benefit of your students be strong enough to take a break, put yourself first and put your oxygen mask on.

1 http://www.huffingtonpost.com/joe-robinson/dealing-with-stress_b_4097921.html