R.E.A.L. Projects – Critique, assessment and tuning

This post is part of an ongoing series written by our PBL Lead Laura Jackson, about how we are implementing R.E.A.L. Projects in school.

As I said when I ended my previous post, I feel very excited when I see the staff project folders with exciting projects in them. There is a summary of all of these at the end of this post and I’m sure you’ll agree that these projects will give our students an exciting variety of different learning experiences during our projects week. I think it’s important to detail how we got to this place in such a short space of time.

Back in February, all staff spent time with our PBL coach, Cara, planting “seeds of passion”- to try to find a way to ignite little sparks of exciting energy to then develop and create into Rigorous, Engaging, Authentic Learning experiences.

These words are vital but how can we apply rigour? How can we ensure projects are engaging? How can authenticity be validated? All of these need to be encapsulated within a learning experience. What do they actually mean? How do we do it?

The Innovation Unit says:

 

 

We also had to decide how to assess each project and also projects had to be “tuned” to make sure they had been given critique, and that and questions the project leader or the critique group may have had about the project were acknowledged and addressed if needed.

Project Tuning

One of the most important parts of project work is the giving and receiving of critique. This happens at all stages of the project but the earliest stage would be at the “tuning” stage. This was also possibly the most challenging part to co-ordinate from my point of view as each project needed to be tuned but also as part of a group, who would be the critique providers. 

My critiqued creation from our session with Cara

In preparation for this, Curriculum Leaders took part in a Critique and Tuning workshop with Cara to allow them to take this process into their Project groups. I then led a critique workshop for the rest of the staff which ensured we all felt it. If you have an emotional connection with something then you feel more passionate about it and believe in it more. If people start changing things, adding bits, taking bits away then it changes YOUR work and YOUR emotional connection with the work.

It was important to me that all staff took part in this session and it was also a light-hearted way to introduce critique to staff, but with a serious message. If critique is given in the wrong way to anyone- staff creating, developing or delivering the project or students who are completing the project, then this would not be useful to anyone and would have a detrimental effect on the person receiving the critique. As adults, we are very protective of our own ideas and values and if they are questioned in the wrong way then it becomes a negative experience and this should not be the case. This is also true of our students we work with day-to-day.  This task made me think about how we self and peer critique in lessons and I intend to use the same session with students towards the end of the academic year, time willing.

Working with Cara has also introduced me to a fantastic book- The Power of Protocols which is extremely useful in terms of gaining the maximum usage out of a set amount of time.  This way of working was also used in a session I attended at School 21 in February when we were discussing enabling conditions for projects in schools. Although I first thought using protocols seemed very regimented, I quickly realised that having set times for different parts of discussion, meeting and critique work is actually extremely useful and stops a lot of time being wasted just chatting or wandering off topic.

To tune projects, there is a need to be very specific and concise to articulate the project to others. In the protocol there are also “norms”

• Hard on the content, soft on the people,
• Share the air (or “step up, step back”)
• Be kind, specific and helpful

This allows all the conversation and discussion taking place to be focused solely on the project and the content.

Within the tuning protocol, there are also different roles :

• Facilitator -Begins the session by reading through the protocol. During the protocol, the facilitator is responsible for keeping the discussion on point and reminding participants to adhere to the norms.
• Timekeeper– Keeps time for each section and lets the facilitator know when time is up.
• Presenter – Shares his/her work with the group.
• Critical Friends – Includes all people present except for the presenter.

During the tuning process, the presenter gets time to present their project idea and the area or aspect they would like critique on. The critical friends then ask clarifying questions to gain a deeper understanding of any aspect of the project or specific area. The critical friends then discuss the project with the presenter taking notes on their discussion without being involved and then they respond to this discussion at the end. The response could take the form of addressing some of the issues discussed by others, drawing conclusions from what has been said or discussed or simply acknowledging the points made by others.  The facilitator introduces each part of the protocol and the timekeeper ensures the protocol timings are adhered to. Timings may differ depending on time constraints- we tuned projects in 15 minute blocks (2 mins- 3 mins- 8 mins- 2 mins) but clearly timings and therefore the timings within each part of the protocol can be changed as needed.

Project Assessment

As well as tuning our projects, we also needed to decide on how our projects would be assessed.

“Academic rigour- head, hand and heart” was my message from the School 21 retreat. Within our discussions in school also, the learning that would take place during the week needed to be meaningful, exciting and also rigorous. The word rigour again:

The Innovation Unit use Buck Institute for Education’s “21st Century Skills” framework to assess PBL work. The Buck Institute website contains a myriad of resources, ideas and information about PBL from all over the world and contains some fascinating work and reading. Their twitter account is definitely worth a follow.

We focused on:

ICT Literacy

• how to find information
• how to use equipment

Cognitive Skills

• Critical Thinking
• Creative Thinking

Metacognitive Skills

• Self- management
• Reflection

Personal Characteristics

• Risk Taking
• Accountability

Interpersonal Skills

• Communication
• Collaboration

As Curriculum Leaders, we discussed our projects with one another and looked for assessment opportunities within these discussions. Once we had these then we put them into the categories above, to allow us to see where the most likely assessment opportunities lay in our range of projects.

From this we then decided on three assessment strands which would be able to be assessed in any project taking place. We also added a learning target to each one to show what would need to be achieved in each assessment strand.

Communication – I can communicate my learning to others in an appropriate way.

Gathering Information – I can source, locate and investigate appropriate information using ICT.

Cognitive Skills – I can reflect and adapt to overcome challenges in my learning.

After further discussion with Cara on a different day, we also decided that to assess all three in one week would be very difficult, especially because critique could lead to multiple drafts. It could also potentially dilute the quality of assessment and critique in the project. Only one strand will be assessed in each project and this is to be decided by the project delivery staff on each project.  This is another benefit of using Cara’s expertise as she works in this field and has experience in this area- I, and we, are still developing our skills all the time and this small tweak means that time will be used effectively and assessment will still be rigorous which is what we require.

From this, short term learning targets have been created within the project plans to individualise the learning target for the projects.

Finally, I would also like to share the variety of projects staff have created and the essential questions driving the projects.

Project Title	Essential Questions
Would you feed your hungry neighbour?	1 in 5 UK mums regularly skip meals to be able to afford to feed their children – is this fair? If your neighbour was hungry would you help them? Almost 1 million people living in the UK needed emergency food supplies from 2013-14, how can we help?
Zombie Apocalypse	Are you a survivor? How would society cope with the breakdown of civilisation? Would you fight or flight?
Flotsam and Jetsam	How can we rethink our current use of material ? What materials can be Salvaged/re-used ? Can another use for a waste material or product be found? How can we create a more sustainable lifestyle ?
Orienteering in the local community	What is orienteering? Why is orienteering a sport that anyone can do? How can we get our local community involved in orienteering?
Mood Music	Can music change your mood? Do we respond differently to different kinds of musical stimulus? Can music help us to relax/prepare for work/get us angry?
Travel Tracks	If you could go anywhere in the world, where would you go?
Maths Movie Making (M3)	What’s the problem with maths? Can I make maths interesting for all students?”
Afternoon Tea	Can we prepare and deliver an event for the local community?
Community	Who were the Durham miners? What can we learn about community from the Durham miners? How has coal shaped our community?
Set, Props and Costume	How can the set, costumes and props enhance the performance of Grease?
Does Fashion have a price?	What is the true price of fashion?
When in Rome…	How have the Romans left their mark on the North East? How did the Romans on Hadrian’s Wall live? How did the famous gladiators fight for survival in the arena? What can we learn from the Romans?
Vikings	Vikings: Bloodthirsty raiders or cultured traders?
Durham Cathedral through your own eyes	What are the secrets of Durham Cathedral?
Kidz for Kidz	What kind of books do 5-6 year olds like to read? How can we make reading enjoyable for young children? How can we encourage and inspire young children to become more independent readers?
Films	Do films dictate our morals today?
Rock Climbing	Could I climb Mount Everest?
Seaham	Will we be able to visit Seaham beach in 50 years time?
Science: Great Beauty or Horrifying Monster?	Do you consider Science beautiful or does it disgust you?
21st Century Britain	What is British?
Trashion	Could you make clothes out of the things you throw away?

I intend to create a daily diary blog during our Projects Week – 29th June 2015 to document the week as it happens.

 

Organising instruction & study: 7 recommendations to improve student learning

This blog is a summary of a Practice Guide by Pashler et al. from 2007, which sets out to provide teachers with specific strategies for instruction and study.

I came across it in a roundabout way via this paper by Dunlosky et al cited in the “What makes great teaching? Review of the underpinning research” by Rob Coe et al.

The central tenet of this particular Practice Guide is that learning depends on memory, which can in turn be strengthened by concrete strategies. These strategies help students to master new knowledge and skills, without forgetting what they have learned.

A note on Practice Guides

The Health Care professions have been using practice guides for some time now to communicate evidence-based advice to medical practitioners.

The recommendations contained within Practice Guides are intended to be:

• Actionable by practitioners
• Coherent in their approach
• Explicitly connected to the level of supporting evidence

Levels of evidence are determined by the types of studies used to draw conclusions, ranging from stronger levels of evidence that come from RCTs, with more moderate levels of evidence coming from non-randomised studies, down to lower levels of evidence that are drawn from the opinions of respected authorities.

Practice Guides are not systematic reviews or meta-analyses that have been subject to detailed literature surveys. Instead they rely more on their authors’ expertise to identify the most important research relative to the recommendations made, in order to characterise its meaning and provide specific, actionable steps. The recommendations contained in this Practice Guide have been agreed by the authors concerned and subjected to independent peer review.

Recommendations

1: Space learning over time – moderate level of evidence

Most of the research in this area has been focused on the acquisition of facts and remembering definitions of terms. To improve retention, students should be exposed to material at least twice, with a delay of weeks to months between exposures. Short delays of less than about 5% of the time between exposure and testing should be avoided. In other words, if you want students to remember material for a test in 6 months time, avoid re-exposure within less than a week or two. “Overshooting” the delay is better than reviewing too soon.

Teaching strategies:

• Regular, in-class review of previously covered material.
• Inclusion of previously covered material in homework assignments.
• Mid-term and final testing that includes cumulative material.

2: Interleave worked examples with problem solving exercises – moderate level of evidence

Experiments and some classroom studies have shown that students learn more when switching between studying examples of worked-out solutions to problems and solving similar problems independently. In the studies cited, alternating and interleaving was more successful than giving students only problems to solve, or a block of worked examples followed by a block of problems to solve. Increasing the amount of variability between successive examples and problems was also beneficial. The scaffolding provided by the worked examples should gradually be removed with time by “fading” more and more stages of the worked examples into problems.

Teaching strategies:

• Provide a worked example for every other problem, e.g. for 10 questions, make questions 1, 3, 5, 7 and 9 worked examples.
• Model the solution to a problem with the class, and then ask students to solve the next problem independently (just one!)
• Ask some students to present their solutions, while others explain the steps (a worked example in its own right) followed by another problem to solve independently.

3: Combine graphics with verbal descriptions – moderate level of evidence

Adding visual representations, e.g. graphs, diagrams or other graphic formats to text descriptions can lead to better learning than just using text. Any accompanying text should be positioned as close as possible to the relevant section of the diagram. This can be further improved with the use of verbal descriptions to accompany visual representations, which allow for both elements to be scrutinised simultaneously.

Teaching strategies:

• Provide visual representations to support the explanation of processes or concepts.
• Highlight the relevant parts of the visual representation while describing processes or concepts.
• Using simplified diagrams that show the relevant parts, rather than more complex representations is sometimes more beneficial.
• Share multiple visual representations, e.g. pictures, models, real objects etc. to illustrate how a single concept can be depicted in different ways.

4: Connect and integrate abstract and concrete representations of concepts – moderate level of evidence

Here, the research seems to suggest that teaching concepts using only concrete representations supports initial understanding but doesn’t support transfer to novel, but relevant contexts. Whereas, using only abstract representations initially can take longer to develop initial understanding, this greater initial difficulty is compensated for via improved application to different situations.

One proposal suggested to utilise the benefits of both is “concreteness fading” with initial concrete examples being gradually and systematically replaced by more abstract representations. Another is to explicitly identify and draw students’ attention to the relationship between the concrete and abstract components in representations of the same concept.

Teaching strategies:

• Show the same idea in multiple forms to show that deep structure is constant despite surface changes.
• Connect abstract ideas to relevant concrete representations and situations.
• Highlight relevant features across both abstract and concrete representations.
• Avoid using the same type of example repeatedly, i.e. examples all from one area, e.g. “sports”
• Avoid knowledge becoming “inert” by allowing time to draw connections between multiple, interleaved examples that vary in their concreteness or abstractness.
• Anchor new ideas in stories or scenarios that are familiar and interesting.

5a: Use pre-questions to introduce a topic – low level of evidence

Pre-questions are thought to activate prior knowledge and focus students’ attention on the material to be learned.

Despite recommending it as way to improve student learning, the panel deemed the level of evidence for quizzing to be low, as most of the research had been completed with college students, or based on laboratory experiments carried out on reading from written text, rather than tested as a component of regular classroom instruction.

The research does seem to suggest, however, that when pre-questions are used to preview the content of assigned material, there will likely be gains in learning of the pre-questioned material, providing students don’t read selectively based on the content of the pre-questions used.

Teaching strategies:

• Direct students’ attention to important facts and concepts by using pre-questions to introduce new topics.
• Prepare several pre-questions that students can attempt immediately on entering the lesson as part of the “do now”

5b: Use quizzes to re-expose students to information – strong level of evidence

The act of practising recalling information from memory enhances learning, reduces the rate of forgetting and cements information to memory.

Laboratory experiments across a wide range of materials and ages have repeatedly demonstrated that testing promotes remembering of material on a later test, and is almost always more powerful than spending additional time studying material.

Teaching strategies:

• Take every opportunity to prompt students to retrieve information.
• Use closed book quizzes after teaching material, prior to final testing.
• Ensure corrective feedback is provided following testing to ensure errors don’t remain.
• Use websites, e.g. http://www.quia.com to share or create quizzes.

6a: Teach students how to use delayed judgement of learning techniques to identify concepts that need further study – low level of evidence

The evidence in support of this recommendation comes mainly from experimental research in the laboratory, rather than in the classroom.

Without training, most learners cannot accurately assess what they know and what they don’t, and typically overestimate how well material has been mastered – “the illusion of knowing.” Knowing what you have and haven’t mastered accurately, is therefore essential in identifying what you still need to spend time studying, which in turn increases the likelihood of performing better when tested.

The “cue-only delayed judgement of learning procedure” is thought to be a key technique for breaking this illusion, which works as follows:

1. Students should test their mastery of material after a meaningful delay.
2. Students should only have access to “the cue” and not the answer when testing whether they know concepts or not, i.e. multiple choice questions should not be used for this purpose.
3. Students should judge how likely they are to get the answer right, as well as answering the question.

A similar technique, the “delayed keyword technique” supports students to judge how well they have retained material they have read after a delay, for example a section of a textbook or a chapter of a book, by asking them to generate keywords or sentences that summarise the main points.

Teaching strategies:

• Pre-prepare 10 questions (for example) that capture the core content to be learned.
• Give the students the questions one at a time, asking them to use a scale of 1 to 100 to judge how likely they feel they would be able to answer the question correctly tomorrow
• Ask students to review the material, use a text or ask the teacher to find out and record the answer to any question they did not score as 100.
• Use repeatedly over the course of the year, teaching students how to use this technique independently.
• Teach students to use the “delayed keyword technique” to generate four key terms and definitions following assigned reading out of class, followed by re-reading if they are unable to do this.

6b: Use tests and quizzes to identify content that needs to be learned – low level of evidence

The evidence in support of this recommendation comes mainly from experimental research of college students and laboratory tests, rather than in the classroom.

As previously stated: Inaccurate judgements by learners of what they have and haven’t learned well can mean subsequent study is focused on the wrong items.

Quizzing is thought to help students identify which items are not well learned, as does re-reading material when a test is interspersed between readings.

Teaching strategies:

• Closed book quizzes following presentation of material.
• Very short “spot check” quizzes covering material from the previous night or prior classwork.

7: Help students build explanations by asking and answering deep questions – strong level of evidence

The evidence base includes over a dozen experimental studies each, in both school and college settings, plus a large number of laboratory experiments.

Shallow knowledge is concerned with basic facts or skills, whereas deep knowledge is when learners are able argue with reason and logic, explore relationships between facts or concepts or answer “why?”

Interventions that specifically train students how to ask deep level questions while studying new material, e.g. classroom discussion, provision of exemplar materials and modelling how to ask and answer questions, have been shown to improve the rate and depth of student questions, as well as their comprehension of the material.

Teaching strategies:

• Identify and prepare deep level questions that require deep level responses.
• Ask questions that challenge students’ prior beliefs and assumptions.
• Model the process of asking and answering deep questions.
• Model and encourage students to “think aloud”.
• Encourage students to respond to explanations by their peers.
• Allow plenty of time to answer deep level questions.

---

The IES Practice Guide: Organizing Instruction and Study to Improve Student Learning by Pashler et al (2007) is available here.