Learning Theory

Retrieval or Why Asking Questions and Answering From Memory is the Best Way to Learn

‘It is beyond question that activities that promote effective encoding, known as elaborative study tasks, are important for learning (6). However, research in cognitive science has challenged the assumption that retrieval is neutral and uninfluential in the learning process (7–11). Not only does retrieval produce learning, but a retrieval event may actually represent a more powerful learning activity than an encoding event. This research suggests a conceptualization of mind and learning that is different from one in which encoding places knowledge in memory and retrieval simply accesses that stored knowledge. Because each act of retrieval changes memory, the act of reconstructing knowledge must be considered essential to the process of learning.’
– Karpicke & Blunt. 2011. * 

‘Research on retrieval practice suggests a view of how the human mind works that differs from everyday intuition. Retrieval is not merely a read-out of the knowledge stored in one’s mind; the act of reconstructing knowledge itself enhances learning. This dynamic perspective on the human mind can pave the way for the design of new educational activities based on consideration of retrieval processes.’
– Karpicke & Blunt. 2011. *

‘The traditional model: it penalizes you for experimentation and failure but it does not expect mastery. We encourage experimentation and failure, but we do expect mastery.’
– Salman Khan. 2011. *

Index: 0 – Objectives | 1 – Introduction | 2 – The Importance of Learning, Teaching, and Learning Myths | 3 – Retrieval vs Repetion & Mind-maps | 4 – Asking Questions | 5 – Flipping The Classroom | 6 – Desirable Difficulty | 7 – Summary | 8 – Quiz | 9 – Further Reading & References  | K | L | M | N O P Q R S T U V W X Y ZÅÄÖ

0.

Objectives ^

• Understanding current evidence for best practices when it comes to learning theoretical material
• The ability to differentiate between high quality experimental studies on best practices for learning compared to less rigorous qualitative learning theories
• Understanding the difference between learning activities that promote encoding, known as elaborative study tasks, compared to retrieval
• Familiarity with how to make lectures more retrieval based and the evidence supporting increased learning when doing so
• Understanding the concept of flipping the classroom and its advantages
• Familiarity with the concept of “desirable difficulties“

1.

Introduction ^

TL;DR

This page was created as a beginners guide to evidence based teaching. It is structured as following: looking at one of the many learning myths: which is the practical utility of different learning styles (i.e. visual, auditorial, kinetic). Spoiler alert: adapting learning to accommodate different learning styles, such as visual material for visual learners, has no support for improved learning outcomes when reviewing the literature.

Understanding learnings myths are important because how we think we learn best can be counterintuitive to the methods that actually work best as is the case with retrieval. People who study with retrieval-based methods are much less confident before an exam compared to those who practice by repetition or mind-maps, yet the retrieval-group outperforms the other two groups by a significant margin. The reason behind this is probably (if I would guess) that when you retrieve information from memory you are focused on everything that you can’t remember as opposed to re-reading the material or creating mind-maps with the material in front of you. It’s that uncomfortable act of retrieving that makes you not only create and enhance neural pathways that promotes superior learning, but that also helps you double-down on the parts you’re struggling with instead of getting a (false) sense of confidence when re-reading the material or making mind-maps with the material in front of you. One of the best ways of practicing retrieval is by creating questions and answering them from memory.

Besides looking at the studies supporting retrieval this page also covers how to incorporate retrieval-based methods in lectures. It has a short section on the advantage of flipping the classroom. After which the next section mentions some of the research on desirable difficulties that has wide ranging implications for how we teach, and the importance of giving students sufficient time to think, and letting them get things wrong rather than being quick with hints and help when they are struggling. Finally, the page ends with a summary, a short quiz, and further reading & references.

2.

The Importance of Learning, Teaching, and Learning Myths ^

How we teach and learn, all the way from kindergarden to higher education, is an area of particular interest to me and should be to you too. As physicians we need to be life long learners, furthermore it is part of our role, whether we acknowledge it or not, to also be teachers as we need to explain medical reasoning and theory to colleagues and patients, often on a daily basis. Teaching is also a great way for our own learning and improvement as it encourages us to be up to date and think about and relate to questions that inevitably arise when teaching, especially when we teach in a more formal setting.

In the world of teaching and academia, again: all the way from kindergarden to higher education, there are a lot of inferior learning methods (myths) that experimental studies have debunked. Yet the myths seem to live on and are practiced widely in all spheres of teaching due to a mismatch between the evidence-based science on learning and (incorrect) intuitive/traditional assumptions about best practices for learning. I have encountered teachers propagating almost all of the myths mentioned, both when I was studying psychology at Stockholm university, and later also as a medical student at Karolinska Institutet.

One of the most common myths are that of learning styles: asserting that different people learn information in different ways (visual, auditorial, kinetic). This concept has gained wide acceptance and influence in academia and has led to the “meshing hypothesis”: according to which instruction is best provided in a format that matches the preferences of the learner (e.g. for a “visual learner,” emphasizing visual presentation of information).

While most students express preferences about how they prefer information to be presented to them, and while there is also plentiful evidence arguing people differ in aptitude for processing different types of information, there is no evidence that supports the theory that information tailored to students specific learning styles produces better learning outcomes. Not only is there not any compelling evidence among the few experimental studies available on the subject, one review by Pashler et al (2009) found that the evidence is also contradictory in several of the experimental studies available:

“However, we found virtually no evidence for the interaction pattern mentioned above, which was judged to be a precondition for validating the educational applications of learning styles. (…) Although the literature on learning styles is enormous, very few studies have even used an experimental methodology capable of testing the validity of learning styles applied to education. Moreover, of those that did use an appropriate method, several found results that flatly contradict the popular meshing hypothesis.” *

3.

Retrieval vs Repetition & Mindmaps ^

Another common misconception about learning is that activities that promote effective encoding, known as elaborative study tasks, such as creating mind-maps (also called concept maps by the study below) produce deep learning while retrieval, the act of retaining information from memory, only produces shallow learning without creating a deeper understanding of the material that is retrieved. The assumption that retrieval only produces shallow learning that is the product of previous learning or previous experience is one of the reasons old exam-questions (and/or answers) are not shared by some teachers and educators (and one of my pet peeves! SHARE your questions-data-bank people it’s one of the best ways to promote deep learning!). One of the best studies on the subject, published in Science 2011 by Karpicke and Blunt, questions that assumption and provides ample evidence and reasoning for why retrieval not only produces deep learning but is also superior when it comes to learning in comparison with encoding methods such as mind-mapping.

In the study from 2011 two experiments were done, the first consisted of 80 students divided into four groups, where each group applied a different learning practice method before doing a science text based exam. The four groups practiced either by study (study the material once), repeated study (studied the text in four consecutive study periods), concept mapping (studied the text in an initial study period and then created a concept map), or retrieval practice (studied the text in an initial study period then practiced recalling as much of the information as they could on a free recall test, after recalling once the students re-studied the text and recalled again). The total amount of learning time was exactly matched in the concept mapping and retrieval practice conditions. The exam consisted of both verbatim questions (questions where the answer was explicitly apparent in the text) and inference questions (questions that required the students to make implicit connections between different parts of the text). The study showed the following results:

[Image credit: Karpicke & Blunt. ‘Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping’. 2011.*]

Not only does retrieval practice produce better learning than concept mapping as the image above demonstrates, it also outperforms the encoding/elaborative practice of concept mapping both when it comes to verbatim and inference questions, suggesting that retrieval practice is not just as good, but that it’s also superior to concept mapping as it relates to producing deep learning. This is counterintuitive as the image above also shows: the students in the retrieval practice group predicted they would do less well than the students in the other groups while the results showed the opposite results. My own guess for why retrieval practice makes you less confident than other methods of studying is the following: when you retrieve information from memory you are focused on everything that you can’t remember as opposed to re-reading the material or creating concept maps with the material in front of you. It’s that uncomfortable act of retrieving that makes you not only create and enhance neural pathways that promotes superior learning, but that also helps you double-down on the parts you’re struggling with instead of getting a (false) sense of confidence when re-reading the material or making concept maps with the material in front of you (see also: The Dunning-Kruger Effect).

In a second within-subject design experiment, 120 students got to study for enumerated texts (list of concepts, eg. muscle tissues) and sequence texts (e.g. a text describing the sequence of events involved in the process of digestion). For each individual learner the relative effectiveness of retrieval practice and elaborative mapping was assessed as they each got to create a concept map of one science text and retrieval of the second text. Adding to the ingenious design half of the students took a final short-answer test as in experiment one above and half took the final exam in which they created concept maps of the two tests, without viewing the texts on the final test. Initial learning time was once again exactly matched for elaborative (concept mapping) and retrieval practice. The results speak for them selves:

[Image credit: Karpicke & Blunt. ‘Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping’. 2011.*]

Even when the final test involved using memory to construct a concept map, practicing retrieval during the learning period produced better performance than engaging in elaborative study by creating concept maps as the main method of learning. Once again the students prediction for how they would perform based on method of study for each text was opposite to their actual results as the concept mapping practice generated higher confidence than the retrieval practice. As the authors poignantly conclude in the study:

‘Research on retrieval practice suggests a view of how the human mind works that differs from everyday intuition. Retrieval is not merely a read-out of the knowledge stored in one’s mind; the act of reconstructing knowledge itself enhances learning. This dynamic perspective on the human mind can pave the way for the design of new educational activities based on consideration of retrieval processes.’

4.

Asking Questions ^

One of the best ways of practicing retrieval is by asking and answering questions from memory, which is one of the reasons there is such an emphasis on quizes on this learning platform. Besides creating quiz-cards on paper there are many other tech solutions such as quizlet, google-forms (in quiz mode), and kahoot (to name a few of my own favorites) that can be utilized for learning. I encourage you to develop your own habit of creating and collecting good questions for your own learning and teaching endeavours.

It doesn’t matter what format the questions are: multiple choice questions do not produce less learning than short-answer or hybrid questions, according to another study by Karpicke from 2013 (N = 372)*.  You can even use concept mapping as a retrieval tool and it will produce as effective results as paragraph retrieval/recall *. It does not matter if the retrieval is overt (writing the answer) or covert (just thinking of the answer in your head but not producing it): they both yield the same learning outcome as shown by another study debunking the myth that physically writing an answer enhances learning *. And repeated retrieval with long intervals between each test (spaced test) produce greater long-term retention relative to repeated retrieval with no spacing between tests *.

In short: asking relevant questions to whatever it is you’re trying to learn and answering from memory is one of the best practices we have as it relates to learning.

5.

Flipping the Classroom ^

Flipping the classroom

Salman Khan and the Khan Academy is a platform that proposes a model for “Flipping the classroom”, instead of lectures in class the students get the lectures in videoformat as homework and do the assignments in class where there are instructors available to help if they have difficulties with solving problems or any questions. The benefits of this are many, for one the students can pause and play the lectures in the pace that best suits them. For another, the teachers expertize is better used for problem solving and question answering than repeating the same lecture every semester. By flipping the classroom it allows teachers to turn the traditional model on its head, creating higher expectations on students than students “just passing” the exams, as Khan puts it:

‘The traditional model: it penalizes you for experimentation and failure but it does not expect mastery. We encourage experimentation and failure, but we do expect mastery.’ *

Retrieval Based Lectures

However, for those who still insist on the 45-60 minute lecture format there is a much better way to produce lecture-based learning than having a teacher speaking for a hour straight. Another study published in Science in 2011 (great year for evidence-based learning experiments!) by Deslauriers et al, looked at two groups (N = 267 & N = 271) attending an introductory physics course. * The first group was taught by 3 hours of traditional lectures by experienced highly rated instructors while the second group got 3 hours of instruction taught by a trained but inexperienced instructor using instruction based on research on cognitive psychology and physics education. The results shown below: the second experimental group produced more than twice the learning (!):

[Image credit: Deslauriers et al. ‘Improved Learning in a Large-Enrollment Physics Class’. 2011. *]

Take a moment to appreciate just how much the bell curve is shifted to the right on the result image above for the experimental group. And the fact that the experimental groups result-distribution no longer even holds the shape of a bell curve, with so many students acing the test in that group, or almost acing it (scores of 11 & 12)!

So what the did the instruction based on research on cognitive psychology and physics education consist of? Well as the paper states:

‘A typical schedule for a class was the following: CQ1, 2 min; IF, 4 min; CQ2, 2 min; IF, 4 min; CQ2 (continued), 3 min; IF, 5 min; Revote CQ2, 1 min; CQ3, 3 min; IF, 6 min; GT1, 6 min; IF with a demonstration, 6 min; GT1 (continued), 4 min; and IF, 3 min. The time duration for a question or activity includes the amount of time the students spent discussing the problem and asking numerous questions. There was no formal lecturing; however, guidance and explanations were provided by the instructor throughout the class. The instructor responded to student-generated questions, to results from the clicker responses, and to what the instructor heard by listening in on the student-student discussions. Students’ questions commonly expanded upon and extended the material covered by the clicker questions or small-group tasks.’

As for the abbreviations:

• CQ = in-class clicker questions with student-student discussion
• IF = targeted in-class instructor feedback
• GT = small-group active learning tasks

And, additionally, the instructional approach used in the experimental group also included pre-class reading assignments and pre-class reading quizzes. The experiment instruction consisted, in other words of a lot of incorporated retrieval practice. It’s compelling, when seeing the results the experimental method produced, to view Khan’s mantra: ‘We encourage experimentation and failure, but we do expect mastery’ as a real possibility / opportunity and not just hyperbole, isn’t it?

6.

Desirable Difficulty ^

The final concept that I will mention briefly is that of “desirable difficulties”. Retrieval itself is a form of desirable difficulty as recall is more difficult than just re-reading a text or turning it into a mind-map and therefor also produces better longterm learning.

However, there is another aspect of desirable difficulties I want to highlight:  some of the most interesting research right now is on how teachers giving hints or helping students too much when faced with problems can, while helping in short-term learning, impede longterm learning instead of helping students long term retention. One of the desirable difficulties is known as the “generation effect”: struggling to generate an answer on your own, even a wrong one, enhances subsequent learning. *

In fact, some studies suggests that retrieval success does not produce more learning than does retrieval failure followed by feedback. * The take home message here is: let the students take their time when answering a question in a live setting, even if they answer wrong. Encourage trying to answer itself more than the act of getting it right. The same goes for practical practice, let the students hold the probe and don’t be too eager to show them by taking the probe away, instead let them make their mistakes trying to figure out the correct way of handling the probe.

7.

Summary ^

In summary when learning and teaching, besides physical activity, nothing produces better longterm results than retrieval practice. Forget the traditional lecture-based format, incorporate retrieval practices in all your formal teaching endeavours. One of the best ways to do this is by asking questions! Create and collect good questions, make quizzes, and encourage spaced repetition for long term knowledge. Encourage answers, even wrong ones!

Encourage experimentation and failure from yourself and your students in the short-term, but expect mastery in the long-term!

8.

QUIZ ^

Q1. Which of the following methods produces the best learning outcomes when reading a text?

A. Concept mapping

B. Study (reading once)

C. Retrieval practice

D. Re-reading

Click for answer.

Q1: Answer C is correct!

Q2. Which statement about learning with retrieval practice is false?

A. It only works overtly (you have to produce/write the answer)

B. It works just as well with multiple choice questions as short-answer form

C. It works even if you get the answer wrong

D. Concept mapping can be used as a retrieval practice

Click for answer.

Q2: Answer A is false! Retrieval works equally well when practicing overtly and covertly.

Q3. Which study group, in the experiments mentioned above, think they will perform worst on the test they are studying for?

A. The concept map group

B. The read once group

C. The re-reading group

D. The retrieval practice group

Click for answer.

Q3: Answer D is correct! Those who use retrieval practice for learning thought they would perform the worst while, in reality, they performed the best.

Q4. What happened in the experimental instruction group compared to the traditional lecture based control group?

A. The experimental group did only a little better on the final exam than the control group

B. The experimental group did twice as well on the final exam as the control group

C. The control group did only a little better on the final exam than the experimental group

D. The control group did twice as well on the final exam as the experimental group

Click for answer.

Q4: Answer B is correct!

9.

Further Reading & References ^

Video

• TED: Let’s use video to reinvent education by Salman Khan (YouTube 20 min)

References

• Blunt & Karpicke. ‘Learning With Retrieval-Based Concept Mapping’. 2014.
  • https://psycnet.apa.org/record/2014-05951-001
• Deslauriers et al. ‘Improved Learning in a Large-Enrollment Physics Class’. 2011.
  • https://www.science.org/doi/10.1126/science.1201783
• Karpicke & Blunt. ‘Retrieval Practice Produces More Learning than Elaborative Studying with Concept Mapping’. 2011.
  • https://www.science.org/doi/10.1126/science.1199327
• Karpicke et al. ‘Spaced Retrieval: Absolute Spacing Enhances Learning Regardless of Relative Spacing’. 2011
  • https://pubmed.ncbi.nlm.nih.gov/21574747/
• Kornell et al. ‘The generation effect in monkeys’. 2007.
  • https://pubmed.ncbi.nlm.nih.gov/17680937/
• Kornell et al. ‘Retrieval attempts enhance learning, but retrieval success (versus failure) does not matter’. 2015
  • https://pubmed.ncbi.nlm.nih.gov/25329079/
• Pashler et al. ‘Learning Styles: Concepts and Evidence’. 2009.
  • https://journals.sagepub.com/doi/full/10.1111/j.1539-6053.2009.01038.x
• Smith & Karpicke. ‘Retrieval practice with short-answer, multiple-choice, and hybrid tests’. 2014.
  • https://pubmed.ncbi.nlm.nih.gov/24059563/
• Smith et al. ‘Covert retrieval practice benefits retention as much as overt retrieval practice.‘ 2013.
  • https://psycnet.apa.org/record/2013-22764-001

K. ^ L. ^ M. ^ N. ^ O. ^ P. ^ Q. ^ R. ^ S. ^ T. ^ U. ^ V. ^ W. ^ X. ^ Y. ^ Z. ^ Å. ^ Ä. ^ Ö. ^