Evidence-based reform in education: Promise and pitfalls

Tags: Educational Researcher, research subjects, metacognitive knowledge, studies, SRL, study strategies, educational researchers, Sharon A. Valente, National Center for Education Statistics, review, special education, self-regulated learning, Johns Hopkins University, rigorous evaluation, correlational studies, declarative knowledge, students need, legal consent, research subject, participation, the research process, IRB, procedural knowledge, test scores, rigorous research, students questions, test score, educational psychology students, randomized experiment, Media Education Foundation, Educational attainment, peer review process, J. Q. Adams, Educational Testing Service, MWERA, References Adams, Mid-Western Educational Research Association, School Districts, Ashland University, achievement gap, policy makers, educational psychology, gifted program, research designs, definitive research, replicable programs, National Center for Educational Statistics, Division Chairs, Educational research, research findings
Content: Volume 18, Number 1­Winter 2005
· Official Publication of the Mid-Western Educational Research Association ·
The Ohio State University at Newark
On the Cover The Ohio State University at Newark Students seeking a Big 10 education in an intimate, liberal arts setting will find it on The Ohio State University Newark (OSUN) campus. Located in central Ohio, the 225-acre campus serves as a home to 2,300 Ohio State University students. In addition to a variety of professional programs leading to bachelor's and master's degrees, OSUN offers a slate of classes that ultimately lead to one of 170 Ohio State degrees. The College of Education offers an undergraduate degree in education, initial teacher licensure as a part of a Masters in Education, and ongoing professional development through its Master of Arts in Education programs. OSUN students enjoy small class sizes and individual attention from distinguished faculty and staff.
Call for Manuscripts
The Mid-Western Educational Researcher is a scholarly journal that publishes research-based articles addressing a full range of educational issues. The journal also publishes literature reviews, theoretical and methodological discussions that make an original contribution to the research literature, and feature columns. There are four issues of the journal published annually. The journal is accepting manuscripts for review and possible publication. Manuscripts are submitted to blind reviews by three researchers with knowledge of the literature in the appropriate area. The editors will review the manuscript and make the final decision. The review process requires approximately four months. Manuscripts are accepted from faculty, students, and professionals working in educational or non-educational settings. Membership in the MWERA is not required in order to submit a manuscript for review. The editors encourage the submission of revised papers that have been presented at the annual meetings of the MWERA, AERA, and other professional organizations. Manuscripts may be submitted for review as hard copy or electronically. Hard Copy Submission. Submit four (4) copies of the manuscript with a cover letter to Deborah Bainer Jenkins, Co-Editor. Manuscripts should conform to the style and format described in the Publication Manual of the American Psychological Association, 5th edition. All manuscripts should be typed, double-spaced, and on 8Ѕ x 11 paper with 1Ѕ inch margins on all sides. An abstract of less than 100 words should accompany the manuscript. The author's name, contact information, and affiliation should appear on the title page only. Submissions typically are less than 20 pages in length. A disk file (3Ѕ inch diskette, MS Word) is also required with the submission. Electronic Submission. Submit the manuscript to Deborah Bainer Jenkins, Co-Editor, at [email protected] as an e-mail attachment. Indicate in the subject line that this is a MWERJ manuscript. As with hard copy, the manuscript should conform to APA style, be produced in MS Word, and be limited to 20 pages, including abstract and references, and contain full contact information for the author(s). All manuscripts, whether submitted in hard copy or electronically, will be acknowledged upon receipt. Please note that authors are responsible to submit manuscripts that are free of grammatical and mechanical errors. The editors reserve the right to make minor modifications in order to produce a more concise and clear article. Questions regarding the journal or the submission of feature columns should be directed to the co-editors listed below.
Deborah Bainer Jenkins, Ph.D. Professor and Director State University of West Georgia 228 Education Annex Carrollton, GA 30118 (770) 836-4461 [email protected]
Adrian Rodgers, Ph.D. Assistant Professor The Ohio State University at Newark 1179 University Drive Newark, OH 40355 (740) 366-9261 [email protected]
The Mid-Western Educational Researcher (ISSN 1056-3997) is published quarterly by the MidWestern Educational Research Association through The Ohio State University. The Summer issue serves as the annual meeting program. Non-profit postage paid at Columbus, Ohio, with permission of the College of Education, Donna Evans, Dean. POSTMASTER: Send address change to Jean W. Pierce, Dept. EPCSE, Northern Illinois University, DeKalb, IL 60115.
MWER Co-Editors Deborah L. Bainer Jenkins State University of West Georgia Adrian Rodgers The Ohio State University at Newark MWER Editorial Advisory Board Louise Fleming Ashland University Tom Ganser University of Wisconsin Whitewater Richard Lipka Pittsburg State University Jackie Rickman Educational Consultant Ayres D'Costa The Ohio State University Isadore Newman University of Akron Anni D'Antonio Stinson University of Wisconsin Whitewater Layout and Design Judy E. Teska JET Desktop Solutions
Volume 18, Number 1 · Winter 2005 ISSN 1056-3997 Theory, Research, Practice, Caring, and Spirituality: Zen and the Art of Educational Administration A. William Place, University of Dayton 2 Evidence-Based Reform in Education: Promise and Pitfalls Robert E. Slavin, Johns Hopkins University 8 Closing the Performance Gap in a 4th Wave and Post-Modern Society: Lessons from the Field J. Q. Adams, Western Illinois University 14 Mid-Western Educational Research Association 2005 Annual Meeting Call for Proposals 19 The IRB and Classroom Research Jeffrey B. Hecht, Northern Illinois University 23 Metacognitive Knowledge Monitoring in Post-Secondary Education: The Consequences of Poor Knowledge Monitoring and a Program to Facilitate It Randy M. Isaacson, Indiana University South Bend 29 Twenty Techniques for Teaching with Case Studies Mary R. Sudzina, University of Dayton 37
MWER Publication Address Deborah Bainer Jenkins State University of West Georgia 228 Education Annex Carrollton, GA 30118 Phone: (770) 836-4461 e-mail: [email protected]
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
MWER Subscription & MWERA Membership Information Jean W. Pierce LEPF Department Northern Illinois University DeKalb, IL 60115 Phone: (815) 753-8470 Fax: (815) 753-8750 e-mail: [email protected] 1
Presidential Address Theory, Research, Practice, Caring, and Spirituality: Zen and the Art of Educational Administration A. William Place University of Dayton Abstract Educational theory should be the foundation for most practice and research, but this is not a one-way relationship. Good inductive research may help to develop theory instead of coming from it. The relationship of theory to research can be different in qualitative research than in quantitative research. Practice can and should also inform and be informed by theory and research. Theory, research, and practice are very important to the field of education, as is the nexus of these constructs. The author of this paper considers the nexus of theory, research, and practice and delves deeper into these constructs to suggest a foundational nature of caring and spirituality.
In my search for a topic to present at the presidential address, I reviewed previous presidential addresses and was quite impressed and humbled by the work my predecessors have given us. Much like the doctoral student, I have struggled to focus on a doable topic and still come up with one that will add to the discussion of scholars. Additionally, as we tell our students, the topic needs to be one for which you have a passion. The MWERA conference themes for this year (Theory and Practice: Two Sides of the Same Coin) and last year (Research and Practice: Building Bridges) are a good starting point. While the research--theory--praxis connection is crucial, I took 2 years away from academe to reconnect with practice and for me personally the old adage that "people don't care how much you know until they know how much you care" is worth further exploration. Robert Pirsig (1974) wrote his book, Zen and the Art of Motorcycle Maintenance as a reflection on his life in an effort to make an inquiry into values. He did not offer a factual account of Zen Buddhist practice or apply it to the repair of small engines, nor will this work cover Zen or apply it to education. Pirsig (1974) compares a motorcycle to a system because it is designed with a structure to achieve certain performance objectives. He suggests that a study of "motorcycle maintenance is really a miniature study of the art of rationality itself" (p. 84). Pirsig notes that each part of the motorcycle is conceived to be a part of the whole: "This structure of concepts is formally called a hierarchy and since ancient times has been a basic structure for all Western knowledge. Kingdoms, empires, churches, armies have all been structured into hierarchies" (p. 87). This notion fits squarely in what some traditionalists in education would affirm and there are those who want to stop there and avoid emotion and all that `touchy feely stuff.' The author would like to thank Joseph Watras for his time and comments, which strengthened this paper.
Some traditional administrators and researchers would say that rationality is all we need to consider, but to further consider the alternative I will return to Zen and the Art of Motorcycle Maintenance and the analogy to education. Pirsig (1974) described human advancement and believed we are moving forward and in a positive (or upward) direction. When comparing ancient times to modern times Pirsig claimed, "From that agony of bare existence to modern life can be soberly described only as upward progress and the sole agent for this progress is quite clearly reason itself" (p.112). Yet Pirsig's discussion went beyond traditional rationality and brought in "a new spiritual rationality--in which the ugliness and the loneliness and the spiritual blankness of dualistic technological reason would become illogical" (p. 323). Pirsig (1974) struggled with the inadequacy of what he referred to as the `church of reason' (the university) and in a Chautauqua he explored, a way by which reason may be expanded to include elements that have previously been unassimilable and thus have been considered irrational. I think it is the overwhelming presence of these irrational elements crying for assimilation that creates the present bad quality, the chaotic, disconnected spirit of the twentieth century. (p. 230) He further explored: "the basic fault that underlies the problem of stuckness is traditional rationality's insistence upon `objectivity,' a doctrine that there is a divided reality of Subject and object. For true science to take place these must be rigidly separate" (1974, p. 253). He wrote all this before qualitative research became widely recognized and people like Michelle Fine started to explore "the hyphen at which Self-Other join in the politics of everyday life, that is, the hyphen that both separates and merges personal identities with our inventions of Others" (1994, p. 70). Yet Pirsig (1974) acknowledged, "to build a factory, or fix a motorcycle, or set a nation right without getting stuck, then classi-
2
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
cal structured dualistic subject-object knowledge, although necessary, isn't enough. You have to have some feeling for the quality of the work" (p. 255). Quality was very important to Pirsig (1974). He devotes much discussion to `Quality' which "couldn't be independently related with either the subject or the object but could be found only in the relationship of the two with each other" (p. 215). Quality for him is different from Classic Knowledge or Romantic Knowledge. Quality is a `preintellectual awareness' which defies definition. He notes that, "When traditional rationality divides the world into subjects and objects it shuts out Quality, and when you're really stuck it's Quality, not any subjects or objects that tells you where you ought to go" (1974, p. 253). Pirsig's `Quality' has at least some things in common with my use of spirituality because even though he argued that `Quality' can not be defined, he felt it was known. Spirit has a similar characteristic. For some, "Leading with spirit is not something easily learned by reading a book or attending leadership workshops. We even question whether or not it can be acquired. We do know however, that we can recognize leaders with spirit" (Creighton, 1999, p. 146). Pirsig (1974) used working on a motorcycle, which was his passion, to offer a personal exploration of his life and as a way of integrating his well-being into work. Likewise, the education of students is my passion and this article is an attempt to offer a personal exploration of educational leadership and a way of integrating some of the essential constructs that so often are in separate worlds. The integration of theory, research, and practice is very important to the field of education. Educational theory should be the foundation for most practice and research, but this is not a unilateral relationship. Good inductive research may help to develop theory instead of coming from it. Therefore, the relationship of theory to research can be different in qualitative research than in quantitative research. Practice can and should inform and be informed by theory and research. The 2004 Mid-Western Educational Research Association's conference theme--Theory and Practice: Two Sides of the Same Coin--is similar to the 2003 conference theme--Research and Practice: Building Bridges. In a sense theory, research, and practice are the faith, hope, and love of education. Theory is similar to faith because it supports research and practice and when strong it can move mountains; but when it is weak it seems irrelevant. Research is similar to hope in that when done correctly it gives direction to actions and practice. Good research improves the world and truly does provide hope for a better future. Practice can be viewed as love because most forms of effective education are based in some sort of a caring ethic (Hoyle, 2002; Noddings, 1993). Like all metaphors this one is not complete and is limited. For example, practice also includes technical knowledge of content and pedagogy. Therefore, rather than expand on the faith, hope and love metaphor, I propose a theoretical model to explore these constructs.
These critical constructs in the best of worlds intersect and are tightly interconnected (see Figure 1). Caring and Spirituality, to me, are more than adding a couple of circles to the Venn diagram because those two concepts should be all encompassing, not only the entire three circles, but the whole page and beyond. Some people in today's world have separated themselves from what they do and in the process end up not caring about their jobs. Pirsig (1974) asks "if in that strange separation of what man (sic) is from what man does we may have some clues as to what the hell has gone wrong in this twentieth century" (p. 25). One of our doctoral students wrote in her qualifying paper that "to truly care, I must feel that what I care about is an extension of myself" (V. Browning, personal communication, 2002). Some educators have become `burnt out' and find it easier not to care about their jobs or the students. Others draw strength from each interaction, even after years in the field. When I reconnected with the field, I knew that I was truly blessed and felt that the students and community in which I was principal, helped me to grow immeasurably. Every now and then someone would try to compliment me and say how good it was that I went to a high school from a university setting as if being a university professor was so far above a practitioner. My reaction was usually to thank that person, but point out that I was the one benefiting from the experience. I try to recognize how society and organizations have influenced my perspective. Link (1992) notes "An aboriginal Australian woman said: `If you have come to help me you are wasting your time. But if you have come because your liberation is bound up with mine, then let us work together' " (p. 250). So it is with theory, research, and practice; the three are bound up with each other and all are supported by caring and spirituality. Talking about caring or love makes some people uncomfortable. Perhaps this is because of their abstract nature, which falls outside what some have defined as scientific or research. Others (Marshal, Patterson, Rogers, & Steele, 1996) have moved beyond such simplistic notions that would hold us back and have studied administrators that integrated caring into their work. They "describe how school administrators operating from an ethic of care conduct their daily practice and how that practice differs from administrators operating solely from traditional leadership models" (p. 271). Or perhaps another possible cause of the discomfort could be because some have distorted what this construct of caring or love should be about in education. Hoyle (2002) argues that, "If you can't love, you can't lead." This is not some statement of a hopeless romantic. There is nothing romantic about my argument claiming that without love in organizations, violence, intolerable stress, and poor quality will continue.... The type of love I espouse for this book is unselfish, loyal, and benevolent concern for the good of another. (p. xii)
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
3
Spirituality Caring Spirituality Caring
Caring Spirituality Caring Spirituality
Spirituality Caring Spirituality Caring
Caring Spirituality Caring Spirituality
Theory
Research
Practice
Spirituality Caring Spirituality Caring
Caring Spirituality Caring Spirituality
Figure 1: Critical constructs and their intersection.
Spirituality Caring Spirituality Caring
Caring Spirituality Caring Spirituality
This concern for the good of another is truly caring about the other person and making the other person's well-being your concern. This is far from the sexual love of a predator. That is a perversion within education that should do more than make us uncomfortable; it should move us to action that results in the removal of these individuals from our field, but it should not cause us to remove caring from our vocabulary or the field. To use the existence of a few sick individuals (that need to be locked up rather than being put in charge of children) as a reason to make professionals uncaring robots would be another tragedy. Caring is an administrative and leadership issue--as Hoyle (2002) states, "When the system fails the workers, the workers fail the system and find no joy or love in their endeavors" (p. 102). What I believe Hoyle was emphasizing was that administrators need to care about all employees and make sure that employees know that there are people, in the system, who care about them. That does not exclude the possibility that sometimes there is a need for tough love. For example, if an employee is hurting students, that employee may need to be fired. Being a caring and loving per-
son, does not mean that administrators are not going to do the tough part of their jobs. In fact, if you are a caring and loving person then you are going to do the tough things that need to be done--such as dealing with the individuals that need to be dealt with to stop them from hurting themselves or others. Caring and love are uncomfortable enough, but to really stretch this work I decided to go contrary to the old wisdom that warns to stay away from topics like politics or religion. Although a distinction can and should be made between religion and spirituality, they are closely related. I use spirituality here because it is a broader term than religion. Personally my religion is how I connect to spirituality, but that compatibility and reconciliation is my own. While I have placed spirituality as an underlying foundation to the model in Figure 1, it should be recognized that one aspect of this is what Noddings (2003) indicates in her discussion of the "compatibility of philosophy, feminism, and faith... the task of reconciliation is largely a personal one" (p. 213). It must be acknowledged that in the academy where many feel "confined by the rationalistic structures of higher
4
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
education" (Tisdell, 2000/2002, p. 80) there are atheists some of whom "find spirituality irrelevant to their lives" (Tisdell, 2000/2002, p. 83). Yet as one participant in that study put it, "there are also atheists among the group, yet we somehow seem to delve into spirit. It might be striving to be human...I don't know" (Tisdell, 2000/2002, p. 81). Even if there are members of the community of scholars that have personal frameworks which deny the relevance of the human spirit, there are enough members that hold it in many different ways to be of such vital importance that to exclude it from a model such as this would be a disservice to the community of scholars. Noddings' spirituality differs from my personal reconciliation in that she moved away from her traditional religious background, feeling that, "a whole body of doctrine that flies in the face of both logic and empirical evidence is too much for many of us, but a lively intellect also remains challenged by the existence and grandeur of the universe" (p. 219). She goes on to write "When the spirit soars, it is lifted by something outside itself, but this something need not be a describable god nor need it be a single thing. Many things may trigger this soaring of spirit" (p. 220). For me it is a great leap of faith to believe that the lifting or soaring that many feel is not only outside of the individual, but outside the empirical reality of this world. Once that initial great leap of faith is made, the specific doctrines, that for some fly in the face of logic and empirical evidence, are not difficult for me personally to accept. More important than my own beliefs is the point that I am trying to convey as I try to present a model that includes spirituality. This spirituality is broader than the Judeo-Christian view I happen to hold because it includes the full range of those that may not even believe in a God or gods, but allows for the plausibility of some form of spirituality that goes beyond the immediate strict empirical view that all reality can be sensed or measured. In fact, I would agree with Asma (2004) who makes the point that spirituality is more complex than the question of whether or not God exists. In a discussion about the recent PBS miniseries called The Question of God, Asma states, "melodramatic dichotomies presented here are defused when one introduces religions outside Judeo-Christian tradition" (p. B16). Stressing the limited inclusion of other perspectives Asma continues noting that "Buddhism gets mentioned twice in the conversation segment" (p. B16). If this model is to be constructive in helping the worlds of practice and universities to include any form of spirituality, we must allow for all forms to be accepted. Janesick (2004) provides advice and exercises for novice qualitative researchers, to sharpen one's awareness of the role of the researcher.... Nearly everything written by the Chinese master painters was aimed not just at the technique of painting but also at the painter's spiritual resources in order to express the spirit, or chi, the breath of Tao. The chi is looked upon as an underlying harmony. (p. 103)
Although the personal spirituality Noddings (2003) describes is not the same as mine, she makes the point that I am trying to emphasize here which is that "spirituality--if it is treasured--must be nurtured. To find out what nurtures it is one of life's great tasks, one terribly neglected in today's schools" (p. 221). Neither Noddings nor I support teaching religion or that there is one right way to understand the world: "Religious claims to knowledge are all suspect and with Buber, I fear they lead us away from connection and into the uneasy or even violent separation so characteristic of battles over dogma" (Noddings, 2003, p. 224). However, nurturing spirituality does not require that religion be either established or prohibited. The overlap of caring and spirituality is more salient here than the overlap with religion. One of the many ways the overlap can be expressed is through the work of the scientist-priest Pierre Teilhard de Chardin. He describes the continuing Human Evolution as `progress toward' that overlap in the following passage, The slow progress of energies must reach a peak `from which life will never slip back'. To overcome every obstacle, to unite our beings without loss of individual personality, there is a single force which nothing can replace and nothing destroy, a force which urges us forward and draws us upwards: this is the force of love. (Chardin, 1968, p. 15) Palmer (1993) takes a different approach to integrating spirituality into the larger picture. He notes that educators often feel disconnected and, beneath the broken surface of our lives there remains--in the words of Thomas Merton--"a hidden wholeness." The hope of every wisdom tradition is to recall us to that wholeness in the midst of our torn world... That, I think, is why the spirituality of education is now being explored in so many "unlikely" places. Perhaps the ancient communal act called teaching and learning can be renewed by drawing upon spiritual wisdom. (p. x). Wisdom is not the same as knowledge or even massive attainment of knowledge or education. Palmer (1993) cautions that "spiritual traditions have been used to obstruct inquiry rather than encourage it... Authentic spirituality wants to open us to truth--whatever truth may be... [it] encourages us to welcome diversity and conflict, to tolerate ambiguity, and to embrace paradox" (p. xi). This approach is not often easy, but is the best type of leadership and a necessary if not sufficient basis for theory or research which can inform and be informed by practice. Bolman and Deal (1995) view spirituality as crucial and assert, "Leading with soul returns us to ancient spiritual basics--reclaiming the enduring human capacity that gives our lives passion and purpose" (p. 6). Merriam and Muhamad (2000/2002) in a study of older adult learners in Malaysia and the influence of cultural values of that country found that one of three major themes was "Learning is spiritually and philosophi-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
5
cally driven whether Moslem, Christian, Hindu, or Buddhist, the participants in this study spoke of learning in philosophical and spiritual terms" (p. 51). Crieghton warns us that "We have so thoroughly technicalized and intellectualized the job of organizational leader that there is no place for the real passions and pains that men and women in these jobs feel" (1999, p. 146). Creighton also makes the point that the connection between spirituality and caring is foundational to practice, "Leaders with spirituality demonstrate a true `caring' for all in the organization, so the approach to problem solving is synergistic, not adversarial" (1999, p. 147). That approach to problem solving, which is so crucial to leadership, reinforces the connection of theory, research, and practice. Spirituality and caring are also foundational to research and theory. Collins (1991) stated, "the ethic of caring suggests that personal expressiveness, emotions, and empathy are central to the knowledge validation process" (p. 215). This relationship is explicit for some such as the critical theorists (Tisdell, 2000/2002), but it is important for all researchers. Ethics and institutional review boards remind us of that in some ways. The relationships of spirituality and caring to theory, research, and practice are crucial for the larger questions such as what is important and where do we spend our energy. Noddings (1994) suggest among other things that researchers can and should have an ethic of caring and that there is increased contact "between researchers and teachers, so that collaborative inquiry may be maintained and so that relationships may develop through which all participants are supported in their quest for better ethical selves" (p. 181). Much of the literature dealing with spirit has focused on the connection with practice. That connection is important, but again we should not stop there. Reconnecting with practice as MWERA has been doing is important. I believe this model adds to that effort by making explicit parts of that process that are sometimes left out because caring and spirituality can be difficult to deal with in public settings. As a public school principal, I never spoke of religion and yet one employee wrote a note to me as I left the principalship to return to the university, I am writing a note of thanks for your contribution to ... Schools. You have demonstrated kindness to all. I am sure that there were times that you questioned your value to us. I personally would like to say that you were truly valuable. You were chosen for such a time as this. Many times people are unable to face change, challenge and above all adversity. But, I am learning in the scheme of life we are all seeded with purpose. You came during a time ... where it was necessary to see someone that had the ability to demonstrate respect, kindness and yet discipline. When I would behold you, I saw you in my heart of hearts. I saw beyond the natural mind. What I saw was a man that genuinely cared. You
took strides to demonstrate to our kids that they were worthy of respect. Many times I wanted to embrace you. It was the compassion of the Lord that allowed me to see your heart. The embrace I desired to give you was to let you know that I understood and cared. It was the compassion of the Lord that appreciated you fulfilling His will in your life. It was the embrace that goes beyond human comprehension. Some how we connected by the spirit.... His purpose is yet being fulfilled even when you don't even realize it. He has made you a humble man. This person came from a similar Judeo-Christian tradition religious background although not the same denomination as I am. We did not talk about religion, but still she paid me what I took as the highest compliment I could receive by letting me know that she saw my spirituality in practice. Many practitioners have a deep spirituality that runs through everything they do and gives life to their practice. The memory of an educator that cared is strong for most of us and is part of why many of us chose education as a career. I would again emphasize that including caring and spirituality does not establish religion, but does recognize that for many human beings the spirit can and is a part of the search for truth. Having spirituality may not assure quality, but there does seem to be a relationship. For some, such as myself, with a large number of faults, spirituality may be one reason people do not seem to notice the faults so much. Creighton (1999) as noted earlier recognized spirituality in quality practitioners. Parker Palmer and Nel Noddings have also made the point that this aspect is becoming more prevalent. Acknowledging that for many there can be a spiritual aspect of the search for truth does not diminish the importance of research and theory to practice. Bob Slavin, our keynote speaker at this conference, said in his luncheon address--Evidence-Based Reform in Education, that it is important that education like other fields comes to an "acceptance by practitioners of evidence as the basis for practice." I agree even if the emphasis I would put on randomized trials might not be as great as some, these types of quantitative studies are an important part of the whole portfolio that we need. While discussions of caring and spirituality are only a part of what can be considered when we explore the lives of those in the field of education, they can be qualitatively examined and can be important. Both quantitative evidence as well as qualitative evidence must be part of the whole portfolio that we use to improve education. The intersection of theory, research, and practice is made more viable when the dimensions of caring and spirituality are underlying what we do in education. Whether we are doing research, connecting to theory, or involved in practice, when we are trying to make the world a better place we enhance all three, and if we truly care about any one of those three we cannot ignore their intersection.
6
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
References Asma, S. (2004, September 10). `Mass delusion' or `true myth'? PBS considers "the question of God." The Chronicle of Higher Education, B15-B16. Bolman, L. G., & Deal, T. E. (1995). Leading with soul: An uncommon journey of spirit. San Francisco: Jossey-Bass. Chardin, P. T. de (1968). The divine milieu. New York: Harper & Row. (Original work Le Milieu Divin, copyright 1960). Collins, P. H. (1991). Black feminist thought: Knowledge, consciousness, and the politics of empowerment. New York: Routledge. Crieghton, T. B. (1999). Spiritual development and the principalship. In L. T. Fenwick (Ed.), School leadership: Expanding horizons of the mind and spirit (pp. 145152). Lancaster, PA: Technomic. Fine, M. (1994). Working the hyphens: Reinventing self and other in qualitative research. In N. K. Denzin & Y. S. Lincoln (Eds.), Handbook of qualitative research (pp. 70-82). Thousands Oaks, CA: Sage. Hoyle, J. R. (2002). Leadership and the force of love. Thousand Oaks, CA: Corwin Press. Janesick, V. J. (2004). Stretching exercises for qualitative researchers. (2nd ed.). Thousand Oaks, CA: Sage. Link, M. (1992). Vision 2000. Allen, TX: Tabor. Marshal, C., Patterson, J. A., Rogers, D. L., & Steele, J. R. (1996). Caring as career: An alternative perspective for
educational administration. Educational Administration Quarterly, 32(2), 271-294. Merriam, S. B., & Muhamad, M. (2002). How cultural values shape learning in older adulthood. In S. B. Merriam (Ed.) Qualitative research in practice (pp. 4057). San Francisco: Jossey-Bass. (Reprinted from Adult Education Quarterly, 51, 45-63, by Merriam, S. B., & Muhamad, M., 2000). Noddings, N. (1993). Caring: A feminist perspective. In K. A. Strike & P. L. Ternasky (Eds.), Ethics for professionals in education (pp. 43-53). New York: Teachers College Press. Noddings, N. (1994). An ethic of caring and its implications for instructional arrangements. In L. Stone (Ed.), The education feminism reader (pp. 171-183). New York: Routledge. Noddings, N. (2003). A skeptical spirituality. In R. E. Groenhout & M. Bower (Eds.), Philosophy, feminism, & faith (pp. 213-226). Bloomington, IN: Indiana University Press. Palmer, P. J. (1993). To know as we are known: Education as a spiritual journey. San Francisco: Harper. Pirsig, R. M. (1974). Zen and the art of motorcycle maintenance. New York: Bantam Books. Tisdell, E. J. (2002). Spirituality and emancipatory adult education in women adult educators for social change. In S. B. Merriam (Ed.), Qualitative research in practice. (pp. 62-88). San Francisco: Jossey-Bass. (Reprinted from Adult Education Quarterly, 50, 308-335, by Tisdell, E. J., 2000).
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
7
Keynote Address Evidence-Based Reform in Education: Promise and Pitfalls Robert E. Slavin Johns Hopkins University Abstract In this keynote address presented at the Mid-western Educational Research Association Annual Meeting in October, 2004, the author discusses the increasing interest of federal policy-makers in scientificallybased research. A comparison between education and other disciplines is offered, and a proposal for increased rigor in educational research is proposed.
Education is on the brink of a scientific revolution that has the potential to profoundly transform policy, practice, and research. Consider the following: · In 1998, Congress appropriated $150 million per year to provide schools funds to adopt "proven, comprehensive reform models." This unprecedented legislation, introduced by Congressmen David Obey and John Porter, defined "proven" in terms of experimental-control comparisons on standards-based measures. To my knowledge, this was the first time in history that education funding anywhere has been linked directly to evidence of effectiveness (see Slavin, 1997). Comprehensive School Reform (CSR) funding progressively increased to $310 million annually, and has provided funding to more than 3000 mostly high-poverty schools. · The Bush administration's main education initiative, No Child Left Behind, took the idea of scientifically-based practice to an even higher level. The No Child Left Behind legislation refers to "scientifically-based research" 110 times. It defines "scientifically-based research" as "rigorous, systematic and objective procedures to obtain valid knowledge," which includes research that "is evaluated using experimental or quasiexperimental designs...," preferably with random assignment. "Scientifically-based research" is intended to serve as the basis for a wide array of federally funded programs, especially Reading First programs for reading in grades K-3. · Grover Whitehurst, the current director of the Institute of Education Science (IES) in the U.S. Department of Education, has taken a strong line in support of randomized experiments (Whitehurst, 2002). The U.S. Portions of this paper are adapted from Slavin, R.E. (2003), Evidence-based policies: Transforming educational practice and research. Educational Researcher, 31 (7), 15-21. This paper was written under funding from the U.S. Department of Education (Grant No. OERI-R-117-D40005). However, any opinions expressed are those of the author and do not necessarily represent Department of Education positions or policies.
Department of Education strategic plan for 2002-2007 anticipates having 75% of all OERI-funded research that addresses causal questions use random assignment designs by 2004 (previously, such research was less than 5% of causal research funded by The U.S. Department of Education). As a direct result, Congress significantly increased funding for education research. Research involving random assignment is now under way on early childhood programs, elementary and secondary reading, math, programs for English language learners, teacher professional development, after school remedial programs, and much more. It is important to note that none of these policy developments have yet produced the revolution I am anticipating. These initiatives are too new to have had any impact on practice. Yet these and other developments, if not yet proven, still create the potential for changes with far-reaching consequences. It is possible that these policy reforms could set in motion a process of research and development on programs and practices affecting children everywhere. This process could create the kind of progressive, systematic improvement over time that has characterized successful parts of our economy and society throughout the 20th century, in fields such as medicine, agriculture, transportation, and technology. In each of these fields, processes of development, rigorous evaluation, and dissemination have produced a pace of innovation and improvement that is unprecedented in history (see Shavelson & Towne, 2002). These innovations have transformed the world. Yet education has failed to embrace this dynamic, and as a result, education moves from fad to fad. Educational practice does change over time, but the change process more resembles the pendulum swings of taste characteristic of art or fashion (think hemlines) rather than the progressive improvements characteristic of science and technology (see Slavin, 1989). Welcome to the 20th Century At the dawn of the 21st century, education is finally being dragged, kicking and screaming, into the 20th century.
8
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
The scientific revolution that utterly transformed medicine, agriculture, transportation, technology, and other fields early in the 20th century almost completely bypassed the field of education. If Rip Van Winkle had been a physician, a farmer, or an engineer in the 19th century, gone to sleep, and awoke today, he would be unemployable. If he had been a good primary school teacher in the nineteenth century, he'd probably be a good primary school teacher today. It's not that we haven't learned anything since Rip Van Winkle's time. It's that applications of the findings of educational research remain haphazard, and that evidence is respected only occasionally, and only if it happens to correspond to current educational or political fashions. Early in the 20th century, the practice of medicine was at a similar point. For example, research had long since identified the importance of bacteria in disease, and by 1865 Joseph Lister had demonstrated the effectiveness of antiseptic procedures in surgery. In the 1890s, William Halsted at Johns Hopkins University introduced rubber gloves, gauze masks, and steam sterilization of surgical instruments, and demonstrated the effectiveness of these procedures. Yet it took thirty years to convince tradition-bound physicians to use sterile procedures. If he dropped his scalpel, a physician in 1910 was as likely as not to give it a quick wipe and carry on. Today, of course, the linkage between research and practice in medicine is so tight that no physician would dream of ignoring the findings of rigorous research. Because medical practice is so closely based on medical research, funding for medical research is vast, and advances in medicine take place at breathtaking speed. My father's cardiologist recommended that he wait a few years to have a necessary heart valve operation because he was sure that within that short span of time, research would advance far enough to make the wait worthwhile. As it turned out, he was right. The most important reason for the extraordinary advances in medicine, agriculture, and other fields is the acceptance by practitioners of evidence as the basis for practice. In particular, it is the randomized clinical trial, more than any single medical breakthrough, that has transformed medicine (Doll, 1998). In a randomized clinical trial, patients are assigned at random to receive one treatment or another, such as a drug or a placebo. Because of random assignment, it can be assumed with an adequate number of subjects that any differences seen in outcomes are due to the treatment, not to any extraneous factors. Replicated experiments of this kind can establish beyond any reasonable doubt the effectiveness (or lack thereof) of treatments intended for applied use (see Boruch, 1997). Experiments in Education In education, experiments are not uncommon, but they are usually brief, artificial experiments on topics of theoretical more than practical interest, often involving hapless college sophomores. Far more rare are experiments evalu-
ating treatments of practical interest studied over a full school year or more. I write an educational psychology textbook (Slavin, 2003) that is full of research findings of all kinds, findings that are valuable in advancing theory and potentially valuable to teachers in understanding their craft. Yet the brief experiments, correlational studies, and descriptive studies that yield most of the information presented in my text or any other educational psychology text do not collectively add up to school reform. They are suggestions about how to think about daily teaching problems, not guides to the larger questions educators and policymakers must answer. Imagine that research in cardiology described heart function and carried out small scale laboratory studies, but never developed and tested an artificial heart valve. If this were the case, I'd be an orphan. Imagine that agricultural research studied Plant growth and diseases, but never developed and tested new disease-resistant crops. Educational research has produced many rigorous and meaningful studies of basic principles of practice, but very few rigorous studies of programs and practices that could serve as a solid base for policy and practice, and has had little respect for the studies of this kind that do exist. Because of this, policy makers have rarely seen the relevance of research to the decisions they have to make, and therefore have provided minimal funding for research. This has led to a declining spiral, as inadequate investments in research lead to a dearth of the kind of large-scale, definitive research that policy makers would feel to be valuable, making these policy makers unwilling to invest in large-scale, definitive research. Shifting Policy Perspectives The dramatic changes in federal education policies I mentioned earlier could potentially reverse this declining spiral. If the new funding flowing into research in the U.S. can produce some notable successes, we could have an ascending spiral: rigorous research demonstrating positive effects of replicable programs on important student outcomes would lead to increasing funding for such research which would lead to more and better research and therefore more funding. More importantly, millions of children would benefit in the fairly near term. Once we establish replicable paradigms for development, rigorous evaluation, replication, and dissemination, these mechanisms could be applied to any educational intervention or policy. Imagine that there were programs under way all the time to develop, evaluate, and disseminate new programs in every subject and every grade level, as well as programs on school-to-work transitions, special education, gifted programs, dropout prevention, programs for English language learners, race relations programs, drug abuse prevention, violence prevention, and so on. Every one of these areas lends itself to a development-evaluation-dissemination paradigm, as would many more. Over time, each area would experience the step-by-step, irreversible progress characteristic of medicine and agriculture, because innovations would be held to strict standards of evaluation before being recommended for wide scale use.
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
9
Research Designs The scientific revolution in education will only take hold and produce its desired impacts if research in fact begins to focus on replicable programs and practices central to education policy and teaching, and if it in fact employs research methods that meet the highest standards of rigor. This begs an important question: What kinds of research are necessary to produce findings of sufficient rigor to justify faith in the meaning of their outcomes? Of course, all sorts of research designs are appropriate for various purposes, from description to theory building to hypothesis testing. However, leaders in the current administration and many other educational researchers throughout the world (see Angrist, 2004) have been arguing that nothing less than randomized experiments will do for evaluations of educational interventions and policies. When we want to know the outcome of choosing program X instead of program Y, there is no equivalent substitute for a randomized experiment. Randomized experiments The difference in the value of randomized and wellmatched experiments relates primarily to the problem of selection bias. In a matched experiment, it is always possible that observed differences are due not to treatments, but to the fact that one set of schools or teachers was willing to implement a given treatment while another was not, or that a given set of students selected themselves or were selected into a given treatment while others were not. When selection bias is a possibility at the student level, there are few if any alternatives to random assignment, because unmeasured (often, unmeasurable) pre-existing differences are highly likely to be alternative explanations for study findings. For example, consider studies of after school or summer school programs. If a researcher simply compared students attending such programs to those not attending who were similar in pretest scores or demographic factors, it is very likely that unmeasured factors such as student motivation, parents' support for education, or other consequential factors could explain any gains observed, because the more motivated children are more likely to show up. Similarly, studies comparing children assigned to gifted or special education programs to students with similar pretest scores are likely to miss key selection factors that were known to whoever assigned the students but not measured. If one child with an IQ of 130 is assigned to a gifted program and another with the same IQ is not, it is likely that the children differ in motivation, conscientiousness, or other factors. In these kinds of situations, use of random assignment from within a selected pool is essential. In contrast, there are situations in which it is teachers or schools that elect to implement a given treatment, but there is no selection bias that relates to the children. For example, a researcher might want to compare the achievement gains
of children in classes using cooperative learning, or schools using comprehensive reform models, to the gains made by control groups. In such cases, random assignment of willing teachers or schools is still far preferable to matching, as matching leaves open the possibility that volunteer teachers or staffs are better than non-volunteers. However, the likely bias is much less than in the case of student self-selection. Aggregate pretest scores in an entire school, for example, should indicate how effective the current staff has been up to the present, so controlling for pretests in matched studies of existinG Schools or classes would control out much of the potential impact of having more willing teachers. For external validity, it is crucial to note that the findings of a wellmatched experiment comparing volunteers to non-volunteers apply only to schools or teachers who volunteer, but the potential for bias is moderate (after controlling for pretests and demographic factors). The importance of this discussion lies in the fact that randomized experiments of interventions applying to entire classrooms can be extremely difficult and expensive to do, and are sometimes impossible. My colleagues and I at Johns Hopkins University are doing a randomized evaluation of Success for All, a comprehensive reform model. Recruiting schools for this study was extremely difficult, even though we are offering substantial financial incentives to schools willing to be assigned at random to experimental or control groups. For the cost of doing this randomized study, we (and others) could have done two or three equally largescale matched studies. It is at least arguable that replicated matched studies, done by different investigators in different places, might produce more valid and meaningful results than one definitive, once-in-a-lifetime randomized study. Still, fully recognizing the difficulties of randomized experiments, I think they are nevertheless possible in most areas of policy-relevant program evaluation, and whenever they are possible, they should be used. Reviews of research in other fields have found that matched studies generally find stronger outcomes than randomized studies, although usually in the same direction (e.g., Friedlander & Robins, 1995; Fraker & Maynard, 1987; Ioannidis et al, 2001). Four randomized experiments we are doing at Johns Hopkins University and the Success for All Foundation illustrate the potential and the pitfalls. One of these, which I mentioned earlier, involves randomly assigning 41 schools to Success for All or control conditions for a 3-year experiment. Initially, we offered $30,000 to each school, but we got hardly any takers. Schools were unwilling to take a chance on being assigned to the control group for three years. In spring, 2002, we changed our offer. Schools willing to participate were randomly assigned to use Success for All either in grades K-2 or in 3-5. Recruitment was still difficult, but under this arrangement, we signed up adequate numbers of schools. For another study led by my colleague Bette Chambers, we recruited schools for a third-party study of the Curiosity
10
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Corner preschool model. We offered schools the program for free, to start either in 2003-2004 or 2004-2005 (with random assignment to the two start dates). The 2004-2005 group serves as the control group in 2003-04. This delayed treatment control group design was easy for schools to accept, and we did not have serious recruiting problems. We're doing a nearly identical study of an after-school program, and again, recruitment was not difficult. We recently completed a study of the use of embedded multimedia, video vignettes embedded in beginning reading instruction (Chambers et al., 2004). Again, ten schools were randomly assigned to receive the multimedia materials immediately or one year later. Finally, my colleague Geoff Borman did randomized evaluations of summer school programs, in which individual children were randomly assigned to participate now or later (Borman, Boulay, Kaplan, Rachuba, & Hewes, 2001). In all of these cases, obtaining sufficient volunteers was not difficult. These examples of a diverse set of research problems illustrate that one way or another, it is usually possible to use random assignment to evaluate educational programs. There is no one formula for randomization, but with enough resources and cooperation from policy makers, random assignment is possible. Beyond the benefits for reducing selection bias, there is an important political reason to prefer randomized over matched studies at this point in history. Because of political developments in the U.S., we have a once in a lifetime opportunity to reverse the "awful reputation" that educational research has among policy makers (Kaestle, 1993; Lagemann, 2002). This is a time when it makes sense to concentrate resources and energies on a set of randomized experiments of impeccable quality and clear policy importance, to demonstrate that such studies can be done. Over the longer run, I believe that a mix of randomized and rigorous matched experiments evaluating educational interventions may be healthier than a steady diet of randomized experiments, but right now we need to establish the highest possible standard of evidence, on a par with standards in other fields, to demonstrate what educational research can accomplish. Non-Experimental Research I should hasten to say again that forms of research other than experiments, whether randomized or matched, can also be of great value. Correlational and descriptive research are essential in theory building and in suggesting variables worthy of inclusion in experiments. Our Success for All program, for example, owes a great deal to correlational and descriptive process-product studies of the 1970's and 1980's (see Slavin & Madden, 2001). As components of experiments, correlational and descriptive studies can also be essential in exploring variables that go beyond overall program impacts. In some policy contexts, experiments are impos-
sible, and well-designed correlational or descriptive studies may be sufficient. The experiment, however, is the design of choice for studies that seek to make causal conclusions, and particularly for evaluations of educational innovations. Basing Educational Policy on Evidence Historically, the impact of education research on education practice has been tenuous at best. Innovation takes place, but it is based on fads and politics rather than evidence. At best, education policies are said to be "based on" scientific evidence, but are rarely scientifically evaluated. This distinction is critical. The fact that a program is based on scientific research does not mean that it is in fact effective. For example, imagine an instructional program whose materials are thoroughly based on scientific research, but which is so difficult to implement that in practice, teachers do a poor job of it, or which is so boring that students don't pay attention, or which provides so little or such poor professional development that teachers do not change their instructional practices. Before the Wright brothers, many inventors launched airplanes that were based on exactly the same "scientifically-based aviation research" as the Wright brothers used at Kitty Hawk, but the other airplanes never got off the ground. Worse, any program or policy can find some research somewhere that suggests it might work. Given the current state of research on replicable programs in education, it would be difficult to require that government funds be limited to programs that have been rigorously evaluated, because there are so few such programs. However, programs that do have strong, rigorous evidence of effectiveness should be emphasized over those that are only based on valid principles, and there needs to be a strong effort to invest in development and evaluation of replicable programs in every area, so that eventually legislation can focus not on programs "based on scientifically-based research" but on programs that have actually been successfully evaluated in rigorous experiments. Research Syntheses The evidence-based policy movement is by no means certain to succeed. Education has a long tradition of ignoring or even attacking rigorous research. Researchers themselves, even those who fundamentally agree on methodologies and basic principles, may disagree publicly about the findings of research. These disagreements, which are a healthy and necessary part of the scientific process, will be seized upon by individuals who oppose the entire concept of evidence-based reform as indications that even the experts disagree. For these and many other reasons, it is essential that independent review commissions representing diverse viewpoints be frequently constituted to review the research and produce consensus on what works, in language that all edu-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
11
cators can access. In the area of reading, it is impossible to overstate the policy impact of the National Research Council (Snow, Burns, & Griffin, 1998) and National Reading Panel (1999) reports, which produced remarkable consensus on the state of the evidence in early literacy. Consensus panels of this kind, with deep and talented staff support, should be in operation continually, on a broad range of policy-relevant questions, so that practitioners and policy makers can have a way to cut through all the competing claims and isolated research findings to get to the big picture findings that methodologically sophisticated researchers can agree to represent the evidence fairly and completely. The federally-funded What Works Clearinghouse is carrying out rigorous reviews of research on a range of programs and practices. This effort is just getting under way, but it could become very influential if it gives government funders a basis for favoring well-evaluated practices. Potential Impact of Evidence-Based Policies on Educational Research Up to now, I've spoken primarily about the potential impact of evidence-based policies on education policies and practice. I'd now like to consider the potential impact on educational research. I believe that if evidence-based policies take hold, this will be enormously beneficial for all of educational research, not just research involving randomized or matched experiments. First, I am confident that when policymakers perceive that educational R&D is actually producing programs that are shown in rigorous experiments to improve student outcomes, they will fund research at far higher levels. This should not be a zero-sum game, in which new funds for experiments will be taken from the very limited funds now available for educational research (see Shavelson & Towne, 2002). Rather, I believe that making research relevant and important to policymakers will make them more, not less, willing to invest in all forms of disciplined inquiry in education, be it correlational, descriptive, ethnographic, or otherwise. The popularity of medical research depends totally on its ability to cure or prevent diseases, but because randomized experiments routinely identify effective treatments (and protect us from ineffective treatments), there is vast funding for basic research in medicine, including epidemiological, correlational, and descriptive studies. Researchers and developers will be able to argue convincingly that basic research is essential to tell us what kinds of educational programs are worth evaluating. A climate favorable to evidence-based reform will be one in which individual researchers working on basic problems of teaching and learning will be encouraged and funded to take their findings from the laboratory or the small-scale experiment, or from the observation or interview protocol, to themselves develop and then rigorously evaluate educational treatments. Education is an applied field. Research in education should ultimately have something to do with improving outcomes for children.
Conclusion Evidence-based policies have great potential to transform the practice of education, as well as research in education. Evidence-based policies could finally set education on the path toward progressive improvement that most successful parts of our economy and society embarked upon a century ago. With a robust R&D enterprise and government policies demanding solid evidence of effectiveness behind programs and practices in our schools, we could see genuine, generational progress instead of the usual pendulum swings of opinion and fashion. This is an exciting time for educational research and reform. We have an unprecedented opportunity to make research matter, and to then establish once and for all the importance of consistent and liberal support for high-quality research. Whatever their methodological or political orientations, educational researchers should support the movement toward evidence-based policies, and then set to work to generate the evidence that will be needed to create the schools our children deserve. References Angrist, J. D. (2004). American education research changes tack. Oxford Review of Economic Policy, 20(2), 198212. Borman, G., Boulay, M., Kaplan, J., Rachuba, L., & Hewes, G. (2001). Randomized evaluation of a multi-year summer program: Teach Baltimore. Baltimore: Johns Hopkins University, Center for Research on the Education of Students Placed at Risk. Boruch, R. F. (1997). Randomized experiments for planning and evaluation: A practical guide. Thousand Oaks, CA: Corwin. Chambers, B., Cheung, A., Gifford, R., Madden, N., & Slavin, R. E. (2004). Achievement effects of embedded multimedia in a Success for All reading program. Baltimore: Success for All Foundation. Doll, R. (1998). Controlled trials: The 1948 watershed. British Medical Journal, 317, 1217-1220. Fraker, T., & Maynard, R. (1987). The adequacy of comparison group designs for evaluations of employmentrelated programs. Journal of Human Resources, 22(2), 194-227. Friedlander, D., & Robins, P. K. (1995). Evaluating program evaluations: New evidence on commonly used nonexperimental methods. American Economic Review, 85(4), 923-937. Gallagher, J. (2002). What next for OERI? Education Week, 21(28), 52. Herman, R. (1999). An educator's guide to schoolwide reform. Arlington, VA: Educational Research Service.
12
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Ioannidis, J. P. A. et al. (2001). Comparison of evidence of treatment effects in randomized and nonrandomized studies. Journal of the American Medical Association, 286(7), 821-830. Kaestle, C. F. (1993). The awful reputation of educational research. Educational Researcher, 22(1), 23-26-31. Lagemann, E. C. (2002, April). An elusive science: The troubling history of education research. Paper presented at the annual meeting of the American Educational Research Association, New Orleans. Levin, M. E., & Levin, J. R. (1990). Scientific mnemonics: Methods for maximizing more than memory. American Educational Research Journal, 27, 301-321. Linn, R. L., & Haug, C. (2002). Stability of school-building accountability scores and gains. Educational Evaluation and Policy Analysis, 24(1), 29-36. Mosteller, F., & Boruch, R. (Eds.) (2002). Evidence matters: Randomized trials in education research. Washington, DC: Brookings. National Reading Panel (1999). Teaching children to read. Washington, DC: U.S. Department of Education. Rhine, W. R. (1981). Making schools more effective: New directions from Follow Through. New York: Academic Press. Shavelson, R. J., & Towne, L. (Eds.) (2002). Scientific research in education. Washington, DC: National Academy Press. Slavin, R. E. (1989). PET and the pendulum: Faddism in education and how to stop it. Phi Delta Kappan, 70, 752758.
Slavin, R. E. (1997). Design competitions: A proposal for a new Federal role in educational research and development. Educational Researcher, 26(1), 22-28. Slavin, R. E. (2003). Educational psychology: Theory into practice. (7th ed.). Boston: Allyn & Bacon. Slavin, R. E., & Madden, N.A. (Eds.) (2001). One million children: Success for All. Thousand Oaks, CA: Corwin. Snow, C. E., Burns, S. M., & Griffin, P. (Eds.). (1998). Preventing reading difficulties in young children. Washington, DC: National Academy Press. Southwest Educational Research Laboratory (2002). CSRD database of schools (web site: http://www.sedl.org/csrd/ awards.html) Austin, TX: Author U. S. Congress (2001). No Child Left Behind Act of 2001. Washington, DC: Author. U. S. Department of Education (2002). 2002 application for new grants: Preschool curriculum evaluation research grant program (CFDA No. 84.305J). Washington, DC: Author. U. S. Department of Education (2002b). Strategic plan, 2002-2007. Washington, DC: Author. U. S. Department of Education (2002c). Draft guidance on the Comprehensive School Reform Program (June 14, 2002 update). Washington, DC: Author. Whitehurst, G. (2002). Charting a new course for the U. S. Office of Educational Research and Improvement. Paper presented at the annual meeting of the American Educational Research Association, New Orleans.
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
13
Closing the Performance Gap in a 4th Wave and Post-Modern Society: Lessons from the Field J. Q. Adams Western Illinois University Abstract The USA is undergoing tremendous Cultural Changes as we open the first decade of the 21st century. In this paper the author discusses the need to close the performance gap that exists between White, African American, and Latino/a students. To do so, educators must carefully consider several important cultural forces. The author examines the shift in demographic patterns as well as the impact of the 4th Wave and the emergence of post-modernism on education and society.
The USA is undergoing tremendous cultural changes as we open the first decade of the 21st century. To close the performance gap existing between White, African American, and Latino/a students, educators must carefully consider several important cultural forces. In this paper we will examine the shift in demographic patterns as well as the impact of the 4th Wave and the emergence of post-modernism on education and society. Demographic Changes According to Suarez-Orosco, co-director of the Harvard Immigration Project, as recently as the 1960s eighty-five percent of the immigrants entering the USA were from Europe (Adams, 2001). Since that time there has been a dramatic continental shift. As the twenty-first-century opens eighty to eighty-five percent of immigrants are coming from Latin America, the Afro-Caribbean Basin and Asia. This demographic shift is changing the student population in our schools and increasing the cultural gap between our students and the teachers and administrators who work with them. Source: U. S. Bureau of the Census. Figure 1. Population projections: 2000­2001. This phenomenon is exacerbated by the hyper-segrega- tion many of these new ethnic groups face as they transition
into this nation's communities. Latino/as are more likely to be in schools in which they, a "minority," are the majority. Minority entry into predominately white neighborhoods can trigger "white flight" which also helps to fuel the new voluntary segregation of ethnic minority groups. While the student core is changing, however, the teacher core usually remains the same. Many of these schools have rapidly changed from nearly all white to nearly all minority schools. The cultural gap between veteran white teachers and their minority students that results may contribute to the performance gap of these students. This is especially true if teachers are not able to connect to the different linquistic, religious, and other cultural differences of these ethnic minority students and their parents. Gordon (2004) and other researchers have identified a lowering of expectations by white teachers when they are dealing with African American and Latino/a students under these conditions. Wave Theory For the first time in recorded history there are four distinctive human developmental periods co-existing on the planet. Each wave is distinct and marks one of the major ways in which the six billion people live on the planet today. The 1st Wave, or the Hunter-Gatherers, dominated the planet some 50,000 thousand years ago. They were largely nomadic, lived in clan and tribal societies, and developed rudimentary tools that aided their ability to hunt and forage for food. The fact that this way of life is still successful speaks volumes about the resiliency, creativity, and intelligence of the humans who still practice this way of life. The vast majority of the world still lives in the Agrarian 2nd Wave which in many ways has changed very little in the last 12,000 years. While farming technology has evolved most 2nd Wave societies are still dominated by patriarchal extended families, a bifurcated class system with a large peasant class, and a primary education system geared to maintaining the status quo. The Industrial 3rd Wave is marked by a shift from the country to the city, from the farm to the factory, from extended to nuclear families, and from primary to secondary
14
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
education. The development and spread of technology creates an evolution that consumes natural resources at a rate never experienced before. This has led to an addiction to the consumption of consumer goods which in turn fuels the production of these goods. A by-product of the rise of capitalism is the multibillion dollar marketing and advertising industry that is a seductive distraction for many young people and may play an important role in the choices students make with their time. This Wave also seems to promote democratic principles and the advancement of women and minority rights for equity and social justice. The emergence of the Post-Industrial High Tech 4th Wave has separated the planet into a system of haves and have nots. This digital divide for some countries may take decades to close. For the first time in the history of the planet what a person knows from the neck up is more important than what their body can do from the neck down. Advanced literacy and numeracy skills are absolutely essential for competing within the 4th Wave workforce in which employers will search for there workers on a global level, harvesting the best and the brightest talent from wherever it can be found. The fallout from the 4th Wave is having both inter- and intra-national consequences. The intra-national impact on the family has been profound. The frequency of single parent families, families with two moms or two dads and a variety of other combinations are now common place. This also contributes to the performance gap, a puzzle that will have ominous consequences if education is not able to help these students compete for the jobs that will allow them access to the middle class. As the need for solid educational skills increases, African Americans and Latino/as are graduating from America's high schools with about the same knowledge as the average white eighth-grade student in math and science.
Table 1 NAEP Scores for Grades 8 and 12: Science, Mathematics, and Reading
Grade 8
Grade 12
Science
1996
2000
1996
2000
Blacks
121
122
124
123
Hispanics
129
128
130
128
Whites
159
162
159
154
Mathematics
1996
2000
1996
Blacks
243
247
280
Hispanics
251
253
287
Whites
282
286
310
The Nation's Report Card: Science 2000 and Mathematics 2000
2000 274 283 308
The inter-national implications of Wave Theory will continue to grow in significance since most of the new immigrants are coming from 2nd Wave cultures. These families face not only culture shock but also Wave shock. The cultural distance between the 2nd and 4th Waves is immense but is often not considered by the educational community.
With the exception of Bilingual/ESL programs, most schools have assumed that their only responsibility is the formal education of these students, not their cultural transition. In order for schools to be successful they must effectively deal with the cultural divide that exists within our changing communities. Post-Modern USA Along with the tremendous changes taking place as the result of our changing demographics and our emergence into the 4th Wave, the USA is dealing with a cultural shift from Modernism to Post-Modernism. The late eighteenth through the twentieth century was dominated by the modernist belief in the scientific method, reason over ignorance, order over disorder, high culture/the classics over folk knowledge, and the emergence of egalitarianism as well as meritocracy. However, as the twentieth century became plagued with world wars, eugenics movements, racism, genocide and the threat of nuclear war, philosophers began to question whether these modern ideas have been responsible for these cataclysmic events. This has led to a rejection of a variety of social foundations including the scientific method as the only way of knowing the truth. According to O'Hare and Anderson (1991), the post-modern world is shaped by pluralism, democracy, religious freedom, consumerism, mobility, and increasing access to news and entertainment. Residents of this post-modern world are able to see that there are many beliefs, multiple realties, and an exhilarating but daunting profusion of world views; they see a society that has lost its faith in absolute truth and in which people have to choose what to believe. In the arts composers like John Cage challenge the conventional structures of classical music while Hip Hop goes about redefining world music in both its popularity and message. In architecture the skylines of our cities are changing as traditional landscapes give way to styles that challenge basic modern forms. The Las Vegas strip skyline may be the best example of these new post-modern forms. Many of our African American and Latino students have rejected the traditional rhetoric of the mainstream, choosing instead to listen to the alternative voices within their own ethnic identity or the persuasiveness of "pop culture" (Dyson, 2000). Cutting Through the Clutter Media critic Sut Jhally (1997) states that Americans are bombarded by more than 3600 commercial messages every day. Marketing strategies are designed to flood consumers with their messages using every medium possible including television, radio, newspapers, magazines, billboards, and the internet. This competition to reach the consumer has led advertisers to develop strategies that are able to cut through the clutter. The messages in their ads have become more dramatic, obnoxious, and erotic in order to get our atten-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
15
tion. The objectification of the female body is now common place in ad campaigns that appeal to both males and females. This competition for attention spills over into the educational realm, desensitizing our students because of the overwhelming saturation they are subjected to everyday. It takes powerful messages to cut through the clutter of ads surrounding the average adolescent in our society, especially African American and Latino/a adolescents. Companies like Coca Cola, PepsiCo, Ralph Lauren, Nike and others spend billions in advertisement dollars to capture the attention of young people in today's society.
In Billions
50
45
40
35
30
25
20
15
10
5
0
Direct Mail
Newspapers
Television
Radio
Cable TV
Magazines
Internet
OutdoorBusiness Papers
Sources: PwC/IAB Internet Ad Revenue Report: McCann Erickson
Figure 2. 2003 U.S. Advertising Spending
Educators find it difficult to compete financially or creatively for the attention of our young people. The typical allure of K-12 education in the public sector is that it is free and compulsory.
The Emergence of Ethnic Identity in social space
Today's schools are prime examples of where competition for social space is being waged. As ethnic minority groups grow they are challenging the traditional control of the white middle class. As "white flight" takes place and "hyper-segregation" expands, the characteristics of the tra-
Percentage of High School Graduates with Substantial Credits in Academic Courses, 1982 and 1998 Percentage with four years of English, three years each of social studies and mathematics, and two years of a foreign language
70
60
Asian 50 White
40
Black
Hispanic 30
American
20
Indian/Alaskan Native
10
0 Source: National Center for Education Statistics, Digest of Education Statistics 2001, Table 143. Original data from the National Center for Education Statistics, High School Transcript Study
Figure 3. Rigor of Curriculum
ditional student are being redefined. The new ethnic student possesses cultural dimensions that can lead to dissonance within the teacher and administrative core that is entrenched within most of our schools. This becomes expressed in the overrepresentation of these groups in Special Education programs, in suspensions, and in expulsion rates. Additionally, this phenomenon is expressed in the under-representation of these groups in Gifted, Honors, and Advanced Placement Programs.
Distribution of Advanced Placement Examinations Compared with the Distribution of the High School Population, by Race/Ethnicity, 1999/2002* 70
60
50
40
Black
Hispanic
30
White
20
10
0
Percent of AP
Percent of High
Examinations
School Examinations
*AP examinations are for 2002; high school population data are for 1999
Sources: AP data are from the College Board; High School population data are from the National Center for Education Statistics, Digest of Education Statistics 2001, Table 42
Figure 4. Rigor of Curriculum
However, in the social and athletic aspects of school, ethnic minority students often compete remarkably well. If they have the abilities they can quickly distinguish themselves in athletics and become popular enough over time to be elected as homecoming king or queen. Time on Task Issues
One of Jawanza Kunjufu's favorite statements is "What you do most is what you do best." According to Kunjufu this is the reason African Americans excel in sports: In this area they work as hard or harder than anyone else (Adams, 1991). The question is why African Americans choose these activities to concentrate their time and energy on. The answer is simple. They have observed a visible chain of success in these endeavors, and these successes lead to family and community status, which can lead to scholarships and the possibility within a small elite to earn millions of dollars at the professional level. This formula seems to work in sports as well as other forms of entertainment in our society. In addition, while in school these students must maintain passing grades in order to stay eligible. In the "High School and Beyond" (Tuma, Geis, & Carroll, 1995) study, researchers found that African Americans spent less time studying than any other ethnic/racial group in the USA. On the other hand, they spent more time watching television.
16
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Percentage of Fourth Graders Watching Six Hours or More of TV per Day by Race/Ethnicity (All Schools), 1992-2000
50 40 30 20 10 0 1992
2000
White Black Hispanic
Source: National Center for Educational Statistics, Digest of Education Statistics 2001, Table 112. Figure 5. Television Watching
Additionally, they spent more time engaging in other social activities that are not academically related but may enhance their social ability skills.
Expectations
Steinberg's (1996) study clearly found ethnic and racial differences in student perception of how they thought their parents would react to low grades. Asian American students felt that if they brought home anything lower than an A- they would be in trouble with their parents. European American students stated anything below a B- would result in parental disapproval. For Latino/as and African Americans the lowest acceptable grade was D+. When these data are combined with time-on-task characteristics there should be little wonder why African Americans and Latino/as find themselves performing significantly below their white and Asian counterparts academically.
Ethnic Propriospect
Propriospect can be defined as the "sum total of ones experiences" (Adams, 2001). The experience of ethnic and racial minorities is significantly different from whites due to their long-term historical association to oppression, racism, and prejudice in our society. This difference in how they see the world may have an impact on their decision making and explain why African Americans and Latino/as do not invest in academics at the same level of intensity as their white and Asian counterparts. White middle- and upper-class children perceive the world through a propriospect of privilege, open-ended opportunity, a belief in individual effort, and access to successful role models in their family, community, and nation. This is not to say that whites do not have obstacles to overcome, but few would argue that these obstacles have been as severe or as long lasting as those African Americans and Latino/as have experienced. Many African Americans and Latino/as simply do not buy into modern education's saccharine view of how to make it in our society. Their folk vision suggests to them that there are viable options other than academics at this stage in their lives which will enable them to make it in American society.
Implications for Educators in the 4th Wave Postmodern Era During the 1980s I was involved in the resettlement efforts of Southeast Asian refugees into our country. Communities all across our nation reached out to help these strangers integrate into our neighborhoods and schools. One group in particular, the Hmong, seemed to have more difficulty in their transition than many of the other groups. In hindsight it is now clear why they had so many problems. The Hmong were from 1st Wave societies. While most of the other refugees were from 2nd and 3rd Wave cultures, the Hmong had more cultural differences to overcome. These were people from the highlands of Laos who were primarily nomadic and who had an oral tradition rather than a written language. This made their transition more difficult. Learning English and adjusting to the rapid pace of a 4th Wave society presented a huge challenge. Hmong cultural traditions created problems in other ways. For example, the Hmong are very particular about their bodies and disrobing in public was strictly taboo, but the physical education requirements in the junior and senior high schools they attended required all students to do this. The Hmong girls in particular simply refused to comply. This resulted in matriculation and graduation requirement issues. In one particular school district students were forced to disrobe or be suspended from school. This is a clear example of the additional hurdles culturally different students can bring into our schools. How do schools build bridges across these cultural gulfs? Should the culture of the school be negotiable as it changes from predominately white and middle class to a population that includes increasing numbers of children of color from a spectrum of Wave and social class experiences? Our schools are now places of identity politics, where ethnic or cultural groups compete for control of the social space. Teachers, administrators, and researchers must become acutely conscious of these changing forces if we are to understand why students make the choices they do in this 4th Wave post-modern milieu. The keys to closing the performance gap lie within the dimensions of our cultural propriospects as well as our pedagogy. Lessons from the Field I have worked in the area of educational reform for over twenty years and here are a few of the lessons that I have learned that may be helpful in closing the existing performance gap: 1. In order to compete for the attention of African American and Latino/a students a tri-focal approach is necessary. Community, family, and school must unite with single-mindedness and high expectations for all students. 2. school districts must commit to continuous ongoing faculty and staff development that emphasizes and champi-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
17
ons cultural literacy and intercultural communication skills. 3. Using the tri-focal approach School Districts must strive to create learning communities that develop the knowledge and will to bridge the gulf between the cultural differences and misunderstandings that may arise when the "other" comes into existence. 4. Schools must transition from vessels of assimilation to models of acculturation that respect the cultural identity of each student while encouraging them to reach their highest academic potential. 5. Schools must practice democratic principles, which take advantage of positive teacher intrusiveness, promote student self-regulation and value service learning. Final Thoughts Our ability as educators to recognize, understand, and learn about the differences as well as the similarities of our culturally diverse students will be the key to closing the performance gap in our schools. A one-size-fits-all approach will not work given the impact of the Post-Modern influences challenging everything from evolution and creationism to what art is. The cultural distance between the early Waves and the 4th Wave will continue to expand as this century continues. The absolute necessity of advanced literacy will mark the lines of stratification between those who have agency and those, as Freire (1970) has described, who are only objects of history. Given our changing demographics we must have a national commitment to close this performance gap or condemn much of our future majority (African Americans and Latino/as) to less than a middle-class existence. References Adams, J. Q. (1991). Dealing with diversity: Telecourse, PBS Adult Learning Service. Alexander, VA. Adams, J. Q. (2001) The social interaction model. In J. Q. Adams & P. Strother-Adams, Dealing with diversity: An anthology. Dubuque, IA: Kendal-Hunt.
Barton, P. (2003). Parsing the achievement gap: Baselines for tracking progress. Princeton, NJ: Educational Testing Service. Braswell, J. S., et al. (2001). The Nation's Report Card: Mathematics 2000 (NCES 2001517). Washington, DC: National Center for Education Statistics. Cota-Robles, E., & Gordon, E. (1999). Reaching the top: A report of the national task force on minority high achievement. New York: The College Board. Dyson, M. E. (2000). Art, Commerce and Race: The Controversial Debate on Music and Violence. The Committee on Commerce, Science and Transportation. U.S. Senate September 13, 2000. Flores-Gonzбles, N. (2002). School kids / Street kids. New York: Teachers College Press. Freire, P. (1970). Pedagogy of the oppressed. New York: Herder and Herder. Gordon, E., et. al. (2004). All students reaching the top: Strategies for closing academic achievement gaps. Naperville, IL: Learning Point. Jhally, S. (1997). Advertising and the end of the world. Northampton, MA: The Media Education Foundation. Mirуn, L. (1996). The social construction of urban schooling: Situating the crisis. Cresskill, NJ: Hampton Press. Ogbu, J. (2003). Black american students in an affluent suburb: A study of academic disengagement. Mawah, NJ: Lawrence Erlbaum Associates. O'Hara, M., & Anderson, W. T. (1991). Welcome to the PostModern World. Networker, September/October. O'Sullivan, C.Y., et al (2003). The Nation's Report Card: Science 2000 (NCES 2003453). Washington, DC: National Center for Education Statistics. Steinberg, L. (1996). Beyond the classroom: Why school reform has failed and what parents need to do. New York: Touchstone Books. Tuma, John, Geis, S. and Carroll, Dennis. (1995). High school and beyond: Educational attainment of 1980 high school sophomores by 1992. Washington, DC: National Center for Educational Statistics (NCES 95304).
18
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Research Association
2005 Annual Meeting Call for Proposals
PROPOSAL DEADLINE: May 1, 2005
October 12-15, 2005
Sharon A. Valente, Program Chair
The Westin Great Southern, Columbus, OH
[email protected]
The 2005 Annual Meeting of the Mid-Western Educational Research Association (MWERA) will be held in Columbus with an exciting program of invited speakers, focused workshops, and peer-reviewed papers presented in a variety of session formats. The 2005 program will center around this year's theme: academic integrity: Responsible Learning and will feature dynamic speakers of interest to both researchers and practitioners. Teachers, administrators, and other school personnel are especially invited to come and share their school-based research and experiences at the 2005 MWERA conference. We will be meeting at the Westin Great Southern in Columbus, a historic landmark hotel, featuring charming guest rooms, excellent meeting
facilities, and a location only a short walk from the quaint shops of German Village and one block from the Columbus City Centre. The hotel facility does have wireless computer access. Columbus is the home to numerous theaters, a symphony, wonderful restaurants, shopping and fun nightlife! If you are looking for a place to sit down and chat with researchers from schools and universities about your ideas and perspectives, the Mid-Western Educational Research Association provides that opportunity with its supportive, collaborative environment. Educational researchers across North America return to MWERA to renew acquaintances, make new contacts, and engage in exciting conversation in a collegial atmosphere. Come and be a part of MWERA in 2005!
General Information The 2005 MWERA Annual Meeting will be held Wednesday, October 12 through Saturday, October 15, at the Westin Great Southern in Columbus, Ohio. This year's theme is Academic Integrity: Responsible Learning. The program will consist primarily of presentations, selected through a peer review process, by divisional program chairpersons. In addition, there will be invited speakers and symposia, panel discussions, special sessions for graduate students and new faculty, a luncheon and other social events open to all attendees. Proposals MUST be submitted electronically over the internet using the form available on the meeting website. Proposals mailed or e-mailed to the Program Chair or Division Chairs will NOT be processed. Specific instructions for electronic submission can be found at the meeting website: http://www.mwera.org Questions about a proposal, the electronic submission process, or the meeting should be directed to the Program Chair: Sharon A. Valente MWERA-2005 Program Chair Ashland University 247 Dauch College Ashland, OH 44805 Office: 419-289-5222 Fax: 419-289-5910 E-mail: [email protected] Any educational professional may submit a proposal for MWERA-2005, whether or not that person is currently a member of MWERA. All Annual Meeting presenters must be members in good standing with MWERA (non-members must join MWERA upon notification of proposal acceptance). To promote broader participation in the program, no one person should appear as a presenter on more than three proposals. All proposals must be posted on the MWERA website no later than midnight EST on May 1, 2005. Submissions will then be forwarded to Division Chairs. Each Division Chair will coordinate a number of volunteers in a system of blind (without author identification) review. Appropriate criteria, depending on the format and type of scholarly work being presented, have been developed and are used for the review process. These criteria include: (a) topic (originality, choice of problem, importance of issues); (b) relevance of topic to the Division and MWERA membership; (c) contribution to research and education; (d) framework (theoretical/conceptual/practical, rationale, literature review, grounding); (e) analyses and interpretations (significance, implications, relationship of conclusions to findings, generalizability or usefulness); and (f) overall written proposal quality (clarity of writing, logic, and organization).
Papers presented at MWERA are expected to present original scholarship, conducted by the author(s), which has not been previously presented at any other meeting or published in any journal. Further, it is a violation of MWERA policy to promote commercially available products or services (except as Exhibits) that go beyond the limits of appropriate scholarly/ scientific communication. Individuals who wish to display educationally related products or services are encouraged to contact Dr. Sharon McNeely, Historian/Archivist, P. O. Box 34421, Chicago, Illinois 60634-0421, 773- 442-5518. All persons presenting at the 2005 Annual Meeting are expected to register for the full meeting, including graduate students. All sessions listed in the program will be open to any registered meeting participant; however, enrollment may be limited, and a small additional fee required, for some workshop sessions. Tickets for the Friday luncheon and speaker are available to all pre-registrants. Ticket availability is not guaranteed for late and on-site registrants. Registration materials for the 2005 Annual Meeting will be published in the MidWestern Educational Researcher, on the MWERA website, and can be obtained by contacting the Program Chair. Presenters whose papers have been accepted to a session with a Session Chair and/or Session Discussant are responsible for submitting a completed version of their conference paper to the Session Chair and Discussant no later than September 16, 2005. Papers not available to the Session Chair and Session Discussant may be dropped from the program. Presenters must also provide complete copies of their papers (or detailed handouts) to attendees at their sessions. Overhead projectors and screens will be provided by MWERA in most presentation rooms. Presenters needing additional A/V equipment are responsible for arranging such with the hotel at the presenter's own additional expense. MWERA reserves the right to reproduce and distribute summaries and abstracts of all accepted proposals, including making such works available in a printed Program Abstract, through the MWERA website, and in press releases promoting the Annual Meeting and the organization. As a condition of acceptance, all authors of papers accepted to the 2005 Annual Meeting explicitly grant MWERA the right to reproduce their work's summary and/or abstract in these ways. Such limited distribution does not preclude any subsequent publication of the work by the author(s).
Authors of accepted proposals assume the ethical and professional responsibility to appear at the Annual Meeting and to participate in their presentation or assigned session. When circumstances preclude the author(s) from doing so, it is the responsibility of the author to arrange a suitable substitute and to notify the Program Chair in advance.
Important Dates
Proposal Submission Deadline May 1, 2005
Notification of Acceptance
July 15, 2005
Papers to Session Chairs/Discussants
September 16, 2005
Registration and Hotel Reservations
September 23, 2005
MWERA 2005 Annual Meeting
October 12-15, 2005
Guidelines for Submitting a Proposal Session Format Descriptions Paper Presentation Paper sessions are intended to allow presenters the opportunity to make short, relatively formal presentations in which they overview their papers to an audience. Three to five individual papers dealing with related topics are grouped into a single session running from 1.5 to 2 hours. The presenter(s) of each paper is (are) allowed approximately 15 minutes to present the highlights of the paper. A single Session Discussant is allowed approximately 15 minutes, following all papers, for comments and Critical Review. A Session Chair moderates the entire session. Presenters are expected to provide complete copies of their papers to all interested audience members. Roundtable Discussion/Poster Roundtable Discussion/poster sessions are intended to provide opportunities for interested individuals to participate in a dialogue with other interested individuals and the presenter(s) of the paper. Presenters are provided a small table around which interested individuals can meet to discuss the paper. Presenters may elect to provide small, table-top poster-type displays, ancillary handouts, or other table-top A/V materials to augment their discussions. Interested individuals are free to move into and out of these discussions/posters as they wish. Presenters are expected to make available complete copies of the paper on which the roundtable discussion/poster was focused. Symposium A symposium is intended to provide an opportunity for examination of specific problems or topics from a variety of perspectives. Symposium organizers are expected to identify the topic or issue, identify and ensure the participation of individual speakers who will participate in the session, prepare any necessary materials for the symposium, and Chair the session. It is suggested, though not required, that the speakers or symposium organizer will provide interested individuals with one (or more) papers relevant to, reflective of, or drawn from the symposium. Workshop Workshops are intended to provide an extended period of time during which the workshop leader helps participants develop or improve their ability to perform some process (e.g. how to provide clinical supervision, using the latest features of the Internet, or conduct an advanced statistical analysis). Organizers may request from 1.5 to 3 hours, and are responsible for providing all necessary materials for participants. Many workshops are scheduled for Wednesday afternoon, although others may be scheduled throughout the conference. Organizers may, if they wish, receive an honorarium based upon the number of paid participants
in their workshop and the fee schedule. Alternative Session The form, topics, and format of alternative sessions are limited only by the imagination and creativity of the organizer. These options are intended to afford the most effective method or approach to disseminating scholarly work of a variety of types. Proposals for alternative sessions will be evaluated on their appropriateness to the topic and audience, their suitability to meet the limitations of time, space, and expense for MWERA, and the basic quality or value of the topic. The organizer of alternative sessions is responsible for all major participants or speakers, developing and providing any necessary materials, and conducting or mediating the session. Because a variety of approaches may be proposed within this category, alternative session proposals should include a brief rationale for the alternative being proposed. Best Practices Forum The "Best Practices" sessions are intended to provide opportunities for individuals or groups to present "best" or "promising" practices impacting both K-12 and higher education. These sessions highlight unique and innovative programs that have demonstrated promise for improving and enhancing educational practice. Presenters will be grouped by similar topics to facilitate discussion between and among the groups and audience. Presenters are expected to make available complete copies of the paper on which the "Best Practices" session focused. Submitted Content Summary Summaries for Paper and Roundtable Discussion/Poster proposals should explicitly address as many of the following as appropriate, preferably in this order: (1) Objectives, goals, or purposes; (2) Perspective(s) and/or theoretical framework; (3) Methods and/or techniques (data source, instruments, procedures); (4) Results and conclusions; and (5) Educational and/or scientific importance of the work. Summaries for Symposium, Workshop, and Alternative Session and Best Practices Forum proposals should explicitly address as many of the following as appropriate, preferably in this order: [1] Descriptive title of the session; [2] Objective, goals and purposes of the session; [3] Importance of the topic, issue, or problem; [4] Explanation of the basic format or structure of the session; [5] Listing of the presenter(s), by number not name for blind review (e.g., Presenter 1), with an explanation of each person's relevant background and role in the session; [6] Anticipated audience and kind of audience involvement. Abstract The abstract should be 100 - 150 words. The abstracts of accepted papers will be published in the MWERA 2005 Annual Meeting Abstracts book, and will be available on the MWERA website. Use clear, precise language, which can be understood by readers outside your discipline.
Session Descriptors Ability Grouping Accountability Accreditation Achievement Action Research Adaptive Testing Administration Admissions Adolescence Adult Education/Development Affective Education African-American Education Aging Anthropology Aptitude Artificial Intelligence Arts Education Asian Education Assessment At-Risk Students Attitude Attribution Bilingual/Bicultural Business Education Career Development Case Studies Certification/Licensure Child Development Classroom Management Classroom Research Clinical Education Cognition Cognitive Processes/Develop Collaboration Community Colleges Comparative Education Compensatory Education Comprehension Computer Applications Computerized Testing Computers and Learning Conceptual Change Constructivism Continuing Education Cooperative Learning Counseling Counselor Training/Supervision Critical Theory Critical Thinking Cross-Cultural Studies Curriculum Data Analysis Decision Making Demography Desegregation Differential Item Functioning Dimensionality Dropouts Early Childhood Economics of Education
Educational Policy Educational Reform Elementary Schools Equating Equity Ethics Ethnicity Evaluation Experimental Design
Performance Assessment Philosophy Physical Education Planning Politics Postsecondary Education Principals Private Education Problem Solving
Facilities Factor Analysis
Professional Development Program Evaluation
Faculty Development Family/Home Education Finance Gay/Lesbian Studies Gender Studies Generalizability Theory Gifted Education Governance High Schools Hispanic Education History Indian Education Indicators/Information Systems Individual Differences Information Processing Instructional Design/Development Instructional Practices Instructional Technology Intelligence International Education/Studies Item Response Theory (IRT) Language Comprehension/Devel Language Processes Law/Legal Leadership Learning Environments Learning Processes/Strategies Life-Span Development Literacy Literature Mainstreaming Mathematics Education Measurement Media Medical Education
Psychometrics Qualitative Research Race Reading Research Methodology Research Utilization Restructuring Retention Rural Education School/Teacher Effectiveness Science Education Self-Concept Social Class Social Context Social Processes/Development Social Studies Education Sociology Special Education Staff Development Standard Setting Statistics Stress/Coping Structural Modeling Student Behavior/Attitude Student Cognition Student Knowledge Student Teaching Studying Supervision Survey Research Teacher Assessment Teacher Characteristics Teacher Cognition Teacher Education/Development Teacher Knowledge
Memory Mentoring Meta-Analysis Metacognition middle schools Military Education Minorities Moral Education/Development Motivation
Teacher Research Teaching Context Technology Testing Test Theory/Development Textbooks Tutoring Urban Education Validity/Reliability
Museum Education
Vocabulary
NAEP
Vocational Education
Networking
Women's Issues
Organization
Theory/Change
Work
Peer Interaction/Friendship Writing
Divisions A - Administration and Leadership This division is concerned with research, theory, development, and the improvement of practice in the organization and administration of education. Chair: Chuck Kline, 5136 Beering Hall, Purdue University, West Lafayette, IN 47906, [email protected] B - Curriculum Studies This division is concerned with curriculum and instructional practice, theory, and research. Chair: Emery Hyslop-Margison, Ball State University, Department of Educational Studies, Muncie, IN 47306, [email protected] C - Learning and instruction This division is concerned with theory and research on human abilities, learning styles, individual differences, problem solving, and other cognitive factors. Chair: Tracey Stuckey-Mickell, Northern Illinois University, 147 Gabel Hall, Dept. of Literacy Education, DeKalb, IL 60115,[email protected] D - Measurement and Research Methodology This division is concerned with measurement, statistical methods, and research design applied to educational research. Chair: Gordon Brooks, Ohio University, 305A McCracken Hall, Athens, OH 45701, [email protected] E - Counseling and Development This division is concerned with the understanding of human development, special education, and the application and improvement of counseling theories, techniques, and training strategies. Chair: John Laux, University of Toledo, MS 119, Dept. of Counseling and MHS, 2801 W. Bancroft, Toledo, OH 55009, [email protected] F - History and Philosophy This division is concerned with the findings and methodologies of historical research in education. Chair: Sandra Spickard Prettyman, University of Akron, 326 Zook Hall, Akron, OH 44325-4208, [email protected] G - Social Context of Education This division is concerned with theory, practice, and research on social, moral, affective, and motivational characteristics and development, especially multicultural perspectives. Chair: Aimin Wang, EDP, 118 McGuffey Hall, Miami University, Oxford, OH 45056, [email protected] H - School Evaluation and Program Development This division is concerned with research and evaluation to improve school practice, including program planning and implementation. Chair: Isadore Newman, University of Akron, 424 Zook Hall, Akron, OH 44325-4208, [email protected] I - Education in the Professions This division is concerned with educational practice, research, and evaluation in the professions (e.g., medicine, nursing, public health, business, law, and engineering). Chair: LeAnn Derby, 2354 Fairchild Dr. Suite 4L8, United States Air Force Academy, Colorado Springs, CO 80840, [email protected] J - Postsecondary Education This division is concerned with a broad range of issues related to twoyear, four-year, and graduate education. Chair: Heath Marrs, Dept. of Psychology, Fort Hays State University, 600 Park St., Hays, KS 67601, [email protected] K - Teaching and Teacher Education This division is concerned with theory, practice, and research related to teaching at all levels and in-service and pre-service teacher education, including field experience supervision and mentoring. Chair: Francine Falk-Ross, Northern Illinois University, Literacy Education, GH 125, DeKalb, IL 60115, [email protected] L ­ Educational Policy and Politics This division is concerned with educational policy as well as political, legal, and fiscal matters related to education. Chair: Kathleen S. Brown, University of Missouri ­ St. Louis, 263 Marillac Hall, 8801 Natural Bridge Road, St. Louis, MO 63121, [email protected]
The IRB and Classroom Research
Jeffrey B. Hecht Northern Illinois University Abstract Scholars conducting research in classrooms face a myriad of ethical issues somewhat unique to the educational setting. While the Code of Federal Regulations (45 CFR 46) generally provides that educational research be classified as exempt from review by Institutional Review Boards, those same regulations provide a host of special conditions under which classroom research must not be considered exempt. Depending on the study, classroom research may also involve issues of power and coercion (especially when the researcher is also the instructor); deception (when part or all of the nature of a study must be hidden from the subjects to avoid a bias in the results); anonymity and/or confidentiality; and compensation (including equivalent alternate assignments, when appropriate). Additional rules designed to protect the confidentiality of student information (FERPA), as well as what kinds of data might be collected and the processes used to collect that data (PPRA) also exist and must be followed. Scholars conducting research in classrooms need to navigate all of these issues as a routine part of their research activities.
" . . . Research conducted in established or commonly accepted educational settings, involving normal educational practices . . . " (45 CFR § 46.101(b)1) "Research involving the use of educational tests . . . survey procedures, interview procedures, or observation of public behavior . . . " (45 CFR § 46.101(b)2) "Research means a systematic investigation, including research development, testing and evaluation, designed to develop or contribute to generalizable knowledge." (45 CFR § 46.102(d)) " . . . adequate provisions are made for soliciting the assent of children, when in the judgment of the IRB the children are capable of providing assent . . . " (45 CFR § 46.408(a)) The provisions of 45 CFR 46 provide a set of minimum requirements by which Institutional Review Boards (IRBs) review research which intends to use personally identifiable information from living humans as subjects. Recognizing that certain types of research typically pose little risk to participating subjects and were already commonplace activities in our society, the regulations established six categories of research that are "exempt from this policy" (45 CFR § 46.101(a)2(b)). These components of the regulation would seem to make most, if not all, educational research conducted in elementary, secondary and post-secondary institutions exempt from the IRB process. However, many exceptions and special situations routinely occur that require the research to be reviewed and approved before being conducted. Specific sections in the regulation call for additional protection to be afforded to children, especially those individuals of diminished capacity (Subpart D, 45 CFR §46.401 through .409) or those who are incarcerated. Other sections require a greater degree of IRB review of proposed research
if the investigator is to record data using either an audio or video recorder (where anonymity is difficult, if not outright impossible, and the conditions for safeguarding confidentiality become paramount). Deception, whether active or passive (by misdirection or omission) are also a concern, especially as it involves the ability of subjects (and their legal guardians) to provide true, informed consent. Questions of apparent conflict within the regulations and of the meaning of critical terms have created further confusion in the research and educational communities. These questions include: · What is a "commonly accepted educational setting?" · When is a practice a "normal educational practice" as compared to "research?" · What does it mean to obtain "parental consent" versus "the assent of the child?" Educational researchers, over the past two decades, have been trying to find a means to better define and work within the scope of these regulations. In 1979 Grothberg described the building resistance by educational researchers to complying with the then-new regulations. In 1983, Lyons wrote to rural educators about these regulations giving suggestions for how to completed IRB proposal forms. The discussion continues almost unabated through to today (see Hecht, 1996; 2003; 2004; Howe and Dougherty, 1993). The intersection of local practice and custom, federal and state regulation, and a desire on the parts of researchers and practitioners alike to advance the field of knowledge while acting in a professional and ethical manner seem to have created more questions than answers. Beyond the IRB Two additional bodies of regulation also have an impact on research conducted in classrooms. The first is known
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
23
as the Family Educational Rights and Privacy Act (20 U.S.C. § 1232g; 34 CFR Part 99), known more commonly by its abbreviation FERPA. The second is the Protection of Pupil Rights Amendment (20 U.S.C. § 1232h; 34 CFR Part 98), known by its abbreviation PPRA. Although neither addresses ethical research concerns directly, both deal with the kinds of information that can be collected, maintained and accessed by educational institutions; how that information might be used; and who can have access to that information. Gaining access to the classroom setting or to data stores within the institution is typically as much (if not more) of a challenge than any IRB ethical research concerns are. FERPA provides students 18 years of age and older (as well as those younger attending a school beyond the high school level) and parents of those students under 18 years of age certain specific protections with regard to their schoolmaintained private records. These rights include the right to inspect and review the records, the right to have inaccurate or misleading information corrected, and the need for written permission prior to the release of any information from a student's record except in specific, fairly narrowly defined circumstances. Notably, research is not among the circumstances listed. Educational institutions must not only ensure that collected information is being held in confidence and only used by appropriate individuals for appropriate uses, but must also annually inform students and parents of their rights under this regulation. Even directory information such as a student's name or address must be publicly withheld by the institution if requested by student. PPRA, also known as the "Grassley Amendment" after Senator Charles E. Grassley (R-IA), applies to programs that receive funding from the U.S. Department of Education. Under PPRA, schools and contractors must make instructional materials available for inspection by parents if those materials will be used in connection with a department funded survey, analysis, or evaluation with children in that school. Furthermore, schools and contractors must obtain written parental consent before minor students are required to participate in any department funded survey, analysis, or evaluation that reveals information concerning: 1. Political affiliations; 2. Mental and psychological problems potentially embarrassing to the student and his or her family; 3. Sex behavior and attitudes; 4. Illegal, anti-social, self-incriminating and demeaning behavior; 5. Critical appraisals of other individuals with whom respondents have close family relationships; 6. Legally recognized privileged or analogous relationships, such as those of lawyers, physicians, and ministers; or 7. Income (other than that required by law to determine eligibility for participation in a program or for receiving financial assistance under such a program).
In effect, what this means for parents is that they no longer have to prove that the specific activity that they find offensive is federally funded. They no longer have to show that it is a research or experimentation program or that it is a psychiatric or psychological test with the primary purpose of revealing private information. They must simply show that the survey, analysis, or evaluation revealed private information, that it was in a federally funded program, and that their consent was not obtained. This is much easier for a parent to demonstrate and will thus provide wider protection for parents and students. (Human Research Report, 1995, p. 6) Many educational institutions, from pre-kindergarten through graduate school, have adopted internal policies that detail how that institution will respond to requests for access by researchers, whether generated internally or externally, to students and their data. Federal regulations and whatever state regulations might also apply, with much variability existing among state regulations, represent the minimum that must be done. The need to be able to rapidly and reasonably respond to a request for research access, together with a desire to implement local standards particular to that institution or geographic area, have led many institutions to seek a single, uniform position on how these rules will be addressed. Rather than separate reviews under IRB, FERPA, PPRA, state and local practices, it is not uncommon for an institution to have internal policies that combine all of the above into a unified view of what is generally accepted and what is not. It is also common for institutions to apply the same uniform rule to all research conducted within its borders (using its faculty, staff or students; within its facilities; or using its equipment or resources), regardless of which agency might have promulgated the rule originally or to which set of funding a particular regulation might specifically apply. It is also common for institutions to invest their local IRB, through the establishment of local policies that go beyond the minimum federal regulations, with the responsibility for seeing that these various regulations are upheld within the context of the federally-mandated IRB role (45 CFR 46) in addition to whatever additional regulations are considered important by the institution. Thus, the IRBs often find themselves considering specifics that go well beyond the typically narrow role of federal regulations. Issues to Consider Balancing among all of these competing mandates can be a challenge for even the most experienced researcher. Convincing an Institutional Review Board of the ethical merit of a particular study can sometimes be particularly difficult, especially if the IRB has little experience in classroom-based research or little knowledge of local schools and settings. It falls upon the researcher, as part of their application to the IRB, to explain these seemingly obvious issues and to provide the rationale under which the issues are considered and appropriately managed in the research.
24
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Differentiating Teaching from Research One issue that frequently confronts IRBs involves research conducted in conjunction with a class. Variations on this theme range from studies conducted within the confines of the classroom solely examining classroom exercises, to studies where the students, acting as researchers, collect data in the field with the hope of writing a paper and presenting their findings. A read of the federal regulation might lead one to believe that, unless the study were to contribute to "generalizable knowledge" (as stated in the regulation), it was not to be considered research. A common although incomplete definition for generalizable knowledge often includes that the results of a research effort be published or intended to be published as a paper in a journal or presented at a meeting. This poses the question: If data is to be collected but the intent is not to publish or present, is the activity thus to be considered research? Rather than solely considering what might be done with the product of the research effort such as whether it will be a paper or a presentation, the researchers and the IRB should, I believe, consider the nature of the research activity itself. The intention of the researcher may not initially be to publish or present his or her data; however once it has been collected those data might look better than initially intended and be suitable for such publication. On the other hand a researcher might have every intention of publishing or presenting findings but, due to unsympathetic reviewers or editors, be unable to find an appropriate outlet for their work. The end product of the research effort may not be under the direct control of the researcher! The research activities, however, are under the investigator's direct control. Whether that investigator is a student or a teacher, in a classroom or in the field, they have a responsibility to behave in an ethical and professional manner. At almost every educational institution procedures exist to insure the fair treatment of students by teachers in classes and other academic ventures. Appropriate codes of conduct can be found for both instructors and students, and mechanisms exist for investigating and adjudicating complaints of teachers against students and students against teachers. Oftentimes the syllabus and class handouts, together with catalogs and other program or institutional documentation, serve as a quasi-contract between a faculty and the students enrolled in a class as to what activities are expected, and what grades will be awarded, in a particular class. Within the confines of a class, then, there appears to be adequate provisions for protecting the rights of all individuals involved. Whether it happens within the physical classroom or outside, both faculty and students have an academic responsibility and obligation to behave in certain ways. Such protections, however, are not found when a faculty or student actively encounters individuals not participating in that class. An outsider is most likely unfamiliar with the requirements of the course, the particular assignment being accomplished, or the protections available through academic channels. Further, if the activity is a re-
search activity--one where a systematic observation or interaction is made of human subjects in a naturally occurring or purposefully manipulated condition--those human subjects may be totally unaware of their participation. In this context, teaching ought to be described as an activity that occurs between and among students and teachers. If, using the definition above, the activity is to be a research activity but is to take place solely among the students and teachers as part of a recognized instructional process-- a process where the students and teachers all know of the design and purpose through a syllabus or handout--the activity may validly be considered as not research for IRB purposes. The activity would therefore be exempt from any further review by an IRB. On the other hand, if the activity is to involve individuals who are not students or instructors participating in the course, or is to involve activities where the students or teachers are unaware of their participation-- such as a case where a faculty member is systematically studying their students' responses to manipulated conditions--or where the nature of the participation is more towards the production of generalizable knowledge than to the benefit of the individual--such as in increased learning--the activity should be considered research. This research is subject to IRB review and approval. Another setting that must be considered are activities that occur in student teaching or corporate internships. Although these activities involve individuals beyond the students and teachers, there is a clear sense and understanding by all involved that such activities are for the training of the students and not for systematic investigation or research. In these cases the IRB encourages clear communication of purposes and intents among all participating individuals so that everyone understands the nature and extent of the activity's interactions. These activities, however, are not research and need not be reviewed by an IRB. On a final note, on occasion an individual involved in a student teaching practicum or corporate internship may want to conduct research as a part of their other experiences. While the thrust of the primary activity is not research that additional activity would be and, therefore, would require a proposal approval by an IRB. The situation becomes even more complicated when the researcher is also the teacher. The researcher seeks to engage in systematic study designed to contribute to general knowledge, while the teacher is a direct practitioner engaged in activities designed with the goal of having students reach certain educational outcomes. Participatory action research methodologies show how these roles can be reinforcing and synergistic, with research informing practice in a continuous cycle of self-examination and reflection. Nothing in any of these regulations would suggest a stifling of creativity on the part of the teacher but thought must be given to some teacher-researchers who are conducting such as extensive inquiry that the above concerns ought apply. This is perhaps the most difficult of situations, as the line blurs around the issue of engaged practice where infor-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
25
mation on student performance is constantly collected, assessed and evaluated in order to inform and change future practice in a continuous improvement process. While activities undertaken solely for the purpose of scholarship with no intent of improving performance are clearly research in nature, other activities share features of both research and practice simultaneously. The role of the teacher is also one that includes a dynamic power relationship, which must be considered as the teacher-researcher seeks student participation in activities not just designed to benefit the student but to benefit the researcher. How much coercion, and of what nature, must be considered. In these situations the combination of any intent to produce generalizable knowledge, together with the degree that such activities might normally take place in that context as a usual part of teaching, help to inform both practitioner and IRB about whether the activity ought be considered as research. What is Exempt? One of the first issues an IRB faces when considering educational research is what it means to be "exempt" under 45 CFR 46 (Weinberger, 1981; Howe and Dougherty, 1993). It is clear that the regulations were crafted so as to not create an undue burden on everyday processes, especially those found in educational settings. By the same token the regulation contains ambiguities and conflicts, especially when utilizing children or other vulnerable populations. One ambiguity is whether a principal investigator should be allowed to decide if their own research is exempt from review. Another ambiguity is what characterizes normal practice in a given educational setting. Unfortunately, no clear single answer exists to any of these questions. Researchers and IRBs across the country have interpreted the regulations differently on each of the issues, crafting specific policies and procedures tailored to the needs of their institution and local community. To a large degree this is how the regulations were intended to operate. There is a convincing argument, however, for certain constants that should apply across all cases. It would not be unreasonable to assume that a principal investigator, and his or her co-investigators, is more in touch with a given research effort than any other person. The investigator understands the background literature on the topic, has considered numerous means for investigation, and is oftentimes financially and viscerally committed to the satisfactory conclusion of the research. This closeness has been shown to create a bias when dealing with research subjects, especially when the investigator is aware of the conditions a given subject will be exposed to such as treatment or control factors. Since experimenter bias is such a powerful force in swaying the results of a study, many researchers will employ the use of a blind, whereby those investigators in direct contact with the subjects are oftentimes ignorant of the specific treatment given to a subject. In this way all subjects will be afforded the same treatment or, if there is variation, at least it will not be attributable to the researcher's knowledge of the research conditions.
The same considerations arise when one is asked to determine whether one's own research should be exempt from a given policy. Although the investigator is the one most familiar with the research effort, he or she is also the most vested into it. A recommended course of action that removes the potential for bias requires an outside person or group, such as an IRB (or other qualified person(s) as designated by the IRB), to act as an impartial reviewer. Such an external review process, in addition to removing any potential suggestion of impropriety, ensures that the researcher and his or her subjects are both protected from any accidental oversights or omissions that could occur. The burden of having an outside person or group review all research to determine if, in fact, the research qualifies for an exemption as outlined is the federal guidelines, does add another step to the process of systematic investigation. The benefit gained is well worth that effort. Deciding what is a normal practice in a given educational setting can be a difficult task. What is common and accepted at one place, or at a certain time, may not be common nor accepted somewhere else, or at another time. On this point IRBs must rely heavily upon the integrity of each researcher and practitioner. As they are the ones most familiar with the research subjects, they must come to understand what is typical and expected for that group. Local customs, norms, and conventions must be considered in each and every case, a process which could add a burden to studies covering larger geographic areas. This also places a burden upon the researcher, however, to educate the IRB as to the conditions of these contexts sufficiently so that claims of exemption can be understood and shared. If the prior suggestion of an external review for all research is implemented the researcher, regardless of the single or multiple roles being enacted, is afforded an opportunity to demonstrate, in a proposal submitted for review, that their proposed activities are identical or closely similar to everyday occurrences. Such a demonstration within the proposal documentation would go a long way to bolstering a claim that a given research project should be considered as exempt from further review. It is burdensome to have to go through such a process--both for the researcher and the institution overseeing the research--yet I believe the benefits far outweigh the added work involved. By opening their research to public scrutiny, as well as reasoned peer review, the researcher removes the potential for criticisms of many kinds. It also involves the subjects and the community as active participants in the research process so that they are not passive individuals from whom one must collect needed data. Consent, Assent, and Access In the past it was sufficient for a researcher to secure the consent of participating research subjects, or the parent or legal guardian if the subject was a minor. This process of consent had developed from legal traditions, including such notions as: the transferral of all relevant information, comprehension on the part of the consent giver, and agreement
26
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
to participate free of undue pressure or coercion. Undergirding these notions is the idea that the subject was legally entitled to give their consent. Individuals who were incarcerated, of diminished capacity, or not yet of the age of majority could not give legal consent. Instead another person, or the state acting as their guardian, was empowered to consider whether or not to consent on behalf of the potential subject. It was clear, however, that even though an individual might not be able to give a legally appropriate form of consent the ethical treatment of each potential research subject requires each person's assent to the research process. Thus, a notion of assent was born whereby most, if not all, research subjects were to be solicited for their active assent to their research participation. In this way it was hoped to protect understanding minors and other individuals from exposure to conditions they did not want even though another had provided legal consent for their participation. For children, especially those exposed to school-based research, this would afford an extra means of protection against overexposure to research, forced participation, and misused research findings (Grotberg, 1979). Unfortunately, recent research has shown that the dual intents of legal consent and subject assent are not always being met. In many cases the forms and scripts used to solicit consent or assent are at a higher reading level than appropriate for the intended subjects (Ogloff and Otto, 1991). This results in many minors not understanding the nature of the research, their role in the effort, or the anticipated risks and benefits. Worse, many children were found to not understand that they could discontinue their participation after they had started if they so desired (Nannis, 1987; Abramovitch, Freedman, Henry, and Van Brunschot, 1995). Thus, while the letter of the regulations appears to be satisfied it has been shown that the spirit of the ethical protection of minor subjects is often not. The implication for educational researchers should be clear. Obtaining consent from the legal parent or guardian must be considered as only the first step in securing the participation of a minor subject. Securing the assent of the minor is the second, critical step. This stage must further involve substantive provisions to insure that the minor understands as fully as is possible the research effort, their role, the risks and benefits and, most importantly, their right not to participate at all and to withdraw at any time (Sanford, 1993). Research has shown repeatedly that minors, to a very young age and even with learning and behavior problems, have the capacity to understand these provisions (Adelman, Lusk, Alvarez, and Acosta, 1985). Educational researchers, who so often interact with minor subjects in the course of research, must make that extra effort to insure that their assent is clearly and capably obtained. Consent and assent should not be confused with access to the subjects. In most cases a researcher, desiring to conduct a study in an educational institution, is required to se-
cure permission from one or more significant persons or groups in order just to gain access to the site where the research subjects are located. In a K-12 school setting these gatekeepers might include the district school board, superintendent, building principal, and classroom teacher. It is entirely within the rights of these individuals to deny a researcher access to a facility and, therefore, any potential research subjects therein, for any number of reasons including FERPA, PPRA, and local regulations or norms. These reasons don't have to be sound, logical, or even in any way related to an ethical consideration of risk and benefit. Since these reasons deal only with the researcher's ability to access the research subjects they also do not constitute a form of either consent or assent. It is still up to the researcher, even after gaining permission to access a particular site and subject pool, to secure IRB approval for their study. What about Technology? The use of new technologies is posing a set of questions for which there are no clear or definitive answers. Computer, audio, and video recording technologies may be used as an integral part of a research project or may just serve as an ancillary means of recording data. While the current regulations allow for an expedited review process for studies using audio recordings nothing is stated about video or computer technologies. It should be apparent that it is not the technology that is necessarily of interest but rather the degree to which a human subject surrenders their privacy or their right to confidentiality (Linowes, 1979). Research processes that involve technologies likely to reduce privacy need be given a more stringent review than those less likely to intrude upon personal space and information. Thus, studies which involve only handwritten notes taken by the researcher without reference to an individual's name or other identification should generally qualify for exempt status. Studies that use an audio recorder where a person's voice might be identified, or a computer record where an individual's name or other identifier is stated may qualify for an expedited review process if appropriate care is taken to guard teach individual's identity and the confidentiality of the individual data presented. Those studies using procedures or technologies where the identification of an individual subject is relatively easy, such as through the use of a video recording or electronic mail, must be afforded a more stringent review. This issue is especially important when considering the latest research being conducted over the Internet. A common belief a few years ago was that computers, electronic mail, and the information sent across Internet were inviolate. Successful and well publicized accounts of computer crime, the reading of e-mail by institutional superiors, and the snooping of Internet-transmitted data have completely shattered this myth. No one should assume that their network interactions are not being viewed by others unless specific steps have been taken to strongly encrypt the information being transmitted.
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
27
This is not to say that these types of data collection methodologies and technologies make the studies more risky for the human participants. The actual degree of risk might, in fact, be very small. Rather, it is the methodology and technology that could be used to ends not well understood by the human subject. IRBs often require the researcher, when using such methods and technologies, to discuss how he or she will maintain the privacy of the individual and the confidentiality of the data that is collected. If, for example, a video tape record is to be kept and shown by the researcher as part of a presentation from the study the researcher must make that intent clearly known to the subjects as part of the informed consent process. If research is to be conducted through e-mail communications over the Internet subjects must be warned that their communications may be intercepted and read by other parties. The increased level of review by the IRB insure that these considerations and protections are made. Summary Educational researchers have long enjoyed a unique place within the construct of human subject protection in Social science research. It had been assumed by many that if the research effort involved only normal educational practices and took place within a school it could be exempt from any kind of review regulation. Recent attention by legislators to the kinds of research that actually takes place in schools, and the difficulty of clearly defining normal educational practices, have focused increased attention on educational research. New technologies, especially the explosive increased use of video cameras and computer networks, are posing new and difficult issues for all researchers. Educational researchers and institutions alike must readdress these issues in a positive, proactive way. Rather than waiting for a revised federal regulation to describe how and when research ought to be accomplished the research community should take steps to address these issues and find workable alternatives and solutions. The mechanism of impartial review established into the IRBs provides a vehicle for such action. It is up to the IRBs, institutions, and researchers to make it happen. References Abramovitch, R., Freedman, J. L., Henry, K., & Van Brunschot, M. (1995). Children's capacity to agree to psychological research: Knowledge of risks and benefits and voluntariness. Ethics and Behavior, 5(1), 25-48. Adelman, H. S., Lusk, R., Alvarez, V., & Acosta, N. K. (1985). Competence of minors to understand, evaluate, and communicate about their psychoeducational problems. Professional Psychology: Research and Practice, 16(3), 426-434. Family Privacy Protection Act, 20 USC § 1232g.
Grotberg, E. H. (1979, April). The federal government and the protection of research subjects: Implications for educational research. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. (ERIC ED 170 368). Hecht, J. B. (2004, October). The IRB in classroom settings. Workshop presented at the Annual Meeting of the MidWestern Educational Research Association, Columbus, OH. Hecht, J. B. (2003, April). Conducting ethically responsible instructional technology research. Paper presented at the Annual Meeting of the American Educational Research Association, Chicago, IL. Hecht, J. B. (1996, June). IRB issues in K-12 educational research. Invited paper presented as part of the symposium "Hot spots in Social and Behavioral Science Research" at the Ethical and Regulatory Issues in Social and Behavioral Science and health services Research meeting, Boston, MA. Howe, K. R., & Dougherty, K. C. (1993). Ethics, Institutional Review Boards, and the changing face of educational research. Educational Researcher, 22(9), 16-21. Linowes, D. F. (1979, April). Privacy and the protection of human subjects in educational research. Paper presented at the Annual Meeting of the American Educational Research Association, San Francisco, CA. Lyons, C. M. (1983). Preparing protocols for Institutional Review Boards. Research in Rural Education, 2(1), 2730. Nannis, E. D. (1987, April). Children's understanding of their participation in psychological research. Paper presented at the Biennial Meeting of the Society for Research in Child Development, Baltimore, MD. New regulations affect research with students. (1995, October). Human Research Report, 10(10), 1-4. Ogloff, J. R., & Otto, R. K. (1991). Are research participants truly informed? Readability of informed consent forms used in research. Ethics and Behavior, 1(4), 239252. Protection of Pupil Rights Act, 20 USC § 1232h. Public Health Service Act, 46 CFR § 46. Sanford, L. (1993). Minor's assent, consent, or dissent to medical research. IRB: A Review of Human Subjects Research, 15(2), 1-7. Weinberger, J. (1981, April). Federal administrative law-- privacy, freedom of information and protection of human subjects--affecting educational research. Paper presented at the Annual Meeting of the American Educational Research Association, Los Angeles, CA.
28
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Metacognitive Knowledge Monitoring in Post-Secondary Education: The Consequences of Poor Knowledge Monitoring and a Program to Facilitate It Randy M. Isaacson Indiana University South Bend Abstract One of the most frustrating teaching dilemmas in post-secondary education is helping students who claim to have mastered the course content but are unable to demonstrate their understanding. These students are often convinced they have a command of the material and may even be able to persuade their instructor that their failure is due to the test. But instructors who carefully question these students realize that most of these students have not mastered the material. This paper will briefly report on research from the last decade on metacognitive knowledge monitoring, and then present a program that teaches both self-regulated learning (SRL) and metacognition. Over the past five years the author has researched the relationship between metacognitive knowledge monitoring (MKM) and classroom learning and has developed a program in his educational psychology class which compels students to regulate their own learning and develops metacognitive skills.
In 1995 a man walked into two Pittsburgh banks in broad daylight with no visible disguise and robbed them. That night the man was caught and was surprised that he had been recognized using surveillance cameras because he was sure that rubbing lemon juice on his face would render him invisible to videotape cameras. In their article "Unskilled and Unaware of It: How Difficulties in Recognizing One's Own Incompetence Leads to Inflated Self-Assessment", Kruger and Dunning (1999) point out that incompetence robs people of the ability to recognize their lack of knowledge or skills which further impedes their ability to succeed. In a variety of studies using different domains such as humor, logic, and grammar, they repeatedly found the least capable people were more likely to significantly overestimate their ability to successfully complete tasks. Their conclusion speaks volumes of the problems teachers regularly address with low achieving students in the classroom. Kruger and Dunning (1999, p. 1123) explain "When people are incompetent in the strategies they adopt to achieve success and satisfaction, they suffer a dual burden: Not only do they reach erroneous conclusions and make unfortunate choices, but their incompetence robs them of the ability to realize it." While we may laugh at the incompetence of the bank robber, we are often frustrated when our students cannot recognize that their strategies for learning are inappropriate and their mastery of their learning falls short of the requirement of the academic tasks. The bank robber and our students share a common deficiency--they have not reflected accurately on their strategies for success. As students progress through school the demands of the learning tasks gradually increase, with a significant increase in the learning curve as students enter higher education. Some students have been prepared for the challenges of post-secondary education by the requirement of their high school classes. Other students have been given direct instruction in
how to study for these new tasks. But many students are not prepared for learning assignments that go beyond the simple memorization of facts or algorithms, and often these students flounder in higher education. Many institutions of higher education are opening their doors to able but poorly prepared students and when these students are faced with academic failure they lack the cognitive or metacognitive skills to modify their learning strategies to overcome these challenges. These students may not be capable of reflecting on their deficiencies. Adding insult to injury, these failing students actually believe they know the material at the level required of the course and the demands of the task. It is not uncommon to hear these students deny responsibility for their failure: "I knew the material. I was pretty sure I was going to well, and I was positive I'd at least passed the test. It isn't my fault that I failed, it was the test." When university instructors hear these comments they often infer that these students are lying and making excuses--but many of these failing students honestly believed they understood the material. For these students the origin of their problems may be a lack of metacognitive awareness; they cannot assess their knowledge of the task at the level at which they will be evaluated. Motivating these students to learn may not be enough. Teaching these students learning skills may not be enough. The key to assisting these students may depend upon the student's ability to accurately assess their level of knowledge. The objective of this paper will be to explore the relationship of knowledge monitoring to Academic Success in college students and elaborate on a classroom application of self-regulated learning (SRL) and metacognitive knowledge monitoring (MKM) that the author is using in his undergraduate educational psychology course.
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
29
Research on Learning During the past twenty years extensive research (Pintrich, 1995) has explored an area of learning and motivation called self-regulated learning. This research has demonstrated that students' ability to control their behavior, motivation, and cognitive study strategies impacts student's academic success at all levels of schooling. A variety of programs have demonstrated that college students can acquire these learning-to-learn skills and that the mastery of these skills increases success in college (Hoffer, Yu, & Pintrich, 1998; Dembo & Jakubowski, 1999). These programs have proposed that teaching students these learning-to-learn skills will result in increased learning in higher education, but little attention has been paid to exploring the factors that trigger student's adjustments of their learning strategies. To use self-regulated learning effectively a student must know when they need to invoke a new learning strategy. If students believe that they understand the material they are studying, they are unlikely to engage in more extensive use of the strategies they are presently using, much less change their learning strategies. Clearly it is critical for students to know when they know and when they do not know, so they can engage or disengage in learning and adjust their learning strategies when these strategies do not result in mastery of the required material. Two areas of research have explored these questions, calibration and metacognitive knowledge monitoring. Research on calibration typically focuses on student's ability to estimate their test scores. Classroom research in college settings (Hacker et al., 2000; Isaacson & Fujita, 2001) has illustrated a striking difference between high performing students and low performing students in their ability to predict their test scores: successful students were able to accurately estimate their grades before taking a test, while students who were on the verge of failure were likely to over-estimate their future performance. After taking a test but before having it graded, successful students made adjustments in their expectations, further demonstrating their cognitive awareness, while failing students did not realize they had done significantly worse than they had predicted. Successful students have also been found to be able to correctly choose from test questions of varying difficulty while less successful students could not (Isaacson & Fujita, 2002). Research on metacognitive knowledge monitoring has demonstrated the relationship between academic success and accurate reflection of understanding at both a broad level and in studies which connect metacognitive awareness to selfregulated learning. The Knowledge Monitoring Assessment (KMA) developed by Tobias and Everson (2000, 2002) has shown that learners of all levels of ability and developmental stages including elementary schools, academically oriented high schools, vocational high schools, college freshman and upper level college students, are affected by their ability to monitor their learning. In studies with students of all ages and abilities, Tobias and Everson have found that students who are able to differentiate between when they know and when they do not know are more likely to excel than students who are not able to distinguish their level of comprehension. These studies by Tobias and Everson have focused on the
correlation between knowledge monitoring and student's academic performance, but there is very little research on whether metacognitive knowledge monitoring changes over time and whether MKM can be taught. We have begun to explore whether metacognitive knowledge monitoring improves over time (Isaacson and Fujita, 2003) and have found evidence that within a classroom environment in which SRL and MKM are encouraged, student's test scores improve, their calibration scores improve, and they modify their SRL (as measured by the Motivated Strategies for Learning Questionnaire [MSLQ], Pintrich, et al.1991). If self-regulated learning and knowledge monitoring skills improve over time in classroom situations and these skills are critical to academic success, what instructional approaches can be used to facilitate these improvements? Metacognition has also been shown to impact self-regulated learning. In an experimental study (Thiede, Anderson, and Therriault, 2003) college students who were instructed to reflect on their comprehension of reading text were found to regulate their study behavior more effectively and perform better on subsequent learning tasks than students who were not encouraged to monitor their learning. It was hypothesized that when students are encouraged to focus on the discrepancy between their current state of learning and their desired learning state they are more likely to use self-regulated learning to remove this discrepancy to achieve their goals. The authors proposed that metacognitive monitoring played a critical role in the connection between academic performance and self-regulated learning in a laboratory environment where it is encouraged but, "...it seems likely that left to their own devices people will not accurately monitor comprehension." (p. 71). If students who are instructed to reflect on their understanding in a laboratory environment are more effective in regulating their learning, I began to wonder what I could do in a college course to encourage my students to be more reflective, and whether this metacognitive monitoring could improve their academic performance. Like many university professors I have experienced the frustration of trying to assist students who claim that they know the material in my course, but they are unable to demonstrate their understanding. I have come to realize that it is not unusual for these students to honestly believe they know the material. After exploring their understanding it is clear that the knowledge of these students is superficial or that they are using the wrong criteria to judge their mastery of the requirements of the course. Over the past decade I have gradually modified my preservice educational psychology class to address many of the challenges faced by these students and their instructors. I believe the heart of the problem for these students revolves around three issues: · Students' inability to monitor their understanding of the required material at the required level. · Students' inability to identify appropriate study/learning strategies for the required task. · Students' tendency to blame failure on external attributions such as the teacher, the test, or the circumstances. The primary focus of this paper is to share what I have learned about the relationship of metacognitive knowledge monitoring to self-regulated learning and elaborate on class-
30
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
room practices that I have developed that reveal and teach this relationship to students. I will explain how I have created a classroom environment which: · Directs students to reflect on their understanding. · Teaches students alternative strategies to master the re- quired tasks. · Encourages students to take responsibility for their suc- cess and self-regulate their learning. Teaching Metacognition: A Case Study I teach two sections of approximately fifty undergraduate teacher education students each semester. The classes are made up of both traditional and non-traditional students who are admitted under a relatively open admissions policy. I use Anita Woolfolk's Educational Psychology (9th ed.) and would characterize the content of the class as being fairly traditional but many of the procedures I use to teach the course are fairly unconventional. I designed an extensive web page (http:// mypage.iusb.edu/~edp250/) that supports the delivery of a variety of course materials. My web page also serves as one method of assisting students to assess their mastery of the material. Each chapter has a variety of materials to support student learning including: · A hierarchical outline of the concepts. · Overheads. · Graphic organizers and cognitive maps. · Practice tests. There is also an additional web-based course manager that is specifically designed for each campus of Indiana University called Oncourse (http://oncourse.iu.edu/ ) that delivers Metacognitive Practice Tests approximately twenty-four hours before each test. This course manager also collects students' biweekly reflection on the application of the content of the class and what they have learned about their own learning. The course structure includes: · Two 75 minute lectures per week. · One 75 minute peer mentor discussion class per week where students are required to complete a 10-question quiz and a Peer Mentor Study Journal. · One test each Friday. Students' final grades are determined by weekly tests, discussion group quizzes, points they earn on the biweekly reflection, a self-assessment paper of their SRL, the points they receive for completing a Peer Mentor Study Journal, and a comprehensive final exam. The course is designed to encourage pre-service teachers to study their own learning, specifically their study skills, self-regulated learning, and metacognitive knowledge monitoring. Accepting a Performance Orientation and Adjusting to It As much as I would like to think most of my students have an intrinsic motivation to learn educational psychology, thirty years of teaching have demonstrated to me that some students, often the non-traditional students, are intrinsically
motivated. Most students though are predominantly performance oriented (Harackiewicz, Barron, and Elliot, 1998). Many students are receptive to the idea that educational psychology can help them become a better teacher, but the primary motivation of these future teachers is to get a good grade or avoid getting a bad grade. I have structured my class around the premise that if I give them choices that will help them to achieve their grade-goals, I can lead them to the water (SRL) and maybe even entice them to drink it. The research on intrinsic motivation and self-determination (Deci, 1980) indicates that allowing choices increases the possibility of a positive motivational orientation. Choice can also be used to encourage students to adopt new learning strategies. Since the students who experience the most difficulty in higher education are often traditional students who are young adults, I have found it valuable to give them choices and opportunities to improve over the course of the semester. Based on the work of Clifford (1991) I have incorporated a number of variations of the theory of academic risk taking into the course curriculum. The basic thesis of academic risk taking is that students are more willing to take academic risks such as changing their study strategies when they are allowed to experiment in evaluative situations in which a single negative outcome does not necessarily result in failure in the course. Frequent testing allows me to create options in which students can substitute certain academic tasks for poor test scores. These options include substituting a good quiz score for a poor test score or earning points for completing a semester-long journal and substituting those points for a poor test score. To encourage students to change their study strategies I require that they reflect on both their understanding and the relationship between how they study and their level of learning. The course is designed to elicit and reward both accurate self-reflection (MKM) and the regulation of their own learning (SRL) to fit the particular academic demands of the task. Every facet of the course "holds up a mirror" for students to assess the impact of their study strategies and metacognition on their learning as measured by a test grades. In this way, the facets of the course supports student efforts to improve their SRL and MKM over the course of the semester. Since students with a performance orientation are motivated primarily by grades, I attempt to create many opportunities for students to improve their grade by regulating their learning (SRL) over the course of the semester. To encourage them to improve their self-awareness of their understanding (MKM) I have created opportunities to reflect on their understanding and their own learning. To entice students to change their study strategies I have developed a testing format that rewards higher level thinking, gives feedback on self-regulated learning, and has a payoff for knowing-whenyou-know. To encourage persistence, I have created opportunities for students to replace early-semester failures with improved grades which are the result of improved self-regulation and self-monitoring. Testing Higher Level Thinking: Variable Difficulty-- Variable Weight Objective Tests Traditional college students are typically resistant to changing the study strategies that resulted in their prior suc-
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
31
cess in high school. Perhaps they are thinking: "These strategies were successful before, why change them now?" To encourage students to choose to change their strategies it is important that they are given evaluative tasks that demand higher level thinking skills. Unfortunately, most evaluative tasks which require higher level thinking tend to be openended written exercises which take time to grade. Students often see this as being quite subjective. This causes two potential problems when the purpose of the task is to assist students in recognizing the effectiveness, or ineffectiveness, of their study skills. First, students may attribute their lack of success on these tasks to the grading of the teacher: "I really knew the material well, but the teacher doesn't like the way I write." These external attributions undermine the feedback the student receives. Second, the time delay involved in getting feedback to large classes interferes with students reflecting on their learning. To encourage students to choose to change how they study they need to be given a task which requires higher level thinking skills but also gives the student immediate feedback (minutes not days) in an objective manner. In an effort to motivate students to examine their selfregulated learning I have developed a number of testing formats that: · Encourage students to study for higher level thinking skills. · Reward students for demonstrating higher level thinking skills. · Give students immediate objective feedback. · Present feedback which allows students to examine the effectiveness of various study strategies such as rehearsal, elaboration, and organization. · Encourage students to take academic risks. · Allows students to reduce the negative impact of not knowing as long as they "know when they don't know." The basic premise of the Variable Difficulty--Variable Weight testing format is to present three levels of test questions that theoretically each require a different level of learning and allow students to choose which questions they wish to take. The original Variable Difficulty--Variable Weight test format has been modified a number of times to further encourage students to reflect on their own metacognitive accuracy. The three levels of objective test questions listed below reflect the latest iteration of the variable difficulty--variable weight test. Many instructors give tests with questions of different value which may encourage students to study for different questions with different study strategies. The unique application of the variable difficulty--variable weight test is that students are allowed to choose questions in a manner that maximizes the impact of metacognitive knowledge monitoring. Over the years I have had a number of alterations of the variable difficulty--variable weight test format to encourage students to reflect on how confident they are in their answer. I have extensive anecdotal evidence from students that being required to think about their level of confidence has actually improved their MKM: empirical evidence is being collected
to verify this change. Three variable difficulty--variable weight test formats have been used in the past five years including a 26/36 test format, an original front/back format and a revised front/back format. The 26/36 test format presents students with 16 Level I (1 point), 16 Level II (2 points), and 4 Level III test questions (3 points) from which they choose 26 questions. In this test format students are encouraged to take higher level test questions but only if they are sure they would get them correct. In the 26/36 test format students could choose all the easiest test questions (i.e., 16 Level I and 10 Level II), but the maximum test score they would achieve would be a C+, even if they answered all 26 questions correctly. To receive the maximum point and earn an A, students had to choose to answer the most difficult questions and answer them correctly. The 26/36 test format was modified for research purposes because we were not sure if students were choosing the right test questions. By that, I mean we were unsure if the 26 questions they were choosing to have graded were anymore likely to be correct than the 10 questions they did not choose. Since we did not know whether they knew the correct answer to the 10 questions they did not answer, we could not evaluate the accuracy of their metacognitive knowledge monitoring. The original front/back test format gave students 1, 2, and 3 points for Level I, II, and III questions "on the front" of their test (consistent with the 26/36 format), but 2, 3, and 4 points "on the back" of their test with a penalty for guessing of 1, 2, and 3 points for wrong answers. This test format gave a strong incentive for "knowing when you know" but it also rewarded and punished students for risk-taking. This added a confounding variable to the research design and created extreme apprehension for some students. I used this test format for two semesters with some students, typically students with good MKM, loving the format and some students absolutely hating the format. While this format was probably too punitive in relation to metacognition, student test results clearly demonstrated the lack of MKM of some students: Some students were so deficient at MKM that they received test scores below zero. The modified front/back format, which is now being used, presents students with: · 15 Level I questions from which they are to choose 10 for the front of their answer sheet for 2 points each, and 5 for the back of their answer sheet for 1 point each. · 15 Level II questions from which they are to choose 10 for the front of their answer sheet for 5 points each, and 5 for the back of their answer sheet for 1 point each. · 5 Level III questions from which they are to choose 3 for the front of their answer sheet for 6 points each and 2 for the back of their answer sheet worth 1 point each. This modified front/back test format is effective for a number of reasons. First, it gives students an incentive for improving the MKM because choosing the right answers for "the front" can improve a test score dramatically. Second, it gives the instructors consisting of the professor and peer mentors a way to demonstrate to students that they lack MKM. An instructor can explain to a student: "You got 4 Level II
32
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Table 1 Levels of questions on the variable difficulty--variable weight test.
Level I Questions
Level I Questions are basic objective test questions that require only knowledge and comprehension. Since these questions are very basic and can be answered correctly using simple rehearsal strategies, they are only worth 2 points (or 1 point) for each correct answer.
Level II Questions
Level II Questions are also objective questions but they require the application of psychological concepts to classroom settings. The typical question describes a classroom problem and asks the students to solve the problem using concept from class. These questions are typically more difficult and require elaboration. They are worth 5 points (or 1 point) for each correct answer.
Level III Questions
Level III Questions are also objective questions but they are much more difficult and are designed to measure the student's understanding of the structure of the material being learned. There are a number of formats for Level III questions but the most common are analogy questions and hierarchy questions. Students are encouraged to study for these questions using cognitive organization study skills. These questions are worth 6 points (or 1 point) for each correct answer.
questions correct on the back. If you had put them on the front, you would have improved your score by 16 points." Third, the revision reduces the tendency of students to blame their failure on the test format. For example, students can miss 12 of the 35 test question and still earn a B+ if they are able to pick the right 12 questions to put on the back of their answer sheet This is very difficult to do, even for the students that have good MKM. Both the 26/36 test format and the front/back test formats encourage students to think carefully about which questions they choose to answer. The test formats also encourage students to learn to study for the questions that are worth the most points. This test format encourages a number of important skills and dispositions that are consistent with self-regulated learning in students: · Students choose which questions they answer. This is a key in developing self-regulated learners. · Choosing test questions requires that students "know when they know." I have been amazed at how important metacognitive knowledge monitoring is to success on these tests and how deficient many students are in this area. Some students may harbor the "multiple guess" beliefs about multiple choice and true-false tests but the front-back test format challenges students to reflect on whether they are just guessing, making educated guesses, or are sure of their test answers. · Students are given immediate feedback not only on how many points they received, but also on which type of questions they get correct and incorrect. This eventually leads them to examine how they study and how they might change to improve their score since test questions at 3 levels encourage students to explore how they study: Level I (rehearsal), Level II (elaboration), and Level III (organization). Since many students believe all studying is the same, it is important to demonstrate that different study approaches are effective for different types of test questions. · Students are asked to keep track of how they devote their study time each week and reflect on the effectiveness of their plan for learning. Those who focus solely on rehearsal can see that effective rehearsal results in im-
proved Level I scores but may not necessarily improve Level II or Level III test scores. · Students realize that "learning" involves more than just memorization. This may be the most important lesson students learn from the variability difficulty--variable weight test format, but for many students this is the most difficult lesson to learn. Encouraging Metacognitive Thinking: Pre-Post Questionnaires As part of the data collection for the research on SRL and MKM, I administer a questionnaire immediately before students take their weekly tests. Before students are given the test they are asked to identify: · How many hours they have studied. · What percentage of their time they devoted to rehearsal, elaboration, and organization. · How many points they believe they will achieve on the test (pre-diction). · Their satisfaction and pride goals. · How confident they are of achieving their satisfaction goals. After they have taken the test, but before it is graded, they are asked how many points they believe they will achieve on the test (post-diction) and how confident they are of achieving their satisfaction goal. To highlight the significance of metacognition, and give the students an incentive to be thoughtful about their metacognitive knowledge monitoring, students are given bonus points for accurately predicting and post-dicting their test scores. After students complete this part of the questionnaire they take their test and questionnaire to another room, have their test graded, and are asked to complete a number of questions on their reaction to their test. These post-grade questions are part of the research study but are also designed to encourage the students to reflect on their success or failure and their attributions for their results. After identifying their test results as a success or failure they are asked a series of attributional questions which help them to examine whether their test score was
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
33
the result of their academic ability, their study efforts, their study strategies, their test taking abilities, or the level of difficulty of the test. This information is not only valuable for research purposes--it also raises attributional questions that can stimulate reflection in students. I believe that to learn metacognitive knowledge monitoring and to contemplate changing their study strategies, students need to consider what factors are having an impact on their learning. Asking students questions about the attributions for their success or failure is the first step in this process. Teaching SRL beyond declarative knowledge: Reflection and application to student learning I believe the most powerful way to teach students to be self-regulated learners is to teach them the details of SRL, demonstrate why SRL is effective, and give them an opportunity to practice the application of SRL on relevant, meaningful, graded tasks. In the first third of the semester I teach my educational psychology students fairly extensive information on the declarative knowledge of self-regulated learning. I continue to emphasize the use of SRL throughout the semester by integrating the concepts into chapters on individual differences, behavioral learning theory, cognitive models of learning, motivation, and other topics. In addition to making the declarative knowledge relevant and the organizational structure apparent, I also give students many opportunities to integrate self-regulated learning into their procedural knowledge. After studying and taking a test on the declarative knowledge of SRL, students are given an assignment which requires them to write an extensive self-assessment paper of their own study skills in college. This assignment requires that they reflect on their own strengths and weaknesses, including data from the Learning and Study Strategies Inventory (LASSI, Weinstein, 1986) and the integration of the information they have collected from a structured Peer Mentor Study Journal that is part of the peer mentor discussion groups. My recent work on calibration (Isaacson and Fujita, 2001), the research by Tobias and Everson (2000), and extensive anecdotal evidence has led me to explore the significance of metacognitive awareness in self-regulated learning and to encourage my students to be more reflective. A quote by Winne and Perry (2000, p. 540) describes my present position, "Metacognitive monitoring is the gateway to self-regulating one's learning because without the cognitive evaluation it creates, there is no standard against which to enact regulation." In the past three years I have attempted to encourage my students to be more reflective about their metacognitive awareness and their study skills. For more than a decade I have used discussion groups led by peer mentors in my educational psychology class. Many of my students had difficulty with higher level thinking questions and when I began teaching students self-regulated learning I found that the peer mentors (undergraduate students who had done well in the class in previous semesters) were the best role models for learning these skills. When it became clear that metacognitive knowledge monitoring was an important
part of the change process, I began to integrate a number of curriculum support materials into the program. Peer Mentor Study Journals As mentioned earlier, I have found that traditional students are typically driven by evaluation and if there is no "payoff" for being reflective it is unlikely that traditional students will think about their thinking. To facilitate metacognitive awareness in my students I have developed a very structured study journal that requires students to keep track of their study time and place (i.e., SR of Behavior), how they improve their motivation and reduce their test anxiety (i.e., SR of Motivation and Affect), the types of study strategies they use (i.e., SR of Cognitive Strategies), and their reactions to their weekly quiz and tests. In addition I have written a structured set of questions to guide their reflection each week including questions on study strategies, knowledge monitoring before tests, choosing test questions, attributions, and other factors. These guided journal entries are written to reflect the weekly concepts the students are studying in class and to highlight the integration of SRL into learning theory and motivation. Students earn points each week on their journal entries to increase their engagement in the task and the total points earned for the journal can be substituted for one test grade. Peer Mentor Group Quizzes Most students volunteer to participate in the peer mentor discussion group primarily because students can earn points which they can substitute for test grades. Each week the peer mentors administer a short quiz to the students at the beginning of the discussion group. The quizzes also use a variable difficulty--variable weight with students identifying 6 of the 10 questions (6 Level I, 3 Level II, and 1 Level III) in which they are more confident. Each quiz is worth 20 points and the five best quiz scores can be substituted for another test score. In the discussion groups the students receive immediate feedback on their quizzes and the format of the quiz elicits metacognitive feedback. Students answer all 10 questions but choose 6 to have scored "on-the-front" (worth 2, 3, and 5 points) with the other 4 questions worth only 1 point. The peer mentors discuss the process of choosing quiz questions emphasizing metacognitive knowledge monitoring. Confidence Rating: Absolutely Sure, Fairly Sure, and Just Guessing A number of years ago one of the peer mentors came upon a strategy to help students choose test questions that has become an integral part of our program. Students were having trouble identifying which questions they should put "on-the-front" of their quizzes and tests. The students needed a way to categorize test questions so that they could keep track of how confident they were of their answers. The peer mentor suggested to the students that, as they were taking the test, they label each question as: Absolutely Sure of their Answer; Fairly Sure of their Answer; or, Unsure or Just Guessing at their Answer. The students in her group found this
34
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
strategy to be very helpful in categorizing their confidence about their answers. We have since adapted this confidence assessment as part of the data collection in our study and have strong anecdotal evidence that asking this question ("Are you absolutely sure, fairly sure, or just guessing at your answer for this question?) has an impact on students' metacognitive knowledge monitoring. From the comments students have made on their reflection, it seems that many students had never considered how confident they were about the answers they were putting on their tests. By the end of the semester many students have shared with me that not only do they ask themselves this question on all their tests (even tests in other classes), they also have adopted this question while they are learning. While only anecdotal, this indicates to me that encouraging students to reflect on their level of mastery of knowledge and skills should be an important part of the learning cycle for all students. Bi-Weekly Reflection In response to NCATE and the INTASC standards, the School of Education at Indiana University South Bend has placed an increasing emphasis on encouraging our teacher education candidates to become reflective practitioners. Recently I have added a course assignment which requires on-line reflections of how students will apply what they are learning in my educational psychology class to their future classroom teaching. I have also added bi-weekly reflections on their own learning with a particular emphasis on self-regulated learning and metacognitive knowledge monitoring. I believe that by consistently raising questions about self-regulation and metacognition I have heightened my students' awareness of these learning issues. The anecdotal responses I have received from students leads me to believe that their first reaction is annoyance, but by the end of the semester most students believe MKM and SRL has improved their learning and they plan on integrating these skills into their own teaching. I have detailed student reflections indicating that many students believe that the course resources such as the test format, journals, confidence ratings, and so on, as well as the bi-weekly reflections have had a positive impact on their MKM, their SRL, and their learning. This qualitative research data is IRB approved and ready for analysis if a colleague were interested in working with me to further explore the SRL, MKM, learning relationship. Metacognitive Practice Tests The research by Tobias and Everson (2000) clearly demonstrates that metacognitive knowledge monitoring is closely tied to academic achievement. Their research shows that there is a significant correlation between metacognitive awareness and achievement for students of all ages at every ability level. I have modified the test format, the Peer Mentor Study Journal, the quizzes, and the bi-weekly reflections to try to emphasize to students the importance of reflecting on the degree to which they have mastered the material they are studying. I have also incorporated the confidence ratings (Absolutely sure; Fairly sure; or Just Guessing) of their quiz answer. But many students need assistance with tasks that will give them feed-
back outside of class on their level of master and their confidence in relation to their MKM. I have recently introduced a new format for the on-line metacognitive practice test that asks students to make this judgement for every practice test question they take every week. Using a web-based course manager, I have delivered a variable difficulty practice test approximately twenty-four hours before their actual test. The students are allowed to take the practice test up to three times using random question selection generated by the software before taking their test. I hope to examine the uses of this practice test to determine which students choose to take advantage of this metacognitive assistant, and eventually to modify the software to allow students to control the level of difficulty of the questions. This software may help us to explore how students decide that they understand the material well enough to disengage from learning before a test. What have we learned, Where should we go? While the empirical evidence is not abundant, the anecdotal evidence from my students is clear: students do change their metacognition from the beginning of a semester to the end of a semester if there is the support and incentive within the course for them to make SRL and MKM changes. The study by Isaacson and Fujita (2003) demonstrates a relationship between improved test scores, calibration, and expected changes in self-regulation. The student reflections attributes those changes to the resources available in the class and the demands created by the test format of the class. Further research needs to explore this relationship from a number of different perspectives. The study by Isaacson and Fujita (2003) demonstrates a relationship between metacognition, self-regulation, and learning. The cause-effect of this relationship is critical to pedagogue but the cause is not clear. Kruger and Dunning (1999) suggest that improvements in expertise make improvements in metacognition possible: they contend that the cause is improved learning and the effect is improved metacognition. But student reflections from my class indicate that when students begin to think more about whether they truly understand the content they are studying they are more likely to change their study strategies which leads to increased learning. I have extensive anecdotal evidence (qualitative data) which supports this relationship and I would like to invite interested colleagues to join me in analyzing this data. I have also increased the power of the data I am collecting to allow us to do finer-grain analysis of the longitudinal data on students. A potential research question might be `Which comes first: changes in metacognition, changes in self-regulation, or changes in test scores? The question of the relationship of metacognition, selfregulation, and learning in post-secondary students raises a number of other questions that are pertinent to instructors of adolescents and adults: · What specific classroom practices assist students in assessing their metacognition and monitoring skills? · What resources can instructors make available to students to help them improve their metacognition?
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
35
· Are some students more likely to benefit from these practices and resources, and what can instructors do to create a classroom environment that makes students aware of the need to change? One of the most challenging aspects of teaching metacognitive knowledge monitoring and self-regulated learning is that the students who are most deficient in MKM and SRL are the least likely to recognize their inadequacy. Students with poor study skills are unaware of alternative approaches to learning and less likely to realize their study strategies are the cause of their poor results. Even more problematic is that they do not even recognize that they are not learning at the necessary level of mastery which leads them to externalize the blame for their academic failure. Over the past decade I have implemented many strategies and course modifications to help students realize their SRL and MKM deficiencies. Each of the strategies mentioned in this paper have been adapted to increase self-awareness of learning in post-secondary students, and I have anecdotal and/or empirical evidence for the efficacy of all of them. But there are still some students whom I have not been able to reach--students for whom a gentle nudge and a "Maybe you don't know this as well as you think." is not enough. Recently I was introduced to a new technology that I am presently testing in my class to see if it can be used to reach even the students who are most adamant and resistant to considering that they do not know when they don't know. An audience response system (The Hyper-Interactive Teaching Technology or H-ITT) with the appropriate software and hardware allows me to include all students in classroom discussions where each student has an opportunity to anonymously respond to classroom questions and then discuss them with classmates before responding to the questions a second time. By presenting students with challenging questions and pairing them up with study partners in class, I give them the opportunity to reflect on their own understanding in an environment which creates disequilibrium but lessens the embarrassment of the teacher telling the student they are wrong in front of the entire class. I have just begun using the H-ITT technology and look forward to adapting it to my class to improve the metacognitive knowledge monitoring of my students. The possible application of pedagogical adaptations using technology creates new methods of assisting students to improve their metacognition, their self-regulation, and their learning. References Clifford, M. (1991). Risk taking: Theoretical, empirical, and educational considerations. Educational Psychologist, 26(3), 263-297. Deci, E. (1980). The psychology of self-determination.. Lexington, MA: Lexington Books. Dembo, M., & Jakubowski, T. (1999, April). An evaluation of a learning-to-learn course. Presented at the Annual
meeting of the American Educational Research Association, Montreal. Hacker, D., Bol, L., Horgan, D., & Rakow, E. (2000). Test prediction and performance in a classroom context. Journal of Educational Psychology, 92(4), 160-170. Harckiewicz, J. M., Barron, K. E., & Elliot, A. J. (1998). Rethinking achievement goals: When are they adaptive for college students and why? Educational Psychologist, 33(1), 1-21. Hofer, B., Yu, S., & Pintrich, P. (1998). Teaching college students to be self-regulated learners. In D. Schunk & B. Zimmerman (Eds.), Self-regulated learning: From teaching to self-reflective practice (pp. 57-85). New York: The Guilford Press. Isaacson, R., & Fujita, F. (2001, April). The effects of goals, expectations, and self-efficacy on self-regulation and performance in college students. Presented at the Annual conference of the American Educational Research Association, Seattle, WA. Isaacson, R., & Fujita, F. (2002, April). Knowing when you know and when you don't know: Metacognitive knowledge monitoring and task selection. Presented at the annual meeting of the Mid-western Educational Research Association, Columbus, OH. Kruger, J., & Dunning. (1999). Unskilled and unaware of it: How difficulties in recognizing one's own incompetence lead to inflated self-assessment. Journal of Personality and Social Psychology, 77(6), 1121-1134. Pintrich, P. (1995). Understanding self-regulated learning. San Francisco: Jossey-Bass. Pintrich, P., Smith, D., Garcia, T, & McKeachie. (1991). A manual for the us of the motivated strategies for learning questionnaire (MSLQ). Technical report #91-B-004. University of Michigan. Thiede, K., Anderson, M., & Therriault, D. (2003). Accuracy of metacognitive monitoring affects learning of texts. Journal of Educational Psychology, 95(1), 66-73. Tobias, S., & Everson, H. (2000). Assessing metacognitive knowledge monitoring. In G. Schraw & J. Impara (Eds.), Issues in measurement of metacognition (pp. 147-222). Lincoln, NE: Buros Institute of Mental Measurement. Tobias, S., & Everson, H. (2002). Knowing what you know and what you don't: Further research on metacognitive knowledge monitoring. College Board Report No. 20023. College Board, New York. Weinstein, C., & Mayer, R. (1986). The teaching of learning strategies. In M. Wittrock (Ed.), Handbook of research on teaching. (pp. 315-327). New York: MacMillan. Winne, P., & N. E. Perry. (2000). Measuring self-regulated learning. In M. Boekaerts, P. R. Pintrich, & M. Zeidner (Eds.), Handbook of self-regulation (pp. 531-566). New York: Academic Press.
36
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Twenty Techniques for Teaching with Case Studies
Mary R. Sudzina University of Dayton
Abstract
Problem-based learning and teaching with case studies are instructional approaches that are increasingly being applied in a variety of disciplines, such as business, law, medicine, and education. Instructors who have experienced traditional, teacher-centered instruction are often looking for ways to successfully integrate case studies, a constructivist, student-centered approach to teaching and learning, into their courses. This paper summarizes a workshop presented at the 2004 MWERA Annual Meeting. That workshop featured a proven 5-step process for case analysis, 20 techniques for facilitating case discussions, and a video of an actual case discussion in progress in which those techniques were demonstrated and modeled.
Introduction The number of disciplines that are including problembased learning (PBL) and case studies in their professional preparation is becoming widespread. Additionally, case studies are now being included on standardized tests to assess competencies for professional certification, accreditation, or licensure, such as the Praxis II - Principles of Learning and Teaching (PLT) exam in teacher education. By definition, PBL presents a rich problem that affords free inquiry by students. The learning is student-centered, active, collaborative, and often incorporates case studies. Case studies are "slices of life" that present real life dilemmas with multilayered issues and perspectives. Cases, which may be 1 page or 100 pages, are applicable to a variety of content areas and offer no one "right" answer. However, effective responses are practical, rooted in the literature or research, and take into consideration all aspects of the problem. Hence, responses are synergetic in nature. Traditional instruction, by contrast, is teacher-centered and teacher-directed. The instructor selects and delivers the content, usually in a lecture format, and evaluates students on their ability to reproduce that content on an exam. There usually is one particular answer that is required. Instructional leaders, trained in the traditional lecture method, often struggle with how to incorporate PBL and cases into their curricula, facilitate case discussions, and assess student outcomes with cases. This paper summarizes an invited interactive workshop presented at the MWERA annual meeting that introduced participants to a proven 5-step process for case analysis, and outlined 20 techniques to enhance facilitating case discussions. Additionally, attendees were shown a videotape of an actual case discussion in progress in which those techniques were demonstrated and modeled by the author. Background Initial use of PBL started in the medical community in the late 1960s to interject active and collaborative learning, relevance, and student motivation into the medical curricula
for students disenchanted with their passive role in memorizing the required material. Although PBL was not initially a popular idea, it has since been adopted by medical schools around the world and has spread beyond the confines of medicine to learning in many different disciplines and professions (see, Evensen & Hmelo, 2000). Similarly, the case method, or teaching with case studies, had its contemporary roots in the Harvard Business School in the 1980s, where Christensen (1987) taught and wrote eloquently about interactive teaching and "the case method:" Reading about problems or memorizing principles does little to prepare the practitioner--architect, doctor, or manger--to apply concepts and knowledge to the complexity of real-life problems. Discussion teaching achieves these objectives better than alternative pedagogies. It puts the students in an active learning mode, challenges them to accept substantial responsibility for their own education, and gives them substantial appreciation of, and experience with, the application of knowledge to practice (p. 3). In 1991, Kay Merseth, also at Harvard, published The Case for Cases in Teacher Education and made a compelling argument for including cases in teacher preparation. A score of casebooks, case commissions, national case competitions, and books on how to apply cases and PBL quickly followed. Case-based teaching was seen to be a superior method to lecture in promoting students' long-term conceptual understandings and retention of information through the applications of theory to practice (Sudzina, 1997). Additionally, students exposed to the case study method appeared to develop superior skills in analytical reasoning and written and oral expressional compared to students exposed only to traditional instruction. An added benefit, from an instructor's perspective, was thinking about teaching creatively as a facilitator of knowledge rather than simply as a provider of information. Teaching with cases also allows the instructor to probe or correct students' misunderstandings or gaps in information that reveal themselves in case analysis and discussion (see, Sudzina, 2000).
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
37
Teaching with Case Studies Getting Started There are many things to consider when beginning teaching with case studies. These include course content to be addressed, classroom setting and time constraints, student considerations, case study sources, case selection, teaching strategies and assignments, and assessment strategies. For a fuller explanation of each of these areas and case resources, see "Guidelines for Teaching with Cases" (Sudzina, 1999, pp. 8-19). A Five-Step Process For Case Analysis Once the decision has been made on the content to be addressed, the case to be used, and the resulting assignment, it is critical to introduce students to a conceptual framework for case analysis. Such a framework serves as a template to guide students in their problem solving and to prepare them for case study discussions. One framework that has been found to be effective across content areas, and at the undergraduate, graduate, and inservice levels of instruction, is the five-step process of McNergney, Herbert, and Ford (see Table 1). Table 1 A five-step process for case analysis, based on the work of John Dewey 1. Identifying the issues and facts in a case 2. Identifying the differing perspectives and values 3. Identifying professional knowledge 4. Formulating and prioritizing actions, both short-term and long-term 5. Considering the consequences of such, actions, both positively and negatively (McNergney, Herbert, & Ford, 1994) This process for case analysis, based on the work of John Dewey, can also be used to assess the thoroughness and effectiveness of students' case analyses and discussions. Twenty Techniques for Teaching with Case Studies Teaching Strategies The heart and soul of case analysis is the case discussion. There are many teaching techniques that have been found to be effective when leading case discussions (see Table 2). While these pedagogical strategies are not new, when used in concert with case content and analysis they bring a high level of student interactivity and engagement to case discussions. These techniques can be used in any teaching situation in which instructor-student interactivity is desired. While many of these techniques need little explanation, an example of how they can be used in case discussions may be helpful. I offer that explanation in Table 3.
Table 2 Twenty techniques for facilitating case discussions o Setting the agenda o Interviewing o Follow up o Role playing o Probing o Elaboration o Asking for information o Devil's Advocate inquiry o Open-ended questioning o Active listening o Alternative perspective o Drawing out students o Clarification o Reinforcing appropriate use of resources o Polling o Redirecting o Compare and contrast o Positive feedback o Summarizing o Closure (Sudzina, 2004) Case Study Video As a follow-up to the introduction to case analysis and the 20 strategies for teaching with case studies, participants viewed the videotape Teaching and Learning with Case Studies: Facilitating Case Discussions and Engaging Students (Sudzina, 2004). This 29-minute professional development video, written and produced by the author, introduces participants to a clip from the video case study, "What to Do About Raymond" from the video series, Becoming a Star Teacher (Rowley & Hart, 1995). The case is about a seventh grade student who falls asleep in class and fails to complete his assignments. The case highlights issues of communication, behavior management, engaging students in relevant material, extra credit, parental involvement, and suspension. The class featured in the video is a sophomore level educational psychology class. All have given their permission to appear in the video. The students have been introduced to the case study method and have seen the video case. They have been assigned to teams of 4 students each and have completed a written analysis of the case to bring to class in preparation for the case discussion. The discussion that takes place is spontaneous and unrehearsed. Briefly, the video opens with the author giving the class a short synopsis of the case and setting the agenda for the case discussion. Excerpts from the "Raymond" case are shown. The author then facilitates the case discussion using a 5-step process based on the work of John Dewey. As she draws out the issues, perspectives, knowledge, actions, and consequences of the case, notations appear in the left hand corner of the video screen identifying the strategies used in the case dis-
38
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
Table 3 Examples of how techniques can be used in case discussions
Setting the agenda
outlining the plan and expectations for the case discussion
Interviewing
asking a student what he/she personally thinks about a certain issue and why
Follow-up
asking for more clarification; i.e., the consequences of a proposed action
Role-play
assigning, or having students volunteer, to assume the values, perspectives, motivations, and emotions of one of the characters in the case; to become that character
Probing
asking for more information or explanation
Elaboration
having students give a fuller explanation of the facts or issues
Asking for information
uncovering overlooked facts, details, or research; i.e., "What's missing from this discussion of the issues?"
Devil's Advocate inquiry
challenging students to disagree with and/or question a course of action that seems plausible, but may not be appropriate in this case situation
Open-ended questioning asking students' opinions in order to jump-start the discussion
Active listening
rephrasing a student response; i.e., "If I heard you correctly, you said..."
Alternative perspective
asking for the opposite or different perspective than the one just given
Drawing out students
calling on students who tend not to participate and asking them to comment. This is particularly effective for moving the discussion around the room.
Clarification
asking for additional information to make clear a student's point of view
Reinforcing appropriate use of resources
giving positive feedback for the use of research or professional sources
Polling
asking who agrees or diasagrees with the last comment, through a show of hands
Redirecting
calling on a student in another part of the room to comment on the laststatement; focusing attention in another part of the room
Compare and contrast
focusing attention on how a this situation/solution is alike and different from others students have encountered
Positive feedback
reinforcing student responses through specific, supportive feedback
Summarizing
asking students to capture, in their own words, what major points were gleaned from the case discussion
Closure
bringing the conversation full-circle by reiterating the goals of the discussion (i.e., setting the agenda) and reinforcing the lessons learned, particularly with the applications of theory and research to practice
cussion. Strategies such as probing, redirecting, role-playing, and Devil's Advocate questioning are illustrated. The video concludes with a short summary of the benefits of using case studies that include improving student competence, problem-solving abilities, and professionalism. Resources and references are listed at the end of the tape. Summary and Conclusions Teaching with case studies is more than the sum of its parts. Just as there is often no one "right" answer in case analysis, so too, there is no one "right" way to teach with case studies. This workshop was intended to spark and inspire instructional leaders' interest in teaching with cases and to suggest teaching techniques to help deliver content in a highly contextual, interactive format. Participants were exposed to the possibilities and benefits of using case studies in their courses, a 5-step conceptual framework for case analysis, and examples of twenty pedagogical techniques that can be used to facilitate case discussions. According to feedback from workshop participants, a highlight of this session was the opportunity to view a video-
tape of an actual case discussion in which the 5-step process for case analysis was used to frame the discussion and each one of the twenty teaching techniques was modeled and demonstrated by the author. The top technique mentioned was the role-play. One faculty member, summarizing the sentiment of many, said her favorite part of the video was "the demonstration [role play] asking to talk to the student, father, and teacher--that really caused the students to walk in another person's shoes--essential to be...empathetic to all those involved in a situation/case." I couldn't agree more. Using a variety of techniques to deliver content is considered best practice and contributes to the professional development of the instructor and student alike. Applying case studies to course content is an effective vehicle to engage students in higher-order thinking as they make the connections between theory and research to real life dilemmas in their areas of concentration. References Christensen, C. (1987). Teaching and the case method. Boston: Harvard Business School.
Volume 18, Number 1 · Winter 2005
Mid-Western Educational Researcher
39
Educational Testing Service. (2002). Praxis II: Principles of learning and teaching examination. Princeton, NJ: Educational Testing Service. Evenson, D. & Hmelo,C. (Eds.). (2000). Problem-based learning: A research perspective on learning interactions. New York: Erlbaum. McNergney, R., Herbert, J., & Ford, R. (1994). Cooperation and competition in teacher education. Journal of Teacher Education, 45(5), 339-345. Merseth, K. (1991). The case for cases in teacher education. Washington, DC: American Association of Colleges for Teacher Education. Rowley, J., & Hart, P. (1995). What to do about Raymond, from Becoming a Star Teacher, (Videotape Series). Teacher-Development.Com.
Sudzina, M. (1997). Case study as a constructivist pedagogy for teaching educational psychology. Educational Psychology Review, 9(2), 199-218. Sudzina, M. (1999). Guidelines for teaching with cases. In M. Sudzina (Ed.), Case study applications for teacher education: Cases of teaching and learning in the content areas. Boston: Allyn & Bacon. (www.abprofessional.com/edresources) Sudzina, M. (2000). Innovative resources and strategies for applying cases to teacher education. In H. Klein (Ed.), Complex demands on teaching require innovation. Madison, WI: Omni Press. Sudzina, M. (2004). Teaching and learning with case studies: Facilitating case discussions and engaging students. (Videotape). New York: Insight Media. (www.insight-media.com)
Call for Graduate Student Reviewers Mid-Western Educational Researcher The Mid-Western Educational Researcher seeks doctoral students to be an energetic, contributing part of its growing list of reviewers. The Mid-Western Educational Researcher is a scholarly journal that publishes research-based articles addressing a full range of educational issues. The journal also publishes literature reviews, theoretical and methodological discussions that make an original contribution to the research literature, and feature columns. It is the official journal of the Mid-Western Educational Research Association (MWERA), a regional affiliate of the American Educational Research Association (AERA). Four issues of the journal are published annually. The editors seek doctoral students who want to gain experience as journal reviewers. Reviewers are electronically sent an abstract of a manuscript in their field of expertise and asked if they can provide a review within four weeks. If they can, a blind copy of the manuscript and a review form are sent. While we prefer electronic reviews and transmission, hard copy is also an option. Please provide your review information to Deborah Bainer Jenkins, Co-Editor, at [email protected] Please send: name, mailing address, email address, telephone number, institutional affiliation, academic status, and areas of interest or expertise.
40
Mid-Western Educational Researcher
Volume 18, Number 1 · Winter 2005
The Mid-Western Educational Research Association Gift Membership A gift membership has been given to you, by Your name is now included as a member in one of the most recognized, well respected, educational research groups in the United States and Canada. Your one year membership includes a subscription to the Mid-Western Educational Researcher, the Association's journal that highlights research articles, features, interviews, and Association news. Members pay reduced registration fees for the annual meeting held in Columbus, Ohio, in October. This conference attracts many nationally recognized leaders in educational research. Enjoy your membership.
Thank you for providing your colleague, student, or friend with a special one year gift membership to the Mid-Western Educational Research Association. It is a gift of professional involvement that is sure to be appreciated throughout the year. To give your gift membership fill out the top portion of this card and use it to inform the recipient of the gift membership; then fill out the bottom portion of this card and mail it with your check to: Jean W. Pierce--Dept EDCSE--Northern Illinois Univ.--DeKalb, IL 60115
Person Receiving Gift Membership
Person Giving Gift Membership
Name
_____________________________
Address
_____________________________
_____________________________
_____________________________
Affiliation
_____________________________
Work Phone _____________________________
Home Phone _____________________________
E-mail
_____________________________
Fax
_____________________________
Division
_____________________________
Preference (optional)
_____________________________ _____________________________ _____________________________ _____________________________ _____________________________ _____________________________ _____________________________ _____________________________ _____________________________ _____________________________
Check one below and make check payable to Mid-Western Educational Research Association.
Professional Membership--$45
Student Membership--$25 Student must be currently enrolled.
Lifetime Membership--$450
The Ohio State University 2500 Kenny Road Columbus, Ohio 43210

File: evidence-based-reform-in-education-promise-and-pitfalls.pdf
Title: Index
Author: Owner
Subject: Index
Published: Mon Aug 28 18:55:23 2006
Pages: 44
File size: 1.12 Mb


A history of bestiality, 140 pages, 1.01 Mb

García Márquez, Gabriel, 2 pages, 0.05 Mb

, pages, 0 Mb
Copyright © 2018 doc.uments.com