ePortfolio, student learning, Association of American Colleges and Universities, ePortfolios, IJeP, learning, Research Agenda, Stanford University, special issue, International Journal, research evidence, Terrel Rhodes Association of American Colleges and Schools, University of Georgia Portland State University, National Institute for Learning Outcomes Assessment, Example Research Questions Category Example Research Questions Assessment, AAC&U, case studies, social science research, qualitative research methodologies, Possibilities Watson, Crowdsourcing ePortfolio Research, Helen Chen, mixed methods, self-reported data, qualitative methods, research question, learning outcomes, References Bryant, rigorous methodologies, National and Community College Surveys of Student Engagement
International Journal of ePortfolio http://www.theijep.com
2014, Volume 4, Number 1, 1-5 ISSN 2157-622X
Editorial: A Call for More Rigorous ePortfolio Research
Terrel Rhodes Association of American Colleges and Schools
Helen L. Chen Stanford University
C. Edward Watson
The University of Georgia Portland State University
This special issue of the International Journal of ePortfolio is entitled "Defining Practice and a Research Agenda: Selections from AAC&U's Fifth Annual E-Portfolio Forum." This article serves as an introduction to the special issue, provides highlights from the E-Portfolio Forum, and concludes with a call concerning new directions for rigorous ePortfolio research.
Framing the Issue The Association of American Colleges and Universities' (AAC&U) fifth annual E-Portfolio Forum, E-PORTFOLIOS: Defining Practice and a Research Agenda, examined the driving question: What is the evidence supporting the theoretical connections between ePortfolio and student outcomes
? The Forum engaged attendees in both sharing their current research and practice and helping define the research agenda moving forward. As one of AAC&U's academic sponsors for the Forum (along with the Association for Authentic, Experiential and Evidence-Based Learning [AAEEBL]), the International Journal of ePortfolio (IJeP), is the leading outlet for research studies on ePortfolio for learning, assessment, personal development and beyond. As such, IJeP has devoted this special issue of the Journal to research articles
from campus practitioners who presented their findings at the 2014 AAC&U E-Portfolio Forum. The ePortfolio community has been growing rapidly in recent years. Well over forty percent of higher education institutions now report using ePortfolios on their campuses. As part of AAC&U's Liberal Education and America's Promise (LEAP) initiative, ePortfolios have been encouraged as a potentially significant strategy for integrating Student learning
, faculty engaged pedagogies and curricular alignment. With the establishment of AAEEBL in 2009, the professional organization for ePortfolio faculty and administrators and its growing set of resources and conferences, AAC&U has a partner in the promotion and exploration of ePortfolios and their value. The ePortfolio community now has a robust locus for engaged learning and practice centered on effective use of digital resources. At the same time, the growing awareness of and interest in ePortfolios related to student learning and student success has increased the demand for evidence that ePortfolios are or can be correlated with measures of improved learning and completion. IJeP is the primary response to many of those calls for evidence. As the primary outlet in the United States for research on the effects of ePortfolios for the variety of audiences
interested in this pedagogical approach, AAC&U has strongly supported the efforts of IJeP to create an online and print venue for disseminating the research on all aspects of ePortfolio use as well as the Journal's call for more rigorous and generalizable research on the multi-faceted outcomes of ePortfolios for students, faculty, and institutions. The 2014 AAC&U E-Portfolio Forum focused one of its two Tracks on research evidence, calling upon the e-Portfolio community to share the research already underway on campuses across the country. The Forum combined presentations of campus practice and findings, a crowdsourcing exercise on new areas of research needed to support continued ePortfolio use, and an analysis of previously published research on ePortfolios in IJeP and other journals. The presenters for each research session also were invited to prepare a paper on their research methodology and findings. The papers were then peer review
ed by IJeP. This issue contains accepted research papers presenting evidence of ePortfolios related to student learning and success. Opening Plenary The Opening Plenary of AAC&U's 2014 EPortfolio Forum focused on the research around neurological bases of learning and how ePortfolios are a robust medium that allows for capturing, demonstrating and integrating student learning through a broad set of practices and modes. A further focus was on the Degree Qualifications Profile (supported through Lumina Foundation funding) that provides a framework designed to assess college/university-level learning regardless of where, when or how it was acquired. The emerging Global Learning Qualifications Framework is the result of extensive research and builds on the progress of qualification frameworks from over 90 countries, the Lumina Degree Qualifications Profile, the Association of American Colleges and Universities (AAC&U) Essential Learning Outcomes
and VALUE Rubrics, and other emerging work in the field. This plenary asked us to place the E-Portfolio Forum work in a
Rhodes, Chen, Watson, and Garrison
A Call for More Rigorous ePortfolio Research 2
neurological and global context for examining research questions as the ePortfolio community moves forward. Findings from the "Crowdsourcing ePortfolio Research" Exercise The crowdsourcing activity, conducted during lunch by Helen Chen on the day of the E-Portfolio Forum, was designed to leverage the collective knowledge and interests of the over 150 ePortfolio community members in attendance. Attendees were asked to respond to the question, "What is your burning research question about ePortfolio?" and to write their question on an index card. These questions were subsequently traded away and through a series of five pairings among attendees, each question was given a score from 1 to 5, based on the scorers' level of interest in reading a journal article about the research question on the card. At the end of the five rounds, the scores on each card were summed into a number ranging from 0 to 25. The cards were then collected at the end of the activity and informally coded and sorted into broad categories by the four authors. The methods and the results from the crowdsourcing activity are by no means scientific nor broadly generalizable; however, they do represent the views of the majority of the attendees at the 2014 EPortfolio Forum whose questions as faculty, staff, administrators as well as vendors, are aligned with many of the interests and concerns of the ePortfolio community more extensively. Table 1 lists the categories ranked from highest to lowest levels of interest together with example questions. Of the 14 topics that emerged from the categorization of the 156 questions that were collected, the top three areas of interest were Institutional, Faculty, and Student Buy-In and Adoption (22%), Assessment (18%), and Student Learning (17%). The latter two topics were purposely broad and additional exploration of the nuances of these questions is needed. When individual questions were ranked by score, the categories that represent possible emerging areas of research interest are Retention/Transfer and Career/Post-Graduation ePortfolios. Examples of the questions that received the highest scores are below: · In what ways can reflective ePortfolios affect student persistence/retention? (Category: Retention/Transfer; Score: 24) · Do employers value ePortfolios as a substitute for traditional resumes? How do we build better reflective authors (ePortfolio students)? (Category: Career/Post Graduation ePortfolios; Score: 23)
· Are there institutional data that prove that ePortfolio use leads to higher retention/graduation rates? (Category: Retention/Transfer; Score: 23) Closing Plenary During the closing plenary session, Watson (2014) began by reviewing the learning theory in support of ePortfolios. He suggested that reflection, feedback, Social learning
, knowledge integration, and cognitive processing were key among the reasons why we have confidence in the efficacy of ePortfolios; however, he focused the remainder of his talk on questions regarding the evidence in support of these theoretical connections between ePortfolios and various student outcomes. General Findings from Survey of Published Research Literature Through two lenses, Watson considered research regarding ePortfolios. First, he shared the literature review work of Bryant and Chittum (2013). They conducted a thorough review of the literature published in peer-reviewed journals between 1996 and 2012 considering ePortfolios. Four general categories emerged across 118 articles with the majority of the publications being descriptive or self-report in nature. Approximately 15 percent (n = 18) provided empirical data considering student outcomes, and only two of the 118 "utilized valid and reliable measures in addition to a comparison/control group" (p. 193). While a diversity of research strategies have been employed to examine ePortfolios, Bryant and Chittum found that very little research had been published that meets the most rigorous standards expected of Educational Research
. Landscape, Limitations, and Possibilities Watson also provided insight into the range of submissions to the International Journal of ePortfolio since its launch in 2011. Looking at the totality of all submissions to the journal, the most common methodologies employed by authors included case studies
, focus groups, surveys, and rubrics. Many of the most popular topics mirror the learning theories that have led a number of practitioners to ePortfolio. They include reflection, assessment, rubrics, feedback, career portfolios, technology narratives, and integrative learning, and the results have typically provided case narratives, self-reported attitudinal data, self-perceptions, and qualitative student experiences with ePortfolios. Looking across the range of submitted manuscripts, many suffer from small sample sizes, a lack of rigor
Rhodes, Chen, Watson, and Garrison
A Call for More Rigorous ePortfolio Research 3
Crowdsourcing Categories Ranked According to Interest and Example Research Questions
Example Research Questions
Assessment Student Learning Career/Post Graduation ePortfolios Buy-in/Adoption Institutions/Departments Retention/Transfer Buy-in/Adoption - Faculty Buy-in/Adoption - Students Reflection Integrative Learning How to Build Scalability Identity Technology General Education
How do you balance promoting and demonstrating student learning with getting solid institutional data from ePortfolios? Do data support the notion that a campus-wide ePortfolio initiative is worth the effort and money? What kinds of support can help students articulate for themselves and others what they have learned and how they can transfer that learning to other situations in the future? How can ePortfolios enhance the civic learning and citizenship skills
of students? Do employers value ePortfolios as a substitute for traditional resumes? How do we build better reflective authors (ePortfolio students)? Will ePortfolios have an enduring value to students beyond graduation? What are the longitudinal effects of ePortfolio use on competencies post-graduation? How do ePortfolios affect/help create institutional change/learning? How can you create a culture of ePortfolio where one doesn't exist? How do "we" (students/faculty) [move] from a paper portfolio state of mind to an "e" portfolio state of mind? Are there institutional data that prove that ePortfolio use leads to higher retention/graduation rates? What happens when students carry evidence of learning across institutional boundaries? Can institutions handle this? What methods are most effective at engaging faculty in the ePortfolio process? Does the term "ePortfolio" encourage or discourage faculty interest/participation in "ePortfolio"/learning- centric projects? How do we get students interested/invested in doing ePortfolios? Does the use of ePortfolios in a class enhance student motivation? What audiences for student folios are most powerful as incentives for students to take their folios seriously and work on them over time - even years? How best to motivate both faculty and students to embrace reflective learning within the context of ePortfolios, so as to create a campus-wide culture of reflective learning? What incentive would result in authentic responses to ePortfolio reflection prompts? How do ePortfolios support/help develop reflective, metacognitive capacity? To what extent can an integrative knowledge (e)Portfolio process increase students' confidence, security and ownership in their sense of self and identity? Designing a new curriculum, how can we use ePortfolios to help students synthesize their learning outcomes? What is the role of visual narratives and/or visual and written narratives in ePortfolios? Does it help address "audience interference?" (see: film, photography, archival image/footage) How do you help students understand the different audiences and purposes for portfolios? What should we be aware of when scaling/expanding our ePortfolio programs? Ex: ease of use How do you expand an ePortfolio program beyond initial, dedicated faculty? Can ePortfolios help bridge the gap that first-generation college students often experience between their home-culture (of their parents and relatives) and the campus culture (that they now attend)? How much can ePortfolios affect students' dispositions and how much do dispositions affect the portfolio? Would students become more engaged with their ePortfolio if they were available on their mobile devices
with more social spaces for feedback, peer review in real time? Which is more successful - strictly administered platform or free-form, student-initiated ePortfolios? What will link all ePortfolios regardless of platform for the good of humanity? Do ePortfolios produced to assess general education specifically reveal a different level of learning than those produced for program? i.e. Which might be better options to show student learning? How can ePortfolios be used on our campus to effectively (accreditation, faculty/student buy in) assess student learning of general education?
associated with measurement reliability and validity, and/or a lack of randomization. As a result, the findings of these studies often have limited applicability beyond their own context, and may not be true in different settings. Further, some findings may not be entirely true within their own contexts. For example, we know that students do not always prefer the instructional methods that result in the greatest learning gains (Steinberg, 1989), and thus self-reports of learning gains lack the reliability that measurement provides.
In social science
research, it is true that each welldesigned study provides additional insights into the phenomenon under consideration, and there is much value in case studies, self-reported data, and rich narratives of experience; however, given the challenges to generalizability found within the current corpus of peer-reviewed ePortfolio research, it is now time to broaden the scope of current research to include methodologies, such as quantitative and qualitative
methods as well as mixed methods approaches that
Rhodes, Chen, Watson, and Garrison
A Call for More Rigorous ePortfolio Research 4
enable practitioners to confidently adopt practices in their own contexts with an expectation of similar gains across intended student outcomes. This leads to the following call. The "Call" Moving Forward · How do we move beyond perceptions and attitudes to explore how ePortfolios can be used to document evidence of student success and achievement of learning outcomes? · How do we operationalize the questions that emerged from the crowdsourcing exercise and design thoughtful and comprehensive research studies? · How can we strengthen ePortfolio research through the use of both quantitative and qualitative research
methodologies as well as mixed methods and longitudinal approaches? · How can the triangulation of data from multiple sources (e.g., individual institutions as well and initiatives such as the National and Community College
Surveys of Student Engagement) help us tell a stronger and compelling story about the impact of ePortfolios on student learning? · In order to truly evaluate impact, how can we engage and build partnerships with colleagues in student affairs, employers, and other stakeholders in the broader ePortfolio and higher education community? The ePortfolio community is diverse faculty, staff, instructional design
ers, educational technologists, students, researchers but we have a shared interest in these topics and advancing the field of ePortfolio to support student learning. As the ePortfolio discipline/field has matured and as new people join, there is a need to move beyond case studies and anecdotal stories towards more rigorous methodologies and data across individuals as well as institutions and perhaps over time. This may lead to more projects, such as cross-institutional collaborations like the Connect to Learning project, consortia such as the Inter/national coal
ition on ePortfolio Research, and organizations such as AAC&U, AAEEBL, and the ePortfolio Community of Practice that provide venues for both face-to-face and virtual interactions and community-building. References Bryant, L. H., & Chittum, J. R. (2013). Eportfolio effectiveness: A(n ill-fated) search for empirical
evidence. International Journal of ePortfolio, 3(2),
Kuh, G. D., Jankowski, N., Ikenberry, S. O., & Kinzie,
J. (2014). Knowing what students know and can
do: The current state
of student learning outcomes
assessment in US colleges and universities.
Urbana, IL: University of Illinois and Indiana
University, National Institute for Learning
Outcomes Assessment (NILOA).
Steinberg, E. (1989). Cognition and learner control: A
literature review. Journal of Computer-Based
Instruction, 16(4), 117-121.
Watson, C. E. (2014, January). Current trends andFuture Directions
regarding eportfolio research.
Closing plenary presented at the annual ePortfolio
Forum of the Association of American Colleges
and Universities, Washington, DC
TERREL RHODES, PhD, is Vice President for the Office of Quality, Curriculum and Assessment at the Association of American Colleges and Universities (AAC&U) where he focuses on the quality of undergraduate education
, access, general education, eportfolios and assessment of student learning. Before moving into national higher education work, he was a faculty member for twenty-five years. At AAC&U he led the faculty driven assessment of student learning entitled Valid Assessment of Learning in Undergraduate Education (VALUE). Currently, he leads the Quality Collaboratives initiative, working in nine states with twenty 2- and 4-year partner campuses testing the usefulness of the Degree Qualifications Profile (DQP) as a transfer framework focused on student learning mastery rather than seat time or credit accumulation. He is PI for Preparing Critical Faculty for the Future, focused on leadership development
and institutional change lead by women faulty of color in STEM at HBCU's through NSF's HBCU-UP program. Rhodes serves on the editorial board of IJeP.
HELEN L. CHEN, PhD, is a research scientist in the Designing Education Lab in the Department of Mechanical Engineering
and the Director of ePortfolio Initiatives in the Office of the Registrar at Stanford University. Helen is a co-founder and co-facilitator of EPAC, an ePortfolio community of practice (http://epac.pbworks.com). She works closely with Association of American Colleges and Universities as a faculty member for the Institute on General Education and Assessment and is the Co-Director of Research for the Association for Authentic, Experiential and Evidence-Based Learning (AAEEBL). Helen serves on the editorial board for the International Journal of ePortfolio and has research interests in the areas of
Rhodes, Chen, Watson, and Garrison
A Call for More Rigorous ePortfolio Research 5
ePortfolio pedagogy and reflective practices in general education, the disciplines, and co-curricular experiences and how they might inform a redesign of the academic record. She and her colleagues Tracy Penny-Light and John Ittelson are the authors of Documenting Learning with ePortfolios: A Guide for College Instructors (2011, Wiley). C. EDWARD WATSON, PhD, is the Director of the Center for teaching and learning
at the University of Georgia. He is also the Executive Editor of the International Journal of ePortfolio, served on the Planning Committee
for AAC&U's 2014 E-Portfolio Forum, and is a board member of the Association for
Authentic, Experiential and Evidence-Based Learning (AAEEBL). WENDE GARRISON is faculty within the Child and Family Studies program in the School of Social Work at Portland State University. She focuses her teaching efforts on helping students both articulate their stories and share them through portfolios. Wende has collaborated in several important contributions to the ePortfolio field in the last decade, including co-chairing AAC&U's E-Portfolio Forum since its inception and serving on the Editorial and Review boards of the International Journal of ePortfolio.
T Rhodes, HL Chen, CE Watson, W Garrison