Archive | Design Research Blog Posts

RSS feed for this section

All blog posts syndicated related to Design Research

Brief Reflection about Using a Text Mining Approach in a Design-Based Research

A post by Alejandro Andrade

After having the opportunity to explore the use of a text mining approach to analyze information in a design-based research project about using video to support pre-service teachers’ ability to notice (Van Es & Sherin, 2002), I have three major ideas to share. First, these data-mining techniques are flexible and powerful tools, and yet one should be aware of several of their limitations. For instance, the stemmed words in a text document are but proxies of participants’ conceptual engagement, but these might be a rather distal than proximal type of evidence. The bag-of-words approach, the one used in my analysis, overlooks a great deal of information that might have been relevant to help tease apart more nuanced hypotheses. Nonetheless, the approach, however distal it might have been, did provide relevant evidence given the context of the present study, for instance, the relationships between the learning theories and the student analytics, and these latter and the experts’ analytics.

Brief Reflection about Using a Text Mining Approach in a Design-Based Research

Brief Reflection about Using a Text Mining Approach in a Design-Based Research

Second, while the bag-of-words is one text mining approach, it is not the only computerized technique available. Indeed, other more powerful tools can supplement or replace such an approach. For instance, some computerized linguistic analyses exist that can provide measures of coherence and cohesion in text documents. One of such techniques is the Coh-Metrix (Graesser, McNamara, Louwerse, & Cai, 2004), a free online tool that provides more than a hundred different indices with text characteristics. Among others, Coh-Metrix provides information about text easability, referential cohesion, content word overlap, connective incidence, passivity, causal verbs and causal particles, intentional particles, temporal cohesion, etc. With this tool one can supplement the findings about differences and similarities between the students’ and experts’ analytics, for instance.

Third, I believe that the incorporation of computational techniques to the researcher’s toolkit is bound to gain traction in the learning sciences. In particular, as researchers adopt design-based research methodologies (Cobb, Confrey, Disessa, Lehrer, & Schauble, 2003; W. Sandoval, 2014; W. A. Sandoval & Bell, 2004) that demand a sequence of test and refine iterations, they might consider having tools in their belts that can allow swift and reliable understanding of their results. Unlike other slow qualitative coding-scheme-based approaches that require inter-rater reliability, content analytic tools such as the bag-of-words are much faster and consistent. Also, these quantitative tools are useful with only a small group of students or with larger samples of various hundreds or even several thousands of participants. This doesn’t mean that traditional coding schemes are not good, or that we should stop caring about them. On the contrary, I believe that both approaches can work in tandem, where computational techniques provide a first glance at the data for a quick and dirty pass of analysis that can inform the research team on how to adapt and refine the design, and then, when resources allow, researchers can go deep into the data and examine the nuances of student learning interactions.

References

Cobb, P., Confrey, J., Disessa, A., Lehrer, R., & Schauble, L. (2003). Design experiments in educational research. Educational Researcher, 32(1), 9.

Graesser, A. C., McNamara, D. S., Louwerse, M. M., & Cai, Z. (2004). Coh-Metrix: Analysis of text on cohesion and language. Behavior Research Methods, Instruments, & Computers, 36(2), 193-202.

Sandoval, W. (2014). Conjecture mapping: an approach to systematic educational design research. Journal of the Learning Sciences, 23(1), 18-36.

Sandoval, W. A., & Bell, P. (2004). Design-based research methods for studying learning in context: Introduction. Educational Psychologist, 39(4), 199-201.

Van Es, E. A., & Sherin, M. G. (2002). Learning to Notice: Scaffolding New Teachers’ Interpretations of Classroom Interactions. Journal of Technology and Teacher Education, 10(4), 571-596.

Our DML Commons Story at ALTfest



Last week, we had the opportunity to present our experience co-facilitating the DML Commons Design Based Research course at the Academic Learning Transformation Festival (ALTfest) held at Virginia Commonwealth University. The purpose of ALTfest was to explore ideas and share stories of learning transformation in a festival-type atmosphere. Mimi Ito kicked off the day’s events with a keynote address where she shared her work of connected learning with young people. Many of the participants in ALTfest came from higher education backgrounds, and were very interested in applying the principles of connected learning with faculty, in professional development with teachers, and in other spaces. A few quick highlights from our day at ALTfest included a visit to the Makerspace where we met @abchamberlain, and pulled a few prints.

 Our DML Commons Story at ALTfestOur DML Commons Story at ALTfest

We also met the folks from Hack.RVA, a makerspace for adults in Richmond, VA, played with some 3D printed animals, and generally enjoyed the atmosphere that buzzed with creativity and all kinds of artists and makers.

During our presentation on co-facilitating the DML Design Based Research Course, we began with introductions in order to get to better understand who was attending our session, and why. We created an online poll to get a better sense of attendees’ level of experience with MOOCs and DOCCs.  As you can see from the results, most of the attendees had never facilitated a MOOC or DOCC before, and we learned that most folks there were interested in learning more about how to facilitate open, online courses. 


Our DML Commons Story at ALTfest

During our presentation, we provided a brief tour of the DML Commons website, and engaged participants in some of the DML Commons activities we found particularly fun, such as the “What Epistemology Are You?” quiz (which was very popular among the audience!). We also presented some of the design features we found most helpful in increasing and encouraging participation.

Our DML Commons Story at ALTfest

We then engaged participants in a design challenge. We handed out packets with a collection of the elements of the connected course including webinars, readings, webinar activities, blogs, and so on, and then encouraged them to work in groups to apply some of the design principles to their own connected course and content.

Our DML Commons Story at ALTfest

As we visited with each of the small groups, we were impressed by the level of conversation and the ways in which folks were applying some of these ideas to their own settings. For example, one of the groups wanted to know more about how folks were using Twitter in the course. Another group wanted to learn about the advantages of etherpad in comparison to Google documents for engaging viewers. Yet another participant shared that she didn’t like Twitter and thought that it would be a useless design element to include in an open course, and was surprised to find out that much of the DML Commons interaction took place among participants on Twitter, for example through live tweeting sessions. Another participant wondered about the “seamlessness” of the experience. He asked “with so many platforms being used, were participants overwhelmed or confused?”

Finally, many participants shared that creating a connected course was a lot more difficult than they had imagined. We’d agree with them, but add what we shared at the end of our presentation: that because connected courses are co-created and co-constructed, the line between teacher/learner/facilitator becomes blurred. Indeed, it is in the act of “making” the course, that allows for the most meaningful learning happens. Ultimately, the process of co-creating the Design-Based Research Course not only deepened our understanding of a methodology we are truly excited and passionate about, it also helped us to become more connected learners and teachers.

After the presentation, we had an interesting conversation with one of the participants, musing about the layers of distribution the DML Commons course includes. @derekbruff wanted to know more about why we were calling the DML Commons a “DOCC” instead of a “MOOC.” “Weren’t they, in fact, the same thing?” he asked. We argued that they weren’t. First, we wanted to give credit to the Femtechnet scholars for broadening and pushing our understanding of what it means to teach and learn in online spaces. Second, we feel that the co-constructed and distributed nature of the DML Commons is one of its strongest features, and should be foregrounded. This includes the range of diverse range of online spaces that were appropriated to support the rich sharing of everyone involved, the participatory design of the course to shake traditional hierarchies, and the possibility to enrich offline learning spaces through the generated content and practices. Third, we think it’s important to move beyond the term “MOOC” because, frankly, it has a lot of negative connotations and people tend to have knee-jerk reactions to the term.

Through the DML Commons Design-Based Research course, we had the opportunity to prepare and co-facilitate two online synchronous events before presenting together at the ALTfest. Here, the boundaries that we experienced through online channels, such as directing listeners to where to submit comments or to use a backchannel to coordinate and pitch questions to one another, did not apply. However, the online co-facilitation experience gave us a chance to get to know each other’s presentation style and to develop a shared style together.

Community-based DBR: A Few Linguering Questions


As we are wrapping up an exciting 2 weeks filled with vibrant discussions around community-based research  as well as participatory approaches in design-based research, we created a collection of questions that could perhaps provide others interested in similar work with starting points for inspirational insights.


  • What are the kinds of academic contributions participatory research can make? Ask new questions? 
  • Megan Bang mentioned that it is important to make academia more transparent to participants and to provide them with access to the academic infrastructure and tools. How does this relate to your work?
  • What is the value researchers can they offer to communities?
  • Nancy Erbstein’s challenges of working with youth-led research included that decision makers questioned small sample sizes of the youth's work. Nancy recommended mixed methods. How does this relate to opening or closing participation? What do you recommend?
  • As part of her list of sensibilities of community based partnerships, Magen Bang noted the need for  research to engage in strategic transformation of institutional relations. How important is this question for your work?
  • For Phillip Bell, his theoretical approach informed the way participation infrastructure was framed and how DBR played out. How has this been for you and what would you recommend?
  • Is participation and research+practice partnerships part of your larger research trajectory? What would you recommend others to practice who would want to build their career in this area of work?
  • We frequently heard of projects that started with a 3 year trajectory and then continued and continued and continued. How do you consider sustaining engagement in your project planning? What are the tools and techniques you use? What tells you that you are right in continuing to seek funding for a work?
  • Phil Bell and Julian Sefton-Green pointed out the need to be embedded and personally invested in the community one asks to collaborate with. Do we need to put ourselves in harms way to do good work?


 -Anna

Nancy Erbstein & Julian Sefton-Green on Community-based Research

April 21, 10:00am-11:00am PST (12pm CST/1pm EST)

Webinar recording

Facilitators

  • Alicia Blum-Ross, London School of Economics and Political Science
  • Melissa Brough, CLRN @ DML Hub, UC Irvine
  • Anna Keune, Indiana University

Invited Guests

  • Julian Sefton-Green (London School of Economics)
  • Nancy Erbstein (UC Davis)

We had the pleasure to co-facilitate our webinar with two special guests: Nancy Erbstein from the University of California at Davis who spoke on her work in the area of Youth Participatory Action Research (YPAR), and Julian Sefton-Green from the London School of Economics & Political Science and an independent scholar who shared his diverse experience on who research is for and why to involve people. Here are some notes on the webinar.

Nancy Erbstein got started with YPAR during her undergraduate work that related to educational reform, working with renowned researchers in the area and alongside Language Learners (L2). This is when she recognized the “disconnect between students’ experience and the discourse of the national school reform arena.” She wanted to bring the L2 youth voices to the conversation of policy planning and evaluation. Nancy’s experiences working in South Asia around nationalization of the Nepalese school curriculum with ethnic minorities also informed her YPAR work as it stands today. She developed a curriculum that was based on youth-led research and evaluation. With the creation of the youth-led research and evaluation planning curriculum, Nancy and her colleagues asked youth to research their own conditions and bring that activity to bear upon research and evaluation. A critique of the politics of knowledge production led her into this work. She also co-founded the nonprofit organization Youth In Focus to train young underrepresented people in youth-serving organizations to engage youth in Participatory Action Research (PAR), evaluation and planning.

Typically funders and academics turn to scholars who write about this work without specific focus on the people who developed this work in and with communities. Academics have come and studied and adapted this work and written about it, but in fact there is a lot of work that is done in the field that is not documented in ways that are typically recognized by academia. [We need to] acknowledge that young people were engaged in building YPAR, which is now ubiquitous in comparison to the field 20 years ago. – Nancy Erbstein

Julian Sefton-Green started with the intention of being a “proper academic,” but instead became a high-school teacher with a particular interest in media studies and media education. Training at an institution with a history of activist education he had the opportunity to work in and with schools. Julian mentioned that “at the time classroom teachers, along with heads and teacher advisors, had more influence about what happened in schools, especially in regards to assessment and curriculum.” Working with a community of practitioners developed his wish to find ways to theorize the challenges he experienced when working with a community of teachers.

The starting point for [my] academic scholarship was working with teachers and the problem of trying to understand the meanings that young people brought to the classrooms through their consumption of popular media culture and trying to find ways to credentialize and validate that and to explore that. … The point of working had direct and immediate purpose with and for that community of teachers. – Julian Sefton Green

Nancy Erbstein’s projects: increasing access to data on racial and geographic disparities in youth well-being to inform young advocates’ work

While working with young equity advocates and people in decision-making and planning environments in areas including education, public health, youth development, community development, and philanthropy, Nancy engaged three communities in California: an underinvested urban community, a large town with formerly incarcerated youth of diverse sociocultural backgrounds, and a rural area with predominantly Latino youth. Nancy partnered with organizations to design and pilot an educational curriculum for critical use of public data, digital youth well-being analyses and online GIS systems. Most of the funding for this work came from the university campus, and private foundations.

Nancy recalled that one of the main challenges of working worked with young people on YPAR that involves youth with public decision making is that “decision makers would question [the youth] on their small sample sizes. It was difficult for [the youth] to gain access to large quantitative data analysis that policy makers were using to inform their thinking.”

When advocates contacted Nancy, asking for data and requesting her to analyze and map data to use alongside the youth generated research, Nancy recognized that “public data was not accessible and even if available, looking at data in the way [the youth] wanted (e.g. spatially references, disaggregated) was not possible for them.” The questions emerged:

  • How to construct geo-spatial measures of youth well-being that would rely on publicly available data, and would provide analysis of sub-county level (US) that can be made available online through a GIS platform?
  • How to build youth capacity to work in an informed and critical way with such analyses?

What are the limits of policy makers in reading YPAR research?

It is difficult to know what drives policy change. Often it has little to do with research. Sometimes the one compelling story that gets everyone excited about moving a policy agenda makes the difference, other times it is the link between a piece of research and somebody’s active policy agenda or the moving train in a policy environment. If there is a good fit, there may not be so much concern about sample size. (…) Other times, when trying to raise new issues or contentious issues, people will raise questions about (…) smaller sample size or a primary emphasis on qualitative data. – Nancy Erbstein

With this in mind, Nancy encourages the use of mixed methods, and “providing large scale data sets and encouraging young people to point out the downsides of those data sets and the ways in which their research can fill an important gap and/or raise important questions.”

How do you get young people engage?

The work Nancy was involved with “tied to issues that youth were already contending with”, and did not find it challenging to get youth involved. However, sustaining involvement can be challenge, as young people often have a lot going on in their lives. Another issue could be the need for youth to find work to support their family.

As a way to counter this challenge, Nancy suggests offering stipends to young people, especially if their work was supporting the work of the institution. However, she warns that “in classrooms it can be more difficult to integrate this type of work where [youth] engagement is part of a course, but I’ve seen some remarkable successes.” One example of these success stories comes out of the Community To The Classroom project (video of an example produced by youth). Nancy experienced that “young people often want to discuss the dynamics of (…) power they are experiencing.” Here it is particularly important to support adults with the resources needed to open up lasting ways for youth to engage.

Julian Sefton-Green projects: creating videos with youth and screening them to a public facing audience & English National Project Creative Partnerships to foster collaboration among teachers and artists

While working at a youth community center, Julian was commissioned to create a video, along with youths who were living in care, which was intended to be viewed by a public facing audience. It was a positive learning experience for the youth to tell their stories, and they enjoyed creating the film. Against expectations, the screening of the film to the social services and people in local authority did not go over well. Julian recognized that it was a “complete contradiction to the way the youth wanted their story to be told and the awkward response of the people who commissioned the video.” Julian suggested that it is important not to promise the youth a quick change. In his case, young people already knew how difficult these kinds of experiences are. Julian added: “You engage them into political discourse and give them the opportunity to learn how to engage in this kind of discourse. [It is important] not to promise any outcomes, and that the work would achieve anything.”

Julian also worked with English National Project Creative Partnerships to foster collaboration among teachers and artists toward the development of learning activities for young people. The project had a large budget and varied in its outcomes and results. Julian helped “create literature reviews, because what research can contribute is histories and thinking about bodies of knowledge and ideas that have a tradition. (…) The project also created constituency of interest and the research-validated practices gave authority and weight besides the theorization.”

Research at its best creates constituencies of interest. You might not succeed in persuading people in power to invest into young people, but research creates a language and a way to organize and collect around a set of understandings that you can theorize and frame and put into shape and narrative and that can be powerful. – Julian Sefton-Green

What is the importance of theory?

Julian reframed the question, asking instead “what does contribution to academic theory mean?” For him, it means (a) publishing in journals and collecting citations, and (b) framing and explaining a particular practice to move an argument forward. Theorizing is often of greater interest. And Julian elaborated that “theory is a kind of way to explain things to people on the ground.” For him, “it comes down to how clearly one can write. (…) A lot of it comes down to how easily you can express yourself in simple, easy-to-understand language. And that is not the same thing [as the] premium of what it means [to be] an academic.”

Work-Life balance

Nancy made an explicit decision not to be in a mainstream faculty position. At institutions all of the work it takes to pursue rigorous and mutually respectful, equity-oriented, community-engaged research is considered service. So much of her work-time does not directly focus on writing peer reviewed journal articles. In a non-tenured faculty position without a regular teaching load, she has been able to allocate time for community engagement but still work with students through projects and committees. One can be successful working in this line of work while being in a tenure-track positions, but it presents unique challenges, for example, it is often not an easy fit if you want an academic appointment that carries PI status; but this is changing to some extent. Groups such as CCPH and Imagining America are working to support such scholars and scholarship in the US.

Is there a contribution that participatory and community based research can make to educational research?

Collaborative and political work has a way of engaging participants in ways that other forms of research does not. So the direct and immediate impact is there and part of your life as opposed to indirect effect. Direct collaborative community-based research can be better than other forms of research. (…) it is important to stay on the edge to avoid becoming another orthodoxy within academia. – Julian Sefton-Green
What PAR has to offer to knowledge production (is the) potential generation of new kinds of questions that might not emerge on campuses. Complex challenges that people are looking to define and solve, require multiple ways of knowing to tackle those challenges. PAR offers the opportunity to create the kinds of trans-epistemic communities that can work on these issues. – Nancy Erbstein

The MILL: A Makerspace for Pre-Service Teachers at Indiana University

















With Indiana’s pre-service teacher applications in decline and makerspaces on the rise, there is a need for the IU School of Education to update their approaches to attract and prepare a new generation of teachers for the kinds of challenges they may face in their career. At the same time, makerspaces need educators that come into the spaces with ideas for facilitating holistic and diverse making based on educational theories and practices learned at their university. 

Makerspaces in higher education institutions make it possible for pre-service teachers to use the kinds of tools and materials that can regularly be found in makerspaces but have not been traditionally integrated in learning spaces like classrooms and pre-school but are more frequently being integrated in formal and informal learning spaces, such as libraries, museums and schools. Pre-service teachers can try out making models with 3d printers, craft projects with the laser cutter, assemble electronic circuits, or simply design and prototype with common materials like cardboard or legos. By exploring the materials and tools and letting the material guide their ideas for learning opportunities and educational adventures, pre-service teachers can practice those skills that are in demand in makerspaces that all too often do not have curricular based on educational theory. These explorations can support teachers in taking back charge of the design of learning.

During the 2014-2015 school year, Dr. Kylie Peppler led an interdisciplinary faculty committee within the School of Education at Indiana University that began an initiative to design and build a makerspace in the School of Education.  A space suitable for innovating, collaborating, creating, prototyping, learning and general making. The result will be the MILL (Make Innovate Learn Lab) and renovations are set to happen this summer 2015. We are beyond excited about having the new space!

 As the We had the opportunity to tag-along, observe, and participate. In relation to the DML commons unit of co-design and collaboration in DBR, we think this is an interesting example of how interdisciplinary collaboration and pushing of institutional boundaries happened in a shared effort to setting the foundation for making the vision a reality. We outline here key decisions and practices we observed that seemed to have been instrumental for co-constructing a coherent narrative vision for and around the space and for laying the infrastructure for the successful continuation of this collaborative endeavor.

MILL Space Committee includes representatives from Science Ed, Art Ed, Instructional Systems Technology, Learning Sciences, the Office of Instructional Consulting, Education Technology Services & other School of Education departments. This collaborative committee effort underlined the message for the makerspace to serve the needs of several departments in the IU School of Education. The primary objective of the committee is to extend collaboration across school departments and to enrich the vision for the space from diverse educational perspectives coming from students and faculty. A large part of the committee conversations centered around the makerspace as a place to make stuff AND to represent a starting point for shifting the culture in the IU School of Education. In particular, the idea was to encourage thinking about challenges and design opportunities while making and engaging with material prototypes. While the makerspace is primarily aimed at the students, staff and faculty of the IU School of Education, the committee frequently thinking about how the space can extend the reach of collaboration to community partners and local K-12 schools.


The MILL: A Makerspace for Pre-Service Teachers at Indiana University


A cardboard prototype of portable and transparent storage that can slide under a counter

Creativity Labs research team (lead by Dr. Kylie Peppler) gave input into the design of the space. Opportunities to brainstorm and ideate various aspects of the physical space occurred during a graduate level apprenticeship course that also included an excursion to a new Teen Media Space in the Monroe County Public Library, that is designed based on a connected learning model. Many of the suggestions and ideas of the graduate students made it into the blue prints of the makerspace, and will be included in the construction that is planned to start this summer. The features that graduate students contributes include open and transparent storage opportunities and the need for mobile and reconfigurable furniture to expand accessibility. Much of the labs work this year focused on properties of materials. The need for open and inviting storage options to clearly see materials was a key factor in the design.

The MILL: A Makerspace for Pre-Service Teachers at Indiana University



Examples of portable and transparent storage for easy access to materials

The paper prototype above shows the layout of the space that is planned to combine two smaller rooms into one longer space. The prototype helped mediate the furniture selection and configuration process. It also represents a key design features of the space that was milled over by the Creativity Labs team to further substantiate transparency: one entire wall will be glass. Clear and see-through storage can be pushed against the glass, affording passers-by to see materials at a glance, encouraging and calling them to enter.

Architects, Interior designers, Electricians, Representatives of deans council, SOE donors and more – Some additional aspects of collaboration that occurred throughout the design of the space was the involvement of many other stakeholders in and out of the School of Education. University architects, interior designers, electricians, etc. were regularly met with to plan and design the space and the unique furniture needs for the space. Representatives of the IU School of Education Deans council were brought in early to discuss the possibility of faculty research grants for the makerspace. Additionally, potential donors for the School of Education were contacted about sponsorship and future funding for the space. The amount of collaboration among a wide variety of people was essential in spreading the vision for the space and ensuring its future success.


If you want to learn more about Makerspaces in Schools of Education visit the websites of

Please let us know of any other makerspaces in Schools of Education you might have heard of!

Anna Keune & Justin Whiting