My first class

After a very cold walk to the conference center, I've already had two- count them- two coffees.

My first class was a giant brainstorming session. Sitting at a table with 8 other teachers, we were given three questions about teaching writing, and 45 minutes to discuss the questions. Then, we had to create a poster to share what we discussed.

After 45 minutes of sharing, each table took a turn sharing their poster with the whole class. Then the class will vote on the favorite, and that team will share their poster with the whole big group (500+ people). I hope my team wins!

P.S. The teacher I'm sitting next to is from Grand Valley State University! Go Michigan!

Outgoing Reflection

Image source: www.planetpinkngreen.com

Before taking this course, evaluating media was simply a gut feeling. Does it connect with my objective? Do I like it? That’s it. However, taking this course has changed the way I have looked at media to use in my classroom, taking a more critical look at potential resources. It has even resulted in taking a more critical look at technology usage in general.

Through the work in this course, I have come to realize the disconnect between how media is currently used in the classroom, and how it could be used. While I believe most teachers do innately evaluate print media before using it, looking a little more in depth than just content and readability, I think most teachers dramatically lower their expectations when it comes to any form of multimedia.

An example of this lowered expectation comes directly from my own classroom. My school district participates in Project Charlie, a non-profit organization tasked with teaching drug and alcohol prevention to 3^rd and 5^th grade students. The volunteer brought in a video about a cow that all of the other farm animals made fun of due to the fact that she was big, slow, and ugly. In the end, the farm animals needed the cow’s help. The message in the video itself may have been effective. However, the animation and music was straight from the early 80s, and super cheesy. Even if the video had been evaluated, the content may have been deemed appropriate enough to be worthy of showing. But the video was highly ineffective (and slightly inappropriate) in my particular classroom context. The name of the cow happened to be the same name as one of the shyest girls in my classroom. A media evaluation instrument  would have forced the volunteer (who is not a teacher) to consider the classroom context and specific audience viewing the video.

It’s important to evaluate media using an evaluation instrument to consider all affordances and constraints of any piece of media rather than just using a piece of media because it fits with your lesson.

Analysis of Competing Instruments



Image from voiceinrecovery.wordpress.com

      One of the exciting parts of working in a project based learning environment is coming together in the end to see how differently learners took the same assignment and made it their own. Such is true with the Team Wiki evaluation instrument assignment, where the cohort broke into small groups with the task of creating two evaluation instruments: one for traditional media, and one for Web 2.0 media. While every group had the same assignment, and based the instrument off of the same baseline descriptors, the results were remarkably different.
Traditional Instruments
      When looking across instruments created by groups, the first obvious similarity that stood out was the organization of sequence according to the baseline descriptors: learners, learning, context, standards, learning objectives, strategies, cognitive load, accommodations, student assessment, and accessibility. In fact, many instruments also organized the way the baseline descriptors were evaluated, frequently relying on a 3 point rubric in which the levels of proficiency were narratively described. What’s more, several instruments also included a space after each descriptor with space for narrative response.

      Many groups collected a running total of points earned based upon specific scores in each category, unlike the instrument created by my group in which in lieu of points, the categories were identified as proficient, sufficient, or digressing. The groups had a final point total at the end of the instrument to aid in determining the value of the media. One particular group, Team Media Evaluation, also included a narrative description of what the media should look like. In addition to the rubric as created by my group, there was also a broad range of response strategies, including Likert scales, yes or no questions, and fill in the blanks. Shamelle Nash even included a way to collect point values for the narrative section of her instrument.

      Looking at this broad range of evaluation instruments, a few ways to improve the instrument created by my group are now possible. For example, Team Social Studies Rules included a narrative response section after each rating. However, unlike our blank lines, they prefaced the writing space with a guiding question, thoughtfully designed to help the evaluator think deeply about the media in question. A big “ah-ha” moment for me came when viewing the end of Shamelle Nash’s instrument. At the end of the instrument, there was a benchmark goal. It was cut and dry: scores above this level indicate a quality media. No other group, including my own, had something like that. Most of the rest of us simply collected points and left the evaluator to determine what to do with the points.

Web 2.0 Instruments
      Many of the Web 2.0 Instruments were designed in a similar fashion as the instrument designed by my team. For the most part, the Web 2.0 Instrument is a more specialized version of the Traditional Instrument. Most groups started with the same baseline descriptors, then moved into the more media specific descriptors, such as credibility of source and interface design and navigation. Again, many of the groups based the evaluation on a 3 point rubric.

      Team Social Studies Rules had a dramatically different Web 2.0 instrument, which was not only dramatically different from their Traditional instrument, but also very different from the other Web 2.0 instruments. Not only did the group change the content being addressed, but they used a Web 2.0 tool, Google Forms, to deliver the instrument. A question I would have for the group is this: What is the purpose for the instrument? The form clearly collects a wide range of data, which is presumably aggregated into a Google Spreadsheet. I’m wondering if the purpose for the instrument is to collect data on a range of media, developing a database to refer to when looking for effective media. If the purpose is to ascertain the effectiveness of the instrument, I wonder how the user receives the feedback.

      One easy way to improve both the Web 2.0 instrument as well as the Traditional instrument created by my group is to focus more on ease and aesthetics. My first view of the Web 2.0 instrument created by Team Social Studies Rules resulted in a gasp. The instrument was aesthetically pleasing, and seemed like something I could quickly fill out as I was reviewing media.

Instrumentation Process Reflection

Image source: blogs.seattleweekly.com

1.    As my group continues to construct our evaluation instrument, my thoughts and ideas changed from what I began formulating in my mind after viewing several different instruments as identified in my annotated bibliography. I always believed that important characteristics of any media that warrant evaluation included quality of the production, correlation to teaching and learning objectives, as well as appropriate level of content. However, I hadn't really put much thought into considering Bloom's Taxonomy when evaluating media. I certainly never really considered evaluating the extent in which media will be received by a student group in regards to learning context as well as learner background and culture.
      I also found it difficult to think broadly, especially in relation to the baseline requirements. While researching different evaluation instruments while working on my annotated bibliography, I came to realize that many of the instruments in which I felt would be easiest to use and most likely to be put into service in instructional planning were either some form of Likert scale or a checklist. Additionally, most commonly the instruments were standardized in formatting, something I noted added to the ease in which they could be used. Our group decided to build a Likert scaled rubric for the baseline requirements, and  I ended up finding out that some of the categories didn't seem to work well as a scale.

2.   Collaboratively building this instrument across both time and distance has proven to be challenging. While using a wiki as a shared space to collect thinking and track changes was effective, it wasn't the most efficient way to collaborate. For example, the first time visited the wiki to begin work, I had several ideas in my head. However, as I was not the first group member to begin work, a great deal of work was already started, in a direction different than anticipated. While an absolute affirmation for group work is combining multiple perspectives, using an asynchronous method of meeting, some ideas were marginalized or not shared.
      While trying to stick with a format that has great promise in it's ease of use, specifically using a rubric based upon a Likert scale, many categories didn't work well in that particular format. As categories were placed, the Likert scale felt artificial or forced. An example comes in the baseline requirement regarding the consideration of students' background. How can you objectively quantify how well media considers a student's background? While we did find it important to stick with a particular format to aid in the ease of use, it was difficult finding that balance between ease of use and effectiveness of the instrument.

Annotated Bibliography Comparison

Image from www.signandtrade.com

When comparing the resources I chose for my annotated bibliography with the resources as chosen by Cami, one difference that stood out between the majority of articles selected was the type of information that was collected. The articles I was drawn to tended to be more quantitative in their analysis of media. A specific qualifier is identified, and the media is rated based upon quality. Cami selected many articles that were more qualitative in their analysis. Many of her articles simply asked open ended questions to help the evaluator think about the quality of media and come to a decision independently. I also like the fact that she included an evaluation tool that can be taught to and used by students.

I did notice that both Cami and I approached this assignment in a similar fashion: we cast our nets wide. We looked at evaluation of very specific media, hoping to generalize the findings to all media. Cami found an article created to evaluate Flash software. While the evaluation may have been created specifically for Flash, there may be some unique ideas that could be included when we create our own rubrics. We also both looked for resources regarding the evaluation of technology usage in general, which is a much broader focus than, for example, online video.

To be honest, I was a little surprised Cami and I didn’t have any matches. Clearly we were both looking for resources that were current. While I was researching, I was frustrated with the lack of current articles specifically addressing media evaluation. Most articles I found were either out of date, slightly off-topic, or not a tool that could be applied to current media. I don’t recall coming across any of the articles cited by Cami.

Evaluation of Instructional Media Annotated Bibliography

Herring, D. F., Notar, C. E., & Wilson, J. D. (2005). Multimedia software evaluation

form for teachers. Education, 126(1), 100-111.

The article presents the study of a simple tool for quickly assessing a piece of educational software. The tool assesses software based upon four broad categories: content (and it’s ability to support objectives of the lesson), student involvement, ease of use, and design and esthetics. Each category consists of several qualifiers that determine overall instructional quality of the software. The tool itself is a series of yes or no checklists, designed for ease of use.

This particular tool has merit in its ability to quickly assess a piece of software quite objectively and thoroughly, and could easily be adapted to evaluate more than just software. While the instrument would need to be adapted, its overall design could help influence the rubrics being developed in class.

Holden, J. T., & Westfall, P. J. (2007, December). An instructional media selection guide for distance learning. United States Distance Learning Association.                     

When trying to determine the best type of media to use for a particular instructional context, this article helps identify strengths and weaknesses of each of the following types of instructional media: Asynchronous Web-Based Instruction, Audio Conferencing, Audiographics (such as audio conferencing while a shared image space, such as a presentation or an electronic whiteboard are used),  Computer-Based Instruction, Correspondence in Print, Instructional Television, Recorded Audio, Recorded Video, Satellite e-Learning, Synchronous Web-Based Instruction, and Video Teleconferencing.

This article helps narrow down the type of instructional media that may be the most likely to support a particular method of instruction, although once a type of instructional media is identified, this article won’t help evaluate the quality of specific media.

Gibbs, W., Craves, P. R., & Bernas, R. S. (2001). Evaluation of guidelines for multimedia
            courseware. Journal of Research on Technology in Education, 34(1), 2-17.

The evaluation guidelines from this assessment tool cover 14 broad categories, assessed with 14 a simple yes or no questions. The broad categories include: information content; information reliability; instructional adequacy; feedback and interactivity; clear, concise, and unbiased language; evidence of effectiveness; instructional planning; support issues; and interface design.

Two particular areas of assessment that stuck out in this particular tool include evidence of effectiveness and support issues. This tool requires both pre-instructional planning, but also requires the user to critically reflect upon the success of the lesson after lesson delivery. Also, this tool encourages the user to consider any support issues that could arise while using the particular media.

Leacock, T. L., & Nesbit, J. C. (2007). A framework for evaluating the quality of multimedia

learning resources. Educational Technology & Society, 10(2), 44-59.

The LORI (Learning Object Review Instrument), as explained in the article, is an evaluative instrument designed to work with nearly any type of media. The instrument evaluates media across nine dimensions: content quality, learning goal alignment, feedback and adaptation, motivation, presentation design, interaction usability, accessibility, reusability, and standards compliance.

While different media have different qualities that cannot always be addressed with one standardized evaluation tool, I find this singular tool could be easy to use with nearly any type of media. This would likely be the tool I would use as an active classroom teacher.

Incoming Reflection

Image courtesy of www.thewritingloft.com

In my seven years teaching elementary school, I’ve come across many dramatically different media resources. Early in my career I often made the mistake of finding “cool” resources, whether online or installed on the student workstations on the school network, and tried to find ways to fit them into daily instruction. This backwards thinking, taking the resource and adjusting instruction to fit it in, often resulted in lessons where my main objective was not achieved. I’ve also fallen into the trap of using a resource that seems to be absolutely perfect, then when trying to use it with students, come to find out many technical difficulties or programming limitations in which I hadn’t considered.

I have grown from the many media resource failures, and now have a better understanding of how a resource may work in the classroom. I also have a more innate understanding of what to look for when considering media resources. From this course I hope to become more proficient and evaluating and implementing resources, finding an efficient way to locate, evaluate, and implement resources that powerfully supplement and extend learning in my classroom. I’d also like to be able to share my learnings with colleagues of mine who may be more leery of using media resources after a few bad experiences.