Instrumentation Process Reflection

Image source: blogs.seattleweekly.com

1.    As my group continues to construct our evaluation instrument, my thoughts and ideas changed from what I began formulating in my mind after viewing several different instruments as identified in my annotated bibliography. I always believed that important characteristics of any media that warrant evaluation included quality of the production, correlation to teaching and learning objectives, as well as appropriate level of content. However, I hadn't really put much thought into considering Bloom's Taxonomy when evaluating media. I certainly never really considered evaluating the extent in which media will be received by a student group in regards to learning context as well as learner background and culture.
      I also found it difficult to think broadly, especially in relation to the baseline requirements. While researching different evaluation instruments while working on my annotated bibliography, I came to realize that many of the instruments in which I felt would be easiest to use and most likely to be put into service in instructional planning were either some form of Likert scale or a checklist. Additionally, most commonly the instruments were standardized in formatting, something I noted added to the ease in which they could be used. Our group decided to build a Likert scaled rubric for the baseline requirements, and  I ended up finding out that some of the categories didn't seem to work well as a scale.

2.   Collaboratively building this instrument across both time and distance has proven to be challenging. While using a wiki as a shared space to collect thinking and track changes was effective, it wasn't the most efficient way to collaborate. For example, the first time visited the wiki to begin work, I had several ideas in my head. However, as I was not the first group member to begin work, a great deal of work was already started, in a direction different than anticipated. While an absolute affirmation for group work is combining multiple perspectives, using an asynchronous method of meeting, some ideas were marginalized or not shared.
      While trying to stick with a format that has great promise in it's ease of use, specifically using a rubric based upon a Likert scale, many categories didn't work well in that particular format. As categories were placed, the Likert scale felt artificial or forced. An example comes in the baseline requirement regarding the consideration of students' background. How can you objectively quantify how well media considers a student's background? While we did find it important to stick with a particular format to aid in the ease of use, it was difficult finding that balance between ease of use and effectiveness of the instrument.

Annotated Bibliography Comparison

Image from www.signandtrade.com

When comparing the resources I chose for my annotated bibliography with the resources as chosen by Cami, one difference that stood out between the majority of articles selected was the type of information that was collected. The articles I was drawn to tended to be more quantitative in their analysis of media. A specific qualifier is identified, and the media is rated based upon quality. Cami selected many articles that were more qualitative in their analysis. Many of her articles simply asked open ended questions to help the evaluator think about the quality of media and come to a decision independently. I also like the fact that she included an evaluation tool that can be taught to and used by students.

I did notice that both Cami and I approached this assignment in a similar fashion: we cast our nets wide. We looked at evaluation of very specific media, hoping to generalize the findings to all media. Cami found an article created to evaluate Flash software. While the evaluation may have been created specifically for Flash, there may be some unique ideas that could be included when we create our own rubrics. We also both looked for resources regarding the evaluation of technology usage in general, which is a much broader focus than, for example, online video.

To be honest, I was a little surprised Cami and I didn’t have any matches. Clearly we were both looking for resources that were current. While I was researching, I was frustrated with the lack of current articles specifically addressing media evaluation. Most articles I found were either out of date, slightly off-topic, or not a tool that could be applied to current media. I don’t recall coming across any of the articles cited by Cami.