Symposium for Teaching and Learning with Technology

The presenters for the session are Jonathan Mathews, Kyle Peck and Joy Jin Mao. Jonathan Mathews opened the session by telling us about the project--constructs rubrics within a web page, acts as an interactive scoring rubric, and generate a web page with various form of feedback, and it is FREE.

Dr. Mathews uses the rubric in large classes and the tool ensures consistency among TAs, peer review and just-in-time knowledge and skills review. Can be useful in research as well in terms of giving feedback and responding to needs. The rubric was then demoed. Very flexible tool! You can build your own rubric quickly and you can use any grading scheme including decimals. Behind each cell you can put in feedback so there is interactivity. The professor can create the feedback tied to a particular cell. Feedback can also be grouped to deal with one issue at a time or can be aggregated to provide feedback on a whole topic, such as overall style.

He then went through a detailed example of a rubric to cover student papers: title, abstract, conclusions, references, etc. As the professor choses a block of feedback, the rubric automatically calculates the score. This is useful for training TAs in what is expected and leads to consistent marking. Also great for students to know exactly where they had issues. He augments feedback with audio files as well. Many students don't like to read feedback so audio or other media (Captivate, etc.) can be used to provide feedback to the students.

Question: Is the feedback built in ahead of time or customized per student?

Answer: Feedback is automatic but can be customized to address particular issues on a student-by-student address.

Now the professor can grade quickly and provide "expert" feedback to the students. Tries to put the feedback "in the faces" of the students so they are less likely to ignore it. Then played examples of audio and video examples. Media makes it hard to ignore the feedback!

Joy Jin Mao then discussed the peer assessment side of the rubric. Thanks to Drs. Kyle Peck and Jonathan Mathews for their help and support while she worked on her thesis based on peer assessments. There are advantages and disadvantages to peer review. There is a lot of literature about the advantages of peer review. Involving students in assessment helps develop student engagement and responsibility. Technology for virtual peer review -- not clear if technology improves the peer review process. There are many tools such as UCLA's CPR or Phil Davis' product as well as others. Future research will continue!

Dr. Kyle Peck then took the floor. Interested in ways peers can support each other. The key to increasing involvement and making education affordable is scaffolding such that assessment is not only the responsibility of the instructor. The idea is to create a rubric that carries information so the quality of information is better. It can also save time. He and his group thought to create a database such that it is easy to track problem areas. The feedback not only tells students what they did wrong but how to fix it. This is evolving.

Text is part of the feedback process. There are different levels of feedback. For example, this is what an abstract is, this is where your abstract rates, this is how to improve. If your mark is high enough, you don't get all the feedback.

Dr. Peck went on to talk about a project using the rubric template to help determine whether an instructor is ready for online teaching. Some rubrics force uniform responses or blocks across different issues but Dr. Peck's rubric can include binary responses (yes,no) as well as different choices on a continuum.

Jonathan Mathews resumed his PPT slide presentation and said that the rubric is freely available online at the e-Dutton institute. The site includes a tutorial on the use of the rubric.

Students really like audio files with feedback. We've built it. Will you use it? Some website urls of examples of rubrics were provided but, according to Mathews, these rubrics are not at good as his! :-)

Questions from the audience:

Q: Can you provide formative feedback using the rubric?

A: Yes. The order and feedback can be packaged differently for the whole project and the component parts. This is also a way to force them to use the feedback before the final project.

Q: Is an audio tool included?
A: No. You can use the streaming server and complete a link in the course.

Q: Have you observed any difference in students actually using the feedback over time? Any comments from students on the feedback they get?

A: Yes, students complain the feedback is "canned". May move to students marking themselves first and then submitting the papers.

Kyle Peck suggested using the tool to help students define what constitutes a good project, a multimedia project for example. Students have to think about the content.

Q: Do you still use your red pen?

A: No. I just use the rubric and then students are welcome to come see me. In person I can provide more meaningful feedback.

Joy added that sometimes it is difficult to know whether students benefited from the feedback or from the assessment process. This will be a source of more research in the future.

Dr. Mathews thanked attendees for coming and the session concluded.