Finished !!! Many thanks to Bronwyn for being so patient with me and apologies to the rest of the class for lagging so far behind.
Here is a summary of my evaluation report.
Executive summary
The purpose of this evaluation was to assess the effectiveness of the processes and resources that support new academic staff in using Otago Polytechnic’s (OPs) eLearning technologies (such as Moodle, Elluminate and eXe) for blended and distance delivery. This involved gathering information from several sources using a summative, “multiple methods” evaluation design. Eight new staff completed an online questionnaire indicating their use of technology prior to starting at OP, and the eLearning tools they were required to use once they commenced teaching at OP. In their first six months all the participants were required to learn how to use Moodle and most also had to use Elluminate. Most new staff relied on informal supports to learn how to use these technologies, including colleagues, Moodle champions, EDC staff and online Staff Moodle Resources. However, more than half of the participants were not aware of the range of support systems and resources available, and considered that they were not adequately prepared and supported to use OPs eLearning technologies. Five school managers were interviewed to find out about the processes in their school for introducing new staff to eLearning and eLearning technologies. No schools involved in this project had any formal procedures concerning technology support for new staff, and new staff were either directed to EDC staff, the school Moodle champion, technical support advisor within the school, or to colleagues. Recommendations from this evaluation include ensuring that information about support processes and resources is readily accessible to all staff, especially those who work closely with new staff; establishing a formal Induction process to ensure better initial support for new staff; and the need to continue to seek feedback on the effectiveness of the support processes and resources to ensure they are aligned with staff needs and effective practice.
In last weeks elluminate session we has a very useful discussion with Rachel Byars from Otago Polytechnic describing the needs analysis she carried out for her evaluation project for this course last year.
Her evaluation focussed on the suitability of eLearning options in teaching hotel management; her plan and project can be found here in the course guidebook. Rachel reinforced the benefits of thorough planning, asking the right questions, and using a mixed methods approach (triangulation) to gain a wider perspective. It was really valuable to hear how she had planned and carried out her evaluation, the discoveries she made, and how the outcomes have contributed to her teaching practice.
Thank you Rachel for sharing this with us.
For our second evaluation activity, Mareena and I evaluated OIL module 6: ‘Searching for Information’. In this module, students are taken through a step-by-step process showing how to analyse a topic and develop a search strategy to find relevant information, identify the principle resources for information used in academic study, and use search techniques to effectively find information. I know several of our staff at Otago Polytechnic direct their students to the OIL modules, and Module 6 is often used as a preparation exercise before going to the library for a hands-on with library staff. Then they have the online module to refer back to for reference.
The first thing I thought we needed to do was to put together a quick framework that we could evaluate this resource against. This included presentation (layout, readability etc), organization of content (logical sequence, chunking, navigation, links work, etc), resources and activities (information relevant and current, activities are appropriate and engage users, etc), support and help for users (clear guidelines/instructions provided, etc), and overall effectiveness of the module.
Our evaluation findings have already been posted to Mareena's blog, but I'll add them here for completeness.
Overall we agreed it was an effective learning object with a thoughtful design, easy to use interface, and useful resources.
The aims of lessons were clearly outlined at the very beginning of the module. The site allowed optimum readability with sufficient white space. There weren’t any long scrolling pages. The format was uncluttered and colour and fonts were used effectively and consistently. The writing style was clear and easy to understand. At the end of each lesson there were interactive quizzes to complete that provided instant feedback. This sequence was repeated for all lessons.
Lessons were organized in non threatening way with small chunks of information followed by graphic demos and animations which made the comprehension easier. Specific terminologies appeared in purple with a definition when the mouse hovered over the word/phrases. In addition there was also a glossary page at the end of the module.
The navigation menu appeared on the left hand side on each page offering a choice for the learners to complete the lessons at his/her discretion without following a linear order. There were graphical navigation buttons at the end of each page which allowed the learners to go back and forth from the lessons. There was also a handy print option available for selected pages or the whole module.
Issues / concerns:
We found some of the instructions were not very clear. For example, it was suggested there was a 'Help' for each search tool but it wasn't clear whether this was to be found in the module itself or in a real library search.
Some demos took a long time to download and Mareena lost interest and navigated away from the page, possibly because she was using a dial up connection. I’m on broadband and had no difficulty doing any of the activities or viewing any demos. We wondered if this problem was picked up during initial evaluations of the modules. If these modules were intended to be accessible to a wide range of learners then it needed to avoid slow downloads.
Lack of visual and audio technologies were obvious in the site. Highlighted terminologies with definition could have been much better if sound was attached to them. The option of listening to instructions could have been appealing to auditory learners. Catering for a variety of learning styles would help the learners to engage in the lessons. Incorporating different media types such as sound is something to consider in the future.
The module was very big and it would take a couple of hours to work through it. At the last elluminate session Alex commented that the learning units they develop take no more than 20 minutes to work through, in line with users attention spans.
Some sections are very specific to Otago University library services. The searching skills gained in this module should easily transfer to searching any library, but for novice learners it could be quite confusing. All the modules can be edited and republished (guidelines are provided in the Springboard module), so if this resource was to be used extensively in a particular setting it could be worthwhile customizing these with examples from your own library, although Bronwyn suggested this would require a bit of technical know-how to accomplish.
One of our first tasks in the evaluation course gave us an opportunity to have a look at a case study and get a taste of what evaluation is all about. I hooked up with Mareena for this exercise.
We looked at the Engineering Mechanics case study.
Here's the questions and responses.
What are the key issues?
Is eLearning appropriate for these students? Do they have the required abilities and skills? What support and training would students need to access and use eLearning resources.
He has limited time for developing interactive online resources but has found an online resource that looks useful. Does this resource provide knowledge and/or skills that will be useful for the job?
How can the problem be solved? What strategies can be used?
Need to assess students’ knowledge and use of computers; a questionnaire would be a good tool for this.
It would be useful to trial the eLearning resource with students; this could be done by observing the students using the resource and providing a follow-up interview or questionnaire.
Get feedback on the usefulness of the eLearning resource from teaching staff and from industry representatives; discussion group, individual interview or questionnaire.
How can the evaluation process assist in finding a solution? What type of evaluation is appropriate in this situation?
A well planned and carefully designed formative evaluation should provide information relevant to the specific issues being evaluated. In this example, the evaluation is being used to inform recommendations about the usefulness of eLearning in the course and the effectiveness of the eLearning resource in question.
The evaluation process involves clearly identifying the rationale, purpose and context of an evaluation. This information will then influence the type of evaluation and the choice of methods and instruments that will be used to collect data. Analysis of the data will provide information that can be used to make recommendations. A formal report detailing the evaluation process used and the final recommendations can then be prepared.
The type of evaluation used would be a ‘needs analysis’ to determine whether eLearning is appropriate for teaching and learning in this specific course. A thorough evaluation would help to inform teaching staff of the usefulness and effectiveness of eLearning as well as identifying what additional support students and staff might need, and ensure time and effort is not wasted on implementing and developing resources that may not be useful.
What is evaluation and why is it important?
Evaluation as described in the 'evaluation of eLearning' course, involves critically examining an online programme/course or some aspect of the teaching and learning in order to be able to make an informed judgement about its quality and success. Evaluation is an important component of all stages of instructional design and an essential part of monitoring and improving eLearning practice.
How do you evaluate? What is the evaluation process?
It is important to design and plan your evaluation carefully in order to get information that is useful. Planning should balance the most appropriate ways of gathering information with the time, resources and cost involved in carrying out the evaluation.
Evaluation can be carried out to trial a significant change or new innovative approach or resource prior to introducing it into a programme (needs assessment), during the development and delivery of the course (formative evaluation), and at the completion of a programme/course to determine if the learning outcomes have been met (summative evaluation). In my experience, formative evaluation often occurs in a fairly informal manner based on a teachers perceptions and reflections about how good something will be or has been in the classroom. A more formal evaluation is often carried out at the end of a course, using online or paper-based survey tools, to get feedback from students on student satisfaction, the quality and effectiveness of the learning activities, and the performance of the teacher. This information is used to improve the course in the future and for a teacher's performance review.
The first part of the evaluation process involves identifying the specific purpose and context of the evaluation: What am I evaluating and why? What are the key issues to be considered? What type of evaluation is needed? Who should be involved or consulted as part of the evaluation (stakeholders)?
Next, decide on the methods and instruments for collecting relevant data. A "multiple methods" approach involves using a range of different methods of sampling in order to gather information from many different perspectives (known as triangulation). Methods could include observations, questionnaires, checklists and focus groups. You would need to ensure that this phase is reasonably easy to manage with tools/methods that are simple to administer and organise.
Once the data has been collected, it is analyzed and interpreted, and used to make recommendations relevant to the purpose of the evaluation. A formal report detailing the evaluation process and the final recommendations can then be prepared.
Here's a link to Bronwyn's slide presentation about why evaluation is so important (20 mins). I love the chicken hat!
With my hours reduced this semester at work I'm relishing the opportunity to take a couple of papers as part of my ongoing professional development.
One of these is called Natural World Ko Te Tai Ao, a paper in the Certificate in Mata a Ao Maori offered at Otago Polytechnic, which explores the different views that Maori and non-Maori have in relation to the natural world.
The other papers are part of the Graduate Certificate in Applied eLearning from Manukau Institute of Technology; one of these is Evaluation of eLearning for Effective Practice with Bronwyn Hegarty and a second paper is on a Special Topic which is yet to be decided.
No doubt this is all going to keep me very busy.
A short video from Gilly Salmon about designing for online learning:
Facilitating an online environment has its own set of characteristics and challenges.
When designing online learning, it's more important to drive it from the learning challenge rather than the technology perspective.
You also need to design for student activity rather than delivery of content.
Unlike a classroom setting, all the materials need to be prepared and structured in advance.
Posts
