Methods and Sample

Introduction
This evaluation plan describes the procedures that will be used to evaluate Nutrition and Cancer, a module developed by Medeor Interactive. The Nutrition and Cancer module is used as an adjunct instructional tool in FDNS 8530, a required elective for graduate students in the Department of Foods and Nutrition at the University of Georgia. The background of the module, the purposes of the evaluation, the intended decisions and questions, the proposed methods and instrumentation, the selected sample and inherent limitations, logistics and timeline, and the budget for the evaluation are included in this document.

This evaluation plan has been prepared by Kim Gibson, a graduate student in the Department of Instructional Technology and the Department of Foods and Nutrition at the University of Georgia. The plan has been informed by discussions with the client, Dr. Joan Fischer of the Department of Foods and Nutrition, the supervising instructor, Dr. Tom Reeves of the Department of Instructional Technology, and a review of published literature related to the module.

Background
Nutrition and Cancer is a module in the 10-module Nutrition and Medicine physician series. The Nutrition and Medicine physician series was developed by Medeor Interactive, a company created by Dr. Steven Zeisel of the University of North Carolina-Chapel Hill Medical School. The series was developed due to a recognized need for consistent and comprehensive nutrition education in medical schools. Ten modules were developed to provide a consistent nutrition education curriculum that combined the biochemical, pathophysiological, and clinical aspects of diet-disease relationships and lifecycle conditions. Each module focuses on a specific disease state or lifecycle phase. The modules combine video and case-based interactions to utilize a problem-based pedagogy.

The modules are distributed on a CD-Rom platform and are distributed without cost to medical schools. It is currently estimated that 60% of medical schools in the United States now use the Nutrition and Medicine series in some capacity (Medeor Interactive, 2002).

As shown in the following screen capture of the menu interface, the module is divided into 5 sections:
• Molecular Events in Carcinogenesis
• Tumor Initiation
• Tumor Promotion and Progression
• Nutritional Epidemiology
• Dietary Guidelines

Each section presents a different application of nutrition science, clinical nutrition or nutritional epidemiology to the disease process of cancer. The module also contains three practice multiple-choice board examinations to provide practice for medical board examinations.

Using the example of the Molecular Events in Carcinogenesis, screen captures are provided below to further detail the learning activities within a section of the module. As shown in the following screen capture, at the beginning of each section, the learning objectives of that particular section are delineated.

In each section, the student views a case-based video scenario of a patient, Mrs. Sanders, and her family following her recent diagnosis of colon cancer. Some of the video scenarios focus on Mrs. Sanders, however, the patient case that is presented is Jack Sanders, Mrs. Sander’s son, as he deals with his mother’s diagnosis as well as takes steps to lower his own risk of colon cancer. As shown in the following screen capture, the video is divided into Part 1 through Part 6; one part per section as well as a summary part at the end of the module.

The student is asked to pause the video to read the blue-tabbed information areas that represent the patient’s medical chart: History, Exam, Labs, and Notes. The following screen capture displays the History tab of Jack’s chart.

As shown in the following screen captures, each section also provides static and animated images that are accompanied by audio to explain the cancer-related concepts of that section.

An example of a board examination question is shown in the following screen capture. The module provides three sample examinations; each examination includes twenty multiple choice questions. The examination is graded and the correct answers are displayed after the student clicks “Finish”.

The modules in the physician series have also captured the interest of other health disciplines, including nutrition and dietetics educators. As a result, the modules have been used within nutrition graduate level courses at some universities. As a side note, Medeor Interactive has expanded the original physician series to include a nutritionist series and a nursing series. Some nutrition and dietetics educators have indicated that the nutrition science information is covered more in depth in the physician series and as a result have opted to use the physician modules instead of the nutritionist series modules.

The Nutrition and Cancer module of the physician series is used in the FDNS 8350 course, Nutrition and Disease Processes I. The course is coordinated by Dr. Joan Fischer of the Department of Foods and Nutrition. The course is team taught by five faculty members of the Department of Foods and Nutrition, College of Family and Consumer Sciences and three faculty members of the Department of Pharmaceutical and Biomedical Sciences, College of Pharmacy.

The course consists of lectures from these faculty experts who conduct epidemiological, clinical, animal, and/or cellular studies linking diet with heart disease and cancer. This is the first time that the Nutrition and Cancer module will be included in the course as a supplemental instructional tool. The module will be completed by the student outside of the classroom and will count as 5% of the student’s final course grade.

Purposes
The purpose of this evaluation is to provide timely and accurate data for use by the course coordinator with decisions related to the use of the module in future semesters of the course.

The specific sub-purposes:
• To collect information regarding the acceptability, applicability, and effectiveness from the target audience of students enrolled in the course

• To collect information regarding the content accuracy, applicability, and appropriateness from clinical-based and research-based experts in the areas of nutrition and cancer

This evaluation is summative in the fact that it will not be used by the product developers to enhance future versions of the product. The evaluation will be used to evaluate the effectiveness of the Nutrition and Cancer module within a nutrition graduate course.

Audiences
The primary stakeholders of this evaluation will be the course coordinator, Dr. Joan Fischer and the students of the Fall 2002 Nutrition and Disease Processes I course. Relevant secondary stakeholders include supporting faculty of this course and course coordinators of similar courses.

The designer and implementer of this evaluation plan is Kim Gibson, a student enrolled in EDIT 8350 Instructional Product Evaluation, with the guidance of the course instructor, Dr. Thomas Reeves of the Department of Instructional Technology, College of Education.

Decisions and Questions
The evaluation will be used to inform two major decisions to be made by the client and course coordinator, Dr. Joan Fischer. The following decisions are assumed:

• Should the module be used in future semesters of the course?

• Is the module appropriate and applicable to students enrolled in a graduate nutrition course rather than its intended use with students enrolled in medical school?

Questions that will supply data for these decisions are:
1. Student input
Effectiveness
• Does the student perceive he/she learned new information?
• Content-specific questions: 6 pre- and post-test responses. (Appendix A)

Acceptability
• Does the student enjoy using the module?

Content applicability
• Does the student think that the general information provided by the module was relevant to their current research environment or future career plans?
• Does the student perceive that the information or video scenarios have relevancy to other aspects of their lives?

2. Expert input
Content accuracy
• Are the concepts of carcinogenesis and tumor initiation accurate (sections 1 & 2)?
• Are the concepts of tumor promotion and progression accurate (section 3)?
• Are the nutritional epidemiology applications and dietary guidelines accurate (sections 4 & 5)?
• Are the problem-based video scenarios and supporting chart data accurate from a clinical perspective (video parts 1 – 6)?

Content applicability and appropriateness
• Is the content of the module applicable to a dietetics practitioner in a clinical environment?
• Is the content of the module applicable to a nutrition science researcher?
• Is the content appropriate for Master’s level and Doctorate level students in the Department of Foods and Nutrition?


Methods
Multiple methods of data collection will be used to answer these questions. These methods include:

          User knowledge pre-test and post-test
          User questionnaire
          Mini-focus group with subset of students
          Expert reviews

Evaluation Matrix

Sample
The sample for this evaluation will be 10 – 15 students enrolled in FDNS 8530, Diet and Chronic Disease Processes I. The participants in the evaluation have a variety of undergraduate education backgrounds including exercise physiology, hospitality management, nutrition and dietetics, nutrition science, chemistry, biology, and biochemistry. The students also represent a variety of age groups. The students are predominantly female with only one of the 15 students being male.

All 15 students will be asked to complete pre- and post- tests and user questionnaires. A selected group of students (4-5) will be asked to participate in a mini-focus group following administration of the knowledge tests and the student questionnaire. These students will be selected according to their willingness to participate as well as the diversity of their educational backgrounds.

The expert reviewers will include Dr. Joan Fischer, course instructor and faculty head of the Clinical Studies lab in the Department of Foods and Nutrition, a member of the Community Nutrition Intervention lab in the Department of Foods and Nutrition, and a clinical dietitian. These experts were chosen due to their work and/or research experience related to diet and cancer, nutritional epidemiology, and clinical dietetics.

Instrumentation
The instruments used to gather data for the evaluation include: (all documents in MS Word)
              User knowledge pre-test
              User knowledge post-test
              User questionnaire
              Focus group questions
              Expert review checklist - clinical dietetics
              Expert review checklist - nutritional epidemiology
              Expert review checklist - nutrition science

The pre- and post- knowledge tests will consist of six multiple choice questions that are taken directly from examinations on the Nutrition and Cancer module. The users will answer the six questions prior to using the module and then again after using the module. Pre- and post-test answers will be compared.

The user questionnaire will be administered following the completion of the modules by the students. The questionnaire will be administered during the last ten minutes of one class period. The user questionnaire will be reviewed by an experienced evaluator to help assure wording is appropriate.

The focus group questions will be administered to a selected group of students who agree to participate in a mini-focus group after the questionnaire administration.

The expert review checklists and the CD-Rom will be given to the reviewers to complete and return to the evaluator.

Limitations
The limitations of this evaluation include that a relatively small sample of students is involved. Within this small sample, there are students with a variety of educational and experiential backgrounds. It is known that at least one student is actively conducting research which explores diet-cancer relationships, while other students may have never taken a nutrition graduate course. Students may have varying levels of interest in the topic depending on their personal health interests and family health histories. It can be anticipated that these students will differ greatly in terms of their pre-module knowledge related to diet and cancer. As a result of their experiences, the students may also differ in terms of their expectations and interest in the module.

The students will also be attending class lectures about various topics related to nutrition and cancer. Any change in user post-module knowledge may not be directly attributable to the content of the module. It is anticipated the content will also be presented to the students via class lectures, discussions, and student presentations.

Another potential limitation is that the evaluator is also a student in the classroom. It is possible that the evaluator’s personal experiences with the module may influence her evaluation of the module.

The module covers a variety of sub-topics within the field of nutrition: nutrition epidemiology, nutrition intervention, nutrition science, nutritional aspects of gene expression and clinical dietetics. It would be challenging for a single expert to evaluate the accuracy of the information in these areas. As a result, three different experts will be asked to evaluate the module to help ensure that content accuracy is measured across these areas of nutrition.

Another limitation of the evaluation is that none of the instruments have been tested for validity and reliability. Due to the time constraints of the semester, it will not be feasible to conduct validity and reliability testing. The instruments will be reviewed by an experienced evaluator, Dr. Thomas Reeves.

Logistics and Time Line
This evaluation is being designed and implemented by Kim Gibson, with the guidance of Dr. Tom Reeves, Professor in the Department of Instructional Technology at the University of Georgia. Kim Gibson is responsible for the data collection, analysis, and preparation of the final evaluation report.

Students will be asked to complete the user knowledge pre-test during the first week of October prior to receiving the Nutrition and Cancer module. The expert reviewers will be asked to view the module beginning in mid-October and continuing through mid-November until all three reviewers have completed the checklists. The students will be asked to complete a user knowledge post-test and student questionnaire during the second week of November. Following the questionnaire administration, a mini-focus group with a selected group of students will also be conducted during the second week of November. Analysis of the data will be conducted during the remainder of November. The final report will be prepared and submitted by December 2, 2003.

Budget
Note: The following budget is entirely hypothetical and wishful. No wages or incentives will be disbursed.