Assignment 2

Assignment 2: Program Case Study

            The case study for evaluation is a prenatal fitness program designed to promote regular physical activity in Aboriginal women, in hopes of reducing their risk of developing gestational diabetes mellitus (GDM).  69 women from Saskatoon participated in the program, some of whom had a previous case of GDM. The program was held once a week and provided participants with free fitness instruction, as well as the opportunity to socialize with other mothers and receive educational materials and advice on health related issues.    

            To evaluate this program, I would utilize Stufflebeam’s CIPP model.  This would be a summative report as the program was completed in 1997 and the purpose of the evaluation would be to assess whether the program succeeded in meeting its objectives. Stufflebeam’s model is comprehensive and seeks information regarding a program’s context, input, process, and product. It is for these qualities and specific framework that I believe this model would be best suited in supplying stakeholders with the details needed to make informed decisions regarding the program’s merit.      

Evaluation Outline – CIPP Framework

Context – to assess the needs and problems associated with pregnant Aboriginal women and clearly understand the objectives for the program, I would interview program leaders and stake holders to gain further insight into the problem.  In addition, I would assess and confirm the background information that was collected on the women regarding their medical history and risk factors.  It would be valuable if baseline data regarding the women’s fitness levels was collected in order to compare to at the completion of the program.

Input – to determine the cost-effectiveness of the program and evaluate its strategies for implementation, a cost-benefit analysis would have to be conducted in order to determine if the program could be sustained in the long-term. To evaluate the program plans I would seek out similar programs to compare the model to.

Process – to evaluate the program’s activities I would interview or create a questionnaire for the beneficiaries and program facilitators in order to receive their assessment of the program’s implementation.  Questions that I would pose in the assessment would include: How did participants hear about the program? Did the once a week session meet their needs or should it have been held more often?  Did women benefit from the structure of the program including the educational materials provided and the opportunity to socialize with other mothers?      

Product – to identify and assess short and long term outcomes, I would collect both qualitative and quantitative data through interviews, questionnaires, and/or surveys. If baseline data was conducted at the onset of the program, the women would be given a post-test to determine if their fitness level had indeed improved.  The following questions would assist in this product evaluation:  Did the women continue to be physically active following the program’s completion? Did they develop GDM or type 2 diabetes? Did their children? Permitting time, cost, and resources, incorporating the use of a control group would be helpful in increasing the validity of the evaluation.

References

Dyck, R. F. (2003). Description and evaluation of a prenatal exercise program for urban Aboriginal women. Canadian Journal of Diabetes, 27(3), 231-238.

Stufflebeam, D. L. (2003, March). The CIPP Model for Evaluation: An update, a review of the model’s development, a checklist to guide implementation. In Annual Conference of the Oregon Program Evaluators Network (OPEN) Portland, Oregon.

             

Assignment One

ECUR 809

Program Evaluation Module 1: Assignment 1

Lauren Willey

P.A.R.T.Y. Program Evaluation 2006-2008

Program Overview

The Saskatoon Health Region P.A.R.T.Y. (Prevent Alcohol and Risk-Related Trauma in Youth) Program is a prevention and awareness program that is directed at youth to reduce preventable injuries and death.  With a goal of promoting smart choices and safe risks in high school students, this one day, in-hospital program exposes teens to the dangers associated with drinking and driving, riding without use of a seat belt, and not wearing a bike helmet. Put on by the Acquired Brain Outreach Team, the P.A.R.T.Y. Program runs in approximately 30 communities across Saskatchewan.   

Evaluation Model

            Following Stufflebeam’s CIPP model of evaluation, the purpose of this formative assessment was to determine the program’s effectiveness in relation to their short-term goals.  While the CIPP model defines evaluations as providing information regarding context, input, process, and products, this particular evaluation focused solely on the program’s product.  Stufflebeam and McKee (2003) noted that product evaluations could be divided into subcategories. In the case of the P.A.R.T.Y Program, evaluators used an effectiveness model to judge the success of the program. Motivated to assess outcomes and gage the program’s effectiveness in meeting their needs, the evaluators sought to decipher whether students’ attitudes and knowledge around risk taking behavior changed after attending the one day session. It was determined that the P.A.R.T.Y. Program is beneficial in improving students’ attitudes and risk taking behavior in the short term, but that further evaluations would have to be conducted in order to effectively assess the intermediate and long-term outcomes of the program. 

Method

            In order to assess the short-term outcomes of the program, the evaluators chose a quasi-experimental, pre-post design, with no control group.  408 grade 10 students from both urban and rural high schools were given a questionnaire one week prior to attending the program and were provided the same survey one week following their participation. The questions were a mix of attitudinal and knowledge-based questions surrounding injury and risk, with both qualitative and quantitative data being collected.

Strengths of Evaluation

            Although there are few strengths apparent in this program assessment, evaluators attempted to meet certain objectives suggested for an effectiveness evaluation. The first of these is that the results of the questionnaires were analyzed using statistically sound methods, helping contribute to the validity of the results. Secondly, the evaluators compared assessments of other P.A.R.T.Y. programs in an attempt to judge the effectiveness of the Saskatoon Health Region’s.  A last strength is that the evaluators acknowledged the weakness of their design and included recommendations for future modifications.    

Weaknesses of Evaluation

There are many changes that would need to be made to the evaluation and approach in order to make it reliable and beneficial to stakeholders. The primary weakness of the evaluation is that there were no suggestions made for strengthening and improving the program, a vital component of a formative assessment. Instead, the conclusion of the evaluation described the modifications that would need to be made to the assessment’s approach in order to increase the validity of the findings and provide a more accurate appraisal of the program’s outcomes.  These suggestions included (a) using a control school to compare results to; (b) using a one year follow up time frame instead of two weeks, and (c) going beyond the questionnaire by assessing behavioral change using the driving records of those who participated in the program.  The latter recommendation would be beneficial as changes to students’ attitudes and knowledge surrounding risk-taking behaviors does not necessarily translate into a reduction in risk taking behavior.  A final weakness to the report is that there was no mention of the program’s stakeholders or who the audience of the assessment would be.

In conclusion, the evaluation was not well planned or executed. To provide a more reliable and valuable assessment in the future, the evaluation should adhere to the recommendations provided, as well as offer stakeholders feedback for program improvement.

References

Kershaw T., Marko J & Gerwing, J. (2010). P.A.R.T.Y. Program Evaluation 2006-2008. Saskatoon: Saskatoon Health Region.

Stufflebeam, D. (2002). The CIPP model for evaluation. Evaluation models, 279-317.

 http://www.saskatoonhealthregion.ca/your_health/documents/PHO/PartyProgramEvaluationReport.pdf