Assignment #5 - Surveys

For my project I felt as though two surveys would be more useful.  Since I needed to collect information about the Martin Aboriginal Education Initiative from a student and staff perspective I felt two different surveys were necessary to analyze information.  I proceeded to create two surveys with Survey Monkey and gave them to 4 staff members for proofreading purposes.  The only adjustments that were really made from their advice was rewording a few questions and adding a few additional options as answers in the questions that I had not previously thought about beforehand.  I did run into a significant issue regarding the survey with the website I chose to use for data collection.  The site will not allow you to make more than 10 questions in a survey, export the survey to a .pdf or copy and paste the information in the survey without subscribing to their site with a monthly fee.  This became an issue for the purpose of this assignment as I was then unable to remove it from the website once I had created it.  Rather than finding a new site and starting all over again and potentially not having the same outline or choices, I chose to take screen captures of the survey to post to my blog.  This will appear quite disjointed and would be worse if I posted the original surveys as well as the final drafts.  Additionally, I was not aware that I should have posted the original surveys before I had already made the changes to the rough draft.  So there are six images that will appear below.  The first three pertain to the staff survey and the following three pertain to the student survey. I don't think I will be using survey monkey again in the future unless I need a short survey for online purposes only. What a mess!


Staff Survey:

Student Survey:

Assignment #4 Logic Model

Assignment #3 Evaluation Worksheet

Planning a Program Evaluation: Worksheet

Steps in Program Evaluation

Manage the evaluation § Human Subjects Protection § Timeline § Responsibilities § Budget		1. Engage stakeholders
		2. Focus
		§ Describe program-logic model § Define purpose § Determine use/users § Determine key questions § Select indicators § Determine design	3. Collect data
			§ Identify sources § Select method(s) § Pilot test § Set schedule § Determine sample	4. Analyze & interpret
				§ Process data § Analyze § Interpret data § What did you learn? § What are the limitations?	5. Use
					§ Share findings and lessons learned § Use in decision making § Determine next steps
Standards of evaluation: · Utility     · Feasibility     · Propriety     ·Accuracy

           Engage Stakeholders

Who should be involved?

Mount Royal Collegiate, Classroom teacher, Paul Martin Education Program Rep., Aboriginal students, program sponsors

How might they be engaged?

Hearing about interest level and success of students involved and seeing students pursue business opportunities in the future.

           Focus the Evaluation

What are you going to evaluate?  Describe program (logic model).

The Martin Aboriginal Education Program is a course designed to improve the graduation rates of Aboriginal students in Canada by introducing them to entrepreneurship and business opportunities.  They will be able to learn business strategies and methods to give them an advantage in the work force after high school.  If students wish to pursue a business after high school they will be granted $500 to assist with their startup costs. I will evaluate if the information they are receiving will prepare them to seek business opportunities in the future as well as if they are actually planning on seeking business opportunities after high school. In addition, I will evaluate if the funding they receive as a program is adequate and allocated appropriately.

What is the purpose of the evaluation?

The purpose of the evaluation will be to determine if the program is providing the necessary skills to Aboriginal youth who are interested in beginning their own business in the future through a review of their available resources and engagement level.

Who will use the evaluation? How will they use it?

Who/users	How will they use the information?
Classroom Teacher	To adapt instruction to accommodate student learning and success
Martin Aboriginal Education Program	To review the coordination of funding and available resources to the classroom teacher
Program Sponsors	To review the funding and practical knowledge provided to students involved in the program

What questions will the evaluation seek to answer?

Is the funding model for the program adequate?

Are enough resources available to the students and classroom teacher?

Is the program actually seeing Aboriginal students pursuing business careers in the future?

What information do you need to answer the questions?

What I wish to know	Indicators – How will I know it?
If the funding provided to Mount Royal students is adequate.	Reviewing what is being done in the program and how funding is allocated to the students. Need to determine if there are opportunities being missed due to insufficient funds.
Are students gaining valuable information from the class	Interviewing or surveying students
Are teachers seeing an interest level from the students	Interviewing or surveying the teachers
Are students choosing to pursue business ventures in the future?	Reviewing statistics on the number of students, that were part of the program, seeking business opportunities after high school

When is the evaluation needed?

By the end of the school year to allow for adjustments of the program for the 2014 school year.

What evaluation design will you use?

For this evaluation I will be using Scriven’s approach of a blended formative and goal-oriented evaluation.  This will allow for on-going information to help adapt the program in the future but at the same time evaluating if the program is achieving its intended goals.

           Collect the information

What sources of information will you use?

Existing information:	Martin Aboriginal Education Program website, program brochures, course outlines and resources
People:	Teacher, students, school administrators, MAEP administrators, business mentors
Pictorial records and observations:	Surveys, resource documents, and viewing lessons.

What data collection method(s) will you use?

x	Survey	x	Document review
x	Interview	x	Testimonials
x	Observation		Expert panel
	Group techniques		Simulated problems or situations
	Case study		Journal, log, diary
	Tests		Unobtrusive measures
	Photos, videos		Other (list)

Instrumentation: What is needed to record the information?

Online survey, voice recorder, program documents and resource list, and rooms for interviews.

When will you collect data for each method you’ve chosen?

Method	Before program	During program	Immediately after	Later
Survey	x	x		x
Interview	x	x	x
Observation		x
Document Review	x	x
Testimonials		x		x

Will a sample be used? 

No
Yes	x	If yes, describe the procedure you will use.
A sample will be used for interview purposes only.  The class size only consists of 10 students at this point so information should be able to be gathered from each individual student.

Pilot testing: when, where, how? 

This program is already in a pilot testing phase so the evaluation will just add to the review.

           Analyze and Interpret

How will the data be analyzed?

Data analysis methods:	Surveys will be compiled into a spreadsheet for analyzing. Interviews, observations, and testimonials will be audio recorded and listened to afterward to record information.
Who responsible:	The evaluator will be responsible for gathering and analyzing the data, as there will only be a smaller amount of data to collect from the group.

How will the information be interpreted—by whom?

Information will be interpreted by the evaluator with some collaboration with the classroom teacher.

What did you learn? What are the limitations?

Limitations will be getting unbiased and honest responses from the students as they may feel their answers could impact their mark in the class.  Due to the program being early in its stages it may be difficult to get fully informed answers from staff and students.  There may also be trouble getting information from parents due to the lower involvement of parents with this school.

               Use the Information

How will the evaluation be communicated and shared?

To whom	When/where/how to present
MAEP	In the summer through a written report, statistics, and graphs to allow for any alterations to the program for the 2014 school year
Classroom teacher	At the end of the school year through a collection of statistics and graphs indicating areas of success and improvement
Program sponsors	In the summer through a written report, statistics, and graphs to allow for any alterations to the program for the 2014 school year.
School Administrators	At the end of the school year through a collection of statistics and graphs indicating areas of success and improvement.

Next steps?

Gathering and analyzing data for the groups listed above in order to help improve and adjust parts of the program or provide details on the successes of the program to community members who don’t have the opportunity to regularly hear about the dynamics of the program.

           Manage the evaluation                   Standards

x	Human subject’s protection	x	Utility
	Management chart		Feasibility
	Timeline		Propriety
x	Responsibilities	x	Accuracy
x	Budget