Program Evaluation

Heather: Reviewed & condensed student feedback into a final “quality” piece, reviewed & condensed recommended amendments, created table and contributed to the reflection.  

I commented on both self-directed learning Program Evaluations and the embodied learning Program Evaluation.

Bridgit: Identified & reached out to a professional evaluator, reviewed & condensed the professional evaluator’s feedback into a final “quality” piece, and contributed to the reflection. 

I commented on Group 2 Transformational Learning and Group 4 Embodied Learning Program Evaluations.

Allison: Facilitated initial communication/created google docs, reviewed & condensed student feedback into a final “quality” piece, edited table, contributed to reflection and unified reflections, edited and combined all pieces and published to blog. 

I commented on narrative learning.

Will: Identified & reached out to a professional evaluator, reviewed & condensed the professional evaluator’s feedback into a final “quality” piece, and contributed to the reflection. 

I commented on: Narritive Learning Group 5 on 11/7

An Evaluation of a New Supervisor Training Program Design

Heather Allen, Bridgit Bucher, Will Kerch, Allison Wynbissinger

Ball State University

EDAC 634 Fall 2016

Professional Evaluation

Roles, Positions, and Qualifications

Mr. Robby Tompkins has spent twelve years in the field of nonprofit development for the greater Muncie area. Currently, Robby is working as Director of Development for LifeStream Services. LifeStream is a nonprofit organization which provides services and aide to support senior citizens and those living with disabilities.  Robby oversees all LifeStream functions corresponding to fundraising, donor and corporate relations, events, public relations, volunteering, reception, and information technology. In addition to overseeing these functions, he is responsible for supervising the employees that work in these areas.  This includes hiring, firing, training, and staff coordination.

Ms. Rozlyn Hernandez is the Director of the Express Enrollment Center at the Lafayette campus of Ivy Tech Community College. The Express Enrollment Center is a relatively new concept at Ivy Tech, at just over three years in existence. As the inaugural director, she has been instrumental in setting the direction of the department. This has been done by evaluating the experiences of staff and students’ feedback, then updating methods to meet the needs of the students.  Rozlyn studied Education at Purdue University and is well versed in the various ways students learn. On a daily basis, she facilitates training among her staff members. She also currently presents a training and development program that is specific to new supervisors with an emphasis on problem solving, team improvement, and Strengths Based Leadership, among other methods.  

Feedback

One of the major advantages of our program that Mr. Tompkins identified was our ability to use several different learning styles and approaches to give participants the capacity to learn through experience. Being able to reflect on these experiences through journaling was another aspect that was noted by him as something that supervisors could take advantage of. According to the feedback provided, supervisors need to sometimes take time to reflect on decisions and journaling can help with this process. Finally Robby addressed our program’s strength in helping supervisors in the hiring and firing process. This area of supervision is known to be difficult and our program helps supervisors learn which experiences are valuable to learn from.

            As for improvements, the first that was addressed was the fact that our program will not be able to cover all of the experiences that employers might face in their positions. Having a supervisor that can handle the “what if” situations is a great quality that not everyone has. The second major improvement that was provided to our group was the importance of supervisors knowing the job functions of the people they oversee. Robby thought our program emphasizes too much on the experiences of supervisors but neglects the role of the employees. Giving supervisors the perspective of the people they manage will help them better utilize those individuals in an effective and respectful manner.

Rozlyn has had a great deal of success with StrengthsQuest, which is a program that helps identify a learner’s natural strengths so that those strengths can be further improved. Using tools like this may help free up the facilitator’s time for other pursuits during the supervisor training. As an example, the facilitator would have more time to put together on-boarding information, presenting HR policies to new supervisors, developing scenarios for new supervisors to discuss during the training, and coming up with specific questions for supervisors to answer for their journals. She thought the section that covered recruiting, interviewing, hiring, and onboarding would be extremely helpful to new supervisors.

Student Evaluation

After reviewing the seven comments that we received on our blog, by classmates, we found that the overwhelming majority was positive and affirming of the approach we have taken to make our program design a reality. Numerous co-learners praised our attention to detail and the organization of our plan, which we believe will aide in a seamless implementation. More than one person noted that they agreed surveys could be a good tool to use for pre and post assessment, and that getting feedback on our training program, to know whether it is truly useful or not, will be vital. Multiple people commented on the utilization of reflection being a strong strategic approach and that our plan to use journaling as a means to this end was an excellent choice. With that said, there was feedback that some trainees in the program may limit their honest introspection in their journal entries, knowing that someone else was going to read their thoughts. This is a great point to take into consideration, as we would want participants to be fully honest in order to get the most value from the exercise. Because nearly all the student feedback we received was positive, we were in agreement with our classmates; we do believe we have created a strong proposal. While we appreciate the positive reactions, we do wish we had received more critical feedback from our peers, because we know that it is through constructive criticism that we are best able to improve our design.  

Amendments

We believe there is always room for improvement and this program design is no exception and even thought we received a great deal of positive feedback, we wanted to highlight how we would incorporate their recommendations into our design. We do agree with the evaluation that some training participants may be reluctant to be fully honest, knowing that someone will read their thoughts. To take this into consideration, we will assign some journal entries as “public” and give the trainees opportunities to formulate “private” posts as well. These posts will not be read by the facilitator but done in class, to assure the reflection is being completed, but eliminating the need to read the students’ private thoughts.

            We appreciated the feedback regarding the pre-course surveys. One professional recommended StrengthsQuest, which is a tool that allows for learner self-assessment, providing pre-work for participants that will increase engagement by allowing them to assess their individual opportunities. We also agreed with the benefits of moving students around throughout the learning event by switching up partners and groups. This would also allow participants to engage with learners from different backgrounds and experiences. We also believe that allowing trainees to pair and group themselves creates an environment of openness and conversation that might not otherwise occur.

We have changed the exercise where a trainee would share an example of something a supervisor did not handle well, and have decided to create scenario cards, based on the professionals’ feedback.  This will allow us to better direct the course of conversation. Another piece of feedback that was particularly valuable was around the journaling exercises. We have changed this activity so that the facilitator will provide specific questions for the students to reflect on after each session. Finally, we will build in a discussion around job shadowing, as well as allowing time for trainees to discuss what other situations they may encounter in their specific industries that were not yet discussed.

Reflection

For this assignment in particular we were really able to appreciate the work and effort that our group put into making our program design. While it is not a perfect design and there are many things we can do to improve upon its functions we were all proud of it! We again worked together very efficiently and each commented on our gratefulness for the “team” approach we have taken in equally contributing and completing course work. To accomplish this assignment we anticipate one of the more difficult parts was going to be finding professionals to evaluate our project; fortunately though, it was much easier than expected and we identified our evaluators by the first day after we submitted our program design! Before the reviewers had finished reviewing the program, they were asked to emphasize items that they would change as we did not want them to not point out areas for improvement, because then we would miss out on valuable learning opportunities. Some lessons and tips that we would offer for the evaluation process:

·         Feedback is not necessarily a bad thing! If you have something that can be improved you should embrace it and do everything you can to make the necessary changes.

·         Even though positive feedback “feels” nice, the critiques and recommendations are what are most helpful to better develop and expand on your program.  

·         Quickly identify professionals who would be willing to evaluate your design, so that they are not rushed in their assessment and so that you have enough time to finish the assignment.

·         Begin redesigning your project as your feedback comes in, from each evaluator and student, so that the process isn’t overwhelming down the road.

Conclusion

Every person who gave feedback regarding our program design is appreciated and the comments they offered were not only taken into consideration but many were applied to our Supervisor Training Program to better it. Based on these suggestions for improvement, our group has begun making revisions and adjustments, which can be found in the table below.

	Evaluators	Ideas for Improving program design	Revisions/Our responses
1	Robby Tompkins, Life Stream	i) Supervisors need to be able to manage the unexpected, “what-if” moments because everything cannot be taught ii) Supervisors shadowing their subordinates to understand the roles they are managing	i) Build-in time to allow trainees to brainstorm situations that were not covered and resolve as a group ii) Add in time to talk about the importance of trainees’ job shadowing their supervisees. Also discuss case studies that have shown this to be a successful endeavor
2	Rozlyn Hernandez, Ivy Tech	i) Utilizing StrengthsQuest vs. pre-course surveys ii) Building more structure around the journaling activities	i) Change the pre-course survey model to the StrengthsQuest model, to allow self-assessment and to increase engagement before the training course begins ii) Create questions for students to answer in their journal after each lesson to provide opportunity for reflection
3	Classmates (Laverne & Lecia)	i) Create a feedback mechanism to obtain evidence of the course’s success i) Students may not provide authentic journal feedback knowing their responses will be read	i) Through the journaling experience, incorporate questions that are designed to evaluate the course itself ii) Create a means for anonymous feedback by providing not only public but also private journal entries

Per the syllabus, we invited two professionals to evaluate our program design. Here you will find the professionals' original evaluation documents.

Professional Evaluations (Original)

Robby Tompkin’s Evaluation

Roles & Qualifications. I have been in a supervisory role for 12 years in various non-profits. I am the Director of Development at LifeStream Services, which is the Area Agency on Aging for the seven counties in our area here in East Central Indiana. We are a non-profit that receives federal and state funding to provide services and support to seniors and those with disabilities. LifeStream has an annual budget of $7.9 million and we have 84 employees in seven offices throughout the region. I oversee the departments relating to all fundraising, donor and corporate relations, events, public relations, volunteering, reception, and information technology. I am responsible for hiring, supervising, training, and coordinating with the staff in each area.

My past experiences include being the Executive Director at Cornerstone Center for the Arts for four years. There were 66 employees and I was responsible for all aspects of the organizational leadership and hiring/training/supervising of director roles and key positions in finance, facilities, and community relations. Prior to that I served as the Director of Education at Cornerstone for four years, directly hiring and supervising 29 staff members including associate directors, assistants, teachers, and support staff.

My degree is in Non-Profit Administration with concentrations on leadership and development. I am a past founding Board member and past officer for the Rush County Arts Council and the Muncie Arts and Cultural Council. I served on the Board of Directors for Muncie Civic Theatre for six years and served on the hiring committee for two Executive Directors. I was the Festival Director for the Harvest Moon Film Festival from 2007-2009 in Muncie.

I am a past Top 20 Under 40 Award Recipient, a Ball Brothers Foundation Fellow, and Shafer Leadership graduate. In addition, I own a consulting firm that has worked with a dozen other non-profits in the areas of strategic planning, Board development, leadership training, fundraising, and promotions.

What do you like about the program design? The design you have created involves several learning styles and approaches, which I think is a strength in its ability to adapt to different personality and leadership types. The use of sharing examples from the past, role-playing, and mock situational analysis will help those that gain more from example-based learning. Obviously, knowing and doing are completely different things, but I appreciate the thoroughness of your program to not only engage students in examples, but also to journal and reflect upon situations and experiential learning.

Too often, supervisors don’t take the time to pause and make a complete decision after a brief period of consideration. The feeling is that snap judgments and quick decisions have to happen in order to keep progress rolling ahead. Often times, this is caused by poor structure and lack of a clear chain of communicating needs from the staff they supervise and other department heads. I appreciate that your program helps to instill a sense of responsibility with the students in making sure they process it and think through it.

I think your program does a good job of addressing the essentials such as proper interview techniques and documentation for employees. Hiring and firing are often thought to be similar and have a direct correlation, but in my experience both acts are unique skillsets that are imperative for any supervisor to master. Knowing who to bring on the team and knowing when to show them the door are extremely important for the success of any group or business.

I appreciated that you addressed the pressures to fill a position versus waiting for the correct candidate. It can be dreadful to not have a position filled for a long period of time, but the consequences of putting the wrong person in that role is far more devastating than new supervisors often realize. I had to fill a staff accountant position three times in one year because I was in my first year as Executive Director and didn’t want to have to worry about missing a beat. Instead of putting the resume stack aside and seeking out more qualified candidates, I rushed to a decision with the small pile in front of me – twice. A good example of how some experiences need a second round in order to be fully realized (I suppose I did burn my hand three times on the stove as a kid, so maybe it’s just me). The result was a setback on not only my role but the entire organization.

Overall, a solid program design that takes into account the life-span of supervisory roles in most settings.

What do you think should be improved? Why? And How? I had a couple thoughts on this as I read through the program design. First, the training – like all trainings – is very academic and theoretical, of course. The old adage of “knowing is not doing” certainly applies for many aspects of supervision and leading people. There will be certain things that they will not be prepared for that will just have to be learned. I could see how that would be addressed in the training through the reflection and role-playing, but I do want to emphasize that being prepared for the unknown factors is something to be spotlighted. Examples could include employees who have strong emotional reactions, terminating someone who is out for your position and tries to maneuver to make you seem bad at your role in order to get the job, and how to help others with extreme instances like the death of an employee or extreme stresses, etc.

That kind of discussion and “what if” thinking is especially helpful when thinking about keeping the supervisor in a mindset of fairness and consistency with all employees. We all naturally gravitate towards people we like or find enjoyable or intriguing or some such, but when you have employees that are starkly different than your personality, it can get very tricky. At the end of the day, if you would quite frankly determine that in personal life you would not do well with this person, it can be an extremely challenging feat to maintain a clear and truly fair model of supervision. Lawsuits and bad situations often can arise, if organizations and companies aren’t careful about such things.

Another thought that came to mind as I was reading is the importance for supervisors to not only know the role of the employees they work with in the grand scheme and to communicate that, but also to know the job duties of their employees. General Patton may have me on this one, but I have found that actually participating and doing a task that your employee does often is an eye-opening experience at how to manage, coach, and supervise that person. It gives insight into the inner workings of something perhaps overlooked by a supervisor, and it provides vital understanding of the skill level involved for the employee.

Some examples would be job-shadowing each other for a day. I did this with our Receptionist recently, and it was illuminating for me to understand the constant demand of 200 calls a day, the phone system technologies involved in seven locations, managing to greet people as they walked in and the phone still ringing, company vehicle check-out, conference room prep, etc. etc. etc. Her role became real to me because I actually sat there and did it for a time. Not only does it give the supervisor a sense of understanding and appreciation, it will be very helpful to have the staff see the willingness of a supervisor to stop and appreciate what they actually do – beyond the words on a job description.

Again, overall I like the approach your program is making in helping reach people where they are and to connect all styles of leadership and learning with the various activities. To be overly simplistic, I think it’s like riding a bike or anything else, sometimes. Eventually we just have to do it to know what it is fully. I think your program would certainly help them to know what that could be and would help to prepare them well for it.

Rozlyn Hernandez’s Evaluation

Roles & Qualifications. Director of the Express Enrollment Center at Ivy Tech Community College in Lafayette, Indiana and presenter at a new supervisor training and development program at Ivy Tech.

What do you like about the program design? (written feedback)

What do you think should be improved? Why? And How? (written feedback) Regarding, “Techniques to assess experience could include observations, pretesting, questionnaires, and interviews. A checklist or questionnaire that highlights common supervisory skills and background could be utilized. This would give the educator an idea of which topics need the most emphasis,” How about pre work before the training. This can include HR policies, on-boarding information, and I would suggest utilizing a tool like Strengths Quest. This type of pre-work can have supervisors review what they should be familiar with and can lead to follow up questions in the training if possible. These things can be done online where the facilitator can obtain the completed info and go from there. It can also be seen as another type of learning tool for your participants. This can also be nice to where facilitators are not left to “assess” the type of supervisor a participant is but rather where their strengths are or can be in a supervisory role.

With the possibility of utilizing Strengths Quest you can see more information coming through in a way where a questionnaire may or may not help you determine their background as a supervisor. You can also set up the training for participants to move around randomly utilizing everyone’s background experience instead of the facilitator working to develop how the surveys and participants should come together. This will allow maximum interaction among all the participants and keep them engaged by having them move more than sit.

I think utilizing a tool like Strength Quest can help the participant and facilitator know or have an idea as to what kind of learner they can be. It also allows the participant to reflect on the outcome of the Strength based test to see where their strengths are and perhaps reflect on the strengths of their staff.

I am not a fan of the participants coming up with a scenario about a “situation that a past supervisor resolved poorly”. This can take some time for someone to think about and have some bias to it based on that person’s perspective. Consider supplying the situation on a card for each group or short videos of a situation for each group to determine if the supervisor dealt with it appropriately or what they can do better. This allows for different learning scenarios that can be shared among other groups.

I like the journal entries but perhaps pre-made questions that the participant can answer to show what they’ve gained from that part of the training and how they intend to use it with their staff or co-workers. It seems like a lot of work on the facilitator’s part to understand what they wrote. It can also be more of a survey at the end of each training goal to see what the participants liked or didn’t like or felt they needed more of in that area.

Personal interview with Ms. Hernandez. She began the conversation by talking about the journal. While she liked the idea of having a document to hold onto and reflect upon long after the training had ended, she felt that it would be a lot of work for both the new supervisor and the facilitator to manage while trying to cover the rest of the material in the new supervisor program. She suggested that the facilitator give specific topics to write about and emphasize how the new supervisor planned to use the material.

Ms. Hernandez also reiterated the success that her training program has had using packaged tools like StrengthsQuest. It allows participants to be grouped in such a way that a variety of supervisor strengths are able to be utilized in a group setting. She also emphasized the benefit of having the facilitator come up with predetermined scenarios (where a past supervisor handled a situation poorly); not only does it save time, but it can help ensure a variety of discussion material for the new supervisors to cover.

The conversation was concluded by commending the emphasis on hiring new staff. When she had started her position three years ago, there was not much training on how to select the best candidate for the job. Coming up with her own interview questions was challenging in the beginning. She would have appreciated specific training on the best ways to hire new staff.