My name is Jenny Jones, and I am an Associate Professor in the school of social work at Virginia Commonwealth University. I have spent a considerable amount of time engaging in evaluation research with numerous community based organizations. The majority of my work is done in communities of color, specifically within the African American community and my work almost always has a social justice nature to it. And, while this work brings me much joy, partnering with community based organizations to do this work can be very challenging. Sometimes the challenge can be so great it can make you question why you do this work.

Hot Tips: Last year while working with a community organization that provides asset building services to families at risk for homelessness, I was reminded of the core principles of evaluation that keeps me grounded in this work.

1. Ethical behavior: Always be honest, respectful, and true to your craft, regardless of what the organization may ask of you. This sometimes requires you as the evaluator to do some value clarifying, so that you do not try to make your values that of the agency.
2. Interpersonal skills: The ability to connect with the community that you are working with is critical. This requires communicating in a way that makes others feel valued. Particularly, for communities of color it is important that you as the evaluator speak and behave in a way that is respectful of their culture.
3. Flexibility: Always allow yourself to be flexible in the process. Being flexible requires that you let go of your agenda and learn to work within the agency’s agenda.
4. Set clear expectations: Tell the organization up front what you expect of them in thus process, and what you can and cannot do. Never promise what you cannot deliver.

Use humor: When all else fails, use humor! Trust me it works.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

CBO, community based organizations

My name is Kylie Hutchinson.  I am an independent evaluation consultant and trainer with Community Solutions Planning & Evaluation.  I am also a regular facilitator of the Canadian Evaluation Society’s Essential Skills Series course.  One question my students regularly bring up when learning to develop logic models and evaluation plans is the issue of “contribution analysis”, i.e. contribution versus attribution of a program’s activities to long-term outcomes and impacts.

Rad resource: I’ve found it useful to show them the “Output Outcome Downstream Impact Blues” karaoke video.  This short musical clip, written and performed by Terry Smutylo, addresses this issue in a humorous, yet informative way.  Although it was originally written for those working in international development, it is applicable to other evaluation sectors.  You can find Terry’s clip at http://www.idrc.ca/en/ev-65284-201-1-DO_TOPIC.html

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

contribution analysis, logic models

Hello, My name is Michael Duttweiler and I am assistant director for program development and accountability for Cornell University Cooperative Extension.  One of my roles is to provide evaluation planning assistance to local extension educators, many of whom do not have formal backgrounds in program development or evaluation.

Hot Tip: Talking Your Way Into a Logic Model I often rely on development of logic models as an initial step in evaluation planning.  Occasionally, I encounter individuals and groups that either have not been exposed to logic models or that resist them based on past experience.  When that happens, I rely on a simple narrative approach to begin the process.

I start by asking “What do you do in this program?”  Then I ask “What do you expect the participants to do?” Last, I ask “Why are you doing this? What do you expect to achieve?”  Next I turn to the narrative form of logic modeling and ask them to summarize the program using statements in the form “If we  (program actions/ activities), the participants will (learn, apply, change practices), and the results will be (intended outcomes).”  With one or more draft statements in hand, I ask “What are the assumptions behind these statements? What are you assuming about the audience, their resources, your roles, etc.?”  That often leads to revisions in the statements or addition of intervening statements.   Then I ask “What’s it take to do this program?” to begin getting at inputs.  Last I ask “What things tell you this program is on track?” which gets at outputs and indicators.  Somewhere along the way, groups usually call the game saying something like “OK, OK, we’ll do a logic model.”  But, having experienced the discovery of program definition and analysis, they will usually play along and appreciate the visual impact of typical logic model formats that I introduce by way of summary.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

logic models

My name is Rodney Hopson. I am a faculty member in the Department of Educational Foundations and Leadership in the School of Education at Duquesne University. For the last several years (with a host of great colleagues in AEA, with Torres Consulting Group and OMG Center for Collaborative Learning with the support of the National Science Foundation and the Robert Wood Johnson Foundation, I have been developing internship and fellowship programs to support graduate and post-graduate students of color and from traditionally underrepresented communities. Embedded in their practice is the understanding and development of Culturally Responsive Evaluation (CRE) approaches and practices in these same communities.

Rad Resource: Adapting the elements of evaluation practice into one that reflects a culturally responsive one, Karen Kirkhart, social work professor at Syracuse University, and I have presented at the AEA/CDC Summer Institute for the last few years. We use the Frierson, Hood, & Hughes chapter in the 2002 National Science Foundation User-Friendly Handbook for Project Evaluation to guide a deeper understanding of how to integrate CRE throughout the evaluation.

Hot Tip: Situate CRE within elements or a framework of evaluation. Whether you use the CDC Evaluation Steps, or another framework to describe steps in evaluation, the key is to embed CRE throughout. For instance, at each stage of evaluation, CRE should be present from the time we prepare for the evaluation to the dissemination and use of results. Below is the CRE framework (with appreciation for the support of Elizabeth Kahl, Syracuse University, who assisted us on graphic and technical design elements), adapted from Frierson, et.al, 2002:

Rad Resource: Learn more about fellowship and internship programs in culturally responsive evaluation through the RWJF Evaluation Fellowship website or AEA’s Graduate Education Diversity Internship website.

Rad Resource: Develop a better theoretical and practical explanation of CRE by reading about its origins, its aim, and its movement as reflected in chapters in any of the following books and special issues:

• The Role of Culture and Cultural Context in Evaluation (Hood, Hopson, and Frierson Eds., Information Age Publishing)
• The SAGE International Handbook of Educational Evaluation (Ryan and Cousins, Eds, SAGE.)
• Handbook of Social Research Ethics (Mertens and Ginsberg, Eds., SAGE)
• SAGE Handbook of Evaluation (Shaw, Greene, and Mark, Eds., SAGE.)
• The Handbook of Leadership Development Evaluation (Hannum, Martineau and Reinelt, Eds., Jossey-Bass)
• In Search of Cultural Competence in Evaluation (Thompson-Robinson, Hopson, Sen Gupta, Eds, Jossey-Bass)
• Responsive Evaluation (Greene and Abma, Eds., Jossey-Bass)

Want to hear more from Rodney Hopson? He is serving as this week’s Thought Leader on the AEA Thought Leader Discussion Series. Learn more at http://www.eval.org/thought_leaders.asp. This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

culturally responsive evaluation, culture

If it’s Saturday it must be Susan! My name is Susan Kistler. I am the Executive Director for the American Evaluation Association and I contribute each Saturday’s post to the aea365 blog.

Today, I am going completely practical and broadly applicable. The range of opportunities that call for working with photographs is ever increasing, including incorporating into presentations, reports, webpages, and blogs, creating online profiles, and sharing with friends and family.

Rad Resource: drpic is a free online suite of tools for picture editing. It allows for easy cropping and resizing, rotation, contrast control and touch up. Plus you can add text and frame the picture. In particular, the cropping and resizing tools are sophisticated – as long as you realize that you can view the dimensions of your selections in the upper right hand corner of your page. I initially didn’t realize the dimension information was on the page until after multiple uses and had been guessing.

How does it work? Go to http://drpic.com/ and click on the upload prompt, browse for the photo that you wish to alter, click continue, edit your picture right in your browser, then click “Save to Disk” or “Save to Web.” You’re done! No registering, no paying, no hassles.

Hot Tip: Increasingly, if you work online such as with blogs, you may find that you are asked to post a URL of a picture rather than the picture itself. This basically means that you are adding a pointer to where a picture is available on the web. You can do so by using the “Save to Web” option on drpic – it will post your picture for you (careful, it will be public) and give you a share link to use in such instances.

Rad Resource: Stock photography can be expensive. On flickr, you’ll find literally millions of photographs shared under a Creative Commons license that allows for use in reports, online, and the like – as long as you give appropriate attribution to the photographer. There are a variety of Creative Commons licenses applied to photos – from freely usable with attribution only, to not usable in commercial works, to usable but only without modifications. Learn more about Creative Commons licensing, and access the millions of free pictures available online, at http://www.flickr.com/creativecommons/.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

images, photos

My name is Kim Norris and I am the Evaluation Coordinator for University of Maryland Extension’s Food Supplement Nutrition Education (FSNE) program. Included in my work is to assist educators in developing useful strategies for assessing the impact of their work on our target audience, limited-income, and often, low-literacy, populations.

Hot Tip: Utilize Audience Response Technology System in group class settings for immediate, anonymous assessment and feedback.  The “clicker” technology, as we sometimes refer to it, allows questions to be asked both orally and in writing, allows individuals to respond anonymously, and allows for immediate feedback after responses for both educator and audience.  We used these recently with our own educators to ask questions that, in a setting in which anonymity was not guaranteed, could lead to false answers due to high motivation to fall within socially acceptable norms.  Since results are calculated and visible to all on the spot, group responses can be reviewed, analyzed, interpreted, and addressed by the group, thereby increasing potential for empowering participants.

Other advantages of using this easy-to-teach technology include the ability to:

• collect data from larger numbers of people in a shorter amount of time
• eliminate data entry errors by direct transfer of electronic data to a data base
• engage technology-averse populations in computer technology to their benefit
• help low-literacy populations participate in surveys as respondents
• provide confidentiality for respondents

The technology can lead to missing data if not preceded by sample questions and as group sizes become larger or less engaged.  Studies are underway to better understand strengths and limitations of the technology as an educational and evaluation tool.

Rad Resource: A Bibliography of Selected Readings on Audience Response Systems: http://bit.ly/audienceresponsesystems.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, participation, technology

My name is Cindy J Wong. I am a consultant evaluation and a social science researcher in health and human services. Recently, I have been obsessed with digital video technology as a tool in social, organizational, instructional, and evaluative documentation. I recently had the chance of bringing portable video recorders to a visit to a non-profit organization located in South Africa. The organization provides mobile health services and computer education at primary schools in rural areas to address HIV/AIDS. I had an idea about how the technology might be utilized, but I was pleasantly surprised and thrilled, as the members of the organization had more immediate ideas for the technology that involved educational assessment and implemented them. The staff members are continuing to expand the ways in which the technology is utilized in education and instruction.

Rad Resource: Portable video recording devices, such as FlipVideo (http://www.theflip.com/en-us/) record up to 120 minutes of high definition video. These are hand-held battery-operated units that contain retractable USB ports which can be plugged directly into a computer or laptop for download. The cameras include basic software for downloading, organizing and editing of the video clips. The cameras are affordable as the technology goes, and they are widely available in the United States. Movie Maker 2.1 (http://bit.ly/moviemaker2) has been a powerful software program that can integrate with the Flip system. As a Windows User, you may not even realize that this software is on your computer, since it is downloaded through automatic updates (check your Program Folder). MacUsers can similarly use iMovieMaker (http://www.apple.com/ilife/imovie/). File conversion freeware such as Pazera (http://bit.ly/pazera) convert Flip HD file formats to Windows Movie Maker formats with ease.

I hope you enjoyed this Rad Resource, Cheers!

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

empowerment, participatory, technology

Greetings!  We are Tom McQuiston (USW Tony Mazzocchi Center) and Tobi Mae Lippin and Kristin Bradley-Bull (New Perspectives Consulting Group).  We have collaborated for over a decade on participatory evaluation and assessment projects for the United Steelworkers (labor union).  And we have grappled mightily with how to complete high-quality data analysis and interpretation in participatory ways.

Hot Tip: Carefully determine up front what degree of full evaluation team participation there will be in data analysis.  Some practical considerations include:  the amount of team time, energy, interest, and analysis expertise that is available; the levels of data analysis being completed; the degree of project focus on team capacity-building; and the project budget and timeline.  How these and other considerations get weighed is, of course, also a product of the values undergirding your work and the project.

Hot Tip: Consider preparing an intermediate data report (a.k.a. “half-baked” report) that streamlines the analysis process for the full team.  Before the full team dives in, we:  review the raw quantitative data; run preliminary cross-tabs and statistical tests; refine the data report content to include only the — to us — most noteworthy data; remove extraneous columns spit out of SPSS; and assemble the tables that should be analyzed together — along with relevant qualitative data — into reasonably-sized thematic chunks for the team.

Hot Tip: Team time is a precious commodity, so well-planned analysis/ interpretation meetings are essential.  Some keys to success include:

1. Invest in building the capacity of all team members.  We do this through a reciprocal process of us training other team members in, say, reading a frequency or cross-tab table or coding qualitative data and of them training us in the realities of what we are all studying.
2. Determine time- and complexity-equivalent analyses that sub-teams can work on simultaneously.  Plan to have the full team thoughtfully review sub-team work.
3. Stay open to shifting in response to the team’s expertise and needs.  An empowered team will guide the process in ever-evolving ways.

Some examples of tools we have developed — yes, you, too, can use Legos™ in your work — can be found at: http://newperspectivesinc.org/resources.

We never fail to have many moments of “a-ha,” “what now” and “wow” in each participatory process.  We wish the same for you.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

analysis, collaboration, empowerment, participation

My name is Wayne Miller and I am a senior lecturer in the Faculty of Education at Avondale College, Lake Macquarie, New South Wales, Australia.  In December I received my doctorate from the University of Wollongong following acceptance of my thesis titled Practical methods to evaluate school breakfast programs – A case study.   The study reports the use of empowerment evaluation with a national school breakfast program in Australia known as the Good Start Breakfast Club (GSBC).

During the project some eighty GSBC program personnel took part in ten empowerment evaluation workshops to identify key program activities for investigation; gather baseline data about the strengths and weaknesses of the activities; suggest goals and strategies to monitor and improve the activities identified; and to develop evaluation tools designed to provide evidence of success.  Following workshops I asked participants …from your experiences in these initial workshops how valuable do you think the empowerment evaluation method is for collaboratively evaluating the GSBC program?  42/80 indicated ‘very’ to ‘extremely’ valuable with a further 36/80 responding ‘reasonably’ to ‘quite’ valuable.  A regional coordinator commented, the model is definitely in line with the principles of our program and empowering the community.  One beautiful response from an outlier, a total waste of time and all about Miller getting his doctorate!

Toward the end of the project I interviewed 29 program personnel who had been directly involved in the evaluation and I asked them to reflect on the empowerment evaluation process and particularly whether it had adhered to the ten principles of empowerment evaluation.  Respondents made up of volunteers and teaching staff at the ‘coal face’, school principals, GSBC coordinators and executive staff from Red Cross the program managers, and the Sanitarium Health Food Company, the major sponsor, reported both alignment and misalignment with the principles.  Two examples: On the principle of democratic participation defined as active participation by everyone in shared decision-making is valued…respondents acknowledged that the ‘taking stock’ step of the empowerment evaluation had been particularly democratic but that the democratic nature of the evaluation process had been compromised when those who came together to implement Step 3 – ‘Planning for the future’ were handed evaluands from workshop groups who had completed Steps 1 – Develop a mission, vision or unifying purpose for the program and Step 2 -Taking stock.   On the principle of capacity building defined as program staff and participants learn how to conduct their own evaluations…significant gains in evaluation capacity was reported by personnel at the breakfast club level.  Volunteer staff at one site designed and trialled an instrument which provided average nutrient uptake data which were subsequently used to modify food served to improve fibre intake.  A negative aspect reported was that staff turnover at management level mitigated against evaluation capacity building in one region.

Hot Tip: Trustworthy relationships must be established for empowerment to occur.  Community ‘champions’ committed to their communities, who use empowering processes and have good networks and communications skills, are vital partners.  Ongoing commitment by senior management to nurture and support empowered staff by providing them with the resources necessary to remain so, is a ‘must have’ ingredient to avoid empowerment fade.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, trust

My name is Linda F. Delaney and I serve as a program evaluator for my own company (LFD Consulting, LLC) in Marion, Arkansas.  Since 2005, I have worked closed with Dr. David Fetterman as an evaluator for the Minority Sub-recipient Grant Office at the University of Arkansas in Pine Bluff, Arkansas (UAPB).  I also serve as President of the Arkansas Group of Evaluators (AGEs).  I currently conduct program evaluations for 5 local tobacco control organizations in Arkansas.  These agencies implement tobacco control efforts in approximately 20 of Arkansas’ 75 counties.   The nature of this work requires collaboration through empowerment strategies to increase effectiveness.  These tobacco program coordinators must be prepared to function in interdisciplinary groups ranging from the medical/dental profession to government.  It is important that evaluators empower their clients through training them in empowerment evaluation and effective collaboration.  I will be sharing some hot tips about effective training methods.

Hot Tip: Strategically begin with the clarity and the coordination of primary stakeholder missions.  Are they congruent?  Allow the clients to share the collaborative purposes (first).  Remember, what they say is what you hold them accountable for – so ask pertinent questions that solicit the answers needed to drive the conversation in the direction you know that it must go. (Principle:  If I say it, you may doubt it.  If you say it, you believe it – and that is what you are held accountable for.)

Hot Tip: Take stock by finding out who is currently involved in the activities to complete goals and objectives.  Then brainstorm who else can be involved and why others are needed.  Make this conversation in depth.  Determine various technological tools that can help communication and partnering efforts.  (Principle:  If one understands the WHYs, the HOWs will take care of themselves.)

Hot Tip: Develop a bullet-point strategy/plan with deadlines for recruitment and implementation to involve and empower collaborative partners needed to complete planned activities. (Principle:  Together Everyone Achieves More)

Rad Resource:  http://tobaccoprevention.blogspot.com/ is a blogspot where you can find many tips and strategies on collaboration through empowerment.

This week’s posts are sponsored by AEA’s Collaborative, Participatory, and Empowerment Evaluation Topical Interest Group (http://comm.eval.org/EVAL/cpetig/Home/Default.aspx) as part of the CPE TIG Focus Week. Check out AEA’s Headlines and Resources entries (http://eval.org/aeaweb.asp) this week for other highlights from and for those conducting Collaborative, Participatory, and Empowerment Evaluations.

collaboration, empowerment, participation