Hello, my name is Nicole Jackson. I am both an adjunct faculty in the Human Resource Management Certificate program at U.C. Berkeley Extension and a doctoral candidate in Policy, Organization, Measurement, and Evaluation at U.C. Berkeley’s Graduate School of Education. From my previous and current work, I discovered that interviewing is both an art and a science especially when it is used in more formative evaluations. Although considered important, interviews are prone to researcher bias that can impact data collection and reporting. Below I offer some tips to help mitigate forms of research bias during interviews.

Hot Tip #1: Understand how different interview formats may alter findings. The two general categories of interview formats include invidual versus panel interviews and unstructured versus structured interview scripts. Individual or one-on-one interviews as well as unstructured or loose ended-scripts are the most prone to researcher bias. Both of these formats lend easily to loss of control due to different personality types that can affect information collection. Where possible, try to use multiple interviewers or a small panel with a structured interview script to help mitigate and triangulate real-time interview data. Structured interview scripts should always focus on the critical research questions during an evaluation project.

Hot Tip #2: Tailor question types according to personality type and experience level. A variety of question types exist to help evaluators navigate difficult and shy personality types as well as those participants with more or less knowledge and experience. Where possible try to use more open-ended, situational questions with follow-up probes for more shy personalities and those participants with more knowledge and experience. For more difficult personalities, begin with more close-ended (e.g., yes/no) questions and then transition to open-ended question prompts in order to maintain control and focus during the interview.

Hot Tip #3: Never underestimate the role of the interview environment. Nothing can be as frustrating as a distracting interview environment. Always conduct interviews in a quiet, private location with good lighting, appropriate room temperature, and minimum distraction. Have water ready to go to place participants at ease. When using recording technology, always consider Murphy’s Law and have extra notepads and recorders ready on hand. Test all recording equipment during the first two minutes of the interview as a safe-guard.

Hot Tip #4: Be mindful of both verbal and non-verbal language. Experts on interviewing claim that non-verbal communication is just as important as verbal behavior in evaluating the trustworthiness of data. Be aware of how your own body language and those of your participants can alter data collection and assessment. Never use closed poses such as crossed arms while interviewing, which is a sign of defensive behavior. Also, be mindful that non-verbal behaivor is culturally influenced.

Nicole will be conducting a roundtable at evaluation 2010 on improving methods of inquiry to incorporate diverse views and perspectives. Join Nicole and over 2500 colleagues at AEA’s Annual conference this November in San Antonio.

context, interviewing, interviews

My name is Monica Hargraves and I work with Cooperative Extension associations across New York State as part of an evaluation capacity building effort in the Cornell Office for Research on Evaluation (CORE).  My work with Extension is shaped, in part, by insights we gained through a Concept Mapping research project we did in late 2008.  We wanted to explore, from practitioners’ perspectives, what factors contribute to supporting evaluation practice in an organization.

We used Concept Mapping software from Concept Systems, Inc. to gather ideas in response to this prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …” Contributors included county-based educators and Executive Directors, as well as state-level Extension administrators and Cornell staff.  The raw ideas were pared down to a working set of 80, and then participants sorted the ideas into clusters and rated them on two criteria: potential for making a difference, and relative difficulty.

The individual ideas become points on a “Cluster Map” that gives a visual representation of how participants conceptualized the patterns and themes in ideas (see below. For information on the Concept Systems technology and the statistical techniques that underlie it, see www.conceptsystems.com.)  The ratings are useful for thinking strategically about what to do give priority to when trying to improve and sustain evaluation practice in organizations.

Rad Resource: For more detail on the study, including a handout with the individual idea statements and their ratings on potential difference, see http://core.human.cornell.edu/AEA_Conference.cfm#2008

Cluster Map of Ideas in Response to the Prompt: “One specific thing an Extension organization can do to support the practice of evaluation is …”

Lessons Learned:

• Technical assistance and training are not enough! The top-rated cluster in terms of potential for making a difference was “Communicate the Value of Evaluation.”  The ideas there included educating organization leaders, staff, and volunteers on the importance of evaluation (not the how-to), using evaluation results well and demonstrating how they lead to better programming, having an evaluation champion in-house, making evaluation results easy to understand and user-friendly.
• Communication is important. Communication should be used to motivate evaluation and build organizational commitment to it, and as a practical tool for sharing what works, fostering collaborations, and saving time.
• Leadership and Structure matter. The second and third most important clusters were “Set Expectations and Requirements” and “Integrate into Organization Structure”.  Respondents wanted clarity and consistency, and to have evaluation woven into a wide range of organization functions and practices.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

cluster map, data visualization

My name is Michelle Jay and I am an Assistant Professor at the University of South Carolina. I am an independent evaluator and also an evaluation consultant with Evaluation, Assessment and Policy Connections (EvAP) in the School of Education at UNC-Chapel Hill. Currently I serve with Rita O’Sullivan as Directors of AEA’s Graduate Education Diversity Internship (GEDI) program.

Lessons Learned: A few years ago, EvAP served as the external evaluators for a federally-funded Gaining Early Awareness and Readiness for Undergraduate Programs (GEAR UP) state-wide grant housed at University of North Carolina (UNC) General Administration. Part of our work involved assisting project coordinators in 20 North Carolina counties to collect student-level data required for their Annual Performance Review reports as well as for program monitoring, assessment, and improvement. For various reasons, project coordinators experienced numerous difficulties in obtaining the necessary data from their Student Information Management Systems (SIMS) administrators at both the school and district levels. As collaborative evaluators, we viewed the SIMS administrators not only as “keepers of the keys” to the “data kingdom,” but also as potentially vested program stakeholders whose input and “buy-in” had not yet been sought.

Consequently, in an effort to “think outside the box,” the EvAP team seized an opportunity to help foster better relationships between our program coordinators and their SIMS administrators. We discovered that the administrators often attended an annual conference each year for school personnel. The EvAP team sought permission to attend the conference where we sponsored a boxed luncheon for the SIMS administrators. During the lunch, we provided them with an overview of the GEAR UP program and its goals, described our role as the evaluators, and explained in detail how they could contribute to the success of their districts’ program by providing the important data needed by their district’s program coordinator.

The effects of the luncheon were immediate. Program coordinators who had previously experienced difficulty getting data had it on their desks later that week. Over the course of the year, the quality and quantity of the data the EvAP team obtained from the coordinators increased dramatically. We were extremely pleased that the collaborative evaluation strategies that guided our work had served us well in an unanticipated fashion.

Hot Tip: The data needs of the programs we serve as evaluators can sometimes seem daunting. In this case, we learned that fixing “the problem” was less a data-related matter that it was a “marketing” issue. SIMS administrators, and other keepers-of-the-data, have multiple responsibilities and are under tremendous pressure to serve multiple constituencies. Sometimes, getting their support and cooperation are merely a matter of making sure they are aware of your particular program, the kinds of data you require, and the frequency of your needs. Oh, and to know that they are appreciated doesn’t hurt either.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

Administration, Evaluation planning, project management

Hello, we are Katherine Tibbetts and Wendy Kekahio, and both program evaluators doing work within the field of education in Hawai`i. Our work involves using indigenous ways of teaching and learning to inform culturally relevant and responsive ways of conducting research and evaluation studies.  One of our recent projects involved working with Hawaiian-focused charter schools to assess the impact of participation in professional development programs.

The Collaborative Inquiry (CI) project was designed to be culturally relevant and responsive–representing the values of relevance, rigor, respectful relationships, and reciprocity (for more information see, among others, Tibbetts, Faircloth, Villegas and Wheeler (2008)), The CI project extended the conventional purposes of evaluation to prove or improve, by employing a meta-action-research strategy to support the transfer of knowledge and skills learned at the training and assess their impact on teaching and learning. To do this, all participating teachers were required to conduct collaborative inquiry projects. They were encouraged to do their projects in small groups. The charter school teachers’ projects were supported by faculty contracted from a local college of education and culminated in a Ho`ike (demonstration of knowledge or skills).

Hot Tip: Supporting the Inquiry Projects. If you are interested in replicating this approach, it is important to provide ongoing support and scaffolding for the inquiry projects. The simplified action research curriculum and tools provided by the college of education faculty brought what were previously largely abstract concepts to life for the charter school teachers. Multiple “touch points” throughout school year, including visits to the charter schools enabled the college of education faculty to provide advice on the feasibility of project plans to identify potential sources of data that were tailored to each action research project, and helped sustain the momentum of the projects

Hot Tip: Assessing the Impact. As evaluators, our primary challenge was to synthesize information across a wide variety of projects. In the first year, there were 8 projects conducted in 3 different schools with topics spanning nutrition education, behavior management, mathematics, and writing. We approached the analysis as a multiple case study (based loosely on Stake, 2008) and ultimately created a rubric based on the CI project objectives and standards of inquiry. This allowed us to assess and summarize the quality of the inquiry projects.

Rad Resources:

Deloria Jr., V., & Wildcat, D.R. (2001). Power and place: Indian education in America. Golden, CO: Fulcrum Resources.

Hood, S., Hopson, R. K., & Frierson, H. T. (2005). The role of culture and cultural context: a mandate for inclusion, the discovery of truth and understanding in evaluative theory and practice. Greenwich, CT: IAP

Smith, L. T. (1999). Decolonizing methodologies: research and indigenous peoples. London: Zed Books.

Stake, R.E. (2006). Multiple case study analysis. New York: Guilford Press.

Thompson-Robinson, M., Hopson, R., & SenGupta, S. (Eds.). (2004). In Search of Cultural Competence in Evaluation (Vol. 102). Fairhaven, MA: Wiley Periodicals.

Tibbetts, K. A., Faircloth, S., Villegas, M., & Wheeler, L. (2008). Section III: Indigenizing accountability and assessment. In M. K. P. A. Benham (Ed.), Indigenous Educational Models for Contemporary Practice:  In Our Mothers Voice II. New York: Routledge.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

collaboration, collaborative inquiry, indigenous learning

Hi, I’m Robert Brunger. I am an evaluator with the Ounce of Prevention Fund of Florida; a Tallahassee based non-profit organization that has worked since 1989 to improve the lives of Florida’s children and families.

If you are planning to use focus groups to learn more about what’s on the minds of your stakeholders, here are some suggestions to help you make sense later out of what gets said during the focus group itself.

Hot Tip #1: Digital recorders really are “the greatest thing since sliced bread!” They are available for less than $40 from electronic retailers. Spend enough to get a model that will allow you to transfer the audio file from the device to your computer. (Get a couple of spare batteries, too!)

Hot Tip #2: Practice with your digital recorder before your use it in a focus group. They are not complicated, but you will want to avoid any undue “fussing” in the focus group setting. Record some practice conversations to get used to the controls and volume levels.

Hot Tip #3: When it gets to “show time,” introduce the digital recorder in a very matter-of-fact fashion, get it started, and then pay no further attention to it until the meeting is over.

Hot Tip #4: Place your recorder in the middle of the table, or on a stool in the middle of a circle of chairs. A recent EVALTALK poster, Daphne LaDue, has made a persuasive case for using two digital recorders, pointed in different directions, as a way to improve your ability to figure out what’s been said later.

Hot Tip #5: Start the digital recorder(s) and a stopwatch at the same time. Your note-taker (and, yes, you do need a note-taker!) can make periodic marginal notes about elapsed time from the stopwatch that can be very helpful later in getting your notes and the recorded audio file(s) to match.

Hot Tip #6: It’s also helpful to create a seating pattern diagram to accompany your notes, and assign everyone an identifier – first names will work well, or numbers, or some uniquely identifying characteristic (e.g., red blouse woman, black man with beard, etc.). You can use this scheme while taking notes to identify individual speakers.

Hot Tip #7: Consider how badly you will need to have a full transcript prepared, as that can be a real “time sink,” taking five to six hours per hour of recorded material. If you are doing multiple groups, or if many people will be involved in interpreting the results, you probably will need them, but for smaller projects, your own summary of what was said, based on your notes and selected quotes from the audio file(s) may be entirely sufficient.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

analysis, digital recorders, focus groups, notes

My name is Randahl Kirkendall. I work part-time as an Evaluator with Ellen Iverson, Director of Evaluation, for the Science Education Resource Center (SERC) at Carleton College, which works to improve education through projects that support educators.  Our work is funded primarily through NSF grants.  SERC has expertise in geoscience education, workshop leadership, website development and program and website evaluation.

A primary aim of SERC is to help faculty adopt evidence-based teaching behaviors that will enhance student learning. In evaluating the websites at SERC, our interest is in the role of website use in faculty professional development. We use a variety of web analytic tools such as Google Analytics, server-based website statistics, and web page visit logs in combination with data from surveys, interviews, focus groups, and observations to get as complete a picture as possible for how faculty use  websites and the impact that their use has on teaching behavior.

Lesson Learned: One of the things we have learned from user interviews is that people generally have poor recall of how they found a website and used it. While they can explain why they go to a website (motivation), they have difficulty recalling at what section of the website they started, what pages they viewed, and the search strategy they used. Website use analytics and web server logs of individual visits provide a richer picture of user behavior and interests via records of the actual pages that they visited.

Lesson Learned: The SERC websites often don’t work in isolation. Our survey of 2,000+ faculty found that a significant number of users were using the websites to compliment other professional development activities such as attending workshops, exchanging ideas with colleagues, or reviewing literature. Thus, it has been prudent that we collect data on these other possible influences on their teaching behavior.

Cool Trick: We sequence or build evaluations incrementally, partially basing data collection and/or analyses on findings from other data collection methods. For example, we use the findings from user interviews to describe predominant motivations for using a website and any changes in behavior (such as teaching practice changes) that users attribute (at least partially) to website use. Those descriptions become a guide for using website analytic data to map particular patterns of use and to identify web use logs that can provide insight into how users may navigate the website.

Cool Trick: We use pop-up surveys to identify users that we might not otherwise reach. The pop-up asks for an email that we can use to follow-up with them for future surveys and interviews.

Want to learn more about Randahl and Ellen’s work? Join over 2500 colleagues at the AEA Annual Conference this November in San Antonio and check out their session in the conference program.

web analytics

My name is Susan Kistler, and I am the Executive Director for the American Evaluation Association. I contribute each Saturday’s aea365 alert. Summer is winding down and I know that over 2500 of you are finalizing your plans for attending AEA’s annual conference this November in San Antonio. Even if you won’t be able to join us in Texas, evaluators are a traveling bunch – from visiting clients to conducting site visits to providing capacity building and training, evaluators are out and about. So, this week I’m sharing three resources that I have found to be invaluable in preparing for travel.

Hot Tip – SeatGuru: SeatGuru provides airlines seating charts for almost all flights on major airlines. You enter an airline and a flight and it shows you the type of plane and then a seat map indicating which seats are good, get mixed reviews, or are bad, including providing information about such things as ending up beside lines to the lavatories or non-reclining seats. I never choose seats on a long flight without a quick check on SeatGuru first. A few years ago, thanks to SeatGuru, my husband and I shared the only pair of seats that had extra legroom and were two-across in coach from the United States to South Korea! These seats didn’t even appear on the airline’s own online seat charts.

Hot Tip – CompareAirlineFees: This site does exactly what you would expect. It provides a chart comparing airline fees (baggage fees, change fees, unaccompanied minor fees) for US-based airlines. While many sites have such charts, and none are always accurate given the rapidly changing landscape of fees, I find the ones here to be most regularly updated and formatted for easy reading.

Hot Tip – TripAdvisor: TripAdvisor is a group-sourced travel website with reviews of hotels and destination activities as well as ideas for what to do (check out the traveler lists). The key to TripAdvisor is to be a smart consumer of the reviews – look for recommendations that have been reviewed by multiple travelers and give more weight to reviews from people who have provided lots of reviews across different places (click on a contributor’s name to see all of her or his reviews).

The above opinions are my own and do not necessarily reflect those of the American Evaluation Association. See you in San Antonio!

This contribution is from the aea365 Tip-a-Day Alerts, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

conference, flights, Travel

I’m Michael Matteson and I’m pursuing a graduate degree at the University of Wollongong (Australia).

Lessons Learned – What’s the Hawthorne effect? Why would it matter in Empowerment Evaluation? Most people will remember hearing of the “Hawthorne effect “based on a major study of organizational and environmental effects on productivity which gave confusing and contradictory results over a long period. Commentators suggested that any effects of the experiments were a result, not of the researcher’s manipulation of variables, but of staff feeling important because they were being observed. Stephen Draper gives a definition of the Hawthorne effect that I find useful:

An experimental effect in the direction expected but not for the reason expected; i.e. a significant positive effect that turns out to have no causal basis in the theoretical motivation for the intervention, but is apparently due to the effect on the participants of knowing themselves to be studied in conjunction with the outcomes measured (Draper, 2009).

Looking at this in terms of Empowerment Evaluation, I’ve come to feel that the evaluation team’s experience of the evaluator is a major part of their experience of the evaluation. This makes it a legitimate part of the process use of the evaluation, which is the mechanism expected to enable the empowerment result.

If so, it’s important to clarify what the effect is in each situation. This will depend on what the evaluator is doing, including the atmosphere they’re providing, and the extent to which the evaluator’s involvement is part of the positive reinforcement that team members’ experience, along with their own decision-making, in the course of the evaluation.

The Hawthorne effect can be expected to modify results outside of the conscious parameters of the investigation unless consciously allowed for. In the case of data gathering, I have decided to combine observation of classes run by staff who were part of the evaluation team with observation of classes that weren’t, hoping that any Hawthorne effect in the result-gathering will be canceled out by a parallel Hawthorne effect in my observation of the non-participants.

Hot Tip: Impression management, common in focus groups, and based on Irving Goffman’s work, may be relevant here.

Resource: Draper’s article explains the issues involved and the many uses of the Hawthorne experiment’s continuing legacy. Draper, S.W. (2009, Dec 23) The Hawthorne, Pygmalion, Placebo and other effects of expectation: some notes

Lessons Learned: While the most common explanation of the Hawthorne experience is some kind of “people felt important” effect, Paul Blumberg’s 1969 Industrial Democracy: The Sociology of Participation (Schocken) already argued, based on the original research, that the most likely factor was the level of the participants’ involvement in decision-making. This aspect has been consistently ignored.

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org.

empowerment, Hawthorne Effect

Hi, my name is Susan Geier and I am a doctoral student at Purdue University studying research methods, measurement and evaluation. I employ a participatory evaluation approach with the GEMscholar project and have learned much from the Native American college students and the dedicated program staff.

Lessons Learned: I would like to share my three R’s for participatory evaluation:

1. Build Rapport: In addition to conducting formal interviews and assessments, I interacted informally with the students and mentors when time allowed, during meals and in between activities. I spent time learning about Native American history and culture from the project team and students.

2. Demonstrate Relevance: I discussed with the stakeholders and participants possible benefits of the evaluation process and their unique roles in the improvement and success of the program components. For example, when the students expressed interest in helping future GEMscholars, a peer-mentoring option was added to the program. Consequently, students began to see the evaluation process as a mechanism for sharing their experiences and suggestions instead of an outside critique of their lives and activities.

3. Maintain Responsiveness: I provided the stakeholders with information in a timely and accessible format. Often these were oral reports followed by brief documents outlining the changes discussed. We had conversations about those issues that could not be resolved in a timely matter and possible effects on the program. In turn, the project team made ongoing changes, adding components where needed and modifying those elements that were not serving the objectives of the program. Assessments were modified if needed and the process continued.

Hot Tip: Journaling is a useful technique to capture real time reactions to interventions. This is particularly important when working with groups who are being introduced to unfamiliar and/or uncomfortable experiences as part of an intervention. I worked closely with the researcher and program coordinator to develop pertinent guiding questions for the students’ and mentors’ daily reflection journals. This is also a good time to develop an analysis rubric if applicable. Journals can be hand written or online (I provide a link to an online journal using Qualtrics). The journal entries provide a project team with valuable insights about how the program elements are perceived by all involved.

If you want to learn more from Susan, check out the Poster Exhibition on the program for Evaluation 2010, November 10-13 in San Antonio.

College Access Programs, Native American, participatory

My name is Jane Davidson and I run an evaluation consulting business called Real Evaluation Ltd. In my work, I advise and support organizations on strategic evaluation; provide evaluation capacity building and professional development; develop tools and templates to help organizations conduct, interpret, and use evaluations themselves; and conduct independent and collaborative evaluations and meta-evaluations.

Over several years’ working with clients and reviewing (at clients’ request) disappointing evaluation reports, I have noticed several critically important elements that make or break evaluation work but are often missing from evaluators’ methodological toolkits.

Hot tip: Clients find it incredibly frustrating to wade through an evaluation report full of evidence and still be none the wiser at the end whether the documented outcomes (let alone the entire program/policy/etc) are any good or not. A key part of an evaluator’s work is to say clearly and explicitly how practically, educationally, socially, or economically (not just statistically) significant outcomes are (severally, and as a set). This is what makes evaluation ‘e-VALU-ation’!

Hot tip: A useful tool for generating real evaluative conclusions is an evaluative rubric. This is a table describing what different levels of performance, value, or effectiveness ‘look like’ in terms of the mix of evidence on each criterion. Grading rubrics have been used for many years in student assessment. Evaluative rubrics make transparent how quality and value are defined and applied. I sometimes refer to rubrics as the antidote to both ‘Rorschach inkblot’ (“You work it out”) and ‘divine judgment’ (“I looked upon it and saw that it was good”)-type evaluations.

Hot tip: Collaborative development of rubrics is a great way to get stakeholders thinking about how ‘quality’ and ‘value’ should be defined for the work they do. It helps build the evaluative thinking needed to generate, understand, accept, and use evaluation findings.

Rad resources:

• Evaluation Methodology Basics: The nuts and bolts of sound evaluation by E. Jane Davidson (2005)
• Improving evaluation questions and answers: Getting actionable answers for real-world decision makers (AEA e-Library’s most viewed and downloaded item)
• Example rubrics in Nunns, Roorda, et al’s (2010) Evaluation of the Recognised Seasonal Employer Policy
• Example rubric (referred to as a ‘global assessment scale’) developed for the evaluation of the Corangamite Salinity Program (case study #10 in Jessica Dart et al’s 1998 Review of Evaluation in Agricultural Extension, pp. 62-63 – a publication from the Rural Industries Research and Development Corporation)
• AEA conference professional development workshop about how to use rubrics (and other evaluation nuts and bolts) to do actionable evaluations (November 10, 2010)
• Strategic evaluation of the workplace assessment program, a relevant recent chapter from Jane in the Handbook of Workplace Assessment (2010)

This contribution is from the aea365 Daily Tips blog, by and for evaluators, from the American Evaluation Association. Please consider contributing – send a note of interest to aea365@eval.org. Want to learn more from Jane? She’ll be presenting as part of the Evaluation 2010 Conference Program, November 10-13 in San Antonio.

Rubric, Toolkit