Our names are Marc Wheeler and Salem Valentino and we are internal evaluators for Big Brothers Big Sisters of America.

There is a lot of buzz today about infographics.  Many of you may have thought about using infographics in your evaluation reports to try to translate your findings more effectively.  We recently took the plunge and incorporated infographics in our 2013 Youth Outcomes Report .

For example, we created the infographic below to clarify the theoretical connection between the short-term outcomes we currently measure for each youth and those long-term outcomes of interest to many stakeholders. The graphic summarized a large quantity of research literature in a single infographic that was easily interpretable and concise.

Lessons Learned:

1. High-quality infographics require a certain level of expertise; as we didn’t have the relevant experience in house, we contracted with an external graphic designer who delivered great results.
2. Beyond experience, it also takes time to develop your infographics and get them right.  First, you need to figure out the story behind your data.  Then, what are the best ways to illustrate this story, while remaining true to the data?  For instance, we wanted to communicate effect sizes but didn’t want to take up space in our report explaining what they meant to a lay audience.  So instead, we developed the visualization below to better illustrate our story of the magnitude of youth outcomes.  Lastly, infographics require a number of iterations and can benefit from the input of diverse audiences.  Budget your time accordingly.

1. Understanding your project needs will help you choose the right designer. We looked at several resources on the internet to find a graphic designer. For us it was important that our work with the designer was collaborative; we wanted to ensure the quality of the evaluation content. Due to our timeline, it was also important that our designer could design the entire report and not merely the infographics.  You may also want to ask your designer how comfortable they are with Excel or other types of data you will be using in the report.

Rad Resources:

Elissa Schloesser at Visual Voice – our designer’s 5 Steps for Translating Evaluation Findings into Infographics

Visual.ly’s Marketplace service will find a designer for you and help you create an infographic for one price.

Easel.ly is a website where you can create your own infographic for free.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · evaluation reporting · infographics

Greetings! I’m Nichole Stewart, a doctoral student in UMBC’s Public Policy program in the evaluation and analytical methods track. I currently work as an analyst, data manager, and evaluator across a few different sites including Baltimore Integration Partnership, Baltimore Workforce Funders Collaborative, and Carson Research Consulting Inc.

Lessons Learned: The Growing Role of Data Science for the “Little” Data in Program Evaluation. Evaluators are increasingly engaged in data science along every step of the evaluation cycle. Collecting participant-level data and developing indicators to measure program outputs and outcomes is now only a small part of the puzzle. Evaluators are working with more complex data sources (administrative data), navigating and querying data management systems (ETO), exploring advanced analytic methods (propensity score matching), and using technology to visualize evaluation findings (R, Tableau).

Evaluators Also Use Big Data.  Large secondary datasets are appropriate in needs assessments and for measuring population-level outcomes. Community-level data, or data available for small levels of geography, provide context and can be used to derive neighborhood indicators. Evaluators must be able to not only access and manipulate this and other kinds of Big Data but to ultimately learn to use data science to maximize the value of the data.

Rad Resource: The American Community Survey (ACS)  is an especially rich, although recently controversial, Big Data resource for evaluators. The survey offers a wide range of data elements for areas as small as the census block and as specific as the percent of carpoolers working in service occupations in a census tract.

Hot Tips:

• View the latest technical document for the ACS 5-Year estimates for a list of available data elements (p. 43).
• Visit the National Historical Geographic Information System to download Census and ACS data as well as GIS boundary files for mapping.

Rad Resource: The Census Bureau’s OnTheMap application is an interactive web-based tool that provides counts of jobs and workers and information about commuting patterns that I explored in an AEA Coffee Break webinar.

Lessons Learned: Data Science is Storytelling: Below is a map of unemployment rates by census tract from the ACS for Baltimore City and surrounding counties.  This unemployment data is overlaid with data extracted from OntheMap depicting job density and the top 25 work destinations for Baltimore City residents.  The map shows that 1) there are high concentrations of unemployed residents in inner-city Baltimore compared to other areas, 2) jobs in the region are concentrated in Downtown Baltimore and along public transportation lines and the beltway, and 3) many Baltimore City workers commute to areas in the surrounding counties for work.  Alone, these two datasets are robust but their power lies in visualizing data and interpreting relevant intersections between them.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

big data · community mapping · Data · GIS · mapping

I’m Clara Hagens. I work for Catholic Relief Services (CRS) as the Regional Technical Advisor for Monitoring, Evaluation, Accountability and Learning in Asia. I’d like to share with you a guidance document we have developed to support project teams to operationalize monitoring and evaluation (M&E) plans, big and small, in various contexts.

Rad Resource: CRS’ Guidance on Monitoring and Evaluation covers a range of topics related to basic M&E concepts and to designing and implementing M&E activities. The topics include gender in M&E, random and purposeful sampling, developing qualitative data collection tools, M&E in emergencies, and community participation in M&E to name a few. Each topic is grounded in a set of standards to guide our M&E practice. The standards are accompanied by narrative to explain how each can be achieved, tips and good practices, examples, and planning tables and templates.

For example, the Guidance provides standards for Community Participation in M&E that state that M&E systems track the changes most important to communities and communities participate in data collection and in the interpretation of M&E results and includes tips for each step in the process. The standards for Planning and Conducting an Evaluation refer to the importance of developing project-specific evaluation questions related to the relevance, effectiveness, efficiency, impact and sustainability. The Guidance also includes a simple evaluation planning table which helps teams to link the methods for data collection and respondents to the evaluation questions.

(Share Clip)

The CRS Guidance on Monitoring and Evaluation is appropriate for project teams who are looking for additional hands-on support to further engage with their M&E systems. I hope you will find this useful!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

commune · evaluation questions · Monitoring and Evaluation · sustainability

Changes are afoot! My name is Susan Kistler and I am just about to become the American Evaluation Association’s Executive Director emeritus. I’ve written before about my stepping down and my last official day in this position is this coming Monday, May 20. I’ll continue to write for aea365, and will be working on the Evaluation 2013 conference program, but will be moving on to new and exciting opportunities and serving AEA primarily in a volunteer capacity. The new staff team coming on board brings a breadth and depth of knowledge of management and communications that represents a significant increase in AEA’s organizational capacity. I can’t wait to see what new things come about in the months ahead.

Rad Resource – Denise Roosendaal: AEA is searching for a permanent Executive Director (see the position description here). In the meantime, Denise Roosendaal, CAE, is the interim ED and may be reached at droosendaal@eval.org. Denise brings to the position twenty-five years of experience in running nonprofits and associations and she can’t wait to get to know more about AEA and evaluation. If you are attending the Summer Institute next month in Atlanta, be sure to stop by the registration desk and to say “hello.”

Lessons Learned – What to Expect: You likely won’t see major changes in AEA’s programs and services in the near-term and only improvements in the long-term.  aea365 will continue to come out daily under the watchful eye of Sheila Robinson, our lead volunteer curator (she may be reached at aea365@eval.org). The association will have a new phone number (1-202-367-1166) but the old one will forward until the end of June. There will be new voices on the other end of the phones, but with the same commitment to serving AEA. We ask that you are patient in the next few weeks as the new staff gets up to speed. If they don’t know the answer, they’ll find it and get back to you expeditiously.

Get Involved – Share Your Ideas: Have a great idea for something new for AEA? Email Denise at droosendaal@eval.org or put it in the comments on aea365.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, I am Vanessa Hiratsuka, secretary of the Alaska Evaluation Network (AKEN) and a senior researcher at Southcentral Foundation (SCF), a tribally owned and managed regional health corporation based in Anchorage, Alaska, which serves Alaska Native and American Indian people.

As part of Commitment to Quality, a key organizational value, Southcentral Foundation (SCF) prioritizes continuous quality improvement (CQI), quality assurance, program evaluation, and research.

Although the strategies and tools used in CQI, quality assurance, program evaluation, and research are similar, we do different things. One of our challenges is to help staff across the organization understand who does what. Because these four fields differ in aim and audience, exploring the goals of a project (aim) and who will use its findings (audience) provides a useful framework to determine where a project fits.

At SCF, improvement staff work directly with SCF department and clinic processes to develop and implement project performance measures and outcome indicators as well as help staff (audience) improve processes to better meet customer-owner needs and inform business directions (aim).  Quality Assurance staff conduct quality monitoring to ensure programs are complying (aim) with SCF processes and the requirements of our accrediting bodies (internal and external audiences).

SCF internal evaluators measure programs’ performance (aim) and provide feedback to programmatic stakeholders — including staff, leadership, and funders (audience). The SCF research department’s projects address questions of clinical significance to contribute to generalizable knowledge (aim) for use within SCF and for dissemination in the scientific literature around American Indian and Alaska Native health (audience).

Lessons Learned:

-        Define the aim and intended audience early in the process! This helps identify the stakeholders, level of review, and oversight needed during all stages of a project, including development, implementation, and dissemination of findings.

-        Broadly disseminate findings! Findings and recommendations from all disciplines are only useful when they are shared. At SCF, findings are shared at interdivisional committee meetings and with staff who oversee the work of departments. Multipronged dissemination ensures involvement from all levels of SCF and supports innovation and the spread of new knowledge.

-        Project review can be complicated!  At SCF, research projects must be vetted through a tribal concept review phase, an Institutional Review Board review, and finally a tribal review of the proposal.  Later, all research dissemination products (abstracts for presentation, manuscripts, and final reports) are also required to undergo a tribal research review process. These take time, so it is important to understand the processes and timelines and build review time into your project management timelines.

Check out these posts on understanding evaluation:

1. 1.    Gisele Tchamba on Learning the Difference between Evaluation and Research
2. 2.    John LaVelle on Describing Evaluation

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · continuous improvement

Greetings! My name is kas aruskevich and I am principal of Evaluation Research Associates LLC. I live in Fairbanks and work primarily in rural Alaska. Alaska is known for its great natural beauty, extreme temperatures, and unique context of diverse and far-flung communities assessable only by air. Alaska is the largest state in the U.S.

Rural communities often have a small population and rarely have a local evaluator for hire. Consequently, a program evaluator is most often hired from outside the community or region. Helicopter evaluation is a depreciating term used to describe a drop in – evaluate – depart approach. Today’s post talks about methods to strengthen and add depth to evaluations that involve distance between evaluator and evaluand.

Hot Tip: First, context is important. Familiarize yourself with the community and region before you travel. Gather demographic data of the community, leading industry, and cultural composition. Learn about the organization hosting the program, before your first contact. Plan your site-visit around a community event so you can see the community in a broader context.

Rad Resource: The importance of context is discussed in New Directions for Evaluation Fall 2012, Issue 135.

Hot Tip: Next, work to build open communication with program staff. Begin with a teleconference to provide an opportunity to meet staff and organization and discuss program status. Teleconferences also give you a chance to describe your evaluation style and see if you are a ‘fit’ for the organization and the evaluation project.

ALWAYS include participatory methods. I don’t ‘come in’ as the expert with an unchangeable evaluation design, but instead write up suggestions for the evaluation to negotiate before a plan is finalized. As an itinerant evaluator you can’t be on site as often as you might like. Using a participatory evaluation approach, program staff can be involved in the evaluation through taking photos or identifying program participants or stakeholders to interview.

Rad Resource – Read more about participatory evaluation in Cousins and Chouinard’s new book Participatory Evaluation Up Close.

Hot Tip: Lastly, work to build a friendly relationship based on mutual interests with at least one person in the organization or community. After years of conducting evaluations, friendly relationships have evolved into continuing friendships. These friendships have mutual benefits, in-part, they are a bridge for the evaluator to learn community specific cultural protocols–very important to conduct evaluations in cross-cultural settings – which in turn can strengthen the program through appropriate evaluation.

Lesson Learned: Itinerant evaluation can be much more than a helicopter site-visit approach. Regular communication and working together with program staff as a team can expand the evaluative evidence collected and increase report credibility, relevance, and use by the program staff.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · participatory · rural

We are Alexandra Hill and Diane Hirshberg, and we are part of the Center for Alaska Education Policy Research at the University of Alaska Anchorage.  The evaluation part of our work ranges from tiny projects – just a few hours spent helping someone design their own internal evaluation – to rigorous and formal evaluations of large projects.

In Alaska, we often face the challenge of conducting evaluations with very small numbers of participants in small, remote communities. Even in Anchorage, our largest city, there are only 300,000 residents. We also work with very diverse populations, both in our urban and rural communities. Much of our evaluation work is on federal grants, which need to both meet federal requirements for rigor and power, and be culturally responsive across many settings.

Lesson Learned: Using mixed-methods approaches allows us to both 1) create a more culturally responsive evaluation; and 2) provide useful evaluation information despite small “sample” sizes. Quantitative analyses often have less statistical power in our small samples than in larger studies, but we don’t simply want to accept lower levels of statistical significance, or report ‘no effect’ when low statistical power is unavoidable.

Rather, we start with a logic model to ensure we’ve fully explored pathways through which the intervention being evaluated might work, and those through which it might not work as well.  This allows us to structure our qualitative data collection to explore and examine the evidence for both sets of pathways.  Then we can triangulate with quantitative results to provide our clients with a better sense of how their interventions are working.

At the same time, the qualitative side of our evaluation lets us lets us build in measures that are responsive to local cultures, include and respect local expertise, and (when we’re lucky) build bridges between western academic analyses and indigenous knowledge. Most important, it allows us to employ different and more appropriate ways of gathering and sharing information across indigenous and other diverse communities. 

Rad Resource: For those of you at universities or other large institutions that can purchase access to it we recommend SAGE Research Methods.  This online resource provides access to full text versions of most SAGE research publications, including handbooks of research, encyclopedias, dictionaries, journals, and ALL the Little Green Books and Little Blue Books.

Rad Resource: Another Sage-sponsored resource is Methodspace, an online network for researchers. Sign-up is free, and Methodspace posts selected journal articles, book chapters and other resources, as well as hosting online discussions and blogs about different research methods.

Rad Resource: For developing logic models, we recommend the W.K. Kellogg Foundation Logic Model Development Guide.

(Share Clip)

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · culturally responsive evaluation · logic models · mixed methods

Greetings from the Last Frontier. I’m Alda Norris, webmaster for the Alaska Evaluation Network (AKEN) and evaluation specialist for the University of Alaska Fairbanks Cooperative Extension Service (CES).

The faculty and staff I work with at CES are experts in a variety of fields, from horticulture, entomology and forestry to economics, nutrition and child development. That adds up to quite an interdisciplinary organization! Our diversity makes for fantastic collaborations, as well as complicated syntheses. Lucky for me, my PhD is in interpersonal communication, which applies across the board.

Lessons Learned:  Ask people to tell you the inspiration behind their projects. Every group has a story to tell.What common goals bring these people together?Inquiring about the “why” and not just the “what” of a program really benefits capacity building efforts. I got to know CES better while writing a Wikipedia entry. Hearing and reading about the contributions Extension has made in Alaska since the 1930s deepened my understanding of what led up to each of our program’s current priorities and logic models.

• Help yourself with history. Too often we are mired in a static view of where an organization is now, rather than having an appreciation for how it has changed, and continues to change, over time. Even in a “young” state like Alaska, there is rich historical data we can learn from.

• Boost your evaluation planning by gathering information on your/the client organization’s “story” from a variety of sources. Talk to emeritus professors, compare the org chart of today to past decades, and comb through newspaper archives. Becoming familiar with past waves of change is very helpful in understanding the meaning behind current missions, goals and structures (and people’s attachments to them).

Hot tip: Communicate about communication! Add a question about communication preferences to your next needs assessment. Don’t assume you know what level of technology and form(s) of interaction your colleagues and clients are comfortable with. Before you do a survey, figure out what modes of communication the target population values. For example, if oral history is a large part of a sample group’s culture, how well will a paper and pencil form be received?

Rad Resources:

1. The National Communication Association (NCA) can help you step up your message design game. Take advantage of free advice from experts on verbal and nonverbal communication by reading NCA’s newsletter, Communication Currents.
2. AnyMeeting is a freetool that you can use to reach a wider audience. With it, you can host online meetings and make instructional videos, both of which are really handy when working in a geographically diverse setting. AnyMeeting also has screenshare clarity in its recordings that Google Hangouts lacks.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · communication · Extension

Hello! My name is Amelia Ruerup, I am Tlingit, originally from Hoonah, Alaska although I currently reside in Fairbanks, Alaska.  I have been working part-time in evaluation for over a year at Evaluation Research Associates and have spent approximately five years developing my understanding of Indigenous Evaluation through the mentorship and guidance of Sandy Kerr, Maori from New Zealand.  I consider myself a developing evaluator and continue to develop my understanding of what Indigenous Evaluation means in an Alaska Native context.

I have come to appreciate that Alaska Natives are historic and contemporary social innovators who have always evaluated to determine the best ways of not only living, but thriving in some of the most dynamic and at times, harshest conditions in the world.  We have honed skills and skillfully crafted strict protocols while cultivating rich, guiding values.  The quality of our programs, projects, businesses and organizations is shaped by our traditions, wisdom, knowledge and values.  It is with this lens that Indigenous Evaluation makes sense for an Alaska Native context as a way to establish the value, worth and merit of our work where Alaska Native values and knowledge both frame and guide the evaluation process.

Amidst the great diversity within Alaska Native cultures we share certain collective traditions and values.  As Alaska Native peoples, we share a historical richness in the use of oral narratives.  Integral information, necessary for thriving societies and passing on cultural intelligence, have long been passed on to the next generation through the use of storytelling. It is also one commonality that connects us to the heart of Indigenous Evaluation.  In the Indigenous Evaluation Framework book, the authors explain that, “Telling the program’s story is the primary function of Indigenous evaluation…Evaluation, as story telling, becomes a way of understanding the content of our program as well as the methodology to learn from our story.” To tell a story is an honor.  In modern Alaska Native gatherings, we still practice the tradition of certain people being allowed to speak or tell stories.  This begs the question: Who do you want to tell your story and do they understand the values that are the foundation and framework for your program?  

Hot Tip: Context before methods.  It is essential to understand the Alaska Native values and traditions that are the core of Alaska Native serving programs, institutions and organizations.  Indigenous Evaluation is an excellent approach to telling our stories.

Rad Resource: The Alaskool website hosts a wealth of information on Alaska Native cultures and values.  This link will take you to a map of “Indigenous Peoples and Languages of Alaska”

The American Evaluation Association is celebrating Alaska Evaluation Network Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · indigenous learning

I’m Corrie Whitmore, president of the Alaska Evaluation Network (AKEN) and an internal evaluator working for Southcentral Foundation (SCF).  SCF is an Alaska Native owned and operated health care organization serving approximately 60,000 Alaska Native and American Indian people living in Anchorage, the Matanuska-Susitna Valley, and 60 rural villages in the Anchorage Service Unit. SCF has had program evaluation in-house since 2009. We are a small department with two evaluators, so it is important for us to build our skills and keep up to date with changes in evaluation practice by staying engaged with the American Evaluation Association (AEA) and local evaluation practitioners.

The Alaska Evaluation Network (AKEN), which is a great resource for all Alaskans interested in evaluation, was founded in 2012 with an emphasis on improving the quality of evaluation research, theory and practice in Alaska and creating forums for dialogue, relationship–building, learning, and collaboration.

Alaska offers a unique environment for evaluation.  According to the 2010 census, we have 730,000 people spread over an area larger than Texas, New Mexico, and Arizona combined. Population density is low, some communities are only accessible by airplane or boat, and many evaluators work in tribal contexts. Building a community of practice encourages AKEN’s members to support evaluation practices that are responsive to the uniqueness of Alaska’s geographic, social, cultural, and administrative context; encourage effective evaluation; improve evaluation capacity within the state; and advocate for evaluation leadership.

AKEN’s goals are to: increase the understanding of evaluation’s purpose and use in Alaska; build evaluator and organization capacity around evaluation approaches, methods, and cultural competency; promote evaluation as a profession; and support the contribution of evaluation to the generation of theory and knowledge about effective human action in Alaska and the circumpolar north.

To date, AKEN has more than 50 members spread from Fairbanks, Alaska, to Southern California (spanning 3000 miles). This geographic dispersal is a strength – we are committed to including evaluators across the state and those living elsewhere who work in Alaska – and a challenge. To address that reality, all of our meetings have a teleconference or web conferencing option, meeting minutes are posted to our website, and much business is done by email.

Lessons Learned:  Connecting with other evaluators in your region can enrich your work and support capacity building. While it would be great to sit across the table from each other, it can be just as valuable to connect using technology!

Get Involved: by joining AKEN or your own local affiliate. These groups are online at the AEA Affiliate List. If your area doesn’t have an affiliate yet – start one! We’d be glad to share our experience with you.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates