Hello, I am Kerry Bruce, the Director of Results and Measurement at Pact.  I’m currently based in Madagascar and support Pact programs in Africa, Asia and Eastern Europe. I am part of Pact’s central technical team that provides monitoring and evaluation support to more than 20 country offices and more than 70 projects around the world.  In 2012 we started to roll out the use of mobile technology in our programs including evaluation.  This is the second post on mobile technology and focuses on choosing your platform. Check out my first post on this topic: Getting Started with Mobile Phones.

Hot Tips:

• Look at a wide range of available platforms and ask yourself:
  • What is my budget for phones?  Some platforms work better than others with entry level (vs. Android) smartphones.
  • What is my budget for the data collection?  Will my data collection reoccur frequently (on-going evaluation) or is this a one time event?  Each platform has a different pricing structure and each lends itself to different types of data collection.
  • Platform operators will promise you the moon – but will their platform deliver?  Test basic issues such as skip logic, ease of set-up and use, how data download and dashboards work before you buy.  Most platforms have a trial version that you can use and some allow small data collection projects for free.
  • Will I need help to set up my survey, or do I have the skill set to set it up in house?  Some platforms offer survey set up and technical support (useful for complicated data collection exercises) and some are all do-it-yourself.
  • What language will the survey be in and can the platform support it?  This is especially important for non-Latin alphabets.
  • Get a reference.  All these platforms should be able to provide you with a reference from someone who has used them before and can tell you what is good and what needs work.

Lesson Learned: Evaluate two or more platforms before you decide which one to use. 

• Some have recurrent or annual costs and others only charge for the data that you collect.  Others are free up to a certain level of data collection.
• Each platform has its strengths (and weaknesses) – you’ll need to understand what you need it to do and shop around until you find it.
• Just because a platform can not do something today does not mean they won’t be able to do it tomorrow, check back and give feedback.  This technology is rapidly adapting.

Rad Resources: Here is a list of some of the mobile technology platforms that are commercially available today.

*These are platforms I have used.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

technology

I’m Chad Green, Program Analyst at Loudoun County Public Schools in Ashburn, VA. For over seven years I’ve served as an internal evaluator of instructional initiatives sponsored by central office administrators.

Do you have an interest in understanding school-based professional development from a sociocultural learning perspective?  Read on! Years ago I evaluated two school-wide improvement initiatives using an integrated conceptual framework.  The first component was Learning Forward’s original context standards which today serve as its first three standards for professional learning. The purpose of this framework was to constrain the data to essential long-term staff development outcomes.  The second component (Honig, 2008) operationalized the first one into six overlapping sociocultural learning practices, two for each context standard (see below).

(Share Clip)

Framework for High-Quality, School-Based Professional Development

I.   Skillful leadership is evidenced when school and central office staff:

1. Model high quality teaching and learning practices
2. Boundary span to connect staff with new sources of expertise

II.  Professional learning communities are evidenced when school and central office staff:

1. Interact at a high level of collaborative inquiry
2. Engage in joint work on authentic tasks that are meaningful and sustained over time

III. Dedicated resources are evidenced when school and central office staff:

1. Provide access to ongoing, job-embedded learning opportunities that increase the level of participation in shared work practices (i.e., from novice to expert)
2. Develop common conceptual and practical tools (e.g., principles, frameworks, routines, language, protocols, templates, materials)

Lesson Learned:  The patterns that emerged from the data were surprising on two levels.  At a superficial level they revealed a continuum of leadership approaches to program implementation ranging from a top-down, hierarchical structure on one end to a more subtle, heterarchical structure on the other. Coincidentally, these leadership structures aligned with the level of diversity (i.e., complexity) of the school’s student populations.  At a deeper level, the findings suggested a connection between each school’s sources of power and knowledge (i.e., truth).  In the top-down structure, tacit knowledge was concentrated in the principal and specialist roles (i.e., authority) whereas in the heterarchical setting knowledge was more explicit in the form of online repositories of co-created tools and resources.

Hot Tip:  Since then, I have learned that I am much more effective when I help central office administrators integrate their prepackaged conceptual frameworks (i.e., programs) into coherent strategic thinking portfolios which facilitate increased experimentation and interconnectedness system-wide.

Rad Resource: Check out Honig’s journal article on district central office as learning organizations.

Final Word: Both schools’ staff development programs were equally effective in the short run with respect to implementation and outcomes.  Which school structure do you think will be more sustainable in the long run?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

education · Professional Development

Our names are Marc Wheeler and Salem Valentino and we are internal evaluators for Big Brothers Big Sisters of America.

There is a lot of buzz today about infographics.  Many of you may have thought about using infographics in your evaluation reports to try to translate your findings more effectively.  We recently took the plunge and incorporated infographics in our 2013 Youth Outcomes Report .

For example, we created the infographic below to clarify the theoretical connection between the short-term outcomes we currently measure for each youth and those long-term outcomes of interest to many stakeholders. The graphic summarized a large quantity of research literature in a single infographic that was easily interpretable and concise.

Lessons Learned:

1. High-quality infographics require a certain level of expertise; as we didn’t have the relevant experience in house, we contracted with an external graphic designer who delivered great results.
2. Beyond experience, it also takes time to develop your infographics and get them right.  First, you need to figure out the story behind your data.  Then, what are the best ways to illustrate this story, while remaining true to the data?  For instance, we wanted to communicate effect sizes but didn’t want to take up space in our report explaining what they meant to a lay audience.  So instead, we developed the visualization below to better illustrate our story of the magnitude of youth outcomes.  Lastly, infographics require a number of iterations and can benefit from the input of diverse audiences.  Budget your time accordingly.

1. Understanding your project needs will help you choose the right designer. We looked at several resources on the internet to find a graphic designer. For us it was important that our work with the designer was collaborative; we wanted to ensure the quality of the evaluation content. Due to our timeline, it was also important that our designer could design the entire report and not merely the infographics.  You may also want to ask your designer how comfortable they are with Excel or other types of data you will be using in the report.

Rad Resources:

Elissa Schloesser at Visual Voice – our designer’s 5 Steps for Translating Evaluation Findings into Infographics

Visual.ly’s Marketplace service will find a designer for you and help you create an infographic for one price.

Easel.ly is a website where you can create your own infographic for free.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

data visualization · evaluation reporting · infographics

Greetings! I’m Nichole Stewart, a doctoral student in UMBC’s Public Policy program in the evaluation and analytical methods track. I currently work as an analyst, data manager, and evaluator across a few different sites including Baltimore Integration Partnership, Baltimore Workforce Funders Collaborative, and Carson Research Consulting Inc.

Lessons Learned: The Growing Role of Data Science for the “Little” Data in Program Evaluation. Evaluators are increasingly engaged in data science along every step of the evaluation cycle. Collecting participant-level data and developing indicators to measure program outputs and outcomes is now only a small part of the puzzle. Evaluators are working with more complex data sources (administrative data), navigating and querying data management systems (ETO), exploring advanced analytic methods (propensity score matching), and using technology to visualize evaluation findings (R, Tableau).

Evaluators Also Use Big Data.  Large secondary datasets are appropriate in needs assessments and for measuring population-level outcomes. Community-level data, or data available for small levels of geography, provide context and can be used to derive neighborhood indicators. Evaluators must be able to not only access and manipulate this and other kinds of Big Data but to ultimately learn to use data science to maximize the value of the data.

Rad Resource: The American Community Survey (ACS)  is an especially rich, although recently controversial, Big Data resource for evaluators. The survey offers a wide range of data elements for areas as small as the census block and as specific as the percent of carpoolers working in service occupations in a census tract.

Hot Tips:

• View the latest technical document for the ACS 5-Year estimates for a list of available data elements (p. 43).
• Visit the National Historical Geographic Information System to download Census and ACS data as well as GIS boundary files for mapping.

Rad Resource: The Census Bureau’s OnTheMap application is an interactive web-based tool that provides counts of jobs and workers and information about commuting patterns that I explored in an AEA Coffee Break webinar.

Lessons Learned: Data Science is Storytelling: Below is a map of unemployment rates by census tract from the ACS for Baltimore City and surrounding counties.  This unemployment data is overlaid with data extracted from OntheMap depicting job density and the top 25 work destinations for Baltimore City residents.  The map shows that 1) there are high concentrations of unemployed residents in inner-city Baltimore compared to other areas, 2) jobs in the region are concentrated in Downtown Baltimore and along public transportation lines and the beltway, and 3) many Baltimore City workers commute to areas in the surrounding counties for work.  Alone, these two datasets are robust but their power lies in visualizing data and interpreting relevant intersections between them.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

big data · community mapping · Data · GIS · mapping

I’m Clara Hagens. I work for Catholic Relief Services (CRS) as the Regional Technical Advisor for Monitoring, Evaluation, Accountability and Learning in Asia. I’d like to share with you a guidance document we have developed to support project teams to operationalize monitoring and evaluation (M&E) plans, big and small, in various contexts.

Rad Resource: CRS’ Guidance on Monitoring and Evaluation covers a range of topics related to basic M&E concepts and to designing and implementing M&E activities. The topics include gender in M&E, random and purposeful sampling, developing qualitative data collection tools, M&E in emergencies, and community participation in M&E to name a few. Each topic is grounded in a set of standards to guide our M&E practice. The standards are accompanied by narrative to explain how each can be achieved, tips and good practices, examples, and planning tables and templates.

For example, the Guidance provides standards for Community Participation in M&E that state that M&E systems track the changes most important to communities and communities participate in data collection and in the interpretation of M&E results and includes tips for each step in the process. The standards for Planning and Conducting an Evaluation refer to the importance of developing project-specific evaluation questions related to the relevance, effectiveness, efficiency, impact and sustainability. The Guidance also includes a simple evaluation planning table which helps teams to link the methods for data collection and respondents to the evaluation questions.

(Share Clip)

The CRS Guidance on Monitoring and Evaluation is appropriate for project teams who are looking for additional hands-on support to further engage with their M&E systems. I hope you will find this useful!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org . aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

commune · evaluation questions · Monitoring and Evaluation · sustainability

Changes are afoot! My name is Susan Kistler and I am just about to become the American Evaluation Association’s Executive Director emeritus. I’ve written before about my stepping down and my last official day in this position is this coming Monday, May 20. I’ll continue to write for aea365, and will be working on the Evaluation 2013 conference program, but will be moving on to new and exciting opportunities and serving AEA primarily in a volunteer capacity. The new staff team coming on board brings a breadth and depth of knowledge of management and communications that represents a significant increase in AEA’s organizational capacity. I can’t wait to see what new things come about in the months ahead.

Rad Resource – Denise Roosendaal: AEA is searching for a permanent Executive Director (see the position description here). In the meantime, Denise Roosendaal, CAE, is the interim ED and may be reached at droosendaal@eval.org. Denise brings to the position twenty-five years of experience in running nonprofits and associations and she can’t wait to get to know more about AEA and evaluation. If you are attending the Summer Institute next month in Atlanta, be sure to stop by the registration desk and to say “hello.”

Lessons Learned – What to Expect: You likely won’t see major changes in AEA’s programs and services in the near-term and only improvements in the long-term.  aea365 will continue to come out daily under the watchful eye of Sheila Robinson, our lead volunteer curator (she may be reached at aea365@eval.org). The association will have a new phone number (1-202-367-1166) but the old one will forward until the end of June. There will be new voices on the other end of the phones, but with the same commitment to serving AEA. We ask that you are patient in the next few weeks as the new staff gets up to speed. If they don’t know the answer, they’ll find it and get back to you expeditiously.

Get Involved – Share Your Ideas: Have a great idea for something new for AEA? Email Denise at droosendaal@eval.org or put it in the comments on aea365.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello, I am Vanessa Hiratsuka, secretary of the Alaska Evaluation Network (AKEN) and a senior researcher at Southcentral Foundation (SCF), a tribally owned and managed regional health corporation based in Anchorage, Alaska, which serves Alaska Native and American Indian people.

As part of Commitment to Quality, a key organizational value, Southcentral Foundation (SCF) prioritizes continuous quality improvement (CQI), quality assurance, program evaluation, and research.

Although the strategies and tools used in CQI, quality assurance, program evaluation, and research are similar, we do different things. One of our challenges is to help staff across the organization understand who does what. Because these four fields differ in aim and audience, exploring the goals of a project (aim) and who will use its findings (audience) provides a useful framework to determine where a project fits.

At SCF, improvement staff work directly with SCF department and clinic processes to develop and implement project performance measures and outcome indicators as well as help staff (audience) improve processes to better meet customer-owner needs and inform business directions (aim).  Quality Assurance staff conduct quality monitoring to ensure programs are complying (aim) with SCF processes and the requirements of our accrediting bodies (internal and external audiences).

SCF internal evaluators measure programs’ performance (aim) and provide feedback to programmatic stakeholders — including staff, leadership, and funders (audience). The SCF research department’s projects address questions of clinical significance to contribute to generalizable knowledge (aim) for use within SCF and for dissemination in the scientific literature around American Indian and Alaska Native health (audience).

Lessons Learned:

-        Define the aim and intended audience early in the process! This helps identify the stakeholders, level of review, and oversight needed during all stages of a project, including development, implementation, and dissemination of findings.

-        Broadly disseminate findings! Findings and recommendations from all disciplines are only useful when they are shared. At SCF, findings are shared at interdivisional committee meetings and with staff who oversee the work of departments. Multipronged dissemination ensures involvement from all levels of SCF and supports innovation and the spread of new knowledge.

-        Project review can be complicated!  At SCF, research projects must be vetted through a tribal concept review phase, an Institutional Review Board review, and finally a tribal review of the proposal.  Later, all research dissemination products (abstracts for presentation, manuscripts, and final reports) are also required to undergo a tribal research review process. These take time, so it is important to understand the processes and timelines and build review time into your project management timelines.

Check out these posts on understanding evaluation:

1. 1.    Gisele Tchamba on Learning the Difference between Evaluation and Research
2. 2.    John LaVelle on Describing Evaluation

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · continuous improvement

Greetings! My name is kas aruskevich and I am principal of Evaluation Research Associates LLC. I live in Fairbanks and work primarily in rural Alaska. Alaska is known for its great natural beauty, extreme temperatures, and unique context of diverse and far-flung communities assessable only by air. Alaska is the largest state in the U.S.

Rural communities often have a small population and rarely have a local evaluator for hire. Consequently, a program evaluator is most often hired from outside the community or region. Helicopter evaluation is a depreciating term used to describe a drop in – evaluate – depart approach. Today’s post talks about methods to strengthen and add depth to evaluations that involve distance between evaluator and evaluand.

Hot Tip: First, context is important. Familiarize yourself with the community and region before you travel. Gather demographic data of the community, leading industry, and cultural composition. Learn about the organization hosting the program, before your first contact. Plan your site-visit around a community event so you can see the community in a broader context.

Rad Resource: The importance of context is discussed in New Directions for Evaluation Fall 2012, Issue 135.

Hot Tip: Next, work to build open communication with program staff. Begin with a teleconference to provide an opportunity to meet staff and organization and discuss program status. Teleconferences also give you a chance to describe your evaluation style and see if you are a ‘fit’ for the organization and the evaluation project.

ALWAYS include participatory methods. I don’t ‘come in’ as the expert with an unchangeable evaluation design, but instead write up suggestions for the evaluation to negotiate before a plan is finalized. As an itinerant evaluator you can’t be on site as often as you might like. Using a participatory evaluation approach, program staff can be involved in the evaluation through taking photos or identifying program participants or stakeholders to interview.

Rad Resource – Read more about participatory evaluation in Cousins and Chouinard’s new book Participatory Evaluation Up Close.

Hot Tip: Lastly, work to build a friendly relationship based on mutual interests with at least one person in the organization or community. After years of conducting evaluations, friendly relationships have evolved into continuing friendships. These friendships have mutual benefits, in-part, they are a bridge for the evaluator to learn community specific cultural protocols–very important to conduct evaluations in cross-cultural settings – which in turn can strengthen the program through appropriate evaluation.

Lesson Learned: Itinerant evaluation can be much more than a helicopter site-visit approach. Regular communication and working together with program staff as a team can expand the evaluative evidence collected and increase report credibility, relevance, and use by the program staff.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · participatory · rural

We are Alexandra Hill and Diane Hirshberg, and we are part of the Center for Alaska Education Policy Research at the University of Alaska Anchorage.  The evaluation part of our work ranges from tiny projects – just a few hours spent helping someone design their own internal evaluation – to rigorous and formal evaluations of large projects.

In Alaska, we often face the challenge of conducting evaluations with very small numbers of participants in small, remote communities. Even in Anchorage, our largest city, there are only 300,000 residents. We also work with very diverse populations, both in our urban and rural communities. Much of our evaluation work is on federal grants, which need to both meet federal requirements for rigor and power, and be culturally responsive across many settings.

Lesson Learned: Using mixed-methods approaches allows us to both 1) create a more culturally responsive evaluation; and 2) provide useful evaluation information despite small “sample” sizes. Quantitative analyses often have less statistical power in our small samples than in larger studies, but we don’t simply want to accept lower levels of statistical significance, or report ‘no effect’ when low statistical power is unavoidable.

Rather, we start with a logic model to ensure we’ve fully explored pathways through which the intervention being evaluated might work, and those through which it might not work as well.  This allows us to structure our qualitative data collection to explore and examine the evidence for both sets of pathways.  Then we can triangulate with quantitative results to provide our clients with a better sense of how their interventions are working.

At the same time, the qualitative side of our evaluation lets us lets us build in measures that are responsive to local cultures, include and respect local expertise, and (when we’re lucky) build bridges between western academic analyses and indigenous knowledge. Most important, it allows us to employ different and more appropriate ways of gathering and sharing information across indigenous and other diverse communities. 

Rad Resource: For those of you at universities or other large institutions that can purchase access to it we recommend SAGE Research Methods.  This online resource provides access to full text versions of most SAGE research publications, including handbooks of research, encyclopedias, dictionaries, journals, and ALL the Little Green Books and Little Blue Books.

Rad Resource: Another Sage-sponsored resource is Methodspace, an online network for researchers. Sign-up is free, and Methodspace posts selected journal articles, book chapters and other resources, as well as hosting online discussions and blogs about different research methods.

Rad Resource: For developing logic models, we recommend the W.K. Kellogg Foundation Logic Model Development Guide.

(Share Clip)

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · culturally responsive evaluation · logic models · mixed methods

Greetings from the Last Frontier. I’m Alda Norris, webmaster for the Alaska Evaluation Network (AKEN) and evaluation specialist for the University of Alaska Fairbanks Cooperative Extension Service (CES).

The faculty and staff I work with at CES are experts in a variety of fields, from horticulture, entomology and forestry to economics, nutrition and child development. That adds up to quite an interdisciplinary organization! Our diversity makes for fantastic collaborations, as well as complicated syntheses. Lucky for me, my PhD is in interpersonal communication, which applies across the board.

Lessons Learned:  Ask people to tell you the inspiration behind their projects. Every group has a story to tell.What common goals bring these people together?Inquiring about the “why” and not just the “what” of a program really benefits capacity building efforts. I got to know CES better while writing a Wikipedia entry. Hearing and reading about the contributions Extension has made in Alaska since the 1930s deepened my understanding of what led up to each of our program’s current priorities and logic models.

• Help yourself with history. Too often we are mired in a static view of where an organization is now, rather than having an appreciation for how it has changed, and continues to change, over time. Even in a “young” state like Alaska, there is rich historical data we can learn from.

• Boost your evaluation planning by gathering information on your/the client organization’s “story” from a variety of sources. Talk to emeritus professors, compare the org chart of today to past decades, and comb through newspaper archives. Becoming familiar with past waves of change is very helpful in understanding the meaning behind current missions, goals and structures (and people’s attachments to them).

Hot tip: Communicate about communication! Add a question about communication preferences to your next needs assessment. Don’t assume you know what level of technology and form(s) of interaction your colleagues and clients are comfortable with. Before you do a survey, figure out what modes of communication the target population values. For example, if oral history is a large part of a sample group’s culture, how well will a paper and pencil form be received?

Rad Resources:

1. The National Communication Association (NCA) can help you step up your message design game. Take advantage of free advice from experts on verbal and nonverbal communication by reading NCA’s newsletter, Communication Currents.
2. AnyMeeting is a freetool that you can use to reach a wider audience. With it, you can host online meetings and make instructional videos, both of which are really handy when working in a geographically diverse setting. AnyMeeting also has screenshare clarity in its recordings that Google Hangouts lacks.

The American Evaluation Association is celebrating Alaska Evaluation Network (AKEN) Affiliate Week. The contributions all this week to aea365 come from AKEN members. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

affiliates · communication · Extension