Hi! I’m Kirsten Rewey, the Senior Evaluation Associate at ACET, Inc. I have been blogging for just under 2 years on the ACET, Inc. Blog.

Rad Resource – ACET, Inc. Blog: The ACET, Inc. Blog is one way I am able to connect with members of the greater ‘evaluation community.’ And not just other evaluators, but also consumers – those who use evaluation. Through the ACET, Inc. Blog I am able to cover a wide range of evaluation-related topics like capacity building, partnerships, resources (especially free ones), funding opportunities, and evaluation events. The ACET, Inc. Blog also gives me an opportunity to share community events, news about our clients, and personal information about myself and the other ACET staff with our readers. In general, the ACET, Inc. Blog is updated once a week and I publish a new post about once a month.

Hot Tips – favorite posts: Here are some of the ACET, Inc. Blog posts that I’ve found most memorable:

• 4/6/2011 – Top 3 Survey Challenges: I am passionate about developing good surveys that will produce useful data for clients, so in this blog I described the three challenges we most often observe in surveys and how to address each.
• 8/26/2010 – Reviewing Grant Applications, or “What I Learned on my Summer Vacation”: In this article I described three themes I observed while reviewing the evaluation portion of grant applications for the U.S. Department of Education.
• 4/19/2010 – Are Your Goals SMART?: In this article I describe how to develop goals that are SMART – specific, measureable, attainable, relevant, and time-bound.

Lessons Learned – why I blog: My goal with the ACET, Inc. Blog is to reach out to the community so that I can dialog with others about evaluation. Agencies across the U.S. are being challenged to find new and inventive ways to meet evaluation goals and benchmarks with shrinking budgets. I have found that blogging about evaluation topics is one way I can reach out to agencies to help them develop and broaden their evaluation capacity with a minimum investment (the time it takes to read a post).

Lessons Learned: The speed at which I can share information through the ACET, Inc. blog is one of the most appealing features of maintaining a blog. But it can be a challenge to identify a new topic each week.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

I’m Stephanie Evergreen and I blog on the aptly titled “Stephanie Evergreen’s Blog.” Clever, huh?

Rad Resource – evereval.wordpress.com: At this point in its evolution, my blog describes and demonstrates how to visualize data and communicate better as evaluators. I like to think I post new ideas weekly.

Hot Tips – favorite posts: I’ve been posting for about two years. Here are some of the highlights:

• 11/08/11 – Releasing the Evaluation Report Layout Checklist: The checklist I talk about in this post was developed as part of my dissertation and I was keeping it pretty close, only sharing it if people asked me to do so via email. But handling those email requests got cumbersome, so in this post I made the checklist publicly available.
• 10/06/10 – My Graphic Design Circa 2005:  I deeply believe an evaluator should be self-reflective, so I’m happy to make fun of my own bad graphic design. This post features presenting and reporting I did several years ago and all of the ways in which I probably impeded the audience’s ability to comprehend my words.
• 05/03/10 – Nix the Table of Contents: This post exemplifies the somewhat controversial ideas I have about evaluation communication and why I began a blog as an outlet for them.

Lessons Learned – why I blog: The subtitle to my blog is Evaluation Unsanctioned, which explains my impetus for starting to blog in the first place. I have so many random thoughts about evaluation and how evaluators communicate that I needed somewhere appropriate to put them. None are lengthy or scientific enough for a journal article. The conference is only once per year. With a blog, I can give my wayward thoughts a home on a regular basis. This is also how I prepare for ideas to include in my workshops on data visualization and reporting. The blog gives me both a testing ground and a forum to script out my talking points.

Lessons Learned: Post around conference time. When I look at the graph of the number of views of my blog over time, I see major spikes in viewing twice a year that line up precisely with my workshops at the AEA/CDC Summer Institute in June and the AEA annual conference in November. That leads me to believe it is helpful to direct people to the blog as part of my blatant self-promotion and to have some fresh content for people to see when they get there.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

I’m Cameron Norman, and I am the Principal of CENSE Research + Design.

Rad Resource – [CENSEMaking]: CENSEMaking is a play on the name of my research and design studio consultancy and on the concept of sensemaking, something evaluators help with all the time. CENSEMaking focuses on the interplay of systems and design thinking, health promotion and evaluation and weaves together ideas I find in current social issues, reflections on my practice as well as the evidence used to inform it. I aspire to post on CENSEMaking 2-3 times per week, although because it is done in a short-essay format, find the time can be a challenge.

Clipped from: censemaking.com (share this clip)

Hot Tips – favorite posts:

• What is Developmental Evaluation? This post came from a meeting of a working group with Michael Quinn Patton and was fun to write because the original exercise that led to the content (described in the post) was so fun to do. It also provided an answer to a question I get asked all the time.
• Visualizing Evaluation and Feedback. I believe that the better we can visualize complexity the more feedback we provide, the greater the opportunities we have for engaging others, and more evaluations will be utilized. This post was designed to provoke thinking about visualization and illustrate how its been creatively used to present complex data in interesting and accessible ways. My colleague and CENSE partner Andrea Yip has tried to do this with a visually oriented blog on health promoting design, which provides some other creative examples of ways to make ideas more appealing and data feel simpler.
• Developmental Design and Human Services. Creating this post has sparked an entire line of inquiry for me on bridging DE and design that has since become a major focus for my work. This post became the first step in a larger journey.

Lessons Learned – why I blog: CENSEMaking originally served as an informal means of sharing my practice reflections with students and colleagues, but has since grown to serve as a tool for knowledge translation to a broader professional and lay audience. I aim to bridge the sometimes foggy world that things like evaluation inhabit – particularly developmental evaluation – and the lived world of people whom evaluation serves.

Lessons Learned: Blogging is a fun way to explore your own thinking about evaluation and make friends along the way. I never expected to meet so many interesting people because they reached out after reading a blog post of mine or made a link to something I wrote. This has also led me to learn about so many other great bloggers, too. Give a little, get a lot in return and don’t try and make it perfect. Make it fun and authentic and that will do.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

Hi, I’m Susan Barnes, Assistant Director of the National Network of Libraries of Medicine Outreach Evaluation Resource Center (OERC). I manage and, with my colleague Cindy Olney, contribute posts to the OERC Blog.

Rad Resource – OERC Blog: Our blog provides summaries of publications and events related to evaluation of library and health information services for our National Network of Libraries of Medicine (NN/LM) audience. The NN/LM is a network of libraries and information centers that promotes access to quality biomedical and consumer health information managed by the National Library of Medicine. The OERC supports program evaluation in the NN/LM through its publications, classes, and consulting services. The OERC Blog is one of our publications and we include items about using evaluation for library advocacy, practical evaluation approaches, summaries of recent research, and book reviews. It is not a very busy blog; we published 13 posts in 2011.

Hot Tips – favorite posts:

• Can Tweets Predict Citations? January 3, 2012. Article about an investigation of whether tweets predict highly cited articles [they do, at least for articles published in the Journal of Medical Internet Research].
• How Many Interviews Are Enough? September 27, 2011. Presents an evidence-based guideline of twelve interviews in studies involving highly structured interviews of a relatively homogeneous group. The author cautions that more will probably be needed if the topic is diffuse or the population is heterogeneous.
• IdeaScale—Quantified Qualitative Data. September 19, 2011. Description of the IdeaScale software, which is designed to support “crowdsourcing” to collect feedback from communities.
• Can I Hear You Now? September 9, 2011. A TEDtalk , “5 Ways to Listen Better,” presents the RASA acronym –Receive, Appreciate, Summarize, Ask—an approach about interpersonal listening that could be useful for teaching and building interview skills.

Lessons Learned – why I blog: I find blogging to be a useful way to provide my reflections and reactions to articles I’ve read and presentations I’ve attended.

Lessons Learned: I have learned that it is easy to postpone the creation of blog posts, but when I actually work on a post, it’s kind of fun and doesn’t really take too long. I have also learned that blogs receive a lot of spam, and I’m grateful to our IT administrator for installing the Akismet plugin to protect the blog from most spam. Sus

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

Hello, I’m Alexey Kuzmin, president of the Process Consulting Company based in Moscow, Russia. My company’s specialization is program evaluation and evaluation related services. I have two evaluation blogs – in Russian and in English.

Rad Resource – Evaluation Space This is my English blog. Its focus is evaluation – no surprises. The blog concept could be described as sharing thoughts related to my professional life. Everything that appears in my blog is somehow related to what happens to me and my company. New content is posted at least twice a month, and usually – more often.

Hot Tips – favorite posts:

• December 29, 2009 A New Year Post This is the most read article. It is about things that I like and don’t like in my profession. Interestingly, I never expected that it will become most popular.
• Speaking of high quality evaluations… (June 17, 2011) and Comments on the previous post (June 20, 2011)  These are two inter-related posts. In the first one I shared a pretty strange evaluation ToR that I received, and in the second one I described what quite a few colleagues from around the world thought about that ToR.
• Use of proverbs in evaluation: “A word is enough to the wise” (March 31, 2010) I always found it challenging to describe lessons learned by the program. Once I tried to use proverbs to do that. It worked fine for the client and was fun for me.

Lessons Learned – why I blog:

• This is a neat way of sharing what I consider important in evaluation profession and to be explicit about my values while doing that.
• My company website has a link to this blog, so clients can get acquainted with my company and then look at my blog to get better acquainted with me. Those who find the blog first and then decide to learn about the company can easily do that as the blog is indeed linked to the company website.
• Next year I will celebrate my 25^th anniversary in consulting business and 20^th anniversary of my company. It has been quite a journey. Blogging allows me to share experience, which might be interesting and even useful – at least for some people.

Lessons Learned:

• Blogging allows you to be yourself and show who you are.
• Blogging makes you more visible and helps to network.
• Blogging allows you to learn and to share experience.
• Blogging can help you get new clients (I had such experience).
• Blogging is an easy way of keeping record of important thoughts, resources and events. I converted my blogs into PDF books and use them as a resource for my training activities and professional writing.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

Hello world! I’m Jara Dean-Coffey, the Principal and founder of jdcPartnerships, a consulting firm working at the intersection of strategy and evaluation primarily with small to mid size organizations in the social and philanthropic sectors focusing on issues related to equity, access and social justice. The name of our blog is To What End? Because where you are going matters.

Rad Resource –To What End? launched in January 2010. The title speaks to our deeply held belief that without “an end in mind”, our individual and collective efforts (regardless of scale and scope) are less meaningful, less impactful and will not lead to sustained and structural change. We launched the blog to provide a space for our own reflection and to advance our practices as well as share the experiences of our clients.

Clipped from: www.jdcpartnerships.com (share this clip)

Hot Tips – favorite posts: There are more than 74 posts to date which span a range of topics from updates on our work in the field/in-person, synthesis and reflection on conferences/presentations to general musings about concepts, and terms and discussions of interest. Here are four that stand out:

• 7/12/2010 – Be More Strategic With Your Evaluation: This post gave me the opportunity to share a core frame that guides our evaluation work.
• 10/1/2010 – The Difference Between Simple, Complex and Complicated: Making the distinctions helped me in my own work.
• 3/22/2011 – Living Your Theory of Change: This was a great excuse to talk to CompassPoint about the impact of our work with them.
• 5/17/2011 – Words of Wisdom and Wit from Our Clients: It was fun to share insights from our clients.

Lessons Learned – why I blog: I blog because I enjoy sharing our work and that of our clients with others. And every time I see the title page, it reminds me why we do this work and forces me to ask myself “ to what end”. But more specifically, blogging provides me with a space to

• Share and build my own knowledge and expertise
• Demystify evaluation, strategy and the intersection between them
• Reflect on issues, ideas and experiences
• Connect with others that I may not come across in day to day (also why I tweet @jdeancoffey)

Lessons Learned: I am still getting in the hang of having a more consistent blog presence. Truth be told, I have more than 40 titles in the queue but am committed in 2012 to actually writing the preferred 250 to 500 words to fill them out.

This winter, we’re running a series highlighting evaluators who blog. Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

blog · Evaluation

I am Susan Kistler, The American Evaluation Association’s Executive Director, and aea365’s ongoing Saturday contributor.

Hot Tip: We’re looking for an aea365 intern! The aea365 intern is responsible for:

• Recruiting contributors – send invitations, work with leaders of sponsoring groups
• Shepherding contributors – send reminders, ask questions, give thanks
• Providing final copyediting – copyedit and re-vet those contributions requiring an edit
• Uploading contributions – enter posts into the aea365 wordpress-based website

You would have primary responsibility, but a number of other people contribute on a regular basis and shepherd contributing groups as well, so you aren’t working alone.

The obligation requires approximately 5 hours per week of work for six months beginning mid-February and ending in mid-August.

The ideal intern has contributed before to aea365, or at a minimum is a regular reader familiar with the format, breadth, and style of entries. She or he has good writing skills and communications skills and is interested in making connections across the evaluation community. Finally, the work can be done remotely, from anywhere, and thus the intern should be self-directed, organized, and adept at meeting deadlines.

Serving as an aea365 intern is a great way to build your professional network and expand your knowledge of the breadth and depth of the field. The intern will receive ongoing mentoring throughout the term of the internship as well as support in learning how to use wordpress. And, this is a partially paid internship. The selected intern will receive a $2000 stipend, $1000 at the end of the third month and $1000 at the end of the internship. Due to US regulations, the intern must be US-based or have a permit to work in the United States, in order to receive the stipend.

To apply – by February 1, send the following to susan@eval.org: (1) a letter of interest noting your favorite type(s) of aea365 posts and why, and (2) an example aea365 post following the contribution guidelines and demonstrating your writing/editing capacity.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

No tags

Hello! We’re Judy Savageau and Linda Cabral from the Center for Health Policy and Research at the University of Massachusetts Medical School. Recently, we developed a journal club which convenes interested evaluators quarterly to discuss selected article(s) on evaluation research principles or methodologies. The journal club also networks parties from across our medical school’s multiple departments, centers, and campuses that have an interest in evaluation. For each one-hour session, a facilitator guides the discussion based on “questions to consider” which are developed and distributed with the selected reading(s) ahead of time. Some examples of the articles we’ve discussed include:

• Skolits GJ, Morrow JA and Burr EM. Reconceptualizing Evaluator Roles. American Journal of Evaluation 2009;30(3):275-295.

• Smith NL. An Analysis of Ethical Challenges in Evaluation. American Journal of Evaluation 2002:23(2):199-206.

• Morris M. The Good, the Bad and the Evaluator: 25 Years of AJE Ethics. American Journal of Evaluation 2011;32(1):134-151.

• Cohen DJ and Crabtree BF. Evaluative Criteria for Qualitative Research in Health Care: Controversies and Recommendations. Annals of Family Medicine 2008;6(4):331-339.

Skolits, Morrow and Burr’s article, one of the AJE’s most popular articles of 2010, generated some interesting discussion about the different roles which journal club members have assumed as evaluators along with the benefits and challenges those roles entailed. We ask participants to consider the relevance and applicability of the selected reading to their work, then end each journal club with a short debriefing session, taking suggestions for articles and recruiting facilitators for future journal club meetings.

Hot Tips:

• When coordinating with people from different sites, schedule the journal club toward the end of the work day so that participants don’t have to return to their office.
• Alternate the location of the journal club meetings among participants’ sites/campuses to share the travel burden as well as be able to visit new sites and meet potential new collaborators.
• Provide light refreshments to maintain an informal atmosphere for lively discussion.
• Keep the group size fairly small (10-12 people) to ensure active participation among group members.

Rad Resource:

• Use Doodle (http://doodle.com) to identify the best dates/times for people to meet.

Lesson Learned: While there can be challenges in bringing together individuals from different sites, these are balanced by the benefits of getting to know the work of others and learning about the methodologies and strategies they’ve used with varying projects, clients, stakeholders and funding sources. It can also offer opportunities for identifying new evaluation projects to work on together.

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

Evaluation · Professional Development

My name is Joshua Joseph and I’m a Sr. Program Manager at Partnership for Public Service, a non-profit that works to revitalize the federal government by focusing on people and performance.

This post is about recruiting study participants – a companion to Elisabeth Autio’s aea365 post from December of 2011. In line with Elisabeth’s good advice, my aim is to share some things that have worked to (1) put myself in participants’ shoes and (2) make it easier and (3) more attractive for them to say “yes.”

Rad Resource: Getting into someone else’s shoes can be hit or miss. Instead of a Web link, your best resource is probably a person. Find and talk with a few potential participants early on. Tell them what you’re planning and ask what matters to them. Keep it casual, implore them to be brutally honest and then listen. You’ll get insights, tips, do’s and don’ts – all from the perspective of real insiders.

Hot Tip – Making it Easier. Simplifying to minimize participant burden usually helps. How else can we make it easier for participants to say “yes?”

• It’s easier if they aren’t scared. Tell participants how you will safeguard any sensitive information they provide and protect against adverse effects.
• It’s easier if you’re credible and relatable. Describing a study in formal or technical terms isn’t the only way to sound credible. But, occasionally, we all forget. Plain language works just as well…often better.
• It’s easier if they know what to expect. Tell participants what they’ll be doing and why it matters, if you can. Most will appreciate the info.

Hot Tip – Making it More Attractive. Raffles and giveaways can be great perks for participating. So can the promise to share project results. What else can we do?

• It’s more attractive when they laugh. Humor tends to be underappreciated as a recruiting tool. Don’t force it but do look for ways to lighten up your message. It makes us real and humanizes our work, which is usually a good thing.
• It’s more attractive if they’re special. Find what makes participants’ views unique and then tell them.
• It’s more attractive if others are doing it. Most people like to know they’re in good company. Sharing that their peers or colleagues are signing up can do the trick.

Hot Tip – Reading the Room. These are rules of thumb, not absolutes. For example, I worked with a passionate group that wanted a longer, more in-depth survey to share their rich insights. Go figure! So use your good judgment to adapt as needed.

Get Involved: Please share what’s worked (or hasn’t) for you via the comments. Anything counter-intuitive?

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators.

recruiting

I’m Johanna Morariu, co-director at Innovation Network, an evaluation consulting firm that works with nonprofit organizations and foundations.

Rad Resource: Every now and then I come across an article or book that makes me fundamentally rethink what I thought I knew. One that I want to share with you all is the book Proofiness: The Dark Arts of Mathematical Deception by Charles Seife.

While the entire book is a great read—especially for evaluators—the first chapter, Phony Facts, Phony Figures, is especially apt. Through many examples and discussions the author makes the point that when numbers are attached to arguments or “facts,” we as humans are more likely to believe the statement. Seife opens the chapter by asserting, “If you want to get people to believe something really, really stupid, just stick a number on it.” Essentially, humans are wired to believe in numbers—numbers don’t lie.

I couldn’t help but think about how this applied to evaluation consulting: So often the individuals and organizations I work with naturally prioritize the supposedly objective, quantitative data over the seemingly fluffy, qualitative data. This is especially evident in two phases: evaluation design and when presented with findings. Before reading Proofiness, I chalked up this point of view to lack of evaluation experience. But since reading Proofiness (and years spent puzzling over the preference for quantitative data over qualitative), I agree with Seife; We as humans have an innate belief in the infallibility of numbers.

Lesson Learned: Now, when I work with an individual or organization to design an evaluation approach or present qualitative findings, I realize the increased need for me to explain the merit and appropriateness of qualitative designs. Sometimes it is not enough to lay out the pros and cons of a variety of quantitative and qualitative approaches—sometimes the rigor of qualitative approaches needs to be specially called out and affirmed.

Have you read Proofiness? If so, how did the book make you think differently about evaluation?

And if you haven’t read Proofiness, I heartily recommend it!

Do you have questions, concerns, kudos, or content to extend this aea365 contribution? Please add them in the comments section for this post on the aea365 webpage so that we may enrich our community of practice. Would you like to submit an aea365 Tip? Please send a note of interest to aea3365@eval.org. aea365 is sponsored by the American Evaluation Association and provides a Tip-a-Day by and for evaluators

No tags