Quotable: Carissa Gonzalez on an “evaluation revolution” for Public Diplomacy

publicdiplomacycouncil.org

image (not from article) from

Thursday, June 23rd 2016

“Analytics. Metrics. Evaluation. Impact. . . . . Public Diplomacy (PD) pro­fession­als in the field . . . have been historically reluctant to discuss them,” wrote State Department Public Diplomacy officer Carissa Gonzalez in an article, “The Evaluation Revolution in Public Diplomacy,” that appeared in the Fall 2015 issue of The Ambassadors Review.  Gonzalez was a Kathryn W. Davis Fellow in Public Diplomacy under the auspices of the Council of American Ambassadors.  The reluctance, she wrote, stems from “fear that admitting such loaded, mathematical terms into the equation of their work will undermine the relational, long-term, nuanced public outreach they do day in and day out.”

 This praiseworthy article represents the kind of professional writing – aimed to encourage professional discussion and debate – seldom produced in the Foreign Service.  Her thumbnail description of the state of evaluation is "numbers of activities organized, hours logged, or dollars spent, with a heart-tugging anecdote or flashy photo tossed in for good effect."  Here are other key points:

• Granted, the ambitious and elusive goals that Public Diplomacy officers are tasked with achieving, such as “promoting freedom of speech,” “increasing respect for women’s and minority rights,” and “building stable and lasting democracies,” are difficult to measure.

• We don’t have the clarity that concrete indicators provide, such as those that our Consular colleagues collect and analyze daily (i.e. number of visas issued, or average wait times).

• Nor do we enjoy the enviable marketing budgets that private sector firms deploy to seg­ment their markets and promote their brands accordingly.

• However, what my experience as a Public Diplomacy officer in the field has taught me is that understanding our audiences and measuring our effectiveness is not a luxury, but rather a crucial element of what we need to commit to doing in order to perform our jobs with any degree of success.

• What we have now as an organization is not an “evaluation culture,” and by that, I mean a culture that encourages and supports objective, rigorous, honest analysis of the impact (or lack thereof) of the public and press engagement we labor at every day.

• At best, what we have is a “reporting culture” of outputs: one that expects knee-jerk replies to ever increas­ing and overlapping data calls for numbers of activities organized, hours logged, or dollars spent, with a heart-tugging anecdote or flashy photo tossed in for good effect.

• In my view, the most serious challenges facing Public Diplomacy evaluation today are that: --  current evaluation efforts are dispersed across various offices and bureaus; --  there is still no consensus on what we ought to be measuring; and --  there is a misleading conflation between operational evaluation of program logistics (Did everything run smoothly?), and impact evaluation (Did the program achieve the policy impact we intended?).

• Public Diplomacy is badly in need of an evidence base if it is to move into the mainstream of international relations. A broad research agenda—with evaluation at its heart—can help provide that evidence base.”

• We would be wise to heed the common sense adage, “Show me your pocketbook, and I’ll show you your priorities.” In September 2014, the United States Advisory Commission on Public Diplomacy found that the Department of State had been severely under-funding research and evaluation efforts, falling well short of the industry standard for international organizations, foundations, and development agencies of five percent of an organization’s budget.

• As Public Diplomacy officers, we are continually told to “do more with less.” And we have become increasingly adept at doing so . . . . However, this has led to the misconception that PD resources (be they budgets, human capital, or influence) are inexhaustible. 

• The inconvenient truth is that time, money, and energy are not unlimited resources. In order to do our work most efficiently, Public Diplomacy officers need the breathing space and time to design outreach strategies based on solid research, the flexibility to choose not to engage on a certain issue in order to focus efforts on high-priority campaigns, and the increased resources needed to measure outcomes and analyze their impacts on issues of foreign policy concern.      

• Thankfully, PD leadership in Washington has realized the need for better analytics, and bureaus and offices throughout the “PD family” have started establishing their own evaluation units. [PA, IIP, ECA, R/PPR]

• State Department PD leaders are navigating uncharted waters. In all the above cases, PD bureaus and offices are looking to evaluate large scale, multi-country or worldwide campaigns and programs and will depend on PD officers in the field to evaluate their efforts on a post or country level.

• Thus, the success of these Department-wide evaluation endeavors, particularly as they roll-out to the field, will hinge upon the level of coordination between the various PD bureaus and offices and their willing­ness to integrate their data onto one unifying platform.

• The PD officer in the field cannot possibly know what success looks like, nor how to faithfully track and report it, if he/she is receiving dozens of mixed signals through a multitude of channels.  

Mental Shifts Wanted

• Recognize that concrete change takes time. Most private sector and academic literature on the subject of public relations and public diplomacy campaigns describe “intermediate outcomes” as those expected in five years, with “long-range outcomes” expected much further out. (“Short-term” outcomes don’t even exist according to some research.)  Furthermore, researchers argue that fostering interest in an issue and achieving attitude shift is usually the most that public affairs activities can ever hope to produce on their own, and that actual behavioral shifts often take tremendous amounts of time and resources over decades of engagement.

• Create a more risk-tolerant culture. The culture must encourage innovation and empower PD officers in the field to take calculated risks. It must not penalize them for a less-than-glowing outcome. (Otherwise, no reporting or evaluation will ever be honest or useful.)

• Reverse the trend of increasing centralization of PD decision making. Trust the experts on the ground to know more than a central office about the audience.

• Stop over-strategizing and start meaningful evaluation. When planning docu­ments become so detailed that they become unwieldy and inflexible, they cease to be useful.

• What we need is a useful platform upon which the work actually happens, and which collects useful data automati­cally as we use it to get the work done. This is why social media metrics and evaluation is so much more advanced than all other PD evaluation: the platforms that collect the data are the same ones which we use to actually do the work of digital outreach. 

• We could possibly transform the current Public Diplomacy Resource Plan (PDRP) to be more of a dashboard, a one-stop-shop for PD officers in the field to do their research on the local media and public opinion climate; set their goals, audiences, and benchmarks; view the totality of their human and physical resources and allocate them accordingly; and measure their progress over time. 

• The winds of change are coming, whether we are prepared for them or not.

• However, there is no cause to fear or resist this change.

Author: Donald M. Bishop