Last week’s Technology Salon New York City touched on ethics in technology for democracy initiatives. We heard from lead discussants Malavika Jayaram, Berkman Center for Internet and SocietyIvan Sigal, Global Voices; and Amilcar Priestley, Afrolatin@ Project. Though the topic was catalyzed by the Associated Press’ article on ‘Zunzuneo’ (a.k.a. ‘Cuban Twitter’) and subsequent discussions in the press and elsewhere, we aimed to cover some of the wider ethical issues encountered by people and organizations who implement technology for democracy programs.

Salons are off the record spaces, so no attribution is made in this post, but I’ve summarized the discussion points here:

First up: Zunzuneo

The media misinterpreted much of the Zunzuneo story. Zunzuneo was not a secret mission, according to one Salon participant, as it’s not in the remit of USAID to carry out covert operations. The AP article conflated a number of ideas regarding how USAID works and the contracting mechanisms that were involved in this case, he said. USAID and the Office of Transition Initiatives (OTI) frequently disguise members, organizations, and contractors that work for it on the ground for security reasons. (See USAID’s side of the story here). This may still be an ethical question, but it is not technically “spying.” The project was known within the OTI and development community, but on a ‘need to know’ basis. It was not a ‘fly by night’ operation; it was more a ‘quietly and not very effectively run project.’

There were likely ethics breaches in Zunzuneo, from a legal standpoint. It’s not clear whether the data and phone numbers collected from the Cuban public for the project were obtained in a legal or ethical way. Some reports say they were obtained through a mid-level employee (a “Cuban engineer who had gotten the phone list” according to the AP article). (Note: I spoke separately to someone close to the project who told me that user opt-in/opt-out and other standard privacy protocols were in place). It’s also not entirely clear whether, as the AP states, the user information collected was being categorized into segments who were loyal or disloyal to the Cuban government, information which could put users at risk if found out.

Zunzuneo took place in a broader historical and geo-political context. As one person put it, the project followed Secretary Clinton’s speeches on Internet Freedom. There was a rush to bring technology into the geopolitical space, and ‘the articulation of why technology was important collided with a bureaucratic process in USAID and the State Department (the ‘F process’) that absorbed USAID into the State Department and made development part of the State Department’s broader political agenda.’ This agenda had been in the works for quite some time, and was part of a wider strategy of quietly moving into development spaces and combining development, diplomacy, intelligence and military (defense), the so-called 3 D’s.

Implementers failed to think through good design, ethics and community aspects of the work. In a number of projects of this type, the idea was that if you give people technology, they will somehow create bottom up pressure for political social change. As one person noted, ‘in the Middle East, as a counter example, the tech was there to enable and assist people who had spent 8-10 years building networks. The idea that we can drop tech into a space and an uprising will just happen and it will coincidentally push the US geopolitical agenda is a fantasy.’ Often these kinds of programs start with a strategic communications goal that serves a political end of the US Government. They are designed with the idea that a particular input equals some kind of a specific result down the chain. The problem comes when the people doing the seeding of the ideas and inputs are not familiar with the context they will be operating in. They are injecting inputs into a space that they don’t understand. The bigger ethical question is: Why does this thought process prevail in development? Much of that answer is found in US domestic politics and the ways that initiatives get funded.

Zunzuneo was not a big surprise for Afrolatino organizations. According to one discussant, Afrolatino organizations were not surprised when the Zunzuneo article came out, given the geopolitical history and the ongoing presence of the US in Latin America. Zunzuneo was seen as a 21st Century version of what has been happening for decades. Though it was criticized, it was not seen as particularly detrimental. Furthermore, the Afrolatino community (within the wider Latino community) has had a variety of relationships with the US over time – for example, some Afrolatino groups supported the Contras. Many Afrolatino groups have felt that they were not benefiting overall from the mestizo governments who have held power. In addition, much of Latin America’s younger generation is less tainted by the Cold War mentality, and does not see US involvement in the region as necessarily bad. Programs like Zunzuneo come with a lot of money attached, so often wider concerns about their implications are not in the forefront because organizations need to access funding. Central American and Caribbean countries are only just entering into a phase of deeper analysis of digital citizenship, and views and perceptions on privacy are still being developed.

Perceptions of privacy

There are differences in perception when it comes to privacy and these perceptions are contextual. They vary within and across countries and communities based on age, race, gender, economic levels, comfort with digital devices, political perspective and past history. Some older people, for example, are worried about the privacy violation of having their voice or image recorded, because the voice, image and gaze hold spiritual value and power. These angles of privacy need to be considered as we think through what privacy means in different contexts and adapt our discourse accordingly.

Privacy is hard to explain, as one discussant said: ‘There are not enough dead bodies yet, so it’s hard to get people interested. People get mad when the media gets mad, and until an issue hits the media, it may go unnoticed. It’s very hard to conceptualize the potential harm from lack of privacy. There may be a chilling effect but it’s hard to measure. The digital divide comes in as well, and those with less exposure may have trouble understanding devices and technology. They will then have even greater trouble understanding beyond the device to data doubles, disembodied information and de-anonymization, which are about 7 levels removed from what people can immediately see. Caring a lot about privacy can get you labeled as paranoid or a crazy person in many places.’

Fatalism about privacy can also hamper efforts. In the developing world, many feel that everything is corrupt and inept, and that there is no point in worrying about privacy and security. ‘Nothing ever works anyway, so even if the government wanted to spy on us, they’d screw it up,’ is the feeling. This is often the attitude of human rights workers and others who could be at greatest risk from privacy breaches or data collection, such as that which was reportedly happening within Zunzuneo. Especially among populations and practitioners who have less experience with new technologies and data, this can create large-scale risk.

Intent, action, context and consequences

Good intentions with little attention to privacy vs data collection with a hidden political agenda. Where are the lines when data that are collected for a ‘good cause’ (for example, to improve humanitarian response) might be used for a different purpose that puts vulnerable people at risk? What about data that are collected with less altruistic intentions? What about when the two scenarios overlap? Data might be freely given or collected in an emergency that would be considered a privacy violation in a ‘development’ setting, or the data collection may lead to a privacy violation post-emergency. Often, slapping the ‘obviously good and unarguably positive’ label of ‘Internet freedom’ on something implies that it’s unquestionably positive when it may in fact be part of a political agenda with a misleading label. There is a long history of those with power collecting data that helps them understand and/or control those with less power, as one Salon participant noted, and we need to be cognizant of that when we think about data and privacy.

US Government approaches to political development often take an input/output approach, when, in fact, political development is not the same as health development. ‘In political work, there is no clear and clean epidemiological goal we are trying to reach,’ noted a Salon participant. Political development is often contentious and the targets and approaches are very different than those of health. When a health model and rhetoric is used to work on other development issues, it is misleading. The wholesale adoption of these kinds of disease model approaches leaves people and communities out of the decision making process about their own development. Similarly, the rhetoric of strategic communications and its inclusion into the development agenda came about after the War on Terror, and it is also a poor fit for political development. The rhetoric of ‘opening’ and ‘liberating’ data is similar. These arguments may work well for one kind of issue, but they are not transferable to a political agenda. One Salon participant pointed out the rhetoric of the privatization model also, and explained that a profound yet not often considered implication of the privatization of services is that once a service passes over to the private sector, the Freedom of Information Act (FOIA) does not apply, and citizens and human rights organizations lose FOIA as a tool. Examples included the US prison system and the Blackwater case of several years ago.

It can be confusing for implementers to know what to do, what tools to use, what funding to accept and when it is OK to bring in an outside agenda. Salon participants provided a number of examples where they had to make choices and felt ethics could have been compromised. Is it OK to sign people up on Facebook or Gmail during an ICT and education project, given these companies’ marketing and privacy policies? What about working on aid transparency initiatives in places where human rights work or crime reporting can get people killed or individual philanthropists/donors might be kidnapped or extorted? What about a hackathon where the data and solutions are later given to a government’s civilian-military affairs office? What about telling LGBT youth about a social media site that encourages LGBT youth to connect openly with one another (in light of recent harsh legal penalties against homosexuality)? What about employing a user-centered design approach for a project that will eventually be overlaid on top of a larger platform, system or service that does not pass the privacy litmus test? Is it better to contribute to improving healthcare while knowing that your software system might compromise privacy and autonomy because it sits on top of a biometric system, for example? Participants at the Salon face these ethical dilemmas every day, and as one person noted, ‘I wonder if I am just window dressing something that will look and feel holistic and human-centered, but that will be used to justify decisions down the road that are politically negative or go against my values.’ Participants said they normally rely on their own moral compass, but clearly many Salon participants are wrestling with the potential ethical implications of their actions.

What we can do? Recommendations from Salon participants

Work closely with and listen to local partners, who should be driving the process and decisions. There may be a role for an outside perspective, but the outside perspective should not trump the local one. Inculcate and support local communities to build their own tools, narratives, and projects. Let people set their own agendas. Find ways to facilitate long-term development processes around communities rather than being subject to agendas from the outside.

Consider this to be ICT for Discrimination and think in every instance and every design decision about how to dial down discrimination. Data lead to sorting, and data get lumped into clusters. Find ways during the design process to reduce the discrimination that will come from that sorting and clustering process. The ‘Do no harm’ approach is key. Practitioners and designers should also be wary of the automation of development and the potential for automated decisions to be discriminatory.

Call out hypocrisy. Those of us who sit at Salons or attend global meetings hold tremendous privilege and power as compared to most of the rest of the world. ‘It’s not landless farmers or disenfranchised young black youth in Brazil who get to attend global meetings,’ said one Salon attendee. ‘It’s people like us. We need to be cognizant of the advantage we have as holders of power.’ Here in the US, the participant added, we need to be more aware of what private sector US technology companies are doing to take advantage of and maintain their stronghold in the global market and how the US government is working to allow US corporations to benefit disproportionately from the current Internet governance structure.

Use a rights-based approach to data and privacy to help to frame these issues and situations. Disclosure and consent are sometimes considered extraneous, especially in emergency situations. People think ‘this might be the only time I can get into this disaster or conflict zone, so I’m going to Hoover up as much data as possible without worrying about privacy.’ On the other hand, sometimes organizations are paternalistic and make choices for people about their own privacy. Consent and disclosure are not new issues; they are merely manifested in new ways as new technology changes the game and we cannot guarantee anonymity or privacy any more for research subjects. There is also a difference between information a person actively volunteers and information that is passively collected and used without a person’s knowledge. Framing privacy in a human rights context can help place importance on both processes and outcomes that support people’s rights to control their own data and that increase empowerment.

Create a minimum standard for privacy. Though we may not be able to determine a ceiling for privacy, one Salon participant said we should at least consider a floor or a minimum standard. Actors on the ground will always feel that privacy standards are a luxury because they have little know-how and little funding, so creating and working within an ethical standard should be a mandate from donors. The standard could be established as an M&E criterion.

Establish an ethics checklist to decide on funding sources and create policies and processes that help organizations to better understand how a donor or sub-donor would access and/or use data collected as part of a project or program they are funding. This is not always an easy solution, however, especially for cash-strapped local organizations. In India, for example, organizations are legally restricted from receiving certain types of funding based on government concerns that external agencies are trying to bring in Western democracy and Western values. Local organizations have a hard time getting funding for anti-censorship or free speech efforts. As one person at the Salon said, ‘agencies working on the ground are in a bind because they can’t take money from Google because it’s tainted, they can’t take money from the State Department because it’s imperialism and they can’t take money from local donors because there are none.’

Use encryption and other technology solutions. Given the low levels of understanding and awareness of these tools, more needs to be done so that more organizations learn how to use them, and they need to be made simpler, more accessible and user-friendly. ‘Crypto Parties’ can help get organizations familiar with encryption and privacy, but better outreach is needed so that organizations understand the relevance of encryption and feel welcome in tech-heavy environments.

Thanks to participants and lead discussants for the great discussions and to ThoughtWorks for hosting us at their offices!

 If you’d like to attend future Salons, sign up here!

We looked at the role of mobiles in youth financial inclusion at our March 11th Technology Salon in New York City. Tim Nourse, Making Cents; Peter Goldstein, Intermedia; and Jamie Zimmerman, Bankable Frontier Associates; joined as lead discussants.

Though mobile financial services are seen by many as inevitable, some Salon participants felt that, like in so many other ‘mobiles for xxxx’ areas, we were long on enthusiasm and short on evidence and successful examples. Are we just too early in the game, as with so much of ICT4D? Emerging research on youth demand for mobile financial services may help answer some of those questions, but many other questions remain.

What do we mean by youth financial inclusion?

The Salon started with a quick overview of the terms “financial inclusion” and ‘youth.’ One lead discussant emphasized that the idea of ‘youth’ is context specific. According to the UN, “youth” are people between 15 and 24 years old, though in many countries this can extend to age 30 or 35. Segmentation within this wide age range is important when designing programs because of varying needs, demands, and concerns within age subsets. Using a gender lens is also critical, because young women and young men have different needs, concerns, barriers, interests and experiences. Cultural norms about girls’ and young women’s access to and use of assets and resources, financial services, and mobiles also come into play and need to be well-understood. When discussing youth financial inclusion, it’s useful to talk about the age ranges of 15-17 and 18-24, because in most countries 18 is the legal age at which youth can enter into a formal financial system, sign contracts, and purchase a SIM card in their own name. Program design, challenges faced, and workable business models may look quite different for these two age groups.

The term ‘youth financial services’ includes a full range of services (credit, savings, insurance, money transfer and payments) that help youth build assets. In other words, financial services go far beyond mobile money transfers. Most youth in developing nations are engaged in some kind of livelihood or education, and access to financial services can help them achieve goals in both arenas. It is important to reach youth with financial education when they are adolescents, as they are more inclined to form good habits if they are engaged early on. Availability of services at specific transition points in youth’s lifecycles when they are making serious decisions is another key to establishing good long-term financial habits. It can be difficult, however, to convince banking institutions to develop a menu of financial services for youth because few successful business models exist for youth-focused financial products and services. Savings, account balances and demand for credit tend to be lower among youth, so serving the youth market profitably can be difficult. Strategic rationales and successful business cases around expanded access to youth financial services are needed.

Emerging guidelines for good practice in design and implementation of youth-inclusive financial services being developed by Making Cents include:

  • Involve youth in market research and product development
  • Develop products and services that represent the diversity of youth
  • Ensure youth have safe and supportive spaces
  • Provide or link youth with complementary non-financial services
  • Focus on core competencies and collaborate with youth organizations to ensure holistic programs
  • Involve communities to reinforce and enhance the effectiveness of programming
  • Establish a strategic rational and ensure institutional readiness for serving youth

Mobiles and youth financial inclusion

Many have high hopes around the role of mobile phones in enhancing and expanding youth financial services. Mobiles may allow financial institutions to lower costs for financial products and thus enable new and profitable business models. In addition to providing direct services, mobiles might be able to improve the reach and impact of financial education aimed at youth, and encourage particular behaviors and habit formation. For example, SMS reminders are being used to ‘nudge’ youth towards particular actions related to savings and smarter purchases.

A report called “Beyond the Buzz” however, highlight some of the major challenges when it comes to the role of mobile and financial inclusion for the under 18 population. As explained by one lead discussant (also one of the report’s authors), most youth surveyed in Sub Saharan Africa believed mobile money would be far more important for financial inclusion in the future than SMS. Non-profit organization practitioners and financial institutions surveyed for the report expressed strong belief in the potential of mobile money and other mobile services for broadening youth financial inclusion.

Enthusiasm is quite high, though there has been little success thus far, and the evidence on the ground is not very encouraging. Even though most people surveyed felt that mobile money was the future and would change everything, mobiles are actually being used far more commonly for financial education (SMS and nudges) than for providing youth access to financial services.

So what are the obstacles?

Some of the challenges that prevent mobile financial services from taking off include:

  • Age restrictions and regulations. In most countries, a young person cannot obtain an identity card until the age of 18, meaning access to a bank account, a SIM and/or mobile money is restricted. Many young people get around this obstacle by borrowing a handset or asking a parent or guardian for support. When phones do not belong to youth, however, SMS ‘nudges’ for financial education may not reach them. In addition, the lack of a private handset may discourage youth from using mobile to manage their money due to the potential loss of privacy and control over their money. Children under the age of 18 are a protected group, and many countries have regulations around collecting information about or marketing to this population. Child protection policies and legal regulations are a positive thing, however, they can also create barriers to financial education and financial services for under 18s.
  • Lack of data. One discussant noted that age-disaggregated data from mPesa’s mobile money service would probably show that older youth (ages 18-30) are the majority of the mobile money users. The lack of data on youth, however, makes it difficult for non-profit organizations to develop targeted and demand-led financial products and services. Mobile Network Operators (MNOs) have data, yet their data are not easy to access. One Salon discussant told of a project where it took over two and a half years to obtain legal permission from an MNO to access youth data for an RCT on the impact of SMS on youth savings.
  • Industry barriers. Successful and sustainable business models for youth financial services are few and far between. The likelihood of low financial returns from youth make most banks uninterested in approaching the youth ‘base of the pyramid’ market. Institutions that make money from youth financial services are most likely making it from 24 and 25 year olds, not under 18s. Explaining the potential benefits of a long-term business model (that you may need to take a loss earlier on to gain from this segment later) to financial institutions is difficult. In addition, mobile operators are not fully empowered to launch mobile financial services on their own, even if they wanted to, because of government regulations (in some cases, added one Salon participant, because the banking industry actively lobbies government to avoid losing business to MNOs).

Long on enthusiasm and short on examples?

Considering all the obstacles, why are hopes so high when it comes to mobiles and youth financial inclusion? Some consider that MNOs have a fundamental advantage over banks in countries where the majority of people have access to a mobile phone yet have never used a bank or formal financial service. In many parts of the world, banking systems are unavailable and/or inefficient, and people do not trust formal systems or large bureaucracies. When it comes to mobile, however, use and availability of handsets, widespread recognition of mobile operator brands and services, and familiarity with the notion of transferring airtime mean that mobile money is a fairly easy idea for people to grasp and thus it may be easier to generate trust in mobile as a means to access financial services.

The impact of mobile money and mobiles on financial inclusion is difficult to evaluate rigorously, however, noted one Salon participant. The volume of money is very small, so we should have very low expectations in that regard. If 20% of a target population uses a financial service or product, we should be excited because we see an individual having more control over and information on their own financial transactions. This enables them to make better decisions over their finances. Mobile financial services are likely doing more good than harm, even if a large, broad-based impact study is not available. Another Salon participant pointed out, however, that market research to inform good product and service offerings is very much lacking, and a concerted effort is needed to document and research this area.

A large study is being conducted with youth ages 15-19 and 20-24 on youth demand for mobile money and financial services in several African and Asian countries as part of the Financial Inclusion Insights program, said one lead discussant, and data will be available to the public. The majority of youth surveyed for the study said that they did not use a bank because they did not have enough money to do so. In five years, according to the discussant, mobile financial products will be accessible in a wide range of countries and the number of youth using them is increasing. Research shows that urban youth tend to adopt these products more often than older people or rural populations, and there is a male-female gap, where more males are accessing and using them. In general, younger populations have been positive about mobile financial products and services.

An inevitable future?

Despite the dearth of successful business models, evidence, and large-scale sustainable examples, some Salon participants felt that we are entering a new era where financial products and services will be widely available through the mobile phone. As one person explained, it’s a question of moving with the times or becoming obsolete. In Southern African countries, she said, the move is towards rolling out products and services that provide holistic financial inclusion — credit, savings and insurance. In addition, municipal and utility bill paying is getting people accustomed to mobile financial services via MNOs. Banks who are running at a low level of innovation will lose out if they are not capable of providing these kinds of time-saving services through mobile phones.

So what should organizations be doing to prepare youth to widely access and use mobile financial services? Should financial education programs include content about mobile financial services, offerings and fees, and potential risks and benefits for youth of using them? Might mobile gaming be a way of getting around some of the barriers for under 18s, as one Salon participant suggested? In this case, children could practice important concepts around savings and loans, types of bank accounts, fee structures for banking, etc., without assuming any real risk.

Some broader questions linger around mobile financial services for youth as well: What impact does (or will) mobile financial services have on people’s lives and wellbeing? Will they impact how youth invest and manage their money? Will they improve redistribution of resources to households? Will they end up pulling a large segment of the population into unsustainable systems and backfire?  So far there’s no clear answer, but watch this space.


A list of resources, links, projects, organizations and research on the topic is here. Please add anything that’s missing!

Thanks to participants and lead discussants for the great discussions and to Population Council for hosting us at their offices for this Salon. Thanks also to Peter Goldstein for suggesting the topic and to Somto Fab-Ukozor for support with notes and the summary. Salons are held under Chatham House Rule, therefore no attribution has been made in this post. If you’d like to attend future Salons, sign up here!

Earlier this month I attended the African Evaluators’ Conference (AfrEA) in Cameroon as part of the Technology and Evaluation stream organized by Pact with financial support from The Rockefeller Foundation’s Evaluation Office and The MasterCard Foundation.

A first post about ICTs and M&E at the Afrea Conference went into some of the deliberations around using or not using ICTs and how we can learn and share more as institutions and evaluators. I’ve written previously about barriers and challenges with using ICTs in M&E of international development programs (see the list of posts at the bottom of this one). Many of these same conversations came up at AfrEA, so I won’t write about these again here. What I did want to capture and share were a few interesting design and implementation thoughts from the various ICT and M&E sessions. Here goes:

1) Asking questions via ICT may lead to more honest answers. Some populations are still not familiar with smart phones and tablets and this makes some people shy and quiet, yet it makes others more curious and animated to participate. Some people worry that mobiles, laptops and tablet create distance between the enumerator and the person participating in a survey. On the other hand, I’m hearing more and more examples of cases where using ICTs for surveying actually allow for a greater sense of personal privacy and more honest answers. I first heard about this several years ago with relation to children and youth in the US and Canada seeking psychological or reproductive health counseling. They seemed to feel more comfortable asking questions about sensitive issues via online chats (as opposed to asking a counselor or doctor face-to-face) because they felt anonymous. This same is true for telephone inquiries.

In the case of evaluations, someone suggested that rather than a mobile or tablet creating distance, a device can actually create an opportunity for privacy. For example, if a sensitive question comes up in a survey, an enumerator can hand the person being interviewed the mobile phone and look away when they provide their answer and hit enter, in the same way that waiters in some countries will swipe your ATM card and politely look away while you enter your PIN. Key is building people’s trust in these methods so they can be sure they are secure.

At a Salon on Feb 28, I heard about mobile polling being used to ask men in the Democratic Republic of Congo about sexual assault against men. There was a higher recorded affirmative rate when the question was answered via a mobile survey than when the question had been asked in other settings or though other means. This of course makes sense, considering that often when a reporter or surveyor comes around asking whether men have been victims of rape, no one wants to say publicly. It’s impossible to know in a situation of violence if a perpetrator might be standing around in the crowd watching someone getting interviewed, and clearly shame and stigma also prevent people from answering openly.

Another example at the AfrEA Tech Salon, was a comparison study done by an organization in a slum area in Accra. Five enumerators who spoke local languages conducted Water, Sanitation and Hygiene (WASH) surveys by mobile phone using Open Data Kit (an open source survey application) and the responses were compared with the same survey done on paper.  When people were asked in person by enumerators if they defecated outdoors, affirmative answers were very low. When people were asked the same question via a voice-based mobile phone survey, 26% of respondents reported open defecation.

2) Risk of collecting GPS coordinates. We had a short discussion on the plusses and minuses of using GPS and collecting geolocation data in monitoring and evaluation. One issue that came up was safety for enumerators who carry GPS devices. Some people highlighted that GPS devices can put staff/enumerators at risk of abuse from organized crime bands, military groups, or government authorities, especially in areas with high levels of conflict and violence. This makes me think that if geographic information is needed in these cases, it might be good to use a mobile phone application that collects GPS rather than a fancy smart phone or an actual GPS unit (for example, one could try out PoiMapper, which works on feature phones).

In addition, evaluators emphasized that we need to think through whether GPS data is really necessary at household level. It is tempting to always collect all the information that we possibly can, but we can never truly assure anyone that their information will not be de-anonymized somehow in the near or distant future, and in extremely high risk areas, this can be a huge risk. Many organizations do not have high-level security for their data, so it may be better to collect community or district level data than household locations. Some evaluators said they use ‘tricks’ to anonymize the geographical data, like pinpointing location a few miles off, but others felt this was not nearly enough to guarantee anonymity.

3) Devices can create unforeseen operational challenges at the micro-level. When doing a mobile survey by phone and asking people to press a number to select a particular answer to a question, one organization working in rural Ghana to collect feedback about government performance found that some phones were set to lock when a call was answered. People were pressing buttons to respond to phone surveys (press 1 for….), but their answers did not register because phones were locked, or answers registered incorrectly because the person was entering their PIN to unlock the phone. Others noted that when planning for training of enumerators or community members who will use their own devices for data collection, we cannot forget the fact that every model of phone is slightly different. This adds quite a lot of time to the training as each different model of phone needs to be explained to trainees. (There are a huge number of other challenges related to devices, but these were two that I had not thought of before.)

4) Motivation in the case of poor capacity to respond. An organization interested in tracking violence in a highly volatile area wanted to take reports of violence, but did not have a way to ensure that there would be a response from an INGO, humanitarian organization or government authority if/when violence was reported. This is a known issue — the difficulties of encouraging reporting if responsiveness is low. To keep people engaged this organization thanks people immediately for reporting and then sends peace messages and encouragement 2-3 times per week. Participants in the program have appreciated these ongoing messages and participation has continued to be steady, regardless of the fact that immediate help has not been provided as a result of reporting.

5) Mirroring physical processes with tech. One way to help digital tools gain more acceptance and to make them more user-friendly is to design them to mirror paper processes or other physical processes that people are already familiar with. For example, one organization shared their design process for a mobile application for village savings and loan (VSL) groups. Because security is a big concern among VSL members, the groups typically keep cash in a box with 3 padlocks. Three elected members must be present and agree to open and remove money from the box in order to conduct any transaction. To mimic this, the VSL mobile application requires 3 PINs to access mobile money or make transactions, and what’s more, the app sends everyone in the VSL Group an SMS notification if the 3 people with the PINs carry out a transaction, meaning the mobile app is even more secure than the original physical lock-box, because everyone knows what is happening all the time with the money.


As I mentioned in part 1 of this post, some new resources and forthcoming documentation may help to further set the stage for better learning and application of ICTs in the M&E process. Pact has just released their Mobile Technology Toolkit, and Michael Bamberger and I are finishing up a paper on ICT-enabled M&E that might help provide a starting point and possible framework to move things forward.

Here is the list of toolkits, blog posts and other links that we compiled for AfrEA – please add any that are missing!

Previous posts on ICTs and M&E on this blog:

I attended the African Evaluators’ Conference (AfrEA) in Cameroon last week as part of the Technology and Evaluation strand organized by Pact with financial support from The Rockefeller Foundation’s Evaluation Office and The MasterCard Foundation. The strand was a fantastic opportunity for learning, sharing and understanding more about the context, possibilities and realities of using ICTs in monitoring and evaluation (M&E). We heard from a variety of evaluators, development practitioners, researchers, tool-developers, donors, and private sector and government folks. Judging by the well-attended sessions, there is a huge amount of interest in ICTs and M&E.

Rather than repeat what’s I’ve written in other posts (see links at the bottom), I’ll focus here on some of the more relevant, interesting, and/or new information from the AfrEA discussions. This first post will go into institutional issues and the ‘field’ of ICTs and M&E. A second post will talk about design and operational tips I learned /was reminded of at AfrEA.

1) We tend to get stuck on data collection –Like other areas (I’m looking at you, Open Data) conversations tend to revolve around collecting data. We need to get beyond that and think more about why we are collecting data and what we are going to do with it (and do we really need all this data?). The evaluation field also needs to explore all the other ways it could be using ICTs for M&E, going beyond mobile phones and surveys. Collecting data is clearly a necessary part of M&E, but those data still need to be analyzed. As a participant from a data visualization firm said, there are so many ways you can use ICTs – they help you make sense of things, you can tag sentiment, you can visualize data and make data-based decisions. Others mentioned that ICTs can help us to share data with various stakeholders, improve sampling in RCTs (Randomized Control Trials), conduct quality checks on massive data sets, and manage staff who are working on data collection. Using big data, we can do analyses we never could have imagined before. We can open and share our data, and stop collecting the same data from the same people multiple times. We can use ICTs to share back what we’ve learned with evaluation stakeholders, governments, the public, and donors. The range of uses of ICTs is huge, yet the discussion tends to get stuck on mobile surveys and data collection, and we need to start thinking beyond that.

2) ICTs are changing how programs are implemented and how M&E is done — When a program already uses ICTs, data collection can be built in through the digital device itself (e.g., tracking user behavior, cookies, and via tests and quizzes), as one evaluator working on tech and education programs noted. As more programs integrate digital tools, it may become easier to collect monitoring and evaluation data with less effort. Along those lines, an evaluator looking at a large-scale mobile-based agricultural information system asked about approaches to conducting M&E that do not rely on enumerators and traditional M&E approaches. In his program, because the farmers who signed up for the mobile information service do not live in the same geographical community, traditional M&E approaches do not seem plausible and ICT-based approaches look like a logical answer. There is little documentation within the international development evaluation community, however, on how an evaluator might design an evaluation in this type of a situation. (I am guessing there may be some insights from market research and possibly from the transparency and accountability sectors, and among people working on “feedback loops”).

3) Moving beyond one-off efforts — Some people noted that mobile data gathering is still done mostly at the project level. Efforts tend to be short-term and one-off. The data collected is not well-integrated into management information systems or national level processes. (Here we may reference the infamous map of mHealth pilots in Uganda, and note the possibility of ICT-enabled M&E in other sectors going this same route). Numerous small pilots may be problematic if the goal is to institutionalize mobile data gathering into M&E at the wider level and do a better job of supporting and strengthening large-scale systems.

4) Sometimes ICTs are not the answer, even if you want them to be – One presenter (who considered himself a tech enthusiast) went into careful detail about his organization’s process of deciding not to use tablets for a complex evaluation across 4 countries with multiple indicators. In the end, the evaluation itself was too complex, and the team was not able to find the right tool for the job. The organization looked at simple, mid-range and highly complex applications and tools and after testing them all, opted out. Each possible tool presented a set of challenges that meant the tool was not a vast improvement over paper-based data collection, and the up-front costs and training were too expensive and lengthy to make the switch to digital tools worthwhile. In addition, the team felt that face-to-face dynamics in the community and having access to notes and written observations in the margins of a paper survey would enable them to conduct a better evaluation. Some tablets are beginning to enable more interactivity and better design for surveys, but not yet in a way that made them a viable option for this evaluation. I liked how the organization went through a very thorough and in-depth process to make this decision.

Other colleagues also commented that the tech tools are still not quite ‘there’ yet for M&E. Even top of the line business solutions are generally found to be somewhat clunky. Million dollar models are not relevant for environments that development evaluators are working in; in addition to their high cost, they often have too many features or require too much training. There are some excellent mid-range tools that are designed for the environment, but many lack vital features such as availability in multiple languages. Simple tools that are more easily accessible and understandable without a lot of training are not sophisticated enough to conduct a large-scale data collection exercise. One person I talked with suggested that the private sector will eventually develop appropriate tools, and the not-for-profit sector will then adopt them. She felt that those of us who are interested in ICTs in M&E are slightly ahead of the curve and need to wait a few years until the tools are more widespread and common. Many people attending the Tech and M&E sessions at AfrEA made the point that use of ICTs in M&E would get easier and cheaper as the field develops, tools get more advanced/appropriate/user-friendly and widely tested, and networks/ platforms/ infrastructure improves in less-connected rural areas.

5) Need for documentation, evaluation and training on use of ICTs in M&E – Some evaluators felt that ICTs are only suitable for routine data collection as part of an ongoing program, but not good for large-scale evaluations. Others pointed out that the notions of ‘ICT for M&E’ and ‘mobile data collection/mobile surveys’ are often used interchangeably, and evaluation practitioners need to look at the multiple ways that ICTs can be used in the wider field of M&E. ICTs are not just useful for moving from paper surveys to mobile data gathering. An evaluator working on a number of RCTs mentioned that his group relies on ICTs for improving samples, reducing bias, and automatically checking data quality.

There was general agreement that M&E practitioners need resources, opportunities for more discussion, and capacity strengthening on the multiple ways that ICTs may be able to support M&E. One evaluator noted that civil society organizations have a tendency to rush into things, hit a brick wall, and then cross their arms and say, “well, this doesn’t work” (in this case, ICTs for M&E). With training and capacity, and as more experience and documentation is gained, he considered that ICTs could have a huge role in making M&E more efficient and effective.

One evaluator, however, questioned whether having better, cheaper, higher quality data is actually leading to better decisions and outcomes. Another evaluator asked for more evidence of what works, when, with whom and under what circumstances so that evaluators could make better decisions around use of ICTs in M&E. Some felt that a decision tree or list of considerations or key questions to think through when integrating ICTs into M&E would be helpful for practitioners. In general, it was agreed that ICTs can help overcome some of our old challenges, but that they inevitably bring new challenges. Rather than shy away from using ICTs, we should try to understand these new challenges and find ways to overcome/work around them. Though the mHealth field has done quite a bit of useful research, and documentation on digital data collection is growing, use of ICTs is still relatively unexplored in the wider evaluation space.

6) There is no simple answer. One of my takeaways from all the sessions was that many M&E specialists are carefully considering options, and thinking quite a lot about which ICTs for what, whom, when and where rather than deciding from the start that ICTs are ‘good and beneficial’ or ‘bad and not worth considering.’ This is really encouraging, and to be expected of a thoughtful group like this. I hope to participate in more discussions of this nature that dig into the nuances of introducing ICTs into M&E.

Some new resources and forthcoming documentation may help to further set the stage for better learning and application of ICTs in the M&E process. Pact has just released their Mobile Technology Toolkit, and Michael Bamberger and I are finishing up a paper on ICT-enabled M&E that might help provide a starting point and possible framework to move things forward. The “field” of ICTs in M&E is quite broad, however, and there are many ways to slice the cake. Here is the list of toolkits, blog posts and other links that we compiled for AfrEA – please add any that you think are missing!

(Part 2 of this post)

Previous posts on ICTs and M&E:

The NYC Technology Salon on February 28th examined the connection between bigger, better data and resilience. We held morning and afternoon Salons due to the high response rate for the topic. Jake Porway, DataKind; Emmanuel Letouzé, Harvard Humanitarian Initiative; and Elizabeth Eagen, Open Society Foundations; were our lead discussants for the morning. Max Shron, Data Strategy; joined Emmanuel and Elizabeth for the afternoon session.

This post summarizes key discussions from both Salons.

What the heck do we mean by ‘big data’?

The first question at the morning salon was: What precisely do we mean by the term ‘big data’? Participants and lead discussants had varying definitions. One way of thinking about big data is that it is comprised of small bits of unintentionally produced ‘data exhaust’ (website cookies, cellphone data records, etc.) that add up to a dataset. In this case, the term big data refers to the quality and nature of the data, and we think of non-sampled data that are messy, noisy and unstructured. The mindset that goes with big data is one of ‘turning mess into meaning.’

Some Salon participants understood big data as datasets that are too large to be stored, managed and analyzed via conventional database technologies or managed on normal computers. One person suggested dropping the adjective ‘big,’ forgetting about the size, and instead considering the impact of the contribution of the data to understanding. For example, if there were absolutely no data on something and 1000 data points were contributed, this might have a greater impact than adding another 10,000 data points to an existing set of 10 million.

The point here was that when the emphasis is on big (understood as size and/or volume), someone with a small data set (for example, one that fits into an excel sheet) might feel inadequate, yet their data contribution may be actually ‘bigger’ than a physically larger data set (aha! it’s not the size of the paintbrush…). There was a suggestion that instead of talking about big data we should talk about smart data.

How can big data support development?

Two frameworks were shared for thinking about big data in development. One from UN Global Pulse considers that big data can improve a) real-time awareness, b) early warning and c) real-time monitoring. Another looks at big data being used for three kinds of analysis: a) descriptive (providing a summary of something that has already happened), b) predictive (likelihood and probability of something occurring in the future), and c) diagnostic (causal inference and understanding of the world).

What’s the link between big data and resilience?

‘Resilience’ as a concept is contested, difficult to measure and complex. In its most simple definition, resilience can be thought of as the ability to bounce back or bounce forward. (For an interesting discussion on whether we should be talking about sustainability or resilience, see this piece). One discussant noted that global processes and structures are not working well for the poor, as evidenced from continuing cycles of poverty and glaring wealth inequalities. In this view, people are poor as a result of being more exposed and vulnerable to shocks, at the same time, their poverty increases their vulnerability, and it’s difficult to escape from the cycle where over time, small and large shocks deplete assets. An assets-based model of resilience would help individuals, families and communities who are hit by a shock in one sphere — financial, human, capital, social, legal and/or political — to draw on the assets within another sphere to bounce back or forward.

Big data could help this type of an assets-based model of resilience by predicting /helping poor and vulnerable people predict when a shock might happen and preparing for it. Big data analytics, if accessible to the poor, could help them to increase their chances of making better decisions now and for the future. Big data then, should be made accessible and available to communities so that they can self-organize and decrease their own exposure to shocks and hazards and increase their ability to bounce back and bounce forward. Big data could also help various actors to develop a better understanding of the human ecosystem and contribute to increasing resilience.

Can ivory tower big data approaches contribute to resilience?

The application of big data approaches to efforts that aim to increase resilience and better understand human ecosystems often comes at things from the wrong angle, according to one discussant. We are increasingly seeing situations where a decision is made at the top by people who know how to crunch data yet have no way of really understanding the meaning of the data in the local context. In these cases, the impact of data on resilience will be low, because resilience can only truly be created and supported at the local level. Instead of large organizations thinking about how they can use data from afar to ‘rescue’ or ‘help’ the poor, organizations should be working together with communities in crisis (or supporting local or nationally based intermediaries to facilitate this process) so that communities can discuss and pull meaning from the data, contextualize it and use it to help themselves. They can also be more informed what data exist about them and more aware of how these data might be used.

For the Human Rights community, for example, the story is about how people successfully use data to advocate for their own rights, and there is less emphasis on large data sets. Rather, the goal is to get data to citizens and communities. It’s to support groups to define and use data locally and to think about what the data can tell them about the advocacy path they could take to achieve a particular goal.

Can data really empower people?

To better understand the opportunities and challenges of big data, we need to unpack questions related to empowerment. Who has the knowledge? The access? Who can use the data? Salon participants emphasized that change doesn’t come by merely having data. Rather it’s about using big data as an advocacy tool to tell the world to change processes and to put things normally left unsaid on the table for discussion and action. It is also about decisions and getting ‘big data’ to the ‘small world,’ e.g., the local level. According to some, this should be the priority of ‘big data for development’ actors over the next 5 years.

Though some participants at the Salon felt that data on their own do not empower individuals; others noted that knowing your credit score or tracking how much you are eating or exercising can indeed be empowering to individuals. In addition, the process of gathering data can help communities understand their own realities better, build their self-esteem and analytical capacities, and contribute to achieving a more level playing field when they are advocating for their rights or for a budget or service. As one Salon participant said, most communities have information but are not perceived to have data unless they collect it using ‘Western’ methods. Having data to support and back information, opinions and demands can serve communities in negotiations with entities that wield more power. (See the book “Who Counts, the power of participatory statistics” on how to work with communities to create ‘data’ from participatory approaches).

On the other hand, data are not enough if there is no political will to make change to respond to the data and to the requests or demands being made based on the data. As one Salon participant said: “giving someone a data set doesn’t change politics.”

Should we all jump on the data bandwagon?

Both discussants and participants made a plea to ‘practice safe statistics!’ Human rights organizations wander in and out of statistics and don’t really understand how it works, said one person. ‘You wouldn’t go to court without a lawyer, so don’t try to use big data unless you can ensure it’s valid and you know how to manage it.’ If organizations plan to work with data, they should have statisticians and/or data scientists on staff or on call as partners and collaborators. Lack of basic statistical literacy is a huge issue amongst the general population and within many organizations, thought leaders, and journalists, and this can be dangerous.

As big data becomes more trendy, the risk of misinterpretation is growing, and we need to place more attention on the responsible use of statistics and data or we may end up harming people by bad decisions. ‘Everyone thinks they are experts who can handle statistics – bias, collection, correlation’ these days. And ‘as a general rule, no matter how many times you say the data show possible correlation not causality, the public will understand that there is causality,’ commented one discussant. And generally, he noted, ‘when people look at data, they believe them as truth because they include numbers, statistics, science.’ Greater statistical literacy could help people to not just read or access data and information but to use them wisely, to understand and question how data are interpreted, and to detect political or other biases. What’s more, organizations today are asking questions about big data that have been on statisticians’ minds for a very long time, so reaching out to those who understand these issues can be useful to avoid repeating mistakes and re-learning lessons that have already been well-documented.

This poor statistical literacy becomes a serious ethical issue when data are used to determine funding or actions that impact on people’s lives, or when they are shared openly, accidentally or in ways that are unethical. In addition, privacy and protection are critical elements in using and working with data about people, especially when the data involve vulnerable populations. Organizations can face legal action and liability suits if their data put people at harm, as one Salon participant noted. ‘An organization could even be accused of manslaughter… and I’m speaking from experience,’ she added.

What can we do to move forward?

Some potential actions for moving forward included:

  • Emphasis with donors that having big data does not mean that in order to cut costs, you should eliminate community level processes related to data collection, interpretation, analysis, and ownership;
  • Evaluations and literature/documentation on the effectiveness of different tools and methods, and when and in which contexts they might be applicable, including things like cost-benefit analyses of using big data and evaluation of its impact on development/on communities when combined with community level processes vs used alone/without community involvement — practitioner gut feelings are that big data without community involvement is irresponsible and ineffective in terms of resilience, and it would be good to have evidence to help validate or disprove this;
  • More and better tools and resources to support data collection, visualization and use and to help organizations with risk analysis, privacy impact assessments, strategies and planning around use of big data; case studies and a place to share and engage with peers, creation of a ‘cook book’ to help organizations understand the ingredients, tools, processes of using data/big data in their work;
  • ‘Normative conventions’ on how big data should be used to avoid falling into tech-driven dystopia;
  • Greater capacity for ‘safe statistics’ among organizations;
  • A community space where frank and open conversations around data/big data can occur in an ongoing way with the right range of people and cross-section of experiences and expertise from business, data, organizations, etc.

In conclusion?

We touched upon all types of data and various levels of data usage for a huge range of purposes at the two Salons. One closing thought was around the importance of having a solid idea of what questions we trying to answer before moving on to collecting data, and then understanding what data collection methods are adequate for our purpose, what ICT tools are right for which data collection and interpretation methods, what will done with the data/what is the purpose of collecting data, how we’ll interpret them, and how data will be shared, with whom, and in what format.

See this growing list of resources related to Data and Resilience here and add yours!

Thanks to participants and lead discussants for the fantastic exchange, and a big thank you to ThoughtWorks for hosting us at their offices for this Salon. Thanks also to Hunter Goldman, Elizabeth Eagen and Emmanuel Letouzé for their support developing this Salon topic, and to Somto Fab-Ukozor for support with notes and the summary. Salons are held under Chatham House Rule, therefore no attribution has been made in this post. If you’d like to attend future Salons, sign up here!

Screen Shot 2014-03-07 at 3.34.23 AM

As Tom over at Humanosphere wrote a few days ago, there’s a cool initiative happening at Sheffield University that seeks to develop a global research agenda related to the post-2015 sustainable development goals process. (Disclaimer, I’m part of the steering committee.)

ID100: The Hundred Most Important Questions in International Development asks individuals and organizations from across policy, practice and academia to submit questions that address the world’s biggest environmental, political and socioeconomic problems. These will then be shortlisted down to a final set of 100 questions following a debate and voting process with representatives from development organizations and academia who will meet in July 2014. 

The final list of questions will be published as policy report and in a leading academic journal. More on the consultation methodology here. Similar crowdsourced priority-setting exercises have worked in biodiversity conservation, food security and other areas, and they have been instrumental in framing global research priorities for policy development and implementation.

Screen Shot 2014-03-07 at 3.36.06 AMAnyone can submit up to five questions related to key issues in international development that require more exploration. You are encouraged to involve colleagues in the formulation of these questions.

Please submit your questions by March 25th – and check the submission guidelines before formulating questions. More information on the project can be accessed on the ID100 website. Hashtag: #ID100.

This is a guest post by Daniella Ben-Attar (@dbenattar) who consults for international development agencies, NGOs and corporations on areas relating to youth participation, governance, municipal capacity building, ICT4D and peace building.

by Daniella Ben-Attar

Youth in Mali with local authorities.

Youth in Mali with local authorities.

ICTs are increasingly being looked to as holding great promise for improving participatory governance and citizen engagement. Mobile phones have been a game-changer in this sphere, with nearly seven billion mobile-cellular subscriptions worldwide, including 89% penetration in the developing world. Youth are at the center of these developments, both as drivers and consumers of technological innovation.  This is particularly true in developing countries where the young generation is leading the way in the usage of technology to overcome social, political and economic exclusion to begin driving positive change in their communities. The largest cohort in history, youth aged 15-24 number more than 1.2 billion worldwide, with an estimated 87% living in developing countries.  They are almost twice as networked as the global population as a whole, with the ICT age gap more pronounced in least developed countries where young people are often three times more likely to be online than the general population.

The combination of the “youth bulge” and “mobile miracle” has great potential to enable new responses to the longstanding challenge of youth engagement in governance across the developing world. Young citizens are utilizing simple mobile technology to innovate new platforms, tools and mechanisms aiming to amplify their voices and influence government. Youth are being proactive to play a greater role in governance through mobile-based communication avenues, user-generated information, tools tracking government accountability, anti-corruption platforms, crowd-sourcing and more. This is a dramatic shift from the days when the only way to gain the attention of a government official was through slow and cumbersome bureaucratic processes and official meetings in government offices.

A Growing Youth-Local Government Disconnect

Ironically, the impact of these efforts appears to be more pronounced at the national level than at the local level of government. Indeed, ICTs seem to be strengthening communications between youth and central government instead of enhancing connections with the closest level of governance where young citizens can be resources for community development. Applications and innovations in cooperation with government that address local issues have largely been the product of national government bodies. Most youth-led initiatives have not been successful in securing local government partnership, limiting impact. A communications gap has widened between young citizens and their local governments, which are often staffed by individuals with far less digital experience than their youthful constituents. As a result, youth and their local leaders often seem to be speaking in different languages through different media.  Local government deficits in capacity and resources continue to exist as barriers, as well as the need for sensitization to youth engagement as a priority outcome of adopting and shaping ICT-enabled practices.

Most young people using technology as a way to influence governance will tell you a similar story. When expressing themselves through social media outlets and ICT-enabled mechanisms, it is usually the national political figures that are more attuned and responsive. Local leaders are far behind their national counterparts in ICT capacity and usage. National ministers and officials often use Twitter accounts, blogs, SMS and websites to engage with their citizens, who by default are largely young. While this is a positive development, it also elicits frustration from young people who feel that their voices are ignored or unheard by elder leaders at the local level where chances are greatest for tangible impact in their day-to-day lives.

President Kagame of Rwanda is a stark example.  Youth have described how the president directly interacted with young citizens via Twitter and addressed concerns relating to many issues, from police violence towards youth to business ideas for urban tourism.  No such possibilities existed for these same youth to approach the local authority with these locally-based needs.  Even more significant, Kagame merged the national ministries of Youth and ICT in 2012 and appointed a Minister of Youth and ICT.  This is a groundbreaking move both in terms of ICT and youth, with youth ministries commonly grouped with sports or culture. However, these extraordinary national developments are not reflected in the policy and practice of local government in Rwanda.

Digital mapping initiatives have been in the spotlight as a new youth-driven tool drawing attention to local issues often overlooked by government officials.  While communities are benefitting from these processes, youth leaders report that these maps often do not gain the attention of city hall. For example, Kenyan NGO Map Kibera has seen its maps utilized by national ministry committees, better equipped with the capacity and mindset to absorb digital data, while city council has not been responsive to ICT-based approaches. Young leaders in Kandy City, Sri Lanka are working to bridge the “youth-local government ICT gap” which they have identified as a major barrier in engaging youth in local development. These young leaders are training municipal officials in computer skills and creating new ICT platforms for citizen-local government interaction as part of a UN-HABITAT supported youth-led training and education program run by YES – City of Youth.

Building Local Government Capacity for ICT & Youth Engagement

Partnership with local government is viewed by stakeholders as a key missing ingredient in enabling governance technology applications to have tangible results at the community level. The importance of “closing the engagement loop” and early local government buy-in is emphasized time and again by stakeholders in the field as a vital lesson learned through pilot programs. Youth organizations like Youth Agenda and Sisi ni Amani have achieved successful governance results by engaging local leaders as partners from the preliminary stages, highlighting the benefits they can gain through mobile solutions that increase civic engagement, enhance service delivery, fight corruption and bridge between local government and citizens.

Bridging the youth-local government gap will require sensitizing local officials and encouraging them to see the advantages of “listening” to youth ICT platforms, to bring them to where the majority of youth are voicing their opinions, and enable them to take responsive actions. National governments should be encouraged to help local governments be better equipped to address youthful concerns at the local level through capacity building for both youth engagement and ICT4G.  This can be supported by integrating local ICT components in national ICT plans, or increased “decentralization” and integration of both youth and ICT strategies, bolstered by budgetary allocations and devolution of authority. When seeking to utilize ICT to deliver positive governance outcomes for young people, “local gov” must be part of the “ICT4Gov” equation.

This blog post draws on findings from a UN-HABITAT Report entitled “ICT, Urban Governance and Youth” co-authored by Daniella Ben-Attar and Tim Campbell.


Get every new post delivered to your Inbox.

Join 743 other followers