Feeds:
Posts
Comments

Posts Tagged ‘Technology Salon’

Our December 2015 Technology Salon discussion in NYC focused on approaches to girls’ digital privacy, safety and security. By extension, the discussion included ways to reduce risk for other vulnerable populations. Our lead discussants were Ximena BenaventeGirl Effect Mobile (GEM) and Jonathan McKay, Praekelt Foundation. I also shared a draft Girls’ Digital Privacy, Safety and Security Policy and Toolkit I’ve been working on with both organizations over the past year.

Girls’ digital privacy, safety and security risks

Our first discussant highlighted why it’s important to think specifically about girls and digital security. In part, this is because different factors and vulnerabilities combine, exacerbating girls’ levels of risk. For example, girls living on less than $2 per day likely only have access to basic mobile phones, which are often borrowed from parents or siblings. The organization she works with always starts with deep research on aspects like ownership vs. borrowship and whether girls’ mobile usage is free/unlimited and un-supervised or controlled by gatekeepers such as parents, brothers, or other relatives. This helps to design better tools, services and platforms and to design for safety and security, she said. “Gatekeepers are very restrictive in many cases, but parental oversight is not necessarily a bad thing. We always work with parents and other gatekeepers as well as with girls themselves when we design and test.” When girls are living in more traditional or conservative societies, she said, we also need to think about how content might affect girls both online and offline. For example, “is content sufficiently progressive in terms of girls’ rights, yet safe for girls to read, comment on or discuss with friends and family without severe retaliation?”

Research suggests that girls who are more vulnerable offline (due to poverty or other forms of marginalization), are likely also more vulnerable to certain risks online, so we design with that in mind, she said. “When we started off on this project, our team members were experts in digital, but we had less experience with the safety and privacy aspects when it comes to girls living under $2/day or who were otherwise vulnerable. “Having additional guidance and developing a policy on this aspect has helped immensely – but has also slowed our processes down and sometimes made them more expensive,” she noted. “We had to go back to everything and add additional layers of security to make it as safe as possible for girls. We have also made sure to work very closely with our local partners to be sure that everyone involved in the project is aware of girls’ safety and security.”

Social media sites: Open, Closed, Private, Anonymous?

One issue that came up was safety for children and youth on social media networks. A Salon participant said his organization had thought about developing this type of a network several years back but decided in the end that the security risks outweighed the advantages. Participants discussed whether social media networks can ever be safe. One school of thought is that the more open a platform, the safer it is, as “there is no interaction in private spaces that cannot be constantly monitored or moderated.” Some worry about open sites, however, and set up smaller, closed, private groups that were closely monitored. “We work with victims of violence to share their stories and coping mechanisms, so, for us, private groups are a better option.”

Some suggested that anonymity on a social media site can protect girls and other vulnerable groups, however there is also research showing that Internet anonymity contributes to an increase in activities such as bullying and harassment. Some Salon participants felt that it was better to leverage existing platforms and try to use them safely. Others felt that there are no existing social media platforms that have enough security for girls or other vulnerable groups to use with appropriate levels of risk. “We sometimes recruit participants via existing social media platforms,” said one discussant, “but we move people off of those sites to our own more secure sites as soon as we can.”

Moderation and education on safety

Salon participants working with vulnerable populations said that they moderate their sites very closely and remove comments if users share personal information or use offensive language. “Some project budgets allow us to have a moderator check every 2 hours. For others, we sweep accounts once a day and remove offensive content within 24 hours.” One discussant uses moderation to educate the community. “We always post an explanation about why a comment was removed in order to educate the larger user base about appropriate ways to use the social network,” he said.

Close moderation becomes difficult and costly, however, as the user base grows and a platform scales. This means individual comments cannot be screened and pre-approved, because that would take too long and defeat the purpose of an engaging platform. “We need to acknowledge the very real tension between building a successful and engaging community and maintaining privacy and security,” said one Salon participant. “The more you lock it down and the more secure it is, the harder you find it is to create a real and active community.”

Another participant noted that they use their safe, closed youth platform to educate and reinforce messaging about what is safe and positive use of social media in hopes that young people will practice safe behaviors when they use other platforms. “We know that education and awareness raising can only go so far, however,” she said, “and we are not blind to that fact.” She expressed concern about risk for youth who speak out about political issues, because more and more governments are passing laws that punish critics and censor information. The organization, however, does not want to encourage youth to stop voicing opinions or participating politically.

Data breaches and project close-out

One Salon participant asked if organizations had examples of actual data breaches, and how they had handled them. Though no one shared examples, it was recommended that every organization have a contingency plan in place for accidental data leaks or a data breach or data hack. “You need to assume that you will get hacked,” said one person, “and develop your systems with that as a given.”

In addition to the day-to-day security issues, we need to think about project close-out, said one person. “Most development interventions are funded for a short, specific period of time. When a project finishes, you get a report, you do your M&E, and you move on. However, the data lives on, and the effects of the data live on. We really need to think more about budgeting for proper project wind-down and ensure that we are accountable beyond the lifetime of a project.”

Data security, anonymization, consent

Another question was related to using and keeping girls’ (and others’) data safe. “Consent to collect and use data on a website or via a mobile platform can be tricky, especially if we don’t know how to explain what we might do with the data,” said one Salon participant. Others suggested it would be better not to collect any data at all. “Why do we even need to collect this data? Who is it for?” he asked. Others countered that this data is often the only way to understand what people are doing on the site, to make adjustments and to measure impact.

One scenario was shared where several partner organizations discussed opening up a country’s cell phone data records to help contain a massive public health epidemic, but the privacy and security risks were too great, so the idea was scrapped. “Some said we could anonymize the data, but you can never really and truly anonymize data. It would have been useful to have a policy or a rubric that would have guided us in making that decision.”

Policy and Guidelines on Girls Privacy, Security and Safety

Policy guidelines related to aspects such as responsible data for NGOs, data security, privacy and other aspects of digital security in general do exist. (Here are some that we compiled along with some other resources). Most IT departments also have strict guidelines when it comes to donor data (in the case of credit card and account information, for example). This does not always cross over to program-level ICT or M&E efforts that involve the populations that NGOs are serving through their programming.

General awareness around digital security is increasing, in part due to recent major corporate data hacks (e.g., Target, Sony) and the Edward Snowden revelations from a few years back, but much more needs to be done to educate NGO staff and management on the type of privacy and security measures that need to be taken to protect the data and mitigate risk for those who participate in their programs.  There is an argument that NGOs should have specific digital privacy, safety and security policies that are tailored to their programming and that specifically focus on the types of digital risks that girls, women, children or other vulnerable people face when they are involved in humanitarian or development programs.

One such policy (focusing on vulnerable girls) and toolkit (its accompanying principles and values, guidelines, checklists and a risk matrix template); was shared at the Salon. (Disclosure: – This policy toolkit is one that I am working on. It should be ready to share in early 2016). The policy and toolkit take program implementers through a series of issues and questions to help them assess potential risks and tradeoffs in a particular context, and to document decisions and improve accountability. The toolkit covers:

  1. data privacy and security –using approaches like Privacy by Design, setting limits on the data that is collected, achieving meaningful consent.
  2. platform content and design –ensuring that content produced for girls or that girls produce or volunteer is not putting girls at risk.
  3. partnerships –vetting and managing partners who may be providing online/offline services or who may partner on an initiative and want access to data, monetizing of girls’ data.
  4. monitoring, evaluation, research and learning (MERL) – how will program implementers gather and store digital data when they are collecting it directly or through third parties for organizational MERL purposes.

Privacy, Security and Safety Implications

Our final discussant spoke about the implications of implementing the above-mentioned girls’ privacy, safety and security policy. He started out saying that the policy starts off with a manifesto: We will not compromise a girl in any way, nor will we opt for solutions that cut corners in terms of cost, process or time at the expense of her safety. “I love having this as part of our project manifesto, he said. “It’s really inspiring! On the flip side, however, it makes everything I do more difficult, time consuming and expensive!”

To demonstrate some of the trade-offs and decisions required when working with vulnerable girls, he gave examples of how the current project (implemented with girls’ privacy and security as a core principle) differed from that of a commercial social media platform and advertising campaign he had previously worked on (where the main concern was the reputation of the corporation, not that of the users of the platform and the potential risks they might put themselves in by using the platform).

Moderation

On the private sector platform, said the discussant, “we didn’t have the option of pre-moderating comments because of the budget and because we had 800 thousand users. To meet the campaign goals, it was more important for users to be engaged than to ensure content was safe. We focused on removing pornographic photos within 24 hours, using algorithms based on how much skin tone was in the photo.” In the fields of marketing and social media, it’s a fairly well-known issue that heavy-handed moderation kills platform engagement. “The more we educated and informed users about comment moderation, or removed comments, the deader the community became. The more draconian the moderation, the lower the engagement.”

The discussant had also worked on a platform for youth to discuss and learn about sexual health and practices, where he said that users responded angrily to moderators and comments that restricted their participation. “We did expose our participants to certain dangers, but we also knew that social digital platforms are more successful when they provide their users with sense of ownership and control. So we identified users that exhibited desirable behaviors and created a different tier of users who could take ownership (super users) to police and flag comments as inappropriate or temporarily banned users.” This allowed a 25% decrease in moderation. The organization discovered, however, that they had to be careful about how much power these super users had. “They ended up creating certain factions on the platform, and we then had to develop safeguards and additional mechanisms by which we moderated our super users!”

Direct Messages among users

In the private sector project example, engagement was measured by the number of direct or private messages sent between platform users. In the current scenario, however, said the discussant, “we have not allowed any direct messages between platform users because of the potential risks to girls of having places on the site that are hidden from moderators. So as you can see, we are removing some of our metrics by disallowing features because of risk. These activities are all things that would make the platform more engaging but there is a big fear that they could put girls at risk.”

Adopting a privacy, security, and safety policy

One discussant highlighted the importance of having privacy, safety and security policies before a project or program begins. “If you start thinking about it later on, you may have to go back and rebuild things from scratch because your security holes are in the design….” The way a database is set up to capture user data can make it difficult to query in the future or for users to have any control of what information is or is not being shared about them. “If you don’t set up the database with security and privacy in mind from the beginning, it might be impossible to make the platform safe for girls without starting from scratch all over again,” he said.

He also cautioned that when making more secure choices from the start, platform and tool development generally takes longer and costs more. It can be harder to budget because designers may not have experience with costing and developing the more secure options.

“A valuable lesson is that you have to make sure that what you’re trying to do in the first place is worth it if it’s going to be that expensive. It is worth a girls’ while to use a platform if she first has to wade through a 5-page terms and conditions on a small mobile phone screen? Are those terms and conditions even relevant to her personally or within her local context? Every click you ask a user to make will reduce their interest in reaching the platform. And if we don’t imagine that a girl will want to click through 5 screens of terms and conditions, the whole effort might not be worth it.” Clearly, aspects such as terms and conditions and consent processes need to be designed specifically to fit new contexts and new kinds of users.

Making responsible tradeoffs

The Girls Privacy, Security and Safety policy and toolkit shared at the Salon includes a risk matrix where project implementers rank the intensity and probability of risks as high, medium and low. Based on how a situation, feature or other potential aspect is ranked and the possibility to mitigate serious risks, decisions are made to proceed or not. There will always be areas with a certain level of risk to the user. The key is in making decisions and trade-offs that balance the level of risk with the potential benefits or rewards of the tool, service, or platform. The toolkit can also help project designers to imagine potential unintended consequences and mitigate risk related to them. The policy also offers a way to systematically and pro-actively consider potential risks, decide how to handle them, and document decisions so that organizations and project implementers are accountable to girls, peers and partners, and organizational leadership.

“We’ve started to change how we talk about user data in our organization,” said one discussant. “We have stopped thinking about it as something WE create and own, but more as something GIRLS own. Banks don’t own people’s money – they borrow it for a short time. We are trying to think about data that way in the conversations we’re having about data, funding, business models, proposals and partnerships. You don’t get to own your users’ data, we’re not going to share de-anonymized data with you. We’re seeing legislative data in some of the countries we work that are going that way also, so it’s good to be thinking about this now and getting prepared”

Take a look at our list of resources on the topic and add anything we may have missed!

 

Thanks to our friends at ThoughtWorks for hosting this Salon! If you’d like to join discussions like this one, sign up at Technology SalonSalons are held under Chatham House Rule, therefore no attribution has been made in this post.

Advertisements

Read Full Post »

At our November 18th Technology Salon, we discussed how different organizations are developing their ICT for development (ICT4D) strategies. We shared learning on strategy development and buy-in, talked about whether organizations should create special teams or labs for ICT- and innovation-related work or mainstream the ICT4D function, and thought about how organizations can define and find the skill sets needed for taking their ICT-enabled work forward. Population Council’s Stan Mierzwa, Oxfam America’s Neal McCarthy, and Cycle Technologies’ Leslie Heyer joined as lead discussants, and we heard from Salon participants about their experiences too.

Participating organizations were at various stages of ICT4D work, but most had experienced similar challenges and frustrations with taking their work forward. Even organizations that had created ICT4D strategies a couple of years ago said that implementation was slow.

Some of the key elements mentioned by our first discussant as important for managing and strategically moving ICT forward in an organization included:

  • being more informed about where different offices and staff were using ICTs for programmatic work,
  • establishing a standard set of technology tools for organizational use,
  • improved knowledge management about ICTs,
  • publishing on how ICTs were being used in programs (to help with credibility),
  • engaging with different teams and leadership to secure support and resources
  • working more closely with human resources teams who often do not understand ICT4D-related job descriptions and the profile needed.

Our second discussant said that his organization developed an ICT4D strategy in order to secure resources and greater support for moving ICT4D forward. It was also starting to be unwieldy to manage all of the different ideas and tools being used across the organization, and it seemed that greater harmonization would allow for improved IT support for more established tools as well as establishment of other ways to support new innovations.

In this case, the organization looked at ICTs as two categories: technology for development workers and technology for development outcomes. They used Gartner’s ‘pace layered’ model (which characterizes systems of innovation, systems of differentiation, and systems of record) as a way of analyzing the support roles of different departments.

One of the initial actions taken by this organization was establishing a small tech incubation fund that different offices could apply for in order to try something new with ICTs in their programs and campaigns. Another action was to take 10 staff to the Catholic Relief Services (CRS) ICT4D conference to learn more about ICT4D and to see what their peers from similar organizations were doing. In return for attending the conference, staff were required to submit a proposal for the tech incubation fund.

For the development of the strategy document and action plan, the ICT4D strategy team worked with a wider group of staff to develop a list of current ICT-enabled initiatives and a visual heat map of actions and activities across the organization. This formed the basis for discussions on where lots of ICT4D activities were happening and where there was nothing going on with ICTs. The team then discussed what the organization should do strategically to support and potentially consolidate existing activities and what should be done about areas where there were few ICT-related activities – should those areas be left alone or was there a reason to look at them to see if ICT should be incorporated?

Having done that, the organization adapted Nethope’s Organizational Guide to ICT4D to fit its own structure and culture, and used it as a framework for ICT4D strategy discussions with key staff from different teams. The Nethope guide suggests five key functions for strategic, organization-wide ICT4D: lead organizational change, drive knowledge exchange, build a portfolio, manage processes, and develop an advisory service (see below). The aforementioned activities were also clustered according to which of these 5 areas they fell into.

Screen Shot 2015-11-24 at 8.53.12 AM

(Table of contents from Nethope’s Guide.)

The organization felt it was also important to change the image of the IT team. ‘We had to show that we were not going to tie people up with formal committees and approvals if they wanted to try something new and innovative. Being more approachable is necessary or staff will bypass the IT team and go to consultants, and then we open ourselves up to data privacy risks and we also lose institutional knowledge.’

Salon participants agreed that it was important to know how to “sell” an ICT4D-related idea to frontline staff, management and leadership. Some ways to do this include demonstrating the value-add of ICTs in terms of longer-term cost and time efficiencies, showing the benefit of real-time data for decision-making, and demonstrating what peer organizations are doing. Organizations often also need someone at the top who is pushing for change and modernization.

Our third discussant said that her company has been shifting from a commercial product developer to a full-fledged technology company. She outlined the need for strategic thinking along that journey. Initially, the company outsourced activities such as research and data collection. With time, it started to pull key functions in house since systems maintenance and technology has become a core part of the business.

“As a small company, we can be flexible and change easily,” she said. ‘ICT is embedded into our culture and everyone thinks about it.’ One challenge that many ICT4D initiatives face – whether they are happening in a non-profit or a for-profit — is sustainability. ‘People are often fine with paying for a physical product, but when it comes to the web, they are accustomed to getting everything for free, which makes long-term sustainability difficult.’

In order to continuously evolve their strategies, organizations need to have time and space to constantly step back and think about their underlying values and where they see themselves in 5 or 10 years. A more pro-active relationship with donors is also important. Although Salon participants felt that the ICT4D Principles and related processes were promising, they also felt that donors do not have a clear idea of what they are looking for, what exists already, what needs to be created, and what evidence base exists for different tools or kinds of ICT4D. One Salon participant felt that ‘donor agencies don’t know what kinds of tech are effective, so it’s up to you as an implementer to bring the evidence to the table. It’s critical to have the ITC4D support staff at the table with you, because if not these more detailed conversations about the tech don’t happen with donors and you’ll find all kinds of duplication of efforts.’

Another challenge with thinking about ICT4D in a strategic way is that donors normally don’t want to fund capacity building, said another Salon participant. They prefer to fund concrete projects or innovation challenges rather than supporting organizations to create an environment that gives rise to innovation. In addition, funding beyond the program cycle is a big challenge. ‘We need to be thinking about enterprise systems, layered on systems, national systems,’ said one person. ‘And systems really struggle here to scale and grow if you can’t claim ownership for the whole.’

Salon participants highlighted hiring and human resources departments as a big barrier when it comes to ICT4D. It is often not clear what kinds of skills are needed to implement ICT4D programs, and human resources teams often screen for the wrong skill sets because they do not understand the nature of ICT4D. ‘I always make them give me all the CVs and screen them myself,’ said one person. ‘If not, some of the best people will not make it to the short list.’ Engaging with human resources and sharing the ICT4D strategy is one way to help with better hiring and matching of job needs with skill sets that are out there and potentially difficult to find.

In conclusion, whether the ICT4D strategy is to mainstream, to isolate and create a ‘lab,’ or to combine approaches, it seems that most organizations are struggling a bit to develop and/or implement ICT4D strategies due to the multiple pain points of slow organizational change and the need for more capacity and resources. Some are making headway, however, and developing clearer thinking and action plans that are paying off in the short term, and that may set the organizations up for eventual ICT4D success.

Thanks to Population Council for hosting this Salon! If you’d like to join discussions like this one, sign up at Technology Salon.

Salons are held under Chatham House Rule. No attribution has been made in this post.

Read Full Post »

Traditional development evaluation has been characterized as ‘backward looking’ rather than forward looking and too focused on proving over improving. Some believe applying an ‘agile’ approach in development would be more useful — the assumption being that if you design a program properly and iterate rapidly and constantly based on user feedback and data analytics, you are more likely achieve your goal or outcome without requiring expensive evaluations. The idea is that big data could eventually allow development agencies to collect enough passive data about program participants that there would no longer be a need to actively survey people or conduct a final evaluation, because there would be obvious patterns that would allow implementers to understand behaviors and improve programs along the way.

The above factors have made some evaluators and data scientists question whether big data and real-time availability of multiple big data sets, along with the technology that enables their collection and analysis, will make evaluation as we know it obsolete. Others have argued that it’s not the end of evaluation, but rather we will see a blending of real-time monitoring, predictive modeling, and impact evaluation, depending on the situation. Big questions remain, however, about the feasibility of big data in some contexts. For example, are big data approaches useful when it comes to people who are not producing very much digital data? How will the biases in big data be addressed to ensure that the poorest, least connected, and/or most marginalized are represented?

The Technology Salon on Big Data and Evaluation hosted during November’s  American Evaluation Association Conference in Chicago opened these questions up for consideration by a roomful of evaluators and a few data scientists. We discussed the potential role of new kinds and quantities of data. We asked how to incorporate static and dynamic big data sources into development evaluation. We shared ideas on what tools, skills, and partnerships we might require if we aim to incorporate big data into evaluation practice. This rich and well-informed conversation was catalyzed by our lead discussants: Andrew Means, Associate Director of the Center for Data Science & Public Policy at the University of Chicago and Founder of Data Analysts for Social Good and The Impact Lab; Michael Bamberger, Independent Evaluator and co-author of Real World Evaluation; and Veronica Olazabal from The Rockefeller Foundation. The Salon was supported by ITAD via a Rockefeller Foundation grant.

What do we mean by ‘big data’?

The first task was to come up with a general working definition of what was understood by ‘big data.’ Very few of the organizations present at the Salon were actually using ‘big data’ and definitions varied. Some talked about ‘big data sets’ as those that could not be collected or analyzed by a human on a standard computer. Others mentioned that big data could include ‘static’ data sets (like government census data – if digitized — or cellphone record data) and ‘dynamic’ data sets that are being constantly generated in real time (such as streaming data input from sensors or ‘cookies’ and ‘crumbs’ generated through use of the Internet and social media). Others considered big data to be real time, socially-created and socially-driven data that could be harvested without having to purposely collect it or budget for its collection. ‘It’s data that has a life of its own. Data that just exists out there.’ Yet others felt that for something to be ‘big data’ multiple big data sets needed to be involved, for example, genetic molecular data crossed with clinical trial data and other large data sets, regardless of static or dynamic nature. Big data, most agreed, is data that doesn’t easily fit on a laptop and that requires a specialized skill set that most social scientists don’t have. ‘What is big data? It’s hard to define exactly, but I know it when I see it,’ concluded one discussant.

Why is big data a ‘thing’?

As one discussant outlined, recent changes in technology have given rise to big data. Data collection, data storage and analytical power are becoming cheaper and cheaper. ‘We live digitally now and we produce data all the time. A UPS truck has anywhere from 50-75 sensors on it to do everything from optimize routes to indicate how often it visits a mechanic,’ he said. ‘The analytic and computational power in my iPhone is greater than what the space shuttle had.’ In addition, we have ‘seamless data collection’ in the case of Internet-enabled products and services, meaning that a person creates data as they access products or services, and this can then be monetized, which is how companies like Google make their money. ‘There is not someone sitting at Google going — OK, Joe just searched for the nearest pizza place, let me enter that data into the system — Joe is creating the data about his search while he is searching, and this data is a constant stream.’

What does big data mean for development evaluation?

Evaluators are normally tasked with making a judgment about the merit of something, usually for accountability, learning and/or to improve service delivery, and usually looking back at what has already happened. In the wider sense, the learning from evaluation contributes to program theory, needs assessment, and many other parts of the program cycle.

This approach differs in some key ways from big data work, because most of the new analytical methods used by data scientists are good at prediction but not very good at understanding causality, which is what social scientists (and evaluators) are most often interested in. ‘We don’t just look at giant data sets and find random correlations,’ however, explained one discussant. ‘That’s not practical at all. Rather, we start with a hypothesis and make a mental model of how different things might be working together. We create regression models and see which performs better. This helps us to know if we are building the right hypothesis. And then we chisel away at that hypothesis.’

Some challenges come up when we think about big data for development evaluation because the social sector lacks the resources of the private sector. In addition, data collection in the world of international development is not often seamless because ‘we care about people who do not live in the digital world,’ as one person put it. Populations we work with often do not leave a digital trail. Moreover, we only have complete data about the entire population in some cases (for example, when it comes to education in the US), meaning that development evaluators need to figure out how to deal with bias and sampling.

Satellite imagery can bring in some data that was unavailable in the past, and this is useful for climate and environmental work, but we still do not have a lot of big data for other types of programming, one person said. What’s more, wholly machine-based learning, and the kind of ‘deep learning’ made possible by today’s computational power is currently not very useful for development evaluation.

Evaluators often develop counterfactuals so that they can determine what would have happened without an intervention. They may use randomized controlled trials (RCTs), differentiation models, statistics and economics research approaches to do this. One area where data science may provide some support is in helping to answer questions about counterfactuals.

More access to big data (and open data) could also mean that development and humanitarian organizations stop duplicating data collection functions. Perhaps most interestingly, big data’s predictive capabilities could in the future be used in the planning phase to inform the kinds of programs that agencies run, where they should be run, and who should be let into them to achieve the greatest impact, said one discussant. Computer scientists and social scientists need to break down language barriers and come together more often so they can better learn from one another and determine where their approaches can overlap and be mutually supportive.

Are we all going to be using big data?

Not everyone needs to use big data. Not everyone has the capacity to use it, and it doesn’t exist for offline populations, so we need to be careful that we are not forcing it where it’s not the best approach. As one discussant emphasized, big data is not magic, and it’s not universally applicable. It’s good for some questions and not others, and it should be considered as another tool in the toolbox rather than the only tool. Big data can provide clues to what needs further examination using other methods, and thus most often it should be part of a mixed methods approach. Some participants felt that the discussion about big data was similar to the one 20 years ago on electronic medical records or to the debate in the evaluation community about quantitative versus qualitative methods.

What about groups of people who are digitally invisible?

There are serious limitations when it comes to the data we have access to in the poorest communities, where there are no tablets and fewer cellphones. We also need to be aware of ‘micro-exclusion’ (who within a community or household is left out of the digital revolution?) and intersectionality (how do different factors of exclusion combine to limit certain people’s digital access?) and consider how these affect the generation and interpretation of big data. There is also a question about the intensity of the digital footprint: How much data and at what frequency is it required for big data to be useful?

Some Salon participants felt that over time, everyone would have a digital presence and/or data trail, but others were skeptical. Some data scientists are experimenting with calibrating small amounts of data and comparing them to human-collected data in an attempt to make big data less biased, a discussant explained. Another person said that by digitizing and validating government data on thousands (in the case of India, millions) of villages, big data sets could be created for those that are not using mobiles or data.

Another person pointed out that generating digital data is a process that involves much more than simple access to technology. ‘Joining the digital discussion’ also requires access to networks, local language content, and all kinds of other precursors, she said. We also need to be very aware that these kinds of data collection processes impact on people’s participation and input into data collection and analysis. ‘There’s a difference between a collective evaluation activity where people are sitting around together discussing things and someone sitting in an office far from the community getting sound bites from a large source of data.’

Where is big data most applicable in evaluation?

One discussant laid out areas where big data would likely be the most applicable to development evaluation:

Screen Shot 2015-11-23 at 9.32.07 AM

It would appear that big data has huge potential in the evaluation of complex programs, he continued. ‘It’s fairly widely accepted that conventional designs don’t work well with multiple causality, multiple actors, multiple contextual variables, etc. People chug on valiantly, but it’s expected that you may get very misleading results. This is an interesting area because there are almost no evaluation designs for complexity, and big data might be a possibility here.’

In what scenarios might we use big data for development evaluation?

This discussant suggested that big data might be considered useful for evaluation in three areas:

  1. Supporting conventional evaluation design by adding new big data generated variables. For example, one could add transaction data from ATMs to conventional survey generated poverty indicators
  2. Increasing the power of a conventional evaluation design by using big data to strengthen the sample selection methodology. For example, satellite images were combined with data collected on the ground and propensity score matching was used to strengthen comparison group selection for an evaluation of the effects of interventions on protecting forest cover in Mexico.
  3. Replacing a conventional design with a big data analytics design by replacing regression based models with systems analysis. For example, one could use systems analysis to compare the effectiveness of 30 ongoing interventions that may reduce stunting in a sample of villages. Real-time observations could generate a time-series that could help to estimate the effectiveness of each intervention in different contexts.

It is important to remember construct validity too. ‘If big data is available, but it’s not quite answering the question that you want to ask, it might be easy to decide to do something with it, to run some correlations, and to think that maybe something will come out. But we should avoid this temptation,’ he cautioned. ‘We need to remember and respect construct validity and focus on measuring what we think we are measuring and what we want to measure, not get distracted by what a data set might offer us.’

What about bias in data sets?

We also need to be very aware that big data carries with it certain biases that need to be accounted for, commented several participants; notably, when working with low connectivity populations and geographies or when using data from social media sites that cater to a particular segment of the population. One discussant shared an example where Twitter was used to identify patterns in food poisoning, and suddenly the upscale, hipster restaurants in the city seemed to be the problem. Obviously these restaurants were not the sole source of the food poisoning, but rather there was a particular kind of person that tended to use Twitter.

‘People are often unclear about what’s magical and what’s really possible when it comes to big data. We want it to tell us impossible things and it can’t. We really need to engage human minds in this process; it’s not a question of everything being automated. We need to use our capacity for critical thinking and ask: Who’s creating the data? How’s it created? Where’s it coming from? Who might be left out? What could go wrong?’ emphasized one discussant. ‘Some of this information can come from the metadata, but that’s not always enough to make certain big data is a reliable source.’ Bias may also be introduced through the viewpoints and unconscious positions, values and frameworks of the data scientists themselves as they are developing algorithms and looking for/finding patterns in data.

What about the ethical and privacy implications?

Big Data has a great deal of ethical and privacy implications. Issues of consent and potential risk are critical considerations, especially when working with populations that are newly online and/or who may not have a good understanding of data privacy and how their data may be used by third parties who are collecting and/or selling it. However, one participant felt that a protectionist mentality is misguided. ‘We are pushing back and saying that social media and data tracking are bad. Instead, we should realize that having a digital life and being counted in the world is a right and it’s going to be inevitable in the future. We should be working with the people we serve to better understand digital privacy and help them to be more savvy digital citizens.’ It’s also imperative that aid and development agencies abandon our slow and antiquated data collection systems, she said, and to use the new digital tools that are available to us.

How can we be more responsible with the data we gather and use?

Development and humanitarian agencies do need be more responsible with data policies and practices, however. Big data approaches may contribute to negative data extraction tendencies if we mine data and deliver it to decision-makers far away from the source. It will be critical for evaluators and big data practitioners to find ways to engage people ‘on the ground’ and involve more communities in interpreting and querying their own big data. (For more on responsible data use, see the Responsible Development Data Book. Oxfam also has a responsible data policy that could serve as a reference. The author of this blog is working on a policy and practice guide for protecting girls digital safety, security and privacy as well.)

Who should be paying for big data sets to be made available?

One participant asked about costs and who should bear the expense of creating big data sets and/or opening them up to evaluators and/or data scientists. Others asked for examples of the private sector providing data to the social sector. This highlighted additional ethical and privacy issues. One participant gave an example from the healthcare space where there is lots of experience in accessing big data sets generated by government and the private sector. In this case, public and private data sets needed to be combined. There were strict requirements around anonymization and the effort ended up being very expensive, which made it difficult to build a business case for the work.

This can be a problem for the development sector, because it is difficult to generate resources for resolving social problems; there is normally only investment if there is some kind of commercial gain to be had. Some organizations are now hiring ‘data philanthropist’ positions that help to negotiate these kinds of data relationships with the private sector. (Global Pulse has developed a set of big data privacy principles to guide these cases.)

So, is big data going to replace evaluation or not?

In conclusion, big data will not eliminate the need for evaluation. Rather, it’s likely that it will be integrated as another source of information for strengthening conventional evaluation design. ‘Big Data and the underlying methods of data science are opening up new opportunities to answer old questions in new ways, and ask new kinds of questions. But that doesn’t mean that we should turn to big data and its methods for everything,’ said one discussant. ‘We need to get past a blind faith in big data and get more practical about what it is, how to use it, and where it adds value to evaluation processes,’ said another.

Thanks again to all who participated in the discussion! If you’d like to join (or read about) conversations like this one, visit Technology Salon. Salons run under Chatham House Rule, so no attribution has been made in this summary post.

Read Full Post »

The July 7th Technology Salon in New York City focused on the role of Information and Communication Technologies (ICTs) in Public Consultation. Our lead discussants were Tiago Peixoto, Team Lead, World Bank Digital Engagement Unit; Michele Brandt, Interpeace’s Director of Constitution-Making for Peace; and Ravi Karkara, Co-Chair, Policy Strategy Group, World We Want Post-2015 Consultation. Discussants covered the spectrum of local, national and global public consultation.

We started off by delving into the elements of a high-quality public consultation. Then we moved into whether, when, and how ICTs can help achieve those elements, and what the evidence base has to say about different approaches.

Elements and principles of high quality public participation

Our first discussant started by listing elements that need to be considered, whether a public consultation process is local, national or global, and regardless of whether it incorporates:

  • Sufficient planning
  • Realistic time frames
  • Education for citizens to participate in the process
  • Sufficient time and budget to gather views via different mechanisms
  • Interest in analyzing and considering the views
  • Provision of feedback about what is done with the consultation results

Principles underlying public consultation processes are that they should be:

  • Inclusive
  • Representative
  • Transparent
  • Accountable

Public consultation process should also be accompanied by widespread public education processes to ensure that people are prepared to a) provide their opinions and b) aware of the wider context in which the consultation takes place, she said. Tech and media can be helpful for spreading the news that the consultation is taking place, creating the narrative around it, and encouraging participation of groups who are traditional excluded, such as girls and women or certain political, ethnic, economic or religious groups, a Salon participant added.

Technology increases scale but limits opportunities for empathy, listening and learning

When thinking about integrating technologies into national public consultation processes, we need to ask ourselves why we want to encourage participation and consultation, what we want to achieve by it, and how we can best achieve it. It’s critical to set goals and purpose for a national consultation, rather than to conduct one just to tick a box, continued the discussant.

The pros and cons of incorporating technology into public consultations are contextual. Technology can be useful for bringing more views into the consultation process, however face-to-face consultation is critical for stimulating empathy in decision makers. When people in positions of power actually sit down and listen to their constituencies, it can send a very powerful message to people across the nation that their ideas and voices matter. National consultation also helps to build consensus and capacity to compromise. If done according to the above-mentioned principles, public consultation can legitimize national processes and improve buy-in. When leaders are open to listening, it also transforms them, she said.

At times, however, those with leadership or in positions of power do not believe that people can participate; they do not believe that the people have the capacity to have an opinion about a complicated political process, for example the creation of a new constitution. For this reason there is often resistance to national level consultations from multilateral or bilateral donors, politicians, the elites of a society, large or urban non-governmental organizations, and political leaders. Often when public consultation is suggested as part of a constitution making process, it is rejected because it can slow down the process. External donors may want a quick process for political reasons, and they may impose deadlines on national leaders that do not leave sufficient time for a quality consultation process.

Polls often end up being one-off snapshots or popularity contests

One method that is seen as a quick way to conduct a national consultation is polling. Yet, as Salon participants discussed, polls may end up being more like a popularity contest than a consultation process. Polls offer limited space for deeper dialogue or preparing those who have never been listened to before to make their voices heard. Polling may also raise expectations that whatever “wins” will be acted on, yet often there are various elements to consider when making decisions. So it’s important to manage expectations about what will be done with people’s responses and how much influence they will have on decision-making. Additionally, polls generally offers a snapshot of how people feel at a distinct point in time, but it may be important to understand what people are thinking at various moments throughout a longer-term national process, such as constitution making.

In addition to the above, opinion polls often reinforce the voices of those who have traditionally had a say, whereas those who have been suffering or marginalized for years, especially in conflict situations, may have a lot to say and a need to be listened to more deeply, explained the discussant. “We need to compress the vertical space between the elites and the grassroots, and to be sure we are not just giving people a one-time chance to participate. What we should be doing is helping to open space for dialogue that continues over time. This should be aimed at setting a precedent that citizen engagement is important and that it will continue even after a goal, such as constitution writing, is achieved,” said the discussant.

In the rush to use new technologies, often we forget about more traditional ones like radio, added one Salon participant, who shared an example of using radio and face to face meetings to consult with boys and girls on the Afghan constitution. Another participant suggested we broaden our concept of technology. “A plaza or a public park is actually a technology,” he noted, and these spaces can be conducive to dialogue and conversation. It was highlighted that processes of dialogue between a) national government and the international community and b) national government and citizens, normally happen in parallel and at odds with one another. “National consultations have historically been organized by a centralized unit, but now these kinds of conversations are happening all the time on various channels. How can those conversations be considered part of a national level consultation?” wondered one participant.

Aggregation vs deliberation

There is plenty of research on aggregation versus deliberation, our next discussant pointed out, and we know that the worst way to determine how many beans are in a jar is to deliberate. Aggregation (“crowd sourcing”) is a better way to find that answer. But for a trial, it’s not a good idea to have people vote on whether someone is guilty or not. “Between the jar and the jury trial, however,” he said, “we don’t know much about what kinds of policy issues lend themselves better to aggregation or to deliberation.”

For constitution making, deliberation is probably better, he said. But for budget allocation, it may be that aggregation is better. Research conducted across 132 countries indicated that “technology systematically privileges those who are better educated, male, and wealthier, even if you account for the technology access gaps.” This discussant mentioned that in participatory budgeting, people tend to just give up and let the educated “win” whereas maybe if it were done by a simple vote it would be more inclusive.

One Salon participated noted that it’s possible to combine deliberation and aggregation. “We normally only put things out for a vote after they’ve been identified through a deliberative process,” he said, “and we make sure that there is ongoing consultation.” Others lamented that decision makers often only want to see numbers – how many voted for what – and they do not accept more qualitative consultation results because they usually happen with fewer people participating. “Congress just wants to see numbers.”

Use of technology biases participation towards the elite

Some groups are using alternative methods for participatory democracy work, but the technology space has not thought much about this and relies on self-selection for the most part, said the discussant, and results end up being biased towards wealthier, urban, more educated males. Technology allows us to examine behaviors by looking at data that is registered in systems and to conduct experiments, however those doing these experiments need to be more responsible, and those who do not understand how to conduct research using technology need to be less empirical. “It’s a unique moment to build on what we’ve learned in the past 100 years about participation,” he said. Unfortunately, many working in the field of technology-enabled consultation have not done their research.

These biases towards wealthier, educated, urban males are very visible in Europe and North America, because there is so much connectivity, yet whether online or offline, less educated people participate less in the political process. In ‘developing’ countries, the poor usually participate more than the wealthy, however. So when you start using technology for consultation, you often twist that tendency and end up skewing participation toward the elite. This is seen even when there are efforts to proactively reach out to the poor.

Internal advocacy and an individual’s sense that he or she is capable of making a judgment or influencing an outcome is key for participation, and this is very related to education, time spent in school and access to cultural assets. With those who are traditionally marginalized, these internal assets are less developed and people are less confident. In order to increase participation in consultations, it’s critical to build these internal skills among more marginalized groups.

Combining online and offline public consultations

Our last discussant described how a global public consultation was conducted on a small budget for the Sustainable Development Goals, reaching an incredible 7.5 million people worldwide. Two clear goals of the consultation were that it be inclusive and non-discriminatory. In the end, 49% who voted identified as female, 50% as male and 1% as another gender. Though technology played a huge part in the process, the majority of people who voted used a paper ballot. Others participated using SMS, in locally-run community consultation processes, or via the website. Results from the voting were visualized on a data dashboard/data curation website so that it would be easier to analyze them, promote them, and encourage high-level decision makers to take them into account.

Some of the successful elements of this online/offline process included that transparency was a critical aspect. The consultation technology was created as open source so that those wishing to run their own consultations could open it, modify it, and repackage it however they wanted to suit their local context. Each local partner could manage their own URL and track their own work, and this was motivating to them.

Other key learning was that a conscious effort has to be made to bring in voices of minority groups; investment in training and capacity development was critical for those running local consultations; honesty and transparency about the process (in other words, careful management of expectations); and recognize that there will be highs and lows in the participation cycle (be sensitive to people’s own cycles and available time to participate).

The importance of accountability

Accountability was a key aspect for this process. Member states often did not have time to digest the results of the consultation, and those running it had to find ways to capture the results in short bursts and visually simple graphics so that the consultation results would be used for decision making. This required skill and capacity for not only gathering and generating data but also curating it for the decision-making audience.

It was also important to measure the impact of the consultation – were people’s voices included in the decision-making process and did it make a difference? And were those voices representative of a wide range of people? Was the process inclusive?

Going forward, in order to build on the consultation process and to support the principle of accountability, the initiative will shift focus to become a platform for public participation in monitoring and tracking the implementation of the Sustainable Development Goals.

Political will and responsiveness

A question came up about the interest of decision-makers in actually listening. “Leaders often are not at all interested in what people have to say. They are more concerned with holding onto their power, and if leaders have not agreed to a transparent and open process of consultation, it will not work. You can’t make them listen if they don’t want to. If there is no political will, then the whole consultation process will just be propaganda and window dressing,” one discussant commented. Another Salon participant what can be done to help politicians see the value of listening. “In the US, for example, we have lobbyists, issues groups, PACs, etc., so our politicians are being pushed on and demanded from all sides. If consultation is going to matter, you need to look at the whole system.” “How can we develop tools that can help governments sort through all these pressures and inputs to make good decisions?” wondered one participant.

Another person mentioned Rakesh Rajani’s work, noting that participation is mainly about power. If participation is not part of a wider system change, part of changing power structures, then using technology for participation is just a new tool to do the same old thing. If the process is not transparent and accountable, or if you engage and do not deliver anything based on the engagement, then you will lose future interest to engage.

Responsiveness was also raised. How many of these tech-fueled participation processes have led to governments actually changing, doing something different? One discussant said that evidence of impact of ICT-enabled participation processes was found in only 25 cases, and of those only 5 could show any kind of impact. All the others had very unclear impact – it was ambiguous. Did using ICTs make a difference? There was really no evidence of any. Another commented that clearly technology will only help if government is willing and able to receive consultation input and act on it. We need to find ways to help governments to do that, noted another person.

As always, conversation could have continued on for quite some time but our 2 hours was up. For more on ICTs and public consultations, here is a short list of resources that we compiled. Please add any others that would be useful! And as a little plug for a great read on technology and its potential in development and political work overall, I highly recommend checking out Geek Heresy: Rescuing Social Change from the Cult of Technology from Kentaro Toyama. Kentaro’s “Law of Amplification” is quite relevant in the space of technology-enabled participation, in that technology amplifies existing human behaviors and tendencies, and benefits those who are already primed to benefit while excluding those who have been traditionally excluded. Hopefully we’ll get Kentaro in for a Tech Salon in the Fall!

Thanks to our lead discussants, Michele, Tiago and Ravi, and to Thoughtworks for their generous hosting of the Salon! Salons are conducted under Chatham House Rule so no attribution has been made in this post. Sign up here if you’d like to receive Technology Salon invitations.

Read Full Post »

The private sector has been using dashboards for quite some time, but international development organizations face challenges when it comes to identifying the right data dashboards and accompanying systems for decision-making.

Our May 29th, 2015, Technology Salon (sponsored by The Rockefeller Foundation) explored data dashboards and data visualization for improved decision making with lead discussants John DeRiggi, Senior Data Architect, DAI; Shawna Hoffman, Associate Manager, Evaluation and Learning at The MasterCard Foundation; Stephanie Evergreen, Evergreen Data.

In short, we learned at the Salon that most organizations are struggling with the data dashboard process. There are a number of reasons that dashboards fail. They may never get off the ground, they may not deliver what was promised, they may deliver but no one uses them, or they may deliver but the data is poor and bad decisions are made. Using data for better decision-making is an ongoing process – not a task or product to complete and then relegate to automation. Just getting a dashboard up and running doesn’t guarantee that it’s a success – it’s critical to look deeper to see if the data and its visualization have actually improved decisions and how. Like with any ICT tool, user centered design and ongoing iteration are key. Successful dashboards are organized, useful, include targets, and have trends and predictions. Organizational culture and change management are critical in the process.

Points discussed in detail*:

1) Ask whether you actually need a dashboard

The first question to ask is whether a dashboard is needed or possible. One discussant, who specializes in data visualization, noted that she’s often brought in because someone wants to do data visualization, and she then needs to work backwards with the organization through a number of other preparatory steps before getting to the part on data visualization. It’s critical to have data dashboard discussions with different parts of the organization in order to understand real needs and expectations. Often people will say they need a dashboard because they want to make better decisions, noted another lead discussant. “But what kind of decisions, and what information is needed to make those decisions? Where does that information come from? Who will get it?”

2) Define the audience and type of dashboard

People often think that they can create one dashboard that will fulfill everyone’s needs. As one discussant put it, they will say the audience for the dashboard is “everyone – all decision makers at all levels!” In reality most organizations will need several dashboards for different levels of decision-making. It’s important to know who will own it, use it, keep it up, and collect the data. Will it be internal or externally facing? Discussing all of this is a key part of the process of thinking through the dashboard. As one discussant outlined, dashboards can be strategic, analytical or operational. But it’s difficult for them to be all three at once. So organizations need to come to a clear understanding of their data and decision-making needs. What information, if available, would help different teams at different levels with their decision making? One dashboard can’t be everything to everyone. Creating a charter that outlines what the dashboard project is and what it aims to do is a way to help avoid mission creep, said one discussant.

3) Work with users to develop your dashboard

To start off the process, it’s important to clearly identify the audience and find out what they need – don’t assume you know, recommended one discussant. But also, as a Salon participant pointed out, don’t assume that they know either. Have a conversation where their and your expertise comes together. “The higher up you go, the less people may understand about data. One idea is to just take the ‘data’ out of the conversation. Ask decision-makers what questions they are trying to answer, what problems they are trying to solve. Then find out how to collect and visualize the data that helps them answer their questions,” suggested another participant. Create ownership and accountability at all levels – with users, with staff who will input the data, with project managers, with grantees – you need cooperation from all levels noted others. Clear buy-in will also help with data quality. If people see the results of their data coming out in a data visualization, they may be more inclined to provide quality data. One way to involve users is to gather different teams to talk about their data and to create ‘entity relationship models’ together. “People can get into the weeds, and then you can build a vocabulary for the organization. Then you can use that model to build the system and create commonality across it,” said one discussant. Another idea is to create paper prototypes of dashboards with users so that they can envision them better.

4) Dashboards help people engage with the data they’ve collected

A dashboard is a window into your data, said one participant. In some cases, seeing their data visualized can help staff to see that they have been providing poor quality data. “People didn’t realize how bad their data was until they saw their dashboard,” said one discussant. Another noted that people may disagree with what the data tells them in the dashboard and feel motivated to provide better data. On the other hand, they may realize that their data was actually good, and instead they need to improve ineffective programs. A danger is that putting a dashboard on top of bad data shines a light on the data, said one participant, and this might create an incentive for people to manipulate their data.

5) Don’t be over-ambitious

Align the dashboard with indicators that link to strategic goals and directions and stay focused, recommended one discussant. There is often a temptation to over-complicate with tons of data and visuals. But extraneous data leads to misinterpretation or distraction. Dashboards should make complex data available in an accessible way to users, she said. You can always make more visuals if needed, but you want a concise story told in the data and visuals that you’re depicting. Determine what is useful, productive and credible and leave out what is exciting but extraneous. “Don’t try to have 30 indicators.”

6) Be clear about your data categories and indicators

Rolling up data from a large number of different programs into a dashboard is a huge challenge, especially if different sites or programs are using different data models. For example, if one program is describing an activity as a ‘workshop’ and the other uses ‘training session,’ said one discussant, you have a problem. A Salon participant explained that her organization started with shallow but important common denominators across programs. Over time they aim to go deeper to begin looking at outcomes and impact.

7) Think through how you’ll sustain the dashboard and related system(s)

One discussant said that her organization established three different teams to work on the dashboard process: a) Metrics – Where do we have credible representative data? Where do we have indicators but we don’t have data? b) Plumbing: Where are the data sources? How do they feed into each other? Who is responsible, and can this be aggregated up? And c) Visualization: What visual would help different decision makers make their decisions? Depending on where the organization is in its stage of readiness and its existing staff capacities, different combinations of skill sets may be required to supplement existing ones. Data experts can help teams understand what is possible, yet program or management teams and other dashboard users also need to be involved so that they can identify the questions they are trying to answer with the data and the dashboard.

8) Don’t underestimate the time/resources needed for a functional dashboard

People may not realize that you can’t make a dashboard without data to support it, noted one participant. “It’s like a power point presentation… a power point doesn’t just appear out of nowhere. It’s a result of conversations, research, data, design and more. But for some reason, people think a dashboard will just magically create itself out of thin air.” People also seem to think you can create and launch a dashboard and then put it on autopilot, but that is not the case. The dashboard will need constant changes and iteration, and there will be continual work to keep it up. The questions being asked will also likely change over time and so the dashboard may need to shift to take this into consideration. Time will be required to get buy-in for the dashboard and its use. One Salon participant said that in her former organization, they met quarterly to present, use and discuss the dashboard, and it took about 2 years in order for it to become useful and for people to become invested in it. It’s very important, said one participant, to ensure that management knows that the dashboard is not a static thing – it will need ongoing attention and management.

9) Be selective when it comes to the technology

People tend to think that dashboards are just visual, said a Salon participant. They think they are really cool, business solution platforms. Often senior leadership has seen been pitched something really expensive and complicated, with all kinds of bells and whistles, and they may think that is what they need. It’s important to know where your organization is in terms of capacity before determining which technology would be the best fit, however, noted one discussant. She counseled organizations to use whatever they have on hand rather than bringing in new software that takes people 6 months to learn how to use. Simple excel-based dashboards might be the best place to start, she said.

10) Legacy systems can be combined with new data viz capabilities

One discussant shared how his company’s information system, which was set up over 15 years ago, did not allow for the creation of APIs. This meant that the team could not build derivative software products from their massive existing database. It is too expensive to replace the entire system, and building modules to replace some of it would lead to fragmenting the user experience. So the team built a thin web service layer on top of the existing system. This exposed the data to friendly web formats from which developers could build interactive products.

11) Be realistic about “real time” and “data quality”

One question that came up was around the the level of evidence needed to make good decisions. Having perfect data served up into a perfect visualization is utopian, said one Salon participant. The idea is that we could have ‘real time’ data to inform our decisions, she explained, yet it’s hard to quality check data so quickly. “So at what level can we say we’ll make decisions based on a level of certainty – is it when we feel 80% of the data is good quality? Do we need to lower that to 60% so that we have timely data? Is that too low?” Another question was around the kinds of decisions that require ‘real time’ data versus those that could be made based on data that is 3 to 6 months old. Salon participants said this will depend on the kind of program and the type of decision. The sector in which one is working may also determine the level of comfort with real time and with data quality – for example, the humanitarian sector may need more timely data and accept a lower level of verification whereas the development sector may be the opposite.

Another point was that dashboards should include error bars and available metadata, as well as in some cases a link to raw data for those who want to dig into the data and understand what is behind the dashboard. Sometimes the dashboard process will highlight that there is simply not much quality data available for some programs in some countries. This can be an opportunity to work with staff on the ground to strengthen capacity to collect it.

12) Relax

As one discussant said, “much of the concern about data quality is related to our own hang-ups as data nerds and what we feel comfortable putting out there for people to use to make decisions. We always say ‘we need more research.’” But here the context is different. “Stakeholders and management want the answer. We need to just put the data out there with some caveats to help them.” One way to offer more context for a dashboard is creating a dashboard report that provides some narrative alongside the visualization. Dashboards should also show trends, not only what has happened already, she said. People need to see trends towards the future so that decisions can be made. It was also pointed out that a dashboard shouldn’t be the only basis for decisions. Like a car dashboard – these data dashboards signal that something is changing but you still need to look under the hood to see what it is. The dashboard should trigger questions – it should be a launch pad for discussion.

13) Organizational culture is a huge part of this process

The internal culture and people’s attitudes towards data are embedded into how an organization operates, noted one Salon participant. This varies depending on the type of organization – an evaluation focused organization vs. a development organization vs. a contractor vs. a humanitarian organization, for example. Outside consultants can help you to build a dashboard, but it will be critical to have someone managing organizational change on the inside who knows the current culture and where the organization is aiming to go with the dashboard process. The process is getting easier, however. Many organizations are thirsty for data now, noted one lead discussant. “Often the research or evaluation team create a dashboard and send it to the management team, and then everyone loves it and wants one. People are ready for it now.”

More resources on data dashboards and visualization.

Special thanks to our lead discussants and to our hosts for this Salon! If you’d like to join our Salon discussions in the future, sign up at the Technology Salon site.

*Salons run under Chatham House Rule, so no attribution has been made in this post.

Read Full Post »

by Hila Mehr and Linda Raftree

On March 31, 2015, nearly 40 participants, joined by lead discussants Robert Fabricant, Dalberg Design Team; Despina Papadopoulos, Principled Design; and Roop Pal, PicoSatellite eXploration Lab; came together for Technology Salon New York City where we discussed the future of wearables in international development. As follows is a summary of our discussion.

While the future of wearables is uncertain, major international development stakeholders are already incorporating wearables into their programs. UNICEF Kid Power is introducing wearables into the fight against malnutrition, and is launching a Global Wearables Challenge. The MUAC (mid-upper arm circumference) band already exists in international health. Other participants present were working on startups using wearables to tackle global health and climate change.

As Kentaro Toyama often says “technology is an amplifier of human intent” and the Tech Salon discussion certainly resonated with that sentiment. The future of wearables in international development is one that we–the stakeholders as consumers, makers, and planners–will create. It’s important to recognize the history of technology interventions in international development, and that while wearables enable a new future, technology interventions are not new; there is a documented history of failures and successes to learn from. Key takeaways from the Salon, described below, include reframing our concept of wearables, envisioning what’s possible, tackling behavior change, designing for context, and recognizing the tension between data and privacy.

Reframing our Concept of Wearables

Our first discussant shared historical and current examples of wearables, some from as far back as the middle ages, and encouraged participants to rethink the concept of wearables by moving beyond the Apple Watch and existing, primarily health-related, use cases. While Intel, Arm, and Apple want to put chips on and in our bodies, and we think these are the first cases of wearables, glasses have always been wearable, and watches are wearables that change our notions of time and space. In short, technology has always been wearable. If we stay focused on existing, primarily luxury, use cases like FitBit and Apple Watch, we lose our creativity in new use cases for varying scenarios, he said.

In many cases of technology introduction into a ‘developing world’ context, the technology adds a burden rather than contributing ease. We should be thinking about how wearables can capture data without requiring input, for example. There is also an intimacy with wearables that could eliminate or reframe some of the ingrained paradigms with existing technologies, he noted.

In the most common use cases of wearables and other technology in international development, data is gathered and sent up the chain. Participants should rethink this model and use of wearables and ensure that any data collected benefits people in the moment. This, said the discussant, can help justify the act of wearing something on the body. The information gathered must be better incorporated into a personal-level feedback loop. “The more intimate technology becomes, the greater responsibility you have for how you use it,” he concluded. 

In the discussion of reframing our notion of wearables, our second discussant offered a suggestion as to why people are so fascinated with wearables. “It’s about the human body connected to the human mind,” she explained. “What is it to be human? That’s why we’re so fascinated with wearables. They enlarge the notion of technology, and the relationship between machine, human, and animal.”

Envisioning What’s Possible

In discussing the prominent use of wearables for data collection, one participant asked, “What is possible to collect from the body? Are we tracking steps because that is what we want to track or because that is what’s possible? What are those indicators that we’ve chosen and why?”

We need to approach problems by thinking about both our priorities and what’s possible with wearable technology, was one reply. “As consumers, designers, and strategists, we need to push more on what we want to see happen. We have a 7-year window to create technology that we want to take root,” noted our lead discussant.

She then shared Google Glass as an example of makers forgetting what it is to be human. While Google Glass is a great use case for doctors in remote areas or operators of complex machinery, Google Glass at dinner parties and in other social interactions quickly became problematic, requiring Google to publish guidelines for social uses cases. “It’s great that it’s out there as a blatant failure to teach other designers to take care of this space,” she said. 

Another discussant felt that the greatest opportunity is the hybrid space between specialized and the generalized. The specialized use cases for wearables are with high medical value. And then there are the generalized cases. With expensive and new technology, it becomes cheaper and more accessible as it meets those hybrid use cases in-between specialized and generalized to justify the cost and sophistication of technology. Developing far out and futuristic ideas, such as one lead discussant’s idea for a mind-controlled satellite, can also offer opportunities for those working with and studying technology to unpack and ‘de-scaffold’ the layers between the wearable technology itself and the data and future it may bring with it.

Tackling Behavior Change

One of the common assumptions with wearables is that our brains work in a mechanical way, and that if we see a trend in our data, we will change our behavior. But wearables have proven that is not the case. 

The challenge with wearables in the international development context is making sure that the data collected serves a market and consumer need — what people want to know about themselves — and that wearables are not only focused on what development organizations and researchers want to know. Additionally, the data needs to be valuable and useful to individuals. For example, if a wearable tracks iron levels but the individual doesn’t understand the intricacies of nutrition, their fluctuations in iron levels will be of no use.

Nike Plus and its FuelBand has been one of the most successful activity trackers to date, argued one discussant, because of the online community created around the device. “It wasn’t the wearable device that created behavior change, but the community sharing that went with it.” One participant trained in behavioral economics noted the huge potential for academic research and behavioral economists with the data collected from wearables. A program she had worked on looked closely at test-taking behaviors of boys versus those of girls, and wearables were able to track and detect specific behaviors that were later analyzed and compared.

Designing for Context

Mainstream wearables are currently tailored for the consumer profile of the 35-year-old male fitness buff. But how do we think about the broader population, on the individual and community level? How might wearables serve the needs of those in emergency, low resource, or conflict settings? And what are some of the concerns with wearables?

One participant urged the group to think more creatively. “I’m having trouble envisioning this in the humanitarian space. 5-10 years out, what are concrete examples of someone in Mali, Chad, or Syria with a wearable. How is it valuable? And is there an opportunity to leapfrog with this technology?”

Humanitarian disaster contexts often face massive chaos, low literacy rates, and unreliable Internet connectivity, if Internet exists at all. How can wearables be useful in these cases? One participant suggested they could be used for better ways of coordinating and organizing — such as a warning siren signal wearable for individuals in warzones, or water delivery signal wearable for when water arrives — while keeping in mind real restrictions. For example, there are fears today about vaccines and other development agency interventions. This may escalate with wearable devices or edible tracking devices.

No amount of creativity, however, replaces the realistic and sustainable value of developing technology that addresses real needs in local contexts. That’s where human-centered design and participatory processes play a vital role. Wearable products cannot be built in isolation without users, as various participants highlighted.

As one lead discussant said, we too often look at technology as a magic bullet and we need to avoid doing this again when it comes to wearables. We can only know if wearable technology is an appropriate use case by analyzing the environment and understanding the human body. In Afghanistan, she noted, everyone has an iPhone now, and that’s powerful. But not everyone will have a FitBit, because there is no compelling use case.

Appropriate use cases can be discovered by involving the community of practice from day one, making no assumptions, and showing and sharing methodology and processes. Makers and planners should also be wary of importing resources and materials, creating an entire new ecosystem. If a foreign product breaks with no access to materials and training, it won’t be fixed or sustainable. Designing for context also means designing with local resources and tailored to what the community currently has access to. At the same time, international development efforts and wearable technology should be about empowering people, and not infantilizing them.

The value of interdisciplinary teams and systems maps cannot be overlooked, participants added. Wearables highlight our individual-centric nature, while systems thinking and mapping shows how we relate with ourselves, our community, and the world. Thinking about all of these levels will be important if wearables are to contribute to development in a positive way.

Tensions around Privacy, Data, and Unethical Uses

Wearables exist in tension with identity, intimacy, and privacy. As consumers, users, makers, and planners of wearables, we have to think critically and deeply about how we want our data to be shared. One discussant emphasized that we need to involve VCs, industry, and politicians in discussion around the ethical implications of wearable technology products. The political implications and erosion of trust may be even more complex in developing world contexts, making a consortia and standards even more necessary. 

One participant noted the risks of medical wearable technology and the lack of HIPAA privacy requirements in other countries. The lack of HIPAA should not mean that privacy concerns are glossed over. The ethics of testing apply no matter the environment, and testing completely inappropriate technology in a developing context just for the captive audience is ethically questionable.

Likewise, other participants raised the issue of wearables and other types of technology being used for torture, mind control and other nefarious purposes, especially as the science of ‘mind hacking’ and the development of wearables and devices inserted under the skin becomes more sophisticated.

Participants noted the value in projects like the EU’s Ethics Inside and the pressure for a UN Representative on privacy rights. But there is still much headway to be made as data privacy and ethical concerns only grow.

The Future We Wear

The rapid evolution of technology urges us to think about how technology affects our relationships with our body, family, community, and society. What do we want those relationships to look like in the future? We have an opportunity, as consumers, makers and planners of wearables for the international context to view ourselves as stakeholders in building the future opportunities of this space. Wearables today are where the Internet was during its first five mainstream years. Now is the perfect time to put our stake in the ground and create the future we wish to exist in.

***

Our Wearables and Development background reading list is available here. Please add articles or other relevant resources or links.

Other posts about the Salon, from Eugenia Lee and Hila Mehr.

Many thanks to our lead discussants and participants for joining us, and a special thank you to ThoughtWorks for hosting us and providing breakfast!

Technology Salons run under Chatham House Rule, therefore no attribution has been made in this summary post. If you’d like to join future Salons to discuss these and related issues at the intersection of technology and development, sign up at Technology Salon.

Read Full Post »

Our January 21st, 2015, Technology Salon examined the role of technology in addressing structural discrimination in the US. We were hosted by the Brooklyn Community Foundation (BCF), whose Director of Community Leadership, Tynesha McHarris, served as a lead discussant. Courtney D. Cogburn Assistant Professor of Social Work at Columbia University and Senior Advisor to the International Center for Advocates against Discrimination (ICAAD), joined as our second discussant.

Our discussion covered various angles, as summarized below, related to the central point of structural discrimination and racism.

Discrimination has multiple forms.

It can be observed at an individual level and in exchanges between people, but we shouldn’t reduce discrimination to a problem of an individual’s beliefs and behaviors, said our first lead discussant, Courtney. Rather we need to look at it as a complex whole that is embedded into structures and culture, and as a set of historic discriminatory patterns that produce systemic social disadvantage. “If we focus at the individual level, we will focus on individual accountability and individual solutions. Instead of looking at the systemic issues, we’ll encourage individuals to try harder, to be smarter, to stop being criminals. We’ll look at those who turn out OK as being exceptional,” she said.

“’I’m not racist’ doesn’t mean ‘I’m anti-racism.’ I don’t need you to like me, I need you to hate racism,” she continued. Interpersonal relationships won’t end structural and social disadvantage. This issue is also bigger than simple socio-economic status. So ending poverty won’t eradicate racism. In our society, there are two sides of the coin – discrimination or privilege, advantage or disadvantage. Our roles and our accountability here depend on how we came out in the coin toss.”

Empirical data on racism in the US are troubling.

As Courtney outlined:

  • Blacks and Hispanics are more likely to be rejected or receive worse terms on a loan — even when they have comparable financial backgrounds/risk to Whites
  • White men earn roughly 15% more than Black men with comparable education and experience
  • A White male with a felony conviction is more likely to be hired than a Black male with a clean record — all other factors being the same
  • Black women with a college degree have higher risk of premature/low birth weight than every other racial/ethnic group across levels of education
  • Even when accounting for various socio-economic status factors, including quality of health insurance: Blacks have an early onset, worse progression and premature death; they are more likely to be amputated and less likely to receive pain medication; and mothers exposed to discrimination while pregnant have higher stress hormones, which transfer over to their newborn children
  • Black men are 21 times more likely to be killed by a police officer. And Black men, our discussant argued, are also more likely to have their deaths mocked, for example, in the case of Trayvon Martin Halloween costumes and the Mike Brown musical.

In the US we fail to frame this as a human rights issue.

If we observed a similar situation happening elsewhere in the world, we would consider it a human rights issue, commented one participant. How can we change the debate in the US? Another felt we need to move the debate within the US from offering ‘charity’ to talking about ‘justice.’ “We need to talk about privilege and power, and philanthropy is about both – it’s older White men making decisions about money that impact the lives of people of color.”

Can technology help?

Social awareness and empathy are critical, as are data and equity-based policies, noted a discussant. Social awareness can help people see beyond their own realities. That might help more people to support equity-based policy. But we also need data in order to document discrimination. “If you don’t have an empirical base, people say ‘that’s just you, you’re playing the race card, Black people are just more violent.’ So we need data to show patterns of violence and discrimination in order to tackle this at the systemic level.” Newer technology can play a role with helping people empathize and with collecting data and visualizing patterns.

In the past several months, videos and hashtags have played a critical role in documenting racist incidents and engaging people and helping them to empathize, said a lead discussant. State violence and police violence are not new, but people are talking about them right now. “We shouldn’t lose sight of the fact that the #BlackLivesMatter movement was run by Black queer women. They are not being recognized for this, yet it’s been trending all over the world. At the same time, it makes me want to cry. I need a hashtag to tell the world that my life, the lives of Black people matter? Just having to state that is problematic.”

Amplifying stories, such as is happening in the ‘Serial’ podcast, which has caused a case to be re-opened, is another way that technology and media have addressed racism and built empathy through mainstream and social media. Gaming may also be a way to help people empathize, said another participant. Efforts are also being made to use virtual reality technology (oculus rift) to help people empathize with situations that they cannot personally experience. YouTube has been a powerful space for helping bring issues of structural racism and State sponsored violence against certain groups into the public eye both nationally and internationally.

“I’m from Cleveland,” noted another participant, “and I’ve been in touch with organizers working on the Tamir Rice case. Technology in that instance has been so helpful in organizing a disorganized city. We’re talking about Google groups and list serves being used in similar ways to how women used list serves and yahoo groups to organize post-Katrina relief. So technology can include even the bare minimum. Email is so powerful to organize mass amounts of people. The NY Justice League is also doing a lot of this kind of work.”

Tech can also help with transparency and credibility. “We found that putting data up on a website helped us to be credible and more transparent – it also gave us strength, backing and encouragement. We posted video narratives backed up with data, and this became part of our outreach and a focus for wider discussions about structural discrimination and issues like the criminal justice system and school suspensions,” commented one person.

Collecting and visualizing data is another arena where technology is useful in the case of structural discrimination, noted another participant, giving the example of hate crimes. “The New York Police Department only has reports of between 7 and 9 thousand hate crimes per year, yet the Department of Justice records up to 290k hate crimes. That is a data gap. We’ve argued that the absence of data is a structural issue. How can the State protect what it has no knowledge or record of? Tech and better data could fill this gap. Communities could report hate crimes through an app either directly to the police or to an NGO/civil society organization as a liaison if they felt uncomfortable reporting directly due to fear of the police.” Better tracking could help put more resources into preventing and following up on hate crimes.

We also need to be better at translating data for people and building empathy around it, said a discussant. “I think we have enough data. We know what’s going on. The issue is translating data across spaces. How can we create a collective body of data? And then there’s a big problem with framing of the issues. I can tell you all day long that Blacks are more likely to die early or be shot by the police, but if that doesn’t fit within your frame or align with your thinking, the data won’t matter.”

Obviously, tech can’t do it all!

“We can’t ask tech to do the work that our minds and hearts have to do,” said one person.

“In Ferguson, right after Mike Brown was murdered, I was close to the people running the Ferguson Action website. I watched a team of communications and web people trying to figure out how to talk about this with America. They were trying to make it compelling enough so that the world would respond to the death of a child, and that broke my heart a little.” People responded to Ferguson because of raw images of young people resisting that shocked us out of our senses, she added.

“We need organizers, lawyers, and communications strategists in this fight. We can’t expect for tech to take the place of work of courage, of seeing the world as it is, and knowing we have agency to change it. Without that website, without that hashtag, however, would the world have known what was happening?” she asked. “Technology and branding brought it to national and world attention.”

Despite that, however, young Black men are still getting shot. It’s still repeating itself. We need to open minds and hearts because there are fundamental beliefs that our media, our literature, our stories are cemented in. Changing minds and hearts is the work we still need to do, she commented.

Tech may divide communities and limit participation

As much as technology can help to support to organize people around human rights and social justice work, tech can also divide communities, noted one discussant. “In East New York, [during a consultation process in Brooklyn], we found that those who feel the most comfortable and confident using tech have been at an advantage when it comes to pushing for their opinions about what is happening in our communities. So they end up having a greater amount of participation in development processes in Brooklyn.” Often new ways of participating in these debates don’t take into consideration that many older people would prefer to find out about things through a flyer and to participate face to face or in community meetings. There are also people with disabilities and recent immigrants who tend to be left out as government starts to become e-government and consultations are more often conducted online.

We need to better understand technology used by and against our communities

“In doing research in Bed-Sty with young people of color on their social networks and tech use, I realized that I had a lot of assumptions about youth culture, mobile technology, and community relations, and they’ve all been challenged. As researchers, we need to continually ask ourselves how tech can help us document discrimination. We need to look at the hardware, the digital artifacts, the hashtags that help us to locate culture and document conversations,” said one person.

“These young people are on platforms I’ve never heard of,” she continued. “They are negotiating their identity spaces in ways I’ve never heard of before. I’m always asking ‘What are you doing? What does that mean?’ I’m asking about ethical and privacy issues and surveillance. I’m less interested in what they are doing when they are online or on their phones and more concerned about how their community is being watched from the outside and how that connects to what they do in their online spaces. There are lots of video cameras in their neighborhoods. They’ll ask me: will the police see my text messages? There is a concern and an acknowledgement of surveillance via social media and tech.” Another participant shared some detailed research on access to and use of technology among low income New Yorkers conducted by her organization.

What about teaching Black youth to code?

Some participants felt that efforts to teach Black and Latino you to code are critical to resolving structural discrimination. “One of the biggest economic justice issues in our country is young people of color not coding,” said one participant. “Coding lets you create things and you can generate income just by people downloading your app.” Currently the vast majority of coders are White and Asian males, and schools don’t tend to teach brown and black kids how to code, they said.

Others pushed back at this idea. “Young ppl I’m working with are not creating apps. They are trying to figure out how to use tech in their day-to-day lives – to create a resume, to find a job. We should be thinking about what are the possibilities and the challenges of having these devices in our communities. How do they constrain or offer support?” commented one person. Another noted “Access to tech is not as simple as ‘Do you have access to a computer and some will power.’ The problem is that systematically for the first 15 years of a young Black man’s life he gets the message that he is worthless. It’s not just parents, it’s teachers, it’s television, and it’s the structural issues. We need to also create environments that encourage Black youth.”

One discussant said that the default response is to point out individual success stories and to put effort into helping individuals who are disadvantaged, “but how about working more to shake things up at the top and hold the privileged and advantaged more accountable? Rather than ‘How do we help these poor kids,’ we should be asking, ‘how are we helping remedy this at its source?’”

These are social justice issues, commented another person. “So yes, teaching 10 year olds to code is great. But the problem is that this is a structural problem that hits 80 year olds, 60 year olds, 40 year olds, and 20 year olds. There is no quick fix. We need to continue to organize. And apps are not the real structural and systemic change that we need. We need to also talk about funding. There is no funding for a radically different way of thinking. It will take time. But how to put money behind big ideas?” she asked. “Can we seed and support more entrepreneurial work with youth in our communities? Is there a tech opportunity there? Can we use technology to link people to a tenant protection fund and to connect them with tenant rights information and eviction support? What would all that look like and how can we make it happen?”

The Brooklyn Insights report synthesizes input from individuals and community organizations across the boroughs and sets out priorities .

The Brooklyn Insights report synthesizes input from individuals and community organizations across the boroughs and sets out priorities.

We need more participatory design and cross-disciplinary teams

“Are the people we are trying to help at the table with us when we are designing?” asked a participant. “And can we do a better job of helping designers and coders to empathize with the people they are designing for? Can we get people out of their boxes and will they be willing to work in an environment where they feel consistently uncomfortable with their own privilege and power?”

“Are there places besides Tech Salon where these conversations are happening?” asked another person. Participants noted that there are hackathons but felt that normally they don’t lead to much in the way of real change. Doing a hackathon for and by a particular user community and tying developments into services that will still be there on Monday was one suggestion for remedying this gap.

Others asked how we could bring together multi-disciplinary teams that combine a deep knowledge of and experience in a community with social science, data science, law, and the social media capacity to help people empathize. A number of organizations present at the Salon are working with and/or conducting research in communities in Brooklyn or around issues of structural discrimination. Others specialize in creating technology for low-resource communities, and some are funding or otherwise supporting wider efforts to reduce structural discrimination. There was interest in continuing this discussion and addressing together some of the structural discrimination issues felt in Brooklyn communities and in the wider US.

Take a look at the BCF’s Insights report, which summarizes results from community consultations with over 1000 Brooklyn residents and numerous community organizations and offers a good overview of the core issues that Brooklyn communities are facing. And join us for Technology Salon Brooklyn, in collaboration with the Brooklyn Community Foundation, where we’ll meet (in Brooklyn) to discuss some of the issues raised and the role of technology in addressing or exacerbating them.

Thanks to ThoughtWorks for providing breakfast for us and to BCF for hosting! If you’d like to attend Technology Salon Brooklyn or Technology Salon New York City, sign up here to receive invitations!

Read Full Post »

« Newer Posts - Older Posts »