Tag: analytics

Digital Democracy and Cutting out the Middleman in Government

Can information technology in general and text analytics in particular help improve the quality of governance?

We believe they can.  In this article, we discuss one problem/weakness with the present system of governance that makes it very susceptible to corruption.  We then present a solution that relies on analytics to mitigate the problem.

Governance

Governance is a service.  An organization (government) provides people in a geographical area with a service called governance.  The organization that provides the service is for all practical purposes a service company owned by all the people to whom the service is provided.

Services provided by government include collecting money and using it to create infrastructure and services for the common good like roads and schools and city planning and waste disposal.

One weakness in the present approach is as follows.

The goals of the service provider may not always be well-aligned with the goals of the people being served.

When corruption exists, these goals may be very poorly aligned indeed.

Misalignment of Goals

Example 1:  Misalignment of Goals in Road Construction

For example, take the construction of a road.  To the people of the city who use roads, what they want in return for paying out money is better roads.  To the governing body who disburses the money, the goal – where corruption is rife – is high kickbacks.

Does Bangalore really not have enough money to build good roads?  It is very likely that our roads are bad not because we don’t have the money or the means to build roads that last, but because our governing body in charge of road repairs repeatedly doles out road maintenance contracts to people who do the road construction authorities favors in return for the contracts.

Example 2:  Misalignment of Goals in Allocating Budgets for Defence and Education

In an article on why India imports vast quantities of arms, we had described how the Indian government was under-spending on education and over-spending on defense procurement.

That article was based on a World Bank report http://www.imf.org/external/pubs/nft/2002/govern/index.htm that mentioned a study that showed that corrupt governments overspend on defence procurement because of the lack of transparency in such deals.

For example, in 2011 and 2012, India committed close to USD 50 billion to purchases of aircraft and ships alone whereas the expenditure towards education was around 12 billion per annum (woefully inadequate for our country).

Here again, we see a complete misalignment of goals.  People in India need education.  The government, however, when given a choice between putting our money into education or into arms, picks the choice that gives it a higher chance of receiving kickbacks.

Both are examples of something we call man-in-the-middle corruption.

One possible solution is to allow people to allocate portions of their income tax to categories of services that we expect our government to provide us.

Goal Alignment

For example, if I am paying Rs. 20,000 in income tax, I might quite reasonably be allowed to allocate say Rs. 10,000 of it to areas of infrastructure that I feel we need to invest in.  I might allocate of 5000 to education and 5000 to health services.  This would give people some measure of control over the use of our money by the governing body.

Moreover, it would give the governing body a deeper insight into the needs of the people, and also put some pressure on it to allocate all public funds according to a similar ratio.

For this to work, the allocation choices offered to people would have to be meaningful.  Meaningful choices may be determined by public discussion and/or referenda.

Any public discussion on the matter would require the use of debate support tools – text analytics tools that help large numbers of people communicate.

We’ve described one such tool that we call an MCT (Mass Communication Tool) in our lab profile.

In essence, what might be needed are text analytics technologies that can support legislation (proposing legislation, modifying legislation, or conducting a referendum on legislation).

Legislation

Much to the point, at this year’s Coling conference, we came across a paper by a student of the Singapore Management University (Swapna Gottipati), on how one might detect suggestions (thoughtful suggestions) in social media messages.  The paper was titled “Finding Thoughtful Comments from Social Media”.  Unfortunately the paper is not yet available online.

There have been attempts to allow people to propose legislation through online communities that don’t seem to work very well as the following article shows you: http://news.yahoo.com/interactive-white-house-secession-petitions-and-presidential-power-235012490.html

But a more successful attempt at using social media is described in this BBC article Why not let social media run the country?, and I quote: “But Nick Jones, deputy director of digital communications at Downing Street … points to the Red Tape Challenge, which has received more than 28,000 comments since it was launched by the prime minister last year and which has a ‘social media element’.  More than 150 pieces of legislation identified by the public as unnecessary have been so far been scrapped.”

I also really like Clay Shirky’s talk on how the internet will one day transform government.  He talks about how freedom of expression is promoted by social media.  What does freedom of speech do?  Well, it allows more ideas to circulate.  The more ideas there are in circulation, the better things (possibly governance) can become.

He talks about a need for an open-source model for generating agreement on ideas and proposes large scale discussion using something like the GIT version control.

He provides examples of legislation dumped on GITHub and his big takeaway seems to be the idea of collaboration without coordination.

He also talks about the need for openness working in two directions (about participatory legislation and not just legislation being visible to everyone), and about the invention of new methods of argument.  Very interesting.

Feedback

Another use of social media in governance is to collect feedback on government policies and decisions.  In that context I want to mention Project Dreamcatcher, an analytics project with a social media component that was used by the Obama campaign in 2012.  Here is an article on Project Dreamcatcher.  It seems to be an extension of feedback monitoring which has been used for customer service.

Summary

There seem to be new possibilities opening up for the use of technology, possibly text analytics technology, in governance.

How far is 5 from 4? Does art take you from 4 to 5? Does MAGIC happen between 4 and 5?

I was intrigued, not long ago, to see a tweet by Dr. Prabhakar Raghavan that went “Berkeley Study: Half-Star Change In Yelp Rating Can Make Or Break A Restaurant http://tcrn.ch/RAy1iM  nonlinear outcomes from linear(?) inputs“.

Anderson and Magruder at Berkeley observed Yelp ratings and tried to see if they impacted the likelihood of a restaurant being fully booked at peak dinner time (between the hours of 6 and 8 pm).

What they found was that:

An extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently, with larger impacts when alternate information is more scarce

Moving from 3 to 3.5 stars reduces the likelihood of availability from about 90% to 70%. A fourth star reduces the likelihood of  availability further to 45%, and that possibility drops to 20% at 4.5 stars

So, the answer to the first question seems to be “A lot farther than you’d think” or “About as far as 4 is from 1” !

Now I am going to show you something else that is equally amazing!

The above economics result is very similar to what is being talked about in this XKCD cartoon on online star ratings!

Take a close look at the cartoon and you will see that the cartoonist is saying much the same thing as the economics researchers.

How did the artist hit it right on the head?

Just how did he do it?  (maybe the cartoonist read the research paper – the guy’s a scientist so it could have happened).

But could it just be that good artists are also incredibly perceptive and intelligent people?

Does art take you from 4 to 5?

I don’t have the answer to that, but I am going to point you to an intriguing paragraph in the book “Transforming Education through the Arts”.

“We found that compared with typical scientists, Nobel laureates are at least 2 times more likely to be photographers; 4 times more likely to be musicians; 17 times more likely to be artists; 15 times more likely to be craftsmen; 25 times more likely to be writers of non-professional writing, such as poetry or fiction; and 22 times more likely to be performers, such as actors, dancers or magicians.”

(From Page 7 of “Transforming Education through the Arts” in the article “The social and educational costs of neglecting the arts”, citing as the source: “Root-Bernstein and Root-Bernstein 2010: 4”).

So, maybe the answer to the second question is ‘maybe‘.

I say ‘maybe’ because there is another explanation.

In software, we have a saying that goes:

80 percent of the features take 20 percent of the time.  The last 20% of the features take 80% of the time.

However, this 80/20 rule seems to apply to many things in life.

You can be a better singer than 80% of the world quite easily.  It might even be possible to become a better singer than 97% of the world with a bit more work.  But it takes infinitely more effort to get from 97% to 99%.  To reach those levels of perfection takes enormous dedication and persistence.

This is the case in every art, and in every craft, and in every area of science, and in every area of technology.

Products become magical when you put in that enormous effort that takes it from 97% to 99% of perfection.

Dancers become magical when they go from 97% to 99%.

We all heard the story of Steve Jobs obsessing over the details of the iPhone even during sessions of chemotherapy.

Cartoonists become magical when …

You get the idea.

So maybe it’s not about art …

Or about science …

Or about technology …

Maybe it’s about having the strength, resilience and stamina to get you to the very top of the mountain and the obsession to keep going …

Maybe that’s what makes all the difference in any area of human endeavour – the intense effort and drive that only very obsessed people are capable of putting in is the magic (or madness) that takes something from 4 to 5.

Maybe that’s why 5 is so far from 4.

Measuring the efficiency of retail and the possible implications

I read a beautiful BBC article today titled “How much will the technology boom change Kenya?

It is about how information delivered over mobile phones can improve people’s lives.  Here is an example:

“Ms Oguya, 25, is the creator of a mobile phone app called M-Farm, designed to help small-scale farmers maximise their potential.

Ms Oguya herself grew up on a farm. She realised people like her parents had two main problems.

Firstly, they did not always know the up-to-date market price for a particular crop.

Unscrupulous middlemen would take advantage of that and persuade farmers to part with their produce at lower prices.

Using M-Farm, a farmer can now find out the latest prices with a single text message.”

This reminded me first of all about the price of pomegranates in Bangalore.  A kilo of pomegranates costs Rs. 120 in Bangalore city.  The price at which traders buy pomegranates from farmers is Rs. 12 per kilo.  In other words, what the producer gets from the consumer is a mere 1/10 of the price a consumer pays for the product.

A year ago. I had written an article on an anti-corruption blog titled “Why Walmart’s measure of efficiency might be flawed“.

I quote from it:

“Walmart’s definition of efficiency is the cost of a product. They say they increase the efficiency of the entire market by lowering the cost of products (by getting an item manufactured in a low-cost geography).”

“However, in my opinion, there is a better way to measure efficiency, and I feel that retailers highlight lower prices as a measure of efficiency only because it is the only measure possible under the current conditions of lack of transparency in retail.”

“Take for example a $10 product that used to cost $5 to manufacture in the USA (potential profit margin of $5). Now, Walmart has a profit motive to move its manufacture to a geography where it costs less than $1 to manufacture if they can still charge $8 for it. Say, the transportation cost is $1. If the product sells for $8, they still have a profit margin of $6 which is higher than $5.”

“However, if I measure efficiency as the percentage of money paid by a customer that is being delivered to the manufacturer, there has actually been a drop in efficiency. The percentage of the price that went to the manufacturer dropped from 50% to only 12.5% (one dollar out of eight).”

“You would never donate your money to a charity without first asking what percentage of your donation was reaching the beneficiary, would you?”

Now let’s do the math for pomegranates.

By using the ratio of purchase price to sale price as a measure of efficiency, we find that the efficiency of the retail mechanism for pomegranates is a mere 10%.

If farmers became better informed, they might work to discover ways to improve the efficiency – either by bypassing middlemen – I remember the farmer’s market in Raleigh, NC, an amazingly simple idea, that did just that – or by negotiating better prices for themselves.

End consumers might also choose to patronize outlets that pay fairer prices to the producers if they could find out how much was really paid to the manufacturer.

That might also help bring back manufacturing to the USA.