Tag: social media

Digital Democracy and Cutting out the Middleman in Government

Can information technology in general and text analytics in particular help improve the quality of governance?

We believe they can.  In this article, we discuss one problem/weakness with the present system of governance that makes it very susceptible to corruption.  We then present a solution that relies on analytics to mitigate the problem.


Governance is a service.  An organization (government) provides people in a geographical area with a service called governance.  The organization that provides the service is for all practical purposes a service company owned by all the people to whom the service is provided.

Services provided by government include collecting money and using it to create infrastructure and services for the common good like roads and schools and city planning and waste disposal.

One weakness in the present approach is as follows.

The goals of the service provider may not always be well-aligned with the goals of the people being served.

When corruption exists, these goals may be very poorly aligned indeed.

Misalignment of Goals

Example 1:  Misalignment of Goals in Road Construction

For example, take the construction of a road.  To the people of the city who use roads, what they want in return for paying out money is better roads.  To the governing body who disburses the money, the goal – where corruption is rife – is high kickbacks.

Does Bangalore really not have enough money to build good roads?  It is very likely that our roads are bad not because we don’t have the money or the means to build roads that last, but because our governing body in charge of road repairs repeatedly doles out road maintenance contracts to people who do the road construction authorities favors in return for the contracts.

Example 2:  Misalignment of Goals in Allocating Budgets for Defence and Education

In an article on why India imports vast quantities of arms, we had described how the Indian government was under-spending on education and over-spending on defense procurement.

That article was based on a World Bank report http://www.imf.org/external/pubs/nft/2002/govern/index.htm that mentioned a study that showed that corrupt governments overspend on defence procurement because of the lack of transparency in such deals.

For example, in 2011 and 2012, India committed close to USD 50 billion to purchases of aircraft and ships alone whereas the expenditure towards education was around 12 billion per annum (woefully inadequate for our country).

Here again, we see a complete misalignment of goals.  People in India need education.  The government, however, when given a choice between putting our money into education or into arms, picks the choice that gives it a higher chance of receiving kickbacks.

Both are examples of something we call man-in-the-middle corruption.

One possible solution is to allow people to allocate portions of their income tax to categories of services that we expect our government to provide us.

Goal Alignment

For example, if I am paying Rs. 20,000 in income tax, I might quite reasonably be allowed to allocate say Rs. 10,000 of it to areas of infrastructure that I feel we need to invest in.  I might allocate of 5000 to education and 5000 to health services.  This would give people some measure of control over the use of our money by the governing body.

Moreover, it would give the governing body a deeper insight into the needs of the people, and also put some pressure on it to allocate all public funds according to a similar ratio.

For this to work, the allocation choices offered to people would have to be meaningful.  Meaningful choices may be determined by public discussion and/or referenda.

Any public discussion on the matter would require the use of debate support tools – text analytics tools that help large numbers of people communicate.

We’ve described one such tool that we call an MCT (Mass Communication Tool) in our lab profile.

In essence, what might be needed are text analytics technologies that can support legislation (proposing legislation, modifying legislation, or conducting a referendum on legislation).


Much to the point, at this year’s Coling conference, we came across a paper by a student of the Singapore Management University (Swapna Gottipati), on how one might detect suggestions (thoughtful suggestions) in social media messages.  The paper was titled “Finding Thoughtful Comments from Social Media”.  Unfortunately the paper is not yet available online.

There have been attempts to allow people to propose legislation through online communities that don’t seem to work very well as the following article shows you: http://news.yahoo.com/interactive-white-house-secession-petitions-and-presidential-power-235012490.html

But a more successful attempt at using social media is described in this BBC article Why not let social media run the country?, and I quote: “But Nick Jones, deputy director of digital communications at Downing Street … points to the Red Tape Challenge, which has received more than 28,000 comments since it was launched by the prime minister last year and which has a ‘social media element’.  More than 150 pieces of legislation identified by the public as unnecessary have been so far been scrapped.”

I also really like Clay Shirky’s talk on how the internet will one day transform government.  He talks about how freedom of expression is promoted by social media.  What does freedom of speech do?  Well, it allows more ideas to circulate.  The more ideas there are in circulation, the better things (possibly governance) can become.

He talks about a need for an open-source model for generating agreement on ideas and proposes large scale discussion using something like the GIT version control.

He provides examples of legislation dumped on GITHub and his big takeaway seems to be the idea of collaboration without coordination.

He also talks about the need for openness working in two directions (about participatory legislation and not just legislation being visible to everyone), and about the invention of new methods of argument.  Very interesting.


Another use of social media in governance is to collect feedback on government policies and decisions.  In that context I want to mention Project Dreamcatcher, an analytics project with a social media component that was used by the Obama campaign in 2012.  Here is an article on Project Dreamcatcher.  It seems to be an extension of feedback monitoring which has been used for customer service.


There seem to be new possibilities opening up for the use of technology, possibly text analytics technology, in governance.

How far is 5 from 4? Does art take you from 4 to 5? Does MAGIC happen between 4 and 5?

I was intrigued, not long ago, to see a tweet by Dr. Prabhakar Raghavan that went “Berkeley Study: Half-Star Change In Yelp Rating Can Make Or Break A Restaurant http://tcrn.ch/RAy1iM  nonlinear outcomes from linear(?) inputs“.

Anderson and Magruder at Berkeley observed Yelp ratings and tried to see if they impacted the likelihood of a restaurant being fully booked at peak dinner time (between the hours of 6 and 8 pm).

What they found was that:

An extra half-star rating causes restaurants to sell out 19 percentage points (49%) more frequently, with larger impacts when alternate information is more scarce

Moving from 3 to 3.5 stars reduces the likelihood of availability from about 90% to 70%. A fourth star reduces the likelihood of  availability further to 45%, and that possibility drops to 20% at 4.5 stars

So, the answer to the first question seems to be “A lot farther than you’d think” or “About as far as 4 is from 1” !

Now I am going to show you something else that is equally amazing!

The above economics result is very similar to what is being talked about in this XKCD cartoon on online star ratings!

Take a close look at the cartoon and you will see that the cartoonist is saying much the same thing as the economics researchers.

How did the artist hit it right on the head?

Just how did he do it?  (maybe the cartoonist read the research paper – the guy’s a scientist so it could have happened).

But could it just be that good artists are also incredibly perceptive and intelligent people?

Does art take you from 4 to 5?

I don’t have the answer to that, but I am going to point you to an intriguing paragraph in the book “Transforming Education through the Arts”.

“We found that compared with typical scientists, Nobel laureates are at least 2 times more likely to be photographers; 4 times more likely to be musicians; 17 times more likely to be artists; 15 times more likely to be craftsmen; 25 times more likely to be writers of non-professional writing, such as poetry or fiction; and 22 times more likely to be performers, such as actors, dancers or magicians.”

(From Page 7 of “Transforming Education through the Arts” in the article “The social and educational costs of neglecting the arts”, citing as the source: “Root-Bernstein and Root-Bernstein 2010: 4”).

So, maybe the answer to the second question is ‘maybe‘.

I say ‘maybe’ because there is another explanation.

In software, we have a saying that goes:

80 percent of the features take 20 percent of the time.  The last 20% of the features take 80% of the time.

However, this 80/20 rule seems to apply to many things in life.

You can be a better singer than 80% of the world quite easily.  It might even be possible to become a better singer than 97% of the world with a bit more work.  But it takes infinitely more effort to get from 97% to 99%.  To reach those levels of perfection takes enormous dedication and persistence.

This is the case in every art, and in every craft, and in every area of science, and in every area of technology.

Products become magical when you put in that enormous effort that takes it from 97% to 99% of perfection.

Dancers become magical when they go from 97% to 99%.

We all heard the story of Steve Jobs obsessing over the details of the iPhone even during sessions of chemotherapy.

Cartoonists become magical when …

You get the idea.

So maybe it’s not about art …

Or about science …

Or about technology …

Maybe it’s about having the strength, resilience and stamina to get you to the very top of the mountain and the obsession to keep going …

Maybe that’s what makes all the difference in any area of human endeavour – the intense effort and drive that only very obsessed people are capable of putting in is the magic (or madness) that takes something from 4 to 5.

Maybe that’s why 5 is so far from 4.

Intentions and Opinions

In the last blog post, I talked about intention analysis and what it does.

Intention analysis is the identification of intentions from text. Some examples of intentions are:

a) intention to complain
b) intention to inquire
c) intention to issue a directive
d) intention to buy

In this post, I am claiming that Sentiment Analysis needs Intention Analysis.

Yes, the results of sentiment analysis will be inaccurate unless you know that the intent of the speaker is to express an opinion.


When sentiment analysis was initially proposed by researchers, they applied it to the analysis of product reviews.

The intention of a reviewer is obvious. Reviewers have only one intention: the intention to opine (either to praise or to criticize).

However, with the growth of social media, especially Twitter, the same sentiment analysis methods began to be applied to the analysis of twitter streams and other social media streams.

Now that’s where there is a problem.

Not every message on Twitter that mentions a particular product or brand intends to express an opinion about the brand!

Below are a few illustrative examples.

Example 1: “Is the Canon EOS 5 a good camera?”

This sentence is not an expression of positive opinion, but an inquiry about the Canon EOS 5.

In other words, the intent of the speaker is not to express an opinion, but to inquire.

Example 2: “I am looking to buy me a good Canon camera”

Here, the intention of the user is to purchase a product (people only indicate a preference for good things … no one really looks to buy a bad camera).

However, most sentiment analysis tools will identify this sentence as an expression of positive sentiment.

Example 3: “Take me to a good movie.”

Here, the speaker’s intent is to direct someone to do something.

A directive is not an assertion, and so does not always imply an intention to opine.

Example 4: “My good old Porsche for sale (cheap)”

Here, the speaker’s intent is to talk up something they’re selling.

The intent here is not to express sentiment about the brand.


So, what we can learn from the above examples is that sentiment analysis is not meant to be applied without reservations to Social Media Analysis.

In other words, for sentiment analysis to be accurate when applied to social media, it needs to be supported by intention analysis.


We recently released a sentiment analysis API that has the ability to filter out many kinds of intention including the ones listed above. We’d love to get your thoughts on our work. The demo is available at the following URL:

Demonstration of VakSent (a Sentiment Analysis API from Aiaioo Labs)

Do write me at cohan@aiaioo.com with intent to opine!