Month: January 2014

Text Analytics Tools for Deliberative Democracy

In our last post, we spoke about various control mechanisms that can be implemented to support direct democracy (which we  interpreted to mean the control of the allocation of common resources by the people who pooled in).

We also examined how these controls could be used to curtail man-in-the-middle corruption.

In this article, we examine a more sophisticated form of direct democracy called a deliberative democracy.

In a deliberative democracy, in addition to the control mechanisms prescribed for direct democracy, there need to be mechanisms to allow deliberation (discussion) before a referendum or any other action is taken.

I quote from the Wikipedia article on deliberative democracy:

Deliberative democracy holds that, for a democratic decision to be legitimate, it must be preceded by authentic deliberation, not merely the aggregation of preferences that occurs in voting.

In elitist deliberative democracy, principles of deliberative democracy apply to elite societal decision-making bodies, such as legislatures and courts; in populist deliberative democracy, principles of deliberative democracy apply to groups of lay citizens who are empowered to make decisions.

The article on direct democracy had the following to say:

Democratic theorists have identified a trilemma due to the presence of three desirable characteristics of an ideal system of direct democracy, which are challenging to deliver all at once. These three characteristics are participation – widespread participation in the decision making process by the people affected; deliberation – a rational discussion where all major points of view are weighted according to evidence; and equality – all members of the population on whose behalf decisions are taken have an equal chance of having their views taken into account.

(Aside to computer scientists: doesn’t this trilemma remind you of the CAP theorem that applies to database systems? Here’s a simple explanation of the CAP theorem:

So, for example, representative democracy satisfies the requirement for deliberation and equality but sacrifices participation.

Participatory democracy allows inclusive participation and deliberation but sacrifices equality.

And then there is direct democracy which supports participation and equality, but not deliberation.

The problem seems to be that when a large number of people are invited to participate in a deliberation (and given that deliberations take time), it will not be possible to compensate them all for their time. Consequently, only those more interested in the issue being debated (or more likely to benefit from one position or the other) are more likely to participate, biasing the sample in their favour (all sections of the population are no longer equally represented in the discussion/decision).

So, it seems that all the three properties desired in an ideal democratic system – participation, equality and deliberation – cannot be present at the same time in a real democratic system.

But then, a while ago, we began wondering if this trilemma is merely a result of the lack of suitable technology and not really a fundamental property of democracy.  So, we proposed a design for (though we have not yet realized it) a tool that can support the participation of a large number of people in deliberations.  We call it the MCT (Mass Communication Tool).

It could be used as a method to enable direct democracies to support deliberations in which all citizens can participate, ahead of a vote on any subject.

It uses text clustering algorithms to solve the problems of volume as well as numeric asymmetry in the flow of communications between the deliberating participants and the moderators of the communications.

There’s a brief overview of the system in our lab profile.

MCTs are bound to have a huge impact on our experience of representative government.  A typical use case would involve a public figure, (say President Obama), sounding out the electorate before introducing legislation on say healthcare reform.


By first discussing the competing proposals with large numbers of people, it might be possible for the initiator of the discussion to get a sense of what might or might not work and what the response to the legislation was likely to be.


An MCT would have to be capable of supporting a live dialog involving a large number of people.

It would use natural language processing and machine learning to enable a few moderators (for example, the CEO of a company) to interact with a large number of people (for example, all the employees of the company) in real time (for example, during a virtual all-hands meeting), get a synopsis of a large number of concurrent discussions in real time, and participate in a significant fraction of the discussions as they are taking place.

The system would consist of:

  1. an aggregator of messages (built from natural language processing components) that groups together messages and discussions with identical semantic content;
  2. a hierarchical clustering system (built from natural language processing components) that assigns aggregated messages their place in a hierarchy by specificity with more general messages closer to the root of the hierarchy and more specific messages closer to the leaves of the hierarchy;
  3. a summarization system (built from natural language processing components) that creates a summary of the aggregate of all messages in a sub-tree; and
  4. a reply routing system (built from natural language processing components) that routes replies from cluster to cluster based on their relevance to the discussion threads.

Direct Democracy and Implications for Research

Direct democracy can be broadly interpreted to mean the control of the allocation of common resources by the people who pooled in.

One common resource is tax money.

In most countries, those who pay taxes only have a say in whom they can elect to power.

Those who pay taxes rarely have a say in how the tax money is spent.

There is a middleman (someone who works in government) who decides how the tax money is spent.

The problem with having a middleman decide the allocation of common resources, is that the resources could end up being allocated very inefficiently due to man-in-the-middle corruption.

Here is an article about man-in-the-middle corruption:

The way out is to let the people who contributed to the common pool decide on how the resources are allocated.

Control Mechanisms

Tool 1: Apportioning

One way to do this is to embed direct democracy mechanisms into the contribution mechanism.

For example, tax-payers could be given the ability to tie a portion of their tax contribution to expenditure categories.

They could be given the right to apportion, out of every $100 that they have paid in taxes, a certain amount to each of the following major categories: education, healthcare, social security, infrastructure and defence (leaving a certain percentage to the finance minister’s discretion).

It could also be left to the tax-payers to specify how much money the government may borrow on their behalf.

This direct control could very likely have prevented the debt crises of Greece and Ireland (and especially in countries where people are averse to taking on debt), and might also have given people in the USA some control over their government’s borrowings.

Tool 2: Referendum

Another mechanism is the referendum.  It is already being used in all democratic countries, but mainly for the selection of the middleman.

Fortunately, things seem to be moving well beyond that stage.  In India, baby steps are being taken towards bringing about a direct democratic model of government.

A new political party came to power in Delhi on an anti-corruption platform.  The first thing they did was conduct an informal referendum to ask the people of Delhi if they should form a minority government.

So, referendums are one of the mechanisms of direct democracy.

This mechanism can also be used to prevent or reduce man-in-the-middle corruption.

Take the example of a road that needs to be surfaced.  Normally, a government official would have issued contracts based on the bribes paid to him by the contending contractors (rendering a selection on the basis of quality very unlikely).

If instead, the people living on the street that needed to be surfaced had been given all the relevant information needed to make a good choice and asked to select the best contractor for themselves instead, the middle-man would have been eliminated and the quality driven up instead of down.

Now, I am going to talk about some problems that I think affect research funding in the USA (and other countries with government-funded research).  Most research funding in the USA comes from government bodies like the NSF, the NRO and DARPA.

Now the following is only my personal opinion, but I think that research efforts in some fields in those countries might be distorted to some extent by the needs of these funding bodies.

Well, I can only speak for the research areas in which Aiaioo Labs is active.  We focus on a narrow research space – predominantly on text analytics and natural language processing.

In this space, I see a lot of low-hanging fruit that nobody in the USA or Europe ever picks.  And that’s quite inexplicable as these are often problems with very obvious applications to the software products space.  And yet I see no papers from California on them.

There are topics on which all the papers I see are from India (often by students who don’t even publish them in international conferences) and sometimes by researchers from Singapore – completely ignored by the main research community.

At other times, I’ve noticed areas of research that DARPA had spent much money on in the 1970s and that researchers had pursued very enthusiastically in that decade.  I’ve seen those lines of research being abandoned in the 90s (possibly once the funding priorities changed) and not being revived again, though product firms are working on those technologies again in California in 2013.

I find it hard to explain why these areas of research are being ignored, except by the remote possibility that they have passed under the radar of the guys in the Naval Research Office which makes it unlikely that grants will be provided for them.

That is again, if my conjecture is right, a man-in-the-middle problem.  The agenda for research is possibly not being driven by the research community or by the market (the needs of start-ups in California) but by people guessing at what sort of proposals might get funded (and that might be encouraging people to stay with what government knows).

So, I shall propose another direct democracy tool to solve this problem as well:

Tool 3:  Suggestions + Referendum

Here, each of the participants (researchers) bidding for the grants would put in suggestions about what the next important thing to focus on as a research community might be.  Then they could all vote on the suggestions.  The allocation of research grants could then be guided by the suggestions and the votes received by each suggestion.

Controls as Rights

In a sense, you can think of these three control mechanisms as three rights that people who contribute toward a common pool of resources will have in a direct democracy:

1)  The right to apportion

2)  The right to be consulted

3)  The right to suggest

There is a nice article on Wikipedia on direct democracy.  The article talks of two of the control mechanisms proposed in this article – referendum and initiative (which corresponds somewhat to suggestions) – and proposes one that I hadn’t mentioned – the right to recall.  It doesn’t talk about apportioning.

Here is an interesting video of a Mohalla Sabha (it’s an interesting participation mechanism that a political organization is experimenting with in Delhi).

Two Types of Corruption

There are really two types of corruption.

Type 1:  Speed-Money Corruption

In India, bureaucrats sometimes deliberately delay the processing of applications in the hope of getting a bribe.  A citizen who needs, say a water connection, has to pay a bribe to a clerk to get their application looked at.  If you are the victim of a crime, and go to the local police station, the policeman will expect a small bribe to register your complaint.

This seems like a fairly harmless form of corruption.  Some people defend it as necessary in a free market, as a mechanism for the differentiation of services.  It actually seems very like a tip.  But in this case, you have to pay before you get the service.

But if you take a closer look at this form of bribe and think about the economics of it, you see that what is being demonstrated is a form of rent-seeking behaviour.  And you will see that it has the harmful side-effect of encouraging inefficiency.

As time progresses, the processes get slower and slower and newer hurdles and pain points are introduced to make people fork out more and more money, and everyone ends up losing – paying a heavy price for encouraging such practices – because of the resulting inefficiencies.

If there’s one take-away from the above discussion, it would be this:  in this form of corruption, the inefficient people get rewarded, and you end up encouraging inefficiency.

Type 2:  Man-In-The-Middle Corruption

The second kind of corruption is the man-in-the-middle kind of corruption. This is the corruption you encounter when people’s money passes through the hands of a middle-man tasked with procuring services for them.

In the realm of public services, like the construction of roads and schools, that middle-man is government.

In the presence of corruption, the middle-man ends up selecting the service-provider who pays the highest bribes, not the service-provider who does the best job.

This leads to a market where the lowest-quality service provider wins.

Economic theory also suggests that this drives the highest-quality vendors out of the market.

That conclusion follows from the work of George A. Akerlof. He described it in a paper titled ‘The Market for “Lemons”: Quality Uncertainty and the Market Mechanism’.

In the case of India, it is possible that the migration of computer scientists and engineers out of India can be explained in part by the pressures of such a process.

We can also compare the two forms of corruption in terms of how damaging they are.

Impact Analysis

1)  Speed-Money Corruption

Let’s say that a citizen needs to get a job worth X dollars/rupees done.  He needs to pay a bribe that’s usually of the order of 3% to 10% of X.

With the cost of the bribe factored in, the value/expenditure ratio is above 90%.  So, the loss is small.

If we count the money taken as a bribe as a benefit to society (assuming the bribe-taker uses it well – maybe he donates it to charity) then the value/expenditure ratio is 100%.

So, speed money corruption doesn’t hurt too badly.  It leads to a small loss if any per transaction.

The other type of corruption is far more damaging.  Here’s why:

2)  Man-in-the-Middle Corruption

Things are much worse (much more damage is done) when a middleman decides to spend the money that you pooled in (tax money) injudiciously.

So let’s say X dollars/rupees of our tax money is in the hands of a middleman.

Let’s say the middleman takes a bribe from a vendor to buy something worthless (I’ll give you some examples of this shortly).

The middleman who made the purchase would get 3% to 10% of the value of the sale from the crooked vendor (this is called a kickback).

Now assuming the middleman spends the money he gets very well (let’s say he donates it to charity), that is the only value the tax-payer gets for the transaction.

So, for X dollars spent, the taxpayer gets a maximum benefit of X/10.

That means the value/expenditure ratio is less than 10%.

Now, here are some examples of possible cases of man-in-the-middle corruption in India.

There is an article in the BBC titled “India scraps $753 Finmeccanica helicopter deal“.  It is about the purchase by the Indian government of 12 helicopters for transporting VIPs at a cost of $753 million. Each helicopter cost around $60 million.  The kickbacks paid are estimated to be a whopping $67.6 million.

In comparison, the cost of India’s space mission to Mars is $73 million.

So, Indian officials were willing to pay as much for a helicopter as it would cost to fund a space mission to Mars.

The helicopters were also quite useless because there are thousands of military helicopters already in use in India to ferry VIPs.

The argument that the Generals who bought the helicopter put forth was that the helicopter would allow the bodyguard of the Prime Minister to stand upright while accompanying him to landings at high altitudes.

That would have been an expenditure of 753 million dollars for a value of 67.6 million (provided the officials used the bribe money well) which is a less than 10% value/expenditure ratio.

This is why man-in-the-middle corruption is particularly dangerous and damaging.  It can lead to a country’s resources being siphoned off without conferring any benefits upon the country.

Another example I can give you is that of the excessive expenditure over the last few years on the import of arms.

In the years 2011, 2012 and 2013, India spent $12 billion to buy aircraft from Europe, and a further $12 billion to buy 120 Sukhoi planes from Russia, and $10 billion for just ten Boeing transport planes (each plane costs $1 billion – no tenders were issued).  [This World Bank report suggests that corrupt governments overspend on defence procurement because of the lack of transparency in such deals.]

And the Indian government is spending so much money on weapons imports while running a deficit (India is borrowing money to finance I believe 50% of its budget).

What is scarier is that the Indian central government’s annual education budget is only $12 billion, at a time when there are 400 million Indians who don’t get a basic education, and when India graduates less than 100 computer science PhD students each year (thousands of Indian students go overseas for graduate studies).

But it’s actually much worse.  More than 250,000 farmers have killed themselves since the mid-1990s — a figure that may be a fraction of the truth as local authorities willfully misreport “accidental” deaths.  Every second child born in India is malnourished. Nearly two million children under the age of five die every year from preventable illness as common as diarrhoea.  Of those who survive, half are stunted owing to a lack of nutrients. The national school dropout rate is 40 per cent.

The above numbers highlight the dangers of man-in-the-middle corruption and make clear why it is imperative to put a stop to it.  We can see from the above that both types of corruption lead to inefficiency, but the second type of corruption can lead to the large-scale misallocation of tax money (leading to poverty, hunger and death on an unimaginable scale).

We have proposed some methods to prevent this type of corruption in the following articles: