Category: AI and Life

Kabir and Language

Kabir
Image from Wikipedia

Yesterday, I went to a concert of songs belonging to the tradition of a 15th century saint-poet called Kabir, and came across a very interesting song that he is said to have composed.

It went something like this.

The cow was milked

Before the calf was born

But after I sold the curd in the market

and this:

The ant went to its wedding

Carrying a gallon of oil

And an elephant and a camel under its arms

From the perspective of natural language processing and machine learning, the incongruous situations depicted in these poems turn out having an interesting pattern in them, as I will explain below.

I found more examples of Kabir’s “inverted verses” online.

The poems at http://www.sriviliveshere.com/mapping-ulat-bansi.html come with beautiful illustrations as well.

Here are a few more lines from Kabir’s inverted verse:

A tree stands without roots

A tree bears fruit without flowers

Someone dances without feet

Someone plays music without hands

Someone sings without a tongue

Water catches fire

Someone sees with blind eyes

A cow eats a lion

A deer eats a cheetah

A crow pounces on a falcon

A quail pounces on a hawk

A mouse eats a cat

A dog eats a jackal

A frog eats snakes

What’s interesting about all of these is that they’re examples of entity-relationships that are false.

Let me first explain what entities and relationships are.

Entities are the real or conceptual objects that we perceive as existing in the world we live in.  They are usually described using a noun phrase and qualified using an adjective.

Relationships are the functions that apply to an ordered list of entities and return a true or false value.

For example, if you take the sentence “The hunter hunts the fox,” there are two entities (1. the hunter, 2. the fox).  The relationship is “hunts”, it returns true for the two entities presented in that order.

The relationship “hunts” would return false if the entities were inverted (as in 1. the fox and 2. the hunter … as in the sentence “The fox hunts the hunter”).

The relationship and the entity can be stored in a database and hence can be considered as the structured form of an unstructured plain-language utterance.

In fact it is entities and relationships such as these that it was speculated would some day make up the semantic web.

Most of Kabir’s inverted verse seems to be based on examples of false entity relationships of dual arity (involving two entities), and that often, there is a violation of entity order which causes the entity function to return the value false.

In the “cow was milked” song, the relationship that is violated is the temporal relationship: “takes place before”.

In the “ant’s wedding” song, the relationship that is violated is that of capability: “can do”.

In the rest of the examples, relationships like “eats”, “hunts”, “plays”, “dances”, “bears fruit”, etc., are violated.

Other Commentary

In Osho’s “The Revolution”, he talks about Kabir’s interest in and distrust of language, quoting the poet as saying:

I HAVE BEEN THINKING OF THE DIFFERENCE BETWEEN WATER

AND THE WAVES ON IT. RISING,

WATER’S STILL WATER, FALLING BACK,

IT IS WATER. WILL YOU GIVE ME A HINT

HOW TO TELL THEM APART?

BECAUSE SOMEONE HAS MADE UP THE WORD ‘WAVE’,

DO I HAVE TO DISTINGUISH IT FROM ‘WATER’?

And Osho concludes with:

Kabir is not interested in giving you any answers — because he knows perfectly well there is no answer. The game of question and answers is just a game — not that Kabir was not answering his disciples’ questions; he was answering, but answering playfully. That quality you have to remember. He is not a serious man; no wise man can ever be serious. Seriousness is part of ignorance, seriousness is a shadow of the ego. The wise is always non-serious. There can be no serious answers to questions, not at least with Kabir — because he does not believe that there is any meaning in life, and he does not believe that you have to stand aloof from life to observe and to find the meaning. He believes in participation. He does not want you to become a spectator, a speculator, a philosopher.

Notes

This genre of verse seems to have been a tradition in folk religious movements in North India.  In “The Tenth Rasa: An Anthology of Indian Nonsense” by Michael Heyman, Sumanya Satpathy and Anushka Ravishankar, they talk about Namdev, a 13th century saint-poet as having authored such verses as well.

Fraud detection using computers

For a long time, we’ve been interested in using mathematics (and computers) to detect and deter fraud.  It is related to our earlier work on identifying perpetrators of terrorist attacks.  (Yeah, I know it’s not as cool, but it’s some similar math!)

Today, I want to talk about some approaches to detecting fraud that we talked about on a beautiful summer day, in the engineering room at Aiaioo Labs.

That day, in the afternoon, somebody had rung the bell.  A colleague had answered the bell and then come and handed me a sheet of paper, saying that a lady at the door was asking for donations.

The paper bore the letterhead of an organization in a script that I couldn’t read.  However the text in English stated that the bearer was a student collecting money to feed a few thousand refugees living in a refugee camp in Hyderabad (the refugees’ homes had been destroyed in artillery shelling on the India-Pakistan border and that there were a few thousand families without shelter who needed food and medicines urgently).

On the sheet were the names and signatures of about 20 donors who had each donated around 1000 rupees.

Now the problem before us was to figure out if the lady was a genuine student volunteer or a fraudster out to make some quick money.

There was one thing about the document that looked decidedly suspicious.

It was that the amounts donated were all very similar – 1000, 1200, 1300, 1000, 1000, 1000, 1000.

All the numbers had unnaturally high values.

So, I called a friend of mine who came from the place she claimed the refugees (and the student volunteers) were from and asked him to talk to her and tell me if her story checked out.

He spoke to her over the phone for a few minutes and then told me that her story was not entirely true.

She was from the place that she claimed the refugees came from, but she was in fact collecting money for her own family (they had come south because one of them had needed a medical operation and were now collecting money to travel back to their home town).

When we asked her why she had lied, she just shrugged.

We felt it would be fine to help a family in need, so we gave her some money.

However, the whole affair gave us an interesting problem to solve.

How do you tell if a set of numbers is ‘natural’ or if it has been made up by a person intent on making them look natural?

Well, it turns out that statistics can give you the tools to do that.

Method 1

In nature, many processes result in random numbers that follow a certain distribution. And there are standard distributions that almost all numbers found in nature belong to.

For example, on the sheet of paper that the lady had presented, the figures for the money donated should have followed a normal distribution.  There should have been a few high values and a few low values and a lot of the values in the middle.

Since that wasn’t the case I could easily tell that the numbers had been made up.

But you don’t need a human to tell you that.  There are statistical tests that can be done to see if a set of numbers belongs to any expected distribution.

I looked around online and found an article that tells you about methods that can be used to check if a set of numbers belongs to a normal distribution (a distribution that occurs very frequently in nature): http://mathforum.org/library/drmath/view/72065.html

Some of the methods it talks about are the Kolmogorov-Smirnov test, the Chi-square test, the D’Agostino-Pearson test and the Jarque-Bera test.

Details of each can be found at these links (taken from the article):

One common test for normality with which I am personally NOT familiar, is the Kolmogorov-Smirnov test.  The math behind it is very involved, and I would suggest you refer to other resources such as this page

  Wikipedia: Kolmogorov-Smirnov Test
    http://en.wikipedia.org/wiki/Kolmogorov-Smirnov_test 

You can read more about the D'Agostino-Pearson test and get a table that can be used in Excel here:

  Wikipedia: Normality Test
     http://en.wikipedia.org/wiki/User:Xargque#Normality_Test 

 Wikipedia: Jarque-Bera Test
     http://en.wikipedia.org/wiki/Jarque-Bera_test 

One item of note: depending on how your stats program calculates kurtosis, you may or may not need to subtract 3 from kurtosis.

 See: Wikipedia Talk: Jarque-Bera Test
      http://en.wikipedia.org/wiki/Talk:Jarque-Bera_test

On to the next method:

Method 2

Another property of many naturally occurring numbers is that about one third of them start with the number 1 !!!  Surprising isn’t it?!!

Well, it turns out that this applies to population numbers, electricity bills, stock prices and the lengths of rivers.

It applies to all numbers that come from power law distributions (power laws govern the distribution of wealth, connections on facebook, the numbers of speakers of a language, and lot of numbers related to society).

This is called Benford’s law:  http://en.wikipedia.org/wiki/Benford’s_law

(I believe that Benford’s law would have applied to the above case as well – donations would have a power law distribution – if you assumed that all donors donated money proportional to their wealth).

When I read about Benford’s law on Wikipedia (while writing this article), I found that it is already being used for accounting fraud detection.

The Wikipedia says:

Accounting fraud detection

In 1972, Hal Varian suggested that the law could be used to detect possible fraud in lists of socio-economic data submitted in support of public planning decisions. Based on the plausible assumption that people who make up figures tend to distribute their digits fairly uniformly, a simple comparison of first-digit frequency distribution from the data with the expected distribution according to Benford’s Law ought to show up any anomalous results. Following this idea, Mark Nigrini showed that Benford’s Law could be used in forensic accounting and auditing as an indicator of accounting and expenses fraud.[10] In practice, applications of Benford’s Law for fraud detection routinely use more than the first digit.[10]

Method 3

There are also methods that can be used by governments and large organizations to prevent fraud in the issuing of tenders.

More about that in my next article.

In trust we god

in_trust_we_god

Can trust affect the outcome of political events (war), business transactions (pricing) and economic affairs (poverty)?

This is a problem that I’ve been very interested in for many years.

A few years ago I came across papers in economics and game theory that supplied the mathematical tools that we need to analyse such problems.

So, I’ll take each area of interest 1) politics 2) business and 3) economics and explain how trust matters in each case.

1.  Politics

Can the outcome of something like war be determined by trust?

Let’s assume an army of 2 soldiers.

In a war, the benefits to each soldier can be modeled as a bi-matrix (normal-form game) as follows:

soldier 2 fights soldier 2 flees
soldier 1 fights 5, 5
–5, 0
soldier 1 flees 0, -5
0, 0
Normal form or payoff matrix of a 2-player, 2-strategy game

The first of the two numbers in the matrix represents the payoff to soldier 1.

The second of the two numbers in the matrix represents the payoff to soldier 2.

(The soldiers win something (represented by 5 points) if their army wins; they win nothing if their army loses; and they lose their life (represented by -5 points) if they do not flee and their army loses; we assume the army wins if both soldiers do not flee and loses if one or both flee).

If soldier 1 trusts soldier 2 not to flee the battlefield, the best strategy for soldier 1 is to stay and fight as well (since he will then get more benefits than if he flees).

If soldier 1 does not trust soldier 2 to stay on the battlefield (if he suspects that soldier 2 will run away), then the best strategy for soldier 1 is to run away himself (so that he does not remain on the battlefield and get killed).

So, this model shows that if two equal 2 man armies meet on a battlefield, the one whose soldiers trust each other more will win.

2.  Business (Pricing)

There is a very interesting paper by George A. Akerlof (‘The Market for “Lemons”: Quality Uncertainty and the Market Mechanism’).

It tries to explain why the price of a new car in a show room is so much higher than the price of a new car in the second-hand car market.

For example, a car costing $25,000 fresh out of the showroom, might fetch $18,000 if sold as a used car in the used car market.

Akerlof’s paper tries to explain why the price dropped so sharply.

Akerlof suggests that the price drop is a result of the uncertainty surrounding the quality of the car in the used-car market.

A certain percentage of cars in a used-car market will be defective (since anyone can sell a car in an unregulated market, and unscrupulous people would have put defective cars up for sale).

Let’s say 50% of the cars in the used car market are defective.

Now, a person buying a used car a day old will only be prepared to risk paying 50% of the showroom price for the car (because of the 50% chance that the car is worth nothing).

The Price of Trust

This result has the following unintended consequence:

The more a person trusts a seller, the higher the price he will be willing to offer for a car.

I’ll give you an example of that.  (I’m sorry, but this is a bit racist).

When I was a student in North Carolina, and I was looking to buy a used car, I was given the following piece of advice by my fellow students.

They said, “Go for a car that an American is selling because they will tell you about any problems that it has.  Don’t buy a car from an Asian or an Indian unless you know them well.  They won’t tell you if there are any problems.”

I see the same effect even when doing business in India today – a lot of business happens through connections.

Price Sensitivity

It might also explain why Indians are so price sensitive.

Indians are said to be very price-sensitive, preferring the less expensive offerings over more expensive ones that promise better quality (I recall Richard Branson said that at one point while explaining why he didn’t want to enter India).

I think the price sensitivity is a result of Indians not being able to trust promises of higher quality from their countrymen.

Price becomes the only measure that Indian buyers are able to trust to when making a purchasing decision, leading to extreme price-sensitivity in the Indian market.

Hiring and ‘Brain Drain’

Even in hiring, this can have the effect of driving down salaries.

When hiring someone, an Indian firm is likely to offer a lower salary than the market, because they don’t trust in the abilities of the person being hired.

In Akerlof’s paper, he talks about a side-effect of a lack of trust.  He says that good quality cars will just stop being sold on the low-trust markets.

The applies to the job market in India as well:  Indian firms tend to offer lower salaries, which might lead to the best engineers choosing MNCs over Indian firms or leaving Indian shores altogether.

3.  Economics

I’ve described in an earlier blog how man-in-the-middle systems of government can fail to work efficiently if the man-in-the-middle is corrupt.

I’ve described in that post how resources can be wrongly allocated in the presence of corruption.

https://aiaioo.wordpress.com/2013/08/15/who-betrayed-ekalavya-2/

The result of an inefficient allocation of our resources is poverty.

For example, the Indian government has tripled defence spending in the last 10 years – through heavy borrowing – when it is possible to show that we need to allocate whatever money we have to education (see our arguments for that https://aiaioo.wordpress.com/2012/06/04/algorithms-to-combat-poverty/).

World Bank studies (that you can get off an Indian Reserve Bank website) show that corrupt governments spend more on arms (because of how easy it is to hide kickbacks from arms deals) than honest governments.

So, the economic prosperity of a country can be impacted by corruption.

Causes of Corruption

But we can ask a deeper question:  “What causes corruption?”

I’ll try to show right here that it is a lack of trust.

Take for example two players in a bidding war (let’s say that they are bidding for a government contract).

Each has the choice to give a bribe or not to give a bribe.

Player 1 is more likely to give a bribe if player 1 does not trust player 2 to not offer a bribe to the government official.

It’s the same decision matrix that I have used for the case of the 2 soldier army.

So you get it?

Everything depends on trust.

Philosophy

I am probably way out of my depth on this, but the ancient Greeks seem to have had two views on the supreme ideal that man should strive for.

According to the Wikipedia article on Dialectics:

“The Sophists taught arête (Greek: ἀρετή, qualityexcellence) as the highest value, and the determinant of one’s actions in life.”

But there lived in Greece a man who disagreed with that notion:  ”Socrates favoured truth as the highest value, proposing that it could be discovered through reason and logic in discussion: ergo, dialectic.”

But the above models seem to suggest that truth (honesty) results in trust (you know that the guy next to you is honest and won’t lie about the quality of a car or bribe a government official to get ahead of you).

And what the Akerlof paper shows is that trust rewards and promotes quality.

In other words, the two Greek concepts of quality (of the values mankind must uphold for its own good) are probably one and the same.

Related Posts:

1.  Framework for evaluating values

2.  What traffic can reveal about society

3.  Who betrayed Ekalavya?

4.  Can economics change the world?

5.  Is there an algorithm to combat poverty?

6.  Why dance is undervalued

7.  Is 5 very far from 4?

Related Far-out Posts:

1.  Splitting the Truth into Precision and Recall

2.  Does AI have Buddha nature?

[The image in this picture was taken from a circulated Facebook post.  The copyright owner of the image is unknown at this time and if anyone knows him/her I’d like to make sure they’re ok with my using the image and acknowledge them].

Should Cecilia have said “insecure” instead of “unsecure”?

In this funny PhD Comic, the main character – Cecilia (the girl in red) – says:

“Do you realize how unsecure your coffee distribution system is?”

That made me wonder – should she have said ‘insecure’?

Even the WordPress spell-checker has a problem with “unsecure”.

It thinks that “unsecure” is a spelling error.

However, the word “insecure” doesn’t sound as if it were the right term to use in the context of computer security.

That is because the word “insecure” is usually used in the context of a person to mean a person who is not confident and self-assured.

To call a computer “insecure” would be a bit like saying that the computer had self-image issues.

Others have written about this cognitive dissonance as well (see http://english.stackexchange.com/questions/19653/insecure-or-unsecure-when-dealing-with-security for a nice discussion).

Given the problem, the author of the cartoon seems to be justified in using a newly-minted word (one not found in any dictionary) in order to describe the lack of security.

This is also very interesting because it throws some light on how words are born.

Before I can explain what I mean, I’ll need you to take a look the Oxford dictionary’s definitions of the word “insecure” (from the Oxford English Dictionary online search at http://oxforddictionaries.com/definition/english/insecure?q=insecure):

insecure

adjective

  • 1   uncertain or anxious about oneself; not confident:  a rather gauche, insecure young man,  a top model who is notoriously insecure about her looks
  • 2   (of a thing) not firm or fixed; liable to give way or break:  an insecure footbridge 

                 not sufficiently protected; easily broken into:  an insecure computer system

  • 3   (of a job or situation) liable to change for the worse; not permanent or settled:  badly paid and insecure jobsa financially insecure period

There are three ways in which the word “insecure” can be used.

The second usage would have been perfect for the context of computer security.

But the first usage might be conflated with the second in that context.

And that is because (sorry, I no longer recall the references to support this claim) computers appear to the human mind to have human-like characteristics (we say things like “Google tells me that …” or “my computer has gone to sleep”).

So, the only word in the dictionary that can do the job – the word “insecure” – has a conflict of interest.

And therefore, a new word needs to be coined that is not susceptible to the same sort of ambiguity.

And if the new word “unsecure” catches on, then one day, the second sense of the word “insecure” could become extinct in the context of computers.

Oh well, “it’s only words!”

POST EDIT

A friend pointed out that the Google NGram Viewer shows a history of the use of the word “unsecure”: http://books.google.com/ngrams/graph?content=unsecure.

The word seems to have been in use between 1650 and 1850 (there is evidence of use in literature), and has in more recent times simply fallen out of circulation (being eclipsed by “insecure” in around 1750).  Thanks, Prashant.

(You can also search for those early usages in books – http://books.google.com/books?id=WmpCAAAAcAAJ&pg=PA12&dq=%22unsecure%22&hl=en&sa=X&ei=aOcLUq7aA-3iyAHu8YGwAg&ved=0CDMQ6AEwAA#v=onepage&q=%22unsecure%22&f=false)

How algorithms can ‘shape’ our world – Kevin Slavin

In our first blog post of the year, we’d like to share a talk by Kevin Slavin.

It is about how math (especially algorithms) has ‘transitioned from being something that we extract and derive from the world to something that actually starts to shape it.’

One of the posts we most enjoyed writing last year was related to this:

It was about algorithms to combat poverty.

We also wrote about how math might someday shape our values.

Frameworks for evaluating values

I recently came across a very interesting 2001 paper by Daphne Koller that dealt with influence diagrams and how they could be applied to game theory.  I came across the paper while doing some background reading on a talk on decision making in accordance with our core values by a friend of mine, Somik Raha.

Influence diagrams are a formalism (very similar to probabilistic graphical models) that are used for making decisions.

What Somik Raha has attempted to do is come up with a framework for making decisions while also taking one’s values into account (either as constraints or as inputs into the decision model).  To do that he proposes extensions to influence diagrams.

What I found interesting when I thought about Daphne Koller’s work and Somik’s together, is that they could possibly give you a framework to evaluate your values.

Koller’s formalism reduces to a game theoretic model, which can be evaluated to determine the outcome of the decisions made by a group of people.

Plug in a formalism based on Somik’s ideas and you just might be able to create a way to quantity the benefits of values.

The Importance of Values?

I have been thinking a bit about values these days because there has been a horrific gang rape in Delhi, and there have recently been numerous incidents of bad driving where friends of mine have been injured in Bangalore.  Then there is corruption.  Our society seems to be quite happy with inequality and vast differences in the distribution of wealth.  It make me wonder if our values are to blame.

I have often wondered whether some of our problems originate in our value systems and whether the value systems that we consider sacrosanct in India are really very good ones.

Let me take just a couple of values that most Indians would consider to be very good values

  1. Non-violence
  2. Obedience

and let’s discuss them in more detail.

  1.  Non-violence

This value appeals not just to people in India.  You see variants of the value of non-violence appear in Tolstoy’s writings and in Semitic religions, as you can see from the Bible (“turn the other cheek”) and the Quran (“give alms to one who begs from you, even if he comes on the back of a horse”).

The issue with this sort of value is that it makes a person (and those around him/her) extremely vulnerable to injustice.

In India, we restrict the liberty of women – in their choice of clothing, company and lifestyle – for fear that they could be in danger if they violated societal norms.  This shows that none of us want to fight society or cross swords with someone who might make disparaging comments about personal choices.

Moreover, possibly as a result of the value of non-violence, very few Indians if any are taught fighting skills in school.  So, even if a person really wanted to act, say to protect a friend, he or she might not really have the skills to take down an aggressor.

So, instead of protecting and standing up for people who might be vulnerable, we become their tormentors and make their lives more miserable, just so we don’t have to get our hands dirty, or because we don’t have the skills and strength to do squat.

I’ve written about how bribes are openly collected by traffic policemen.  It should be very easy to put a stop to such behavior if you’re willing to fight.

If non-violence is not a core value, then how do we protect people from tearing each other to bits?

We could start with a question like:  non-violence for what purpose?  (turning it into an extrinsic value)

If the answer is something like, “so that the weak feel protected”, why not make protecting the weak our core value?

I’d prefer teaching kids values like “Don’t ever turn your back on a bully” rather than values like “Don’t fight anybody, and just come home safe, child!”

2.  Obedience

Indian parents love to boast that their child is “such an obedient child!”

Is that a good thing?

Obedience is different from politeness or respect.  The latter are mutual but the former is one way.

So, the politics of obedience creates a hierarchy of subservience.

In India, Parents expect complete obedience from Children.

The Police expect complete obedience from People.

The Politicians expect complete obedience from Police.

Teachers expect complete obedience from Students.

Managers expect complete obedience from Employees.

The creation of the hierarchy (through expectations of obedience) can be very dangerous in many ways.

1.  It can stifle creativity and problem-solving ability.  There is a bias against ideas flowing up a hierarchy because those higher up the hierarchy claim their place above those required to be obedient to them on the premise that they are somehow superior to those below them.  A good example is how parliament will not accept that people have a right to demand a bill against corruption (members of the Indian parliament claim that parliament is supreme in a parliamentary democracy – not the citizens that the parliamentarians represent).

2.  It can leave young people ill-equipped to defend their personal spaces.  I read in a paper on rape that many rapists approach victims by testing their boundaries.  They make comments and otherwise violate the intended victim’s personal boundaries.  If these are not strongly resisted, the probability of an assault becomes greater.  Another strategy used by rapists is to move their intended victim to a new location where they are more vulnerable. It is very important for people to be conditioned so that they do not obey an order by an attacker to relocate under any circumstances.

3.  The hierarchies perpetuate the power of stronger (bigger, older or richer) parties by providing social sanction to their dominant position, and so hinder social mobility.

4.  The obedience hierarchy could allow a few people at the top to amass too much power. It might have, for example, prevented cops from disobeying those in power during the Gujarat riots.

5.  Obedience means valuing rules above truth.  Obedience implies not challenging the rules or the status quo.  So there is little scope for discovering if the rules really are good ones for everybody.  People often defend something they assert with a “because I said so.” – that is, you are expected to believe them because of their authority, and not because they can substantiate their assertion.

Obedience as an absolute value is not entirely harmless.  It could be dangerous to us as a society because corrupt politicians can use the pliability and obedience of people around them to get away with evil (remember the activist who was hacked to death on the orders of a corporator from Bangalore, the journalist who was burnt to death in Uttar Pradesh, or the shutdown that the former Chief Minister of Karnataka State ordered when he was about to be investigated for corrupt dealings?).

I’d love to replace “obedience” with something else, perhaps “honesty” and “trustworthiness” and “pride”.

Summary

I understand that we as Indians are very proud of our values but I’ve tried to argue that our values need to be re-examined.

Personally, I’d love to see the day when we replace all our values with just the value of trustworthiness.

Trustworthiness as a value would mean we’d fight for each other.  It would mean we’d protect the weak.  It would mean we’d be on time.  It would mean we’d be honest.  It would mean we’d be capable and skilled and strong.  It would mean we’d be proud of each other.  It might mean we’d never lose another war.

Reading Koller’s and Somik’s work you get the feeling that one day you might be able to evaluate the comparative benefits of two sets of values, and pick the better one, using plain math.

And hopefully, by showing them mathematical proofs, you can convince people to change their values and pick better ones for themselves.