Did anything really happen to silicon values?

I recently came across an article on the Silicon Valley by Bill Davidow.  The article was titled “What Happened to Silicon Values?”

Below is a snippet from the article …

<snip>

When I arrived at Hewlett-Packard in 1965, the company was already a $300 million giant. The company was focused on delivering advanced technology of great value, then servicing and supporting the customer to make sure he derived value from what he bought. Customers trusted Hewlett-Packard. I remember one customer who so trusted the salesman who took care of his account that he let the sales rep purchase what he needed. That period of trust went on for a long time. The salesman told me his secret: He never bought anything for the customer that the customer did not really need.

</snip>

At one point in his article, Bill wonders if the restrictions that Apple builds into its products might be not in keeping with Silicon Valley values (because they deprive customers of choice).

He says: “Apple controls our virtual landscape, bounded by iTunes to the north, the iPhone to the south, the iPad to the east, and the iPod to the west, giving it increasing power to deprive customers of choice.”

I wondered about that for sometime.  Then I realized that giving people certain choices need not always be (depending on what those choices are about) the best thing to do.

For example, Apple products’ limited choices make them easier to use.  Customers should not be burdened by unnecessary choices, and consequently be at risk of making bad ones.

The best service to the client is when you help them make the right choices.

I think that Bill’s anecdote about the HP salesman underscores the selfsame point.  The HP salesman was actually making all the choices for the client.  And it was in making those choices correctly, disinterestedly and fairly and in not leaving the client to her/his own devices that the HP salesman made his highest contributions to the client’s success.  In that respect, the HP story is exactly the same as Apple’s.

I remember having worked at Sun Microsystems as an engineer in the 90s.  Sun was very passionate about open systems – about architectures that people could mix and match at will.  They went down because no one really had a need for or a will to pay for openness for openness’ sake, or for free will.

So, I don’t think anything has happened to the values of the Silicon Valley.

Does AI have Buddha nature?

I read a book last evening … “The Zen of Steve Jobs”.

It’s a nice book, and it makes you think.

There was something in it that I wanted to share with you.

It’s a concept called “Mu”.

Mu

In the book, a Zen teacher – Kobun Chino Otogawa – who is a close friend of Steve’s, asks him a series of questions:

Question 1:  ‘That tree over there.  Does it have Buddha nature?’

Question 2:  ‘What about that flock of birds?’  (that is, does it have Buddha nature?)

Question 3:  ‘What about that billboard?’  (that is, does that billboard have Buddha nature?)

Question 4:  ‘Ask me an answer.’   (the question itself is meaningless)

It turns out that the answer for none of these questions is ‘yes’.  BUT, the answer is not ‘no’ either.

To answer ‘yes’ or ‘no’ to any of these questions is absurd.

The right answer in such a situation, Kobun says, is ‘mu’.

Mu is a meaningless term – a word with no meaning – to be used as a way of saying that you offer no answer.

Classification

Now, this is very very interesting to a researcher working on the machine learning problem called ‘classification’.

For instance, given a question like ‘Does AI have Buddha nature’, a classifier is an AI tool that will look at the features in question, and return one of two answers – ‘Yes’ or ‘No’.  A classifier trained to give just those two answers answer will give no other answer.

This poses problems when we move out of the domain that we built the AI tool for, and into another domain that the AI engine was not meant for.

A good example is a sentiment classifier.  In a lot of early studies on sentiment classification, the training sets used were customer reviews.  In other words, reviews were the sole domain of the classifier.  Since most sentences in reviews were opinions, early sentiment classifiers could label anything as positive or negative and be mostly right about it.

In 2004, social media became big.  But Twitter and Facebook were full of sentences that were not opinions.  Running an old-fashioned sentiment classifier on those sentences could lead to a lot of false positives.

Take the sentence ‘Is that a good phone?’  Does this sentence contain a positive or a negative opinion?  The sentence contains neither.

That’s because the sentence is not an opinion.  The sentence is a question.  So you can’t perform sentiment analysis on it.

So, the answer is ‘mu’.

In classification, we sometimes call this category ‘Other’.

Intentions

One day, we attempted to use intention analysis to identify when a customer might quit.

The intention categories we used to do this were:  ‘criticize, complain, accuse, quit’.

If we could identify these intentions, we would have a way to identify and reengage an aggrieved customer before they gave up on the service provider forever.

To build such a system, we started looking at sentences in complaint forums.

Some of the sentences were easy to categorize.

Example 1: ‘my login id not open’ is obviously a complaint.

Example 2: ‘FRADULENT DEDUCTION OF CHARGES’ is an accusation (it implies bad faith in the service provider).

But what about Example 3:  ‘NEGLIGIENT SERVICE’ ?

In Example 3, the customer is not only reporting a problem but also implying bad faith in the provider.

The interesting thing about this sentence is that it certainly belongs to one or the other of the categories we had created for it.

It could be a complaint.  It could be an accusation.

Though you may not know which one of these categories it belongs to, you can be certain that it belongs to one of the two.

So, the answer in this case cannot be ‘mu’.

You need more than one answer.

Why did the Zen philosophers miss it?

What is interesting is that the Zen philosophers whom Steve Jobs so admired seem to have completely failed to notice the existence of an anti-mu (multiple viewpoints that could all be right) though they pondered “mu” over for thousands of years.

But AI researchers have run splat into all of it within just 100 years of AI research.

So, does AI have Buddha nature?

The answer could be ‘mu’ and ‘yes’ and ‘no’.

Method for moving an ambulance at high speed through crowded traffic

The frequency with which I see ambulances impeded by slow-moving traffic in Bangalore led me to wonder if something can be done to speed up their progress.

Studies have shown that in India, the capacity of a standard two-lane road is approximately 1700 pcu (PCU stands for Passenger Car Unit and is a unit of measurement of road capacity) in each direction, which is pretty good.

However, in cities, there just aren’t enough roads in a given direction, or traffic gets slowed down by conflict at intersections.

Whatever the case may be, the slow movement of traffic on roads can impede the progress of ambulances through a city.  Here, we propose one solution to the problem (a solution that requires a minimum of two-lanes).

Each ambulance carries a device to trigger the traffic lights ahead so that they prevent any vehicles from entering the road ahead of it (in both lanes).

There are openings in the median to allow a vehicle to change lanes (as shown in the image).  Once an ambulance has entered a road, it uses these openings to switch to the lane going in the opposite direction (as shown in the image).  Then it enters the next road in the normal manner, and repeats the process.

This method uses the fact that if traffic is stopped, the vehicles don’t usually fill up the road.  It will work as long as traffic can be stopped sufficiently soon, so that the roads ahead are no more than half-full of cars.

Can an algorithm identify the perpetrators of a bomb attack

On 13th July, 2011, there were simultaneous bomb attacks at different locations in Mumbai.  One day, while I was thinking about it, I suddenly realized that there was an algorithm that could be used to bring the perpetrators to book.

It’s almost a year since the attacks took place, and we thought we’d finally write about our work on the problem (which we’ve shared with many government organizations, including CAIR, though we frankly don’t know if they plan to implement it).

Introduction:

When multiple coordinated attacks take place (like the three bombs that were triggered simultaneously in three locations in Mumbai in 2011), if the attackers had used cell phones to coordinate their attacks, the phone numbers involved can be extracted mathematically from the call records linked to the cell towers in the vicinity of the attacks.

Our calculations indicate that in a city the size of Mumbai and a population of around 20 million people, you would need a minimum of only 3 simultaneous attacks to be able to pinpoint the attackers to within 1 person. If there were only 2 attacks, you would have a much higher uncertainty. The greater the number of coordinated attacks, the easier it is to identify the attackers.

The identification will only be possible if the attackers used their cell phones to coordinate with a central handler, or with each other, or a group of handlers who in turn communicated with each other. It would be possible to support different possible patterns of communication and usage of cell phones and SMS to coordinate between the attackers and confirm the attacks.

History:

A similar method might have already been used in Europe after the Madrid bombings to identify those responsible for the same.
Two years after the attacks, the EU mandated that carriers store some details for 6 months. The information that the carriers need to preserve contains all the details required to triangulate attackers in multiple simultaneous attacks.

Here is some information on the directive from the Wikipedia:

—

On 15 March 2006 the EU formally adopted Directive 2006/24/EC, on “the retention of data generated or processed in connection with the provision of publicly available electronic communications services or of public communications networks and amending Directive 2002/58/EC.”

The Directive requires Member States to ensure that communications providers must retain, for a period of between 6 months and 2 years, necessary data as specified in the Directive.

 to trace and identify the source of a communication;
 to trace and identify the destination of a communication;
 to identify the date, time and duration of a communication;
 to identify the type of communication;
 to identify the communication device;
 to identify the location of mobile communication equipment.

The data is required to be available to competent national authorities in specific cases, “for the purpose of the investigation, detection and prosecution of serious crime, as defined by each Member State in its national law”.

—

Information:

The minimum information that would be needed to obtain an effective fix on an attacker is the following:
a) FROM_NUMBER
b) TO_NUMBER
c) TIME_STAMP
d) CELL_TOWER

Indian carriers tend to store the FROM, TO and TIME information in call logs (used by the carriers for billing and for customer service) for SMS messages for 2 days and for phone calls for 30 days. These can be obtained easily from them by a simple request from police.
The location of the cell tower is determined from signaling logs.

Technology:

Messages used to coordinate teams have certain patterns. For example, a possible pattern in an attack is a confirmation – success/failure – to the central handler after an attack.

Each message would have to originate from a cell tower near the site of a blast and in a short time window after the blast and have the same destination.

Out of all the hundreds of millions of messages sent in a city, only some would satisfy all these constraints.
Simple graph analysis can identify these messages, to very high accuracies in a city of 20 million of the size of Mumbai, provided there were 3 or more coordinated attacks.

The algorithm is so simple that an undergraduate student could build it in a few months.

Acknowledgements:

There are many people I have to thank for their help with this study, including old mentors from other research firms in the area who helped me with suggestions for handling temporal queries, and students/interns who helped speed up the study, and my colleague Sumukh who works on graph/tree search himself.

Algorithms to combat poverty

Causes of Extreme Poverty

An excellent study of poverty and its causes may be found in a book by the name of “The End of Poverty” by Jeffrey Sachs.

One of the examples that Jeffrey uses to explain the phenomenon of extreme poverty is a family farming a plot of land that is too small to feed it.

Because the plot is too small to even feed the family, there is no excess food to sell, and consequently, no way to build up any savings.

Without savings there is no way to make any investments that could result in more crops (like fertilizer) or in developing other skills (like education).

So, a family on a very small plot of land ends up trapped in a cycle of poverty – one that it cannot escape the cycle without external aid.

Extreme Poverty in India

Could such a scenario ever materialize in India?  Could a family ever come to have so little land that it does not suffice for its own needs?

It seems so.

In India, by tradition, land is divided equally among children.  So, as the population grows, the average size of land farmed by a family decreases.

The danger is also difficult to spot ahead of time.

When a person inherits a plot of land too small to support a family on, (s)he is usually already an adult and it is too late for him/her to leave farming for another profession.

That is a model that could explain both the huge disparities in income and the widespread poverty that exist in India.

In some other parts of the world, land holdings are inherited by only one child of a farmer, usually the first-born.  The other children need to “go out into the world” to “earn their fortune” by becoming apprenticed to a tradesman, or by becoming a soldier, or by entering the priesthood.

In such a system (which is called primogeniture), there is a built-in safety mechanism that guards against the danger of  land holdings becoming too small for a family.  In each generation, the land only feeds as many people as it did the generation before.  So, there is always a guaranteed surplus of money with which to pay for the training of other children into other professions.

With the Indian system of inheritance, on the other hand, all of the children of a well-off land-owner could find themselves inheriting holdings that are too small, at a time when they are too old to go back to school.

Whatever be the cause of the problem, it seems reasonable for us to assume that as of today, there are around 400 million people living in abject poverty in India because they have plots of land so small that they cannot feed a family, and no skills other than that of a farmer.

Education as the Solution

One solution to the problem would be some form of retraining to equip the 400 million people with skills that would enable them to obtain alternate employment (by that I mean just anything that does not include laboring on a farm).

The government admits that between 300 and 400 million people in India are illiterate (they cannot write or read).  The number of the innumerate is probably far higher.

The most straightforward solution seems to be to pool some money and set up a number of schools for people in rural areas, staffed with teachers.  Supposing we knew what to teach and were able to find the teachers, would we have solved the problem?  Let’s do the math (starting with some assumptions).

Assume we have 1000 teachers, and that each of them can teach a class of some 100 pupils.

Assume further that it takes 3 years to train a person into a new profession.

With this setup, how many years will it take to retrain 400 million students?

The answer is …

12,000 years.

Now that must come as a shock to some.  What if we instead had 100,000 teachers?

It would still take us 120 years, not counting the time it would take to train 100,000 teachers.

Lack of Government Investment in Education

We played a bit more with the numbers to see what would happen if the government doubled the annual allocation for education.  The allocation for education in 2011 was 11 billion dollars.

For another 11 billion dollars, the government could hire just around another 1 million teachers (assuming an annual salary of around 5 lakh rupees per annum) and solve the problem in around 12 years (not counting the number of years of training the teachers would need).

In fact, the total allocation for education across the board in India has been of the order of 3.5% of GDP.  Most Western countries, with far higher per-capita incomes than India, allocate approximately 8% of GDP to education.  India in reality needs to allocate somewhere close to 10% of its GDP to education.

And just as a digression, every time you read in the news that the Indian government is about to import military equipment for between 5 billion and 11 billion USD – Dr. Manmohan Singh’s government paid that sort of money for Sukhoi fighters in 2011, for an aircraft carrier and planes for it in 2012 and for 10 military transport planes in 2013, while Mr. Modi’s government paid a similar sum for six submarines in 2014, and for 36 fighter planes in 2015 – take a moment to pause and think about the amount of money that is being misspent, because that money is comparable to the federal education budget of India, and a better use for that money would have been education.

So, one obstacle in the way of prosperity in India, is that the government of India refuses to spend the money collected as taxes from Indians on education for Indians.

The Difficulty of Skills Planning

Another problem is that appointing more teachers may make people more literate, and even numerate, but not necessarily more employable.

The reason is that the skills that make a person employable are not just basic reading and writing skills.

Higher skills that would enable the students to earn money – perhaps engineering skills or a craft (a friend of mine recently helped out with a government program that taught web development skills at scale) – would be needed.

One problem with a brute-force approach is that there would be no feedback from the market in the beginning as to what skills would be needed and in what quantities.

An investment would have to be made blindly in teachers equipped to teach certain skills, and it would only be decades later that we would know if we had picked the right skills in the right quantities.

So, there is a chance that this method wouldn’t work out very well for the students in terms of equipping them with marketable skills.

Apprenticeship and Skills Development

But is there another way to skill up such a large number of people?

Well, it turns out that there is.  And the method was used in Europe for hundreds of years.

It is called apprenticeship.

Consider a system where one person trains a small number of people, say 10 people, and each of them in turn trains another 10 people.    Assume further that these ten trainees work for the trainer and get paid while they are being trained.

This is how people learnt a trade in Europe for hundreds of years.

Under this model, if each person trains only once and then needs train people no more, 1 becomes 11 in 3 years.  11 becomes 111 in another 3.  111 becomes 1111 at the end of nine years.  In about thirty years, all 400 million people would have been completely retrained, with no massive initial financial investment and with market forces deciding at every stage what skills might be needed most in the coming years.

So, you have an algorithm that can reskill a large number of people at scale in about 30 years.

The Complexity of an Algorithm

Why does one method work so much better than the other?

The brute-force method of skilling people (using traditional schools and a large number of teachers) is what we call an algorithm of complexity O(n).

What that means is:  should the size of the problem double, the time to solve the problem also doubles.

Apprenticeship is what we call an algorithm of complexity O(log n).  In an algorithm of this sort, when the size of the problem grows exponentially, the time to solve the problem only gets multiplied by the exponent.

Does intuition or logic govern ethics?

I’ve written about intuition and AI before.   Just the other day, I was forced to think the matter through still more thoroughly.

It happened this way.

I was driving down Miller’s Road towards Ulsoor Lake when, on the left hand side, I saw a drunken man slumped on the pavement.  He had passed out completely and his right leg was stretched out a few feet into the road.  I was horrified.  His foot was so far out in the road that he was in grave danger of having his foot run over and crushed by a passing vehicle.  It could happen at any time.

But I absolutely didn’t want to stop and pick up a drunken man.  I felt an incredible reluctance to touch a guy who was dead drunk.  I swerved around the outstretched leg, and then drove past him and away down the road.

Once I had passed the man, it became very very difficult to turn around.

Surely somebody behind me will stop and help the guy.  I’ve gotten past him already.  I will be late if I turn back.  It will be a kilometer’s drive to go back to get to where that man is now.  There’s no easy way to turn around and go back.  Why should I be the one to go all the way back?  It’s too late now.  Maybe a car’s driven over his foot already.   Maybe someone has helped him by now.  I’ll write about this in a blog post, and let everyone know how pathetic I was (we are).  Finally, I thought, if no one else stops to help him, this country deserves to go to hell and it won’t be my fault that it did.

The shocking thing to me was that my intuition said, “Don’t go back.”  So, I began to ask myself what intuition really is.  I had reached Ulsoor Lake and saw the Battle Tank monument on the left.  I passed the gate of the Madras Engineering Group barracks.  Then I stopped my motorbike and thought about it.

Are there times in our lives when we should rely on our intuition less than on logic?

For certain problems in AI,  the best AI algorithms are those that can automatically learn to combine a vast amount of evidence in an ‘intuitive’ manner, without performing explicit logical inference to arrive at the answer.

One example of such a problem is hearing.  We usually manage to recognize words that are spoken to us without really performing logical inference.  Another human endeavor in which intuition might dominate is sports.  Another might be love.

But, when it comes to routine choices, for example, the choice between two modes of transportation to reach a destination, how would a person choose one mode of transportation over the other?  It is to be expected that logical reasoning would be used to pick the alternative that represents the best value in terms of time, cost and convenience.

What about ethical choices?  I recently came across a post on this topic by a blogger friend entitled “What has logic got to do with it” in which the author argues that ethical choices are mere intuition couched in logic (a very Zen perspective on ethics).

But here I was with every nerve in my body telling me to drive away.  What was even more troubling was that just a few years earlier, when I was living in Raleigh, had I seen someone in danger, I would have immediately gone to help.

Anyone would have done the same.

That’s when it struck me that intuition might really be nothing more than your expectation of what anyone in your position might do.  If you expected everyone around you to abandon a man in need, your intuition would dictate that you do the very same thing.  Which brings us to the crux of the problem.  If everyone followed their intuition, everyone would act like the majority and consequently, the majority intuition would become the norm – the only surviving response to a situation.

In this particular situation, that meant that if my intuitive understanding of the situation lead me to drive away, then I could expect everybody else to do exactly the same thing.  If everyone kept driving away, the man could have his leg crushed by a heavy transport vehicle before long.

The danger would increase once darkness fell, and the time was 5 pm.

So, I turned the bike around and drove back the way I had come.  I looked for the man and saw him lying exactly where he had fallen.

No one had stopped to move him.

I was on the other side of the road median (which in Bangalore is a concrete barrier separating the two halves of the road) driving back, when I saw a man on a moped drive right past the drunken man.  It was interesting to see how the moped rider reacted because it was like watching a replay of what I must have done fifteen minutes before.  I saw the moped rider stare at the man lying on the road.  The rider look horrified, swerved to avoid the outstretched leg, and then drove past and away, with not a look backward thereafter.

I drove around the barrier, stopped the motorbike, went up to the fallen man and gently woke him up.  I explained to him slowly that his leg was sticking out in the road, and could get run over.   He seemed to understand, but he couldn’t move.

I told him what I was going to have to do, then put my arms under his arm-pits and pulled him up off the ground and back a few feet away from the road.

When I put him down again, he said, “Thank you, Sir”, and he seemed fully bewildered and surprised.  Then he signed to me that he had had too much to drink and that he had to go to sleep.  He rolled back, his eyes closed and he became oblivious of everything around him once more.  He was just a young lad, probably from Kerala.

When solving simple problems in AI, the kind of things humans solve without thinking, we can use algorithms based on intuition.

Once we start dealing with more complex problems, it appears we need to use more complicated algorithms based on logic, planning and inference.