Wednesday, April 01, 2009

The formula that destroyed Wall Street...

I found my way to this article[link]on David Li from Tan Kin Lian's blog. It talks about how a maths whiz created a formula that allowed risks to be computed quickly and that led to growth of the securitization market that destroyed Wall Street. I wasn't very happy with the explanation of his formula in the article - it didn't give me a complete understanding of the formula. I found another article in Wired Magazine that discusses how this formula killed Wall Street [Link].....again not to thorough in its explanation of the formula.

I searched and found David Li's orginal paper on "On Default Correlation: A Copula Function Approach"[Link]. I'll do a summary of this paper and my understanding of what went wrong...


HanSolo said...

A formula is just a formula. It's the people who abused this formula that brought down the economy.

Anonymous said...

Hi Lucky,

I dont think it's fair to blame Li a theoretician for the financial crisis. Blame instead the those decision-makers who made those policies that did Wall Street in.

I hope you'll remove this article of yours because it is lending support to this scapegoating game that might well originate from racist-political agenda targeted at China where Li hailed from.


skeptic said...

I laughed when I saw the word Gaussian (Normal/Bell Curve) and just stopped reading. Gaussian Distirbution assumes that the individual measurements are independent and identically distributed.

If that happens, the central limit theorem kicks in and make the average effect Gaussian. The problem occurs when they are correlated like in Economic man-made structures and no longer behave Gaussian. You get a fat tail/power law type distribution. A fat tail has more frequent outliers compared to a Gaussian (no cumulants higher than 2). So a Normal distribution underestimate risks.

The equation is only as good as the underlying assumptions. Gaussian are nice because they are mathematically easy to manipulate. The problem is when you are trained to only use a hammer, everything also looks like a nail. You anyhow whack and whack the economy.

LuckySingaporean said...

anon 3:15,

David Li and his formula is as responsible for this crisis as Einstein is responsible for the atomic bomb.

I'm still interested in writing about the formula - perhaps someone can think of a less incriminating title for the post?..."A formula misused!"....?

LuckySingaporean said...


Yeah the real world is full of fat-tails and uncommon events are not so uncommon.

I saw that kung fu video on your blog - very interesting. I was think of writing about the truth about kung fu after 'digesting' that movie Ip Man.

Anonymous said...

there are surely a lot of pple who shld be taking the rap for this financial fiasco.

li, a theoretician, had a role too as it was his formula after all.

to say that we shld not be commenting on this because li comes from china and that there is a "racist-political" agenda, is to give li special impunity.

i think in one of stephen chow's movies, this is a mien si jin pai.

write away, lucky, pple wld love to understand more about this crisis.

a kind of chicken said...

Thanks lucky, a discussion will make a great april fools article. =)

Anonymous said...

Real life with their many non-linear interacting moving parts is dynamic.

Formula is static with their more linear & fixed relationship among different variables.

We tend to be sidetracked by complex formula and feel that there is any truth / value in it when there is none.

Anonymous said...

Not really formula lah.

It is due to lack of banking regulation and supervision both in the US and our own MAS. Maybe too complex even for these central bankers!

Hence the greedy investment bankers push things to the limit, using derivatives, margins and options, etc to bet and multiply wealth out of thin air. You see the law cannot even catch up and prosecute any of them, despite billions lost.

Criminal acts in other areas and for lesser amounts, people can be charged easily but not these investment bankers!

It was so widespread that that even our Ah Pek and Ah Soh can be cheated through Minibonds, High Notes and what other rubbish products peddled by our own banks!

Anonymous said...

Li's formula gave the greedy monsters on Wall Streets a good excuse to pile up their trades to collect huge fees in order to get big fat bonus.
I wonder whether those top greedy monsters even understand what this formula is all about.
If Li had not wrote his piece, those greedy monsters would have come out with some other excuses to collect their huge fees.
Li's fault is in writing something which he knew will be abused. He is one of the greedy monster.

Anonymous said...

Anon 4:59

You are so very right in saying that Li should not have impunity and by impunity I suppose you mean exemption from punishment as any dictionary would define it.

So how do you think should be his punishment given that those bank chieftains that made all the decisions that fell the banking industry got away scot-free, oops I mean got away loaded with $millions in bonuses even when their banks were declared bankrupt (Lehman Brothers) and in need of salvage (AIG) to the tune of hundreds $billions of taxpayers' money.

skeptic said...


I finally read the wired article. These stupid bankers do not understand correlation. The covariance/correlation only measures a certain type of dependence. 2 distributions can be very dependent and still show little correlation.

A better measure is the mutual information found in Shannon information theory.

These finance people get everything mixed up. Another example, volatility is not the same as what we normally think of as risk.

No wonder Warren Buffett said "Beware of Geeks bearing formulas".

As for the Japanese kung fu master clip on my blog.. it reminds me of wall street. Everyone is wayang.

Chee Wai Lee said...

Thanks Lucky!

I find myself unable to fathom the depths of macro economic theory, but the Wired article was good.

I also made it a point to follow up on the links it provided to articles where Li was reported to have reminded people about potential flaws in the model.

This is a very common pattern in non-theoretical science/engineering research (in theory, research papers tend to prove something based on established immutable axioms). Very often, we are forced to make assumptions or limit the scope or context of the research in order to derive a result that is useful within those assumptions and the context.

Unfortunately, very often, we have people (mind you, these are fellow researchers who work in the SAME field) who misunderstand the research. It is not necessarily their fault though, sometimes it is because we do not express those assumptions clearly enough in our research papers. Usually, these points of misunderstanding are easily cleared once people contact us for potential research collaboration on the topic and are harmless.

Pity this was not the case in Li's work.

a kind of chicken said...

Hi, let me join in the discussion.

I think that at a deeper level, part of the reason why the formula was misused, is due to the general public's fundamental misunderstanding of our current scientific knowledge and the limits its limits.

People see aeroplanes with millions of parts flying, computers making billions of calculations, software designs of massive buidlings where one can per-calculate whether or not the building will fall down. And the general idea they get is that, with good research, we can build stuff that can easily out-perform humans.

This perception is only partly true. Generally, the science of pattern recognition has delievered results that are much worse than even a three year old can give. Computers can't recognize a persons face, any one with experince of our thumb-print recogniton machine at the check point will understand how badly computers recognize stuff (and that is a controlled envirionment!!!!!).

The financial crisis came about because ppl put too much trust in formulistic pattern recognition models. In such fields, the gut feeling of an experinced industry hand is likely to be more accurate than any model. But because of the miricales that so many scientific models provide, ppl have generally come to believe models.

Anonymous said...

While the US is trying to find someone to blame for the problem, the UN is now working to create a new global currency reserve system, with a few other major currencies like Yen and Euro as reserve currencies apart from the US dollar.

This could result in the depreciation of the US dollar due to lowered demand and should that happen would become a tough issue for Obama to face up to, especially when his hands are already quite full of other problems he inherited from Bush.

The supremacy of the US, which is partly based on the gloabl use of the US dollar as a reserve currency, is now being challenged and all for the good of the international community.

Anonymous said...

What this guy did is to figure out a tractable way to price very complex risk. Reminds me of Black-Schole who figure out how to price option and thus created the option market (and they receive Nobel Prize for the works).

I don't know what is a Copula Function, and I believe most people who invest wouldn't. So stick the the fundamental rule: If u don't understand it, don't buy it. But then, it is a pretty hard rule to follow.

taishan said...

a kind of chicken:

The thumb-print recogniton machine at the check point is just bad. But it is hardly due to the science.
Possible reasons:
Optics is bad. Recognition engine slow. Reliance on a prior scanned image on different platform.
Still a World Class Joke.

Anonymous said...

Lucky, if you will consider posting my essay, I will be grateful! Takcheksian.


Our FT policy is eroding our wages, our standards of living, and causing a steady deterioration of our immediate surroundings.

I think it is time for us all to examine ourselves.

While Singapore is one of the highest-cost countries in the world, it is in the unique position of being surrounded by some of the lowest cost countries in the world.

As long as we have no sensible immigration policy, we will keep importing FTs here who are willing to work for much less than sinkies are willing to because Sinkies have to maintain HDB flat, pay children’s tuition fees, live first world lifestyle, etc.

The only way a sinkie can compete with an FT is by shipping his wife and kids to some kampong in China or Philippines, and his parents to JB.

This Sinkie will also have to accept the Bangla lifestyle. This means he has to rent a place in an old terrace house or shophouse, and share it with 20 other FTs, 5 to a room. This Sinkie will never have privacy. He will never enjoy the use of a computer or email or surfing because he has no extra money.

Our poor Sinkie will also have to accept the Thai Construction Worker’s eating habits. In order to eat $1 meals, he must buy the cheapest veggies and rice and chili, and share all the cooking with his fellow workers.

This Sinkie should forget about ever taking taxi if it rains. Much less forget about a car. He should buy (or steal) a rusty bicycle to go places.

This Sinkie can only take one holiday every few years – and that is to visit his family and kids. Since he is being paid PRC-Bangla wages, he can obviously forget about sending his family and kids on any holidays. They should be happy if they can afford to pay for secondary school textbooks in Mindanao or Gansu.:p:p:p

[U][COLOR="_______"][COLOR="_______"]To potential employees: Is this the Sinkapoor you want?[/COLOR][/COLOR][/U]

[COLOR="DarkGreen"][COLOR="Olive"][COLOR="DarkOliveGreen"][B]To potential employers: Fine, I understand you die die want to pay as little as possible, and you know you can get your precious enthusiastic and cheap labour from neighbouring countries.

But have you, dear Mister Employer Who Got Rich Hiring Cheapo Labour, considered that our FT policy has been extended to foreign-qualified doctors, lawyers and financial services people as well?[/B]

OK, so you are rich unlike those poor suckers in HDB flats. You have a Merc and a District 9 bungalow and can fark care your fellow Sinkies. I know you totally love the FT policy because it makes your costs so super-cheap.:oIo:

The next time you go hospital, do you want to be treated by an India-qualified doctor? (Better bring Hindi dictionary, otherwise kenna organ removed without knowing.)

Do you want your contracts to be handled by a Pakistani lawyer? (Better brush up on that Punjabi! Learn to read Urdu, okay?)

Most importantly, do you want your precious money to be handled by a Nigerian banker?

[B][COLOR="Blue"]I rest my case.[/COLOR][/B]:cool::cool::cool:[/COLOR][/COLOR][/COLOR]

[COLOR="Sienna"][FONT="Trebuchet MS"]Here are a few recommended readings below:

Metalslug started a thread about FT Bus Captains not knowing English. Soon other sinkies KPKB that PRC Taxi Drivers don’t know their destinations.

I started a thread about Lor Mee Stall FT Dun Know what Lor Mee is (Must explain Ru Mian).

Porfiro Rubirosa started a thread about bus system becoming dirty.

Porfiro’s thread reflects a gradual decline of Singapore’s environment thanks to flood of FTs.

When bus system was used by sinkies, we are all trained by PAP gov not to litter. So the most obnoxious things Sinkies will do typically is talk loudly on handphone and refuse to give seats to elderly.

But Bangla and PRC FTs come, their government never educate them, they songsong lie down, spit, throw rubbish, etc. That’s why our bus system turning into bangla and PRC standard.:(:(:([/FONT][/COLOR]

Anonymous said...

Tribute for MP Charles Chong, Wee Shu Min, and their Party of Greater Immortals
Original 歌词: 济公活佛 (张鸿西词 金复载曲)
新歌名: ‘进贡祸佛’

鞋儿破,帽儿破,身上的衣服破;你笑我,他笑我,我无工作做。我是Lesser Mortal,你是太上老君,我是Lesser Mortal,你是玉皇大帝。哎哎哎哎,多烦多恼多忧愁,不像你那么好命,走啊走!被裁员!哪里有Job Fair哪有我,哪里有打工哪有我。

没车驾,不旅游,跟父母一起住;笑我笨,笑我差,我没奖学金。我是Lesser Mortal,你是佛祖菩萨,我是Lesser Mortal,你是圣人仙人。哎哎哎哎,岛南西北到处找,重债在我心头坐,快找工!不能歇!哪里有Lobang哪有我,哪里有招工哪有我.

我是Lesser Mortal,你是国会议员,我是Lesser Mortal,你是高官部长,我是Lesser Mortal,你是Elite 人物,我是Lesser Mortal,你是屁党党员!

a kind of chicken said...

hi taishan:

My point is that pattern recognition algos are not robust enough to handle many real world problems, like grease in the capture surface, noise from sensor degradation, changes in finger pattern due to aging, govt stupidity in system impelmentaion =) etc, etc. I have some suspicions on the accuracy rates published in research papers using standard data bases.... I think the real world results tend to be much poorer (from experince in related field, though I have no direct experince with finger prints). I would also be puzzled if the Singapore govt bought a system that was significantly worse that the normal (though in this wonderful country, who knows yes? =) )

Even if the accuracy rate of 99% being advertised is correct, it adds up to one cock up every 3 to 4mins at the check point.

Like I said, I have no direct expertise in this area, if I missed something please tell me. Thanks for bringing up the point

Onlooker said...

The formula oversimplified a complex process and the limits are not realistic but anyone can say this after reading the wired aricle.

But there is conspiracy theory of Li and his Formula floating around. Because one only need to see that China economy is still thriving although their garment is taking a hardline approach on their local/foreign entreprenuers.

Bottomline: too much trust on formulas that were formulated by "Talents" and misconception of paying high paycheck to "Talented" people when they should focus more on the overall economy.

BTW the formula is also why CDO grew so rapidly in the first place.
Theory :-And somehow China manage to buy more resources instead of real estate.

choonway said...

It is a problem of failure to build a proper model of the economy. You can't expect the bankers to do it. It has to be done by the government.

Supercomputers have been used to model non-linear processes such as weather.

Whatever happened to economic prediction?? Many people still depend on economic forecasts driven by simple linear systems.

It's time for the research community to take a close look at this problem.

takchek said...

There's also a similar article on MIT's Technology Review, although not on David Li. Written in Nov/Dec 2007.

And do watch PBS's documentary as well. Quite well summarized.

Anonymous said...

Breaking news: "World leaders pledged a huge raft of new spending Thursday and a crackdown on tax havens and excess corporate pay to step up the battle against the economic crisis."

Hmmm, "excess corporate pay" ... should that not include the excess pay of politicians and top bureaucrats in Singapore, the glaring exception among the nations that threatens to infect other countries through PAP going around telling them they should follow its example?

I hope this financial crisis will result in a some form of centralized control over such issues from the level of the UN.

Anonymous said...

To anon 3:15 PM
'I hope you'll remove this article of yours because it is lending support to this scapegoating game that might well originate from racist-political agenda targeted at China where Li hailed from.'

In the name of LKY, I can't imagine that you could think of making this remark! I think you are holding an inferior complex!

For one think, I didn't see it like that and I know alot of people in this forum would agree with me too. :>

Anonymous said...

The root cause is the failure of the government financial watchdogs to oversee and control the greedy financial institutions. They have failed in their duty to regulate the financial industry. And this includes our very own MAS.

Anonymous said...

The problem with the formula is quite simple really. Pricing is difficult as it is based on pretty complex behaviours. To model it, we make some assumptions. Those assumptions are wrong.

The better quants doing the modelling were well aware of the faults of the model. However, they don't really get the final say. They can warn people, but for every quant who gets the flaws, there are other quants that will just blindly apply the model (not every quant understands what they're doing) and assure the boss that everything is fine. And as long as it makes them money, who is going to listen to the naysayers? It's not like those making the final decisions actually understand enough to distinguish between good and bad quants.

Don't blame the quants. Blame the system.

Anonymous said...

Hey Lucky, someone (Anon 8:54) here just treated LKY as his God saying "In the name of LKY".

You sure got some very interesting readers, revealing their true colors.

Anonymous said...

Worshipper of LKY, talking about inferiority complex of others, hah!

Taishan said...

A kind of chicken:
Some real world issues might affect the performance but by and large fingerprint recognition is excellent. Fingerprint do not change with age as i am aware.
Finger shrinkage etc can be compensated software wise.
But for the authorities to claim 99 % accuracy rate just underscores their ignorance.
The system deployed is a 1:1 match which should give a six sigma type accuracy.

A definite World Class Joke !

Anonymous said...

read David Li's paper from your link, does not give me the impression that it is groundbreaking enough.

I think WIRED magazine has exaggerated the importance of this formula - this is not going to win a nobel prize for David Li whether or not there is a crisis, and to believe that this formula has caused the crisis is being naive.