I'd like to add a bit of insider info on "models", per se. I started writing this as a comment on TAE, but it kept growing-
---------------------
The advent of computers got plenty of academics, in all disciplines, excited. Hey, we can use Big Math! And arrive at New Truths! Look at how many things we can throw into the soup, and still calculate! Man, you can correlate everything!
The reality, of course, remains the old bit about "Garbage In, Garbage Out." And the increasing complexity of "models" has made it both much more likely that there will be a bit of garbage in the model somewhere, and hugely less likely that anyone will ever find and remove it. Who actually examines the math- bit by bit?
The reality, of course, remains the old bit about "Garbage In, Garbage Out." And the increasing complexity of "models" has made it both much more likely that there will be a bit of garbage in the model somewhere, and hugely less likely that anyone will ever find and remove it. Who actually examines the math- bit by bit?
In truth; virtually no one, except the author, ever checks the math; not even the academic reviewers will put in the hours, days, necessary to truly proofread these monsters.
I've published some calculations on carbon cycle stuff- with surprising conclusions; and as far as I can tell, no one has ever even checked my multiplication. I got two kinds of reactions- with no checking: "wow, cool!"; and "I don't believe it." Check the math? Check basic assumptions? nah- why?
But "models" still carry a great cachet of believability- in the Congressional hearing, "but the model says..." will trump any expert opinion to the contrary.
But "models" still carry a great cachet of believability- in the Congressional hearing, "but the model says..." will trump any expert opinion to the contrary.
Math does not lie- we so desperately want to believe.
Disraeli nailed it long ago; "There are three kinds of lies: lies, damned lies, and statisics." I greatly fear that models may constitute a 4th kind; with a power even greater than statistics.
This first became obvious to me when I acquired my first Macintosh computer (1984, 128K). MacPaint was such a breakthrough program; absolutely astonishing. I was hopeless as an artist with pen and ink; but here, the computer will keep the lines straight; and- you can go back and fix anything, pixel by pixel if necessary.
One of the first things I did was make a diagram of a biological/genetic project I was working on at the time. Took me half a day (no time at all, compared to finding an artist, teaching them what I wanted it to show, etc.) and looked totally professional when done.
And totally convincing. Wow- this looks like- The Truth. Neat, clean, and logical.
And I knew damn well it was NOT a proven hypothesis; there were plenty of doubts possible, and remaining; but I'd made the diagram up as an argument, and they didn't show. Everybody who looked at it was convinced. And on totally inadequate information.
I was so impressed- and frightened- by the power of the Pretty Diagram that I immediately, made up another:
Just clean up the presentation- and it will look like the absolute truth.
My insider's info: the mathematical models used to describe the dynamics of salmon populations- are junk. And most relevant fisheries biologists, and plenty of ecologists, know it.
Primary evidence- uh, they don't work; the salmon fishery in California is closed, this year, since the number of returning fish dropped to 5% of normal. Catastrophic. In fact, the mathematical ecologists have known for decades that the original salmon models are junk.
Once launched, models take on a life of their own. A major purpose for them is to communicate with lawmakers. Having acquainted the lawmakers with Model A- it's almost impossible for an agency to go back to the legislature the next year, and say "oh, incidentally, we've discovered that Model A is total crap."
So what you do is refine Model A. The problem being, Model A still contains crap; and arguably, crap, multiplied by anything, is still crap.
Models, regardless of the field they are applied to; whether economics, ecology, or global weather patterns- are like all technologies. They have no intrinsic ethics; they are true only to the extent their inputs are accurate, and honest. And that's assuming that the model has any chance of reflecting reality in the first place; by no means a given.
They are a particularly powerful tool for liars- because there are so few people who are qualified to refute them. And they are very seductive even for those with the best of intentions. Young scientists get sucked into the worlds of modeling constantly- becoming enamored of the power, and promise- and gradually becoming apologists for the entire process. And; well; this model right here- which they helped write...
One more technology where the potential for abuse and misuse is immense- mostly unrecognized- and not policed.
The salmon models are blindingly simple, compared to macro-economic models.
One somewhat objective way of looking at it:
Salmon models can be tested, and possibly refined. Did the predictions turn out true-ish?
Global warming models also- can be tested.
Economics models- uh- we're immediately out of the realm of hard numbers, into the worlds of fantasy and opinion. Are the premises true? Opinion. Numbers true? No physicist, or ecologist, would accept the kinds of measurements economists claim as accurate. There are simply too many assumptions, ifs, maybes, blatant guesses, and we hopes in all those economic data. And lies.
Ah, but the model shows- this is true.
And I can make a model to show anything I want.
-----------------------------------------------------------------------
Ok. So why, I hear you cry, is this discussion of any value to me? I come here for information on how to build potty houses, for crying out loud. And the occasional bit of stunningly wonderful poetry.
One of the basic problems we face is the increasingly difficult task of figuring out who is telling us the truth about our world, and its possible futures.
There's really no way of escaping the need to be able to judge your source's veracity. You need to know how to guess; is this person lying? a fool? misinformed?
It's not easy; and is not going to get any easier. Perhaps it will be useful to realize- if they're basing their claims on some complex and unexaminable "model" - and they're quite vehement about how this proves their claim- beware, oh, beware.
------------------------------------
Update, 6/16; an article in the Washington Post today contains this bit of comment, regarding the mortgage bubble collapse- " 'Nobody had models for that,' said David E. Zimmer, then one of the executives at People's Choice, a subprime lender based in Irvine. 'Nobody had predicted people going into default in their first three mortgage payments.' "
So- models- that did not work- were being used to guide actions. Result? Millions of foreclosures- suicides- etc, etc.
---------------------------------------
Update 9/19/08 - mid panic- An article yesterday in the NYT, on "How Wall Sreet Lied To Its Computers" - almost catches up to us-
In my grad studies, I read an article by Karplus. He touched on the same issues, and proposed a "spectrum" of models from black box to white box. Please click here to read pg 50, and view the diagram.
ReplyDeleteEcological models in the open system of the real world tend to be on the "dark grey" to "black" section of the spectrum.
Certainly with the addition of human nature to the equation, any social or economic model is even deeper black.
Excellent post, and I am glad to see that you brought the topic up.
baloghblog- interesting diagram there, and pretty accurate, I think though I have suspicions that economics may be further towards the black end even than politics. Glad somebody is doing some serious analysis. Now we need to make this kind of thing required reading for elected officials... :-)
ReplyDeleteI hope that your basement water hazard has begun to percolate. I studied as a scientist and was exposed to statistics, graphs, even double blind studies and lab measurements, theories and supposedly proven therapies and courses of treatment, but now, almost 40 years later little of that fractured information is useful or legitimate.
ReplyDeleteEven more salient is the beautiful and perfect construct available to practitioners in the media field, especially moving image media, a field I also pursued and made a living in over a ten year period, long ago.
Combine the two, say, in a presentation or refutation of Global Warming and finding the truth is supplanted by deciding which politic to lean toward. As a person with a life long scientific outlook and a complete lack of faith in almost anything I can't personally prove {which is very little, sadly} I believe none of the claims of left or right, or any of the dichotomies of opinion that infect modern life.
My view is way beyond skepticism.
Even those with advanced intelligence and ability and education, working as honestly as possible, make and promulgate great error in the science field and in all other fields. Error comes out of the difficulty of eliminating the factors of presupposition and basic human blindness to the obvious. The examples are infinite.
Remember when it was not yet known that ulcers were caused by a pylobacteria? That smoking caused lung cancer? That seat belts were a really good idea? And so on. Thanks for an excellent post and keep treading water.
I find this very interesting as a math teacher. I was recently discussing the differences between reading in multiple disciplines with one of my colleagues. In history or English, the author has to be taken into consideration for perspective and to identify possible bias. In math, the author is seldom mentioned, let alone analyzed. The difference between high school algebra and economics is so huge, but I wonder how many people just apply their "math reading skills" to anything with numbers and assume it will be enough.
ReplyDeletea blog essay I thought you would like:
ReplyDeletehttp://dharmajoint.blogspot.com/2008/06/zen-and-art-of-monetary-maintenance.html
"Zen and the Art of Monetary Maintenance"
--sgl
Honest to pete most of the "information" out there is crap. I sometimes wonder if it's worth reading it at all. I guess the best thing is to learn what works for my family and just let the rest go.
ReplyDeleteOn a side note I have managed to convince my husband that we don't need most of our major appliances. I plan to do an ice box instead of a frig but I dearly love my freezer. Do you have a freezer or do you use an ice house?