So to put it another way, these 'scientists' purposely made their data unverifiable and thus non-scientific. IMO any adjusted or modified data is unscientific without the original data set and how and why the data was modified the way it was.
If you adjust data to normalise it due to the differing global climatic effects of El Nino or La Nina then that's perfectly reasonable, if someone can corroborate what you did and if you did it right. However with no original data set, and apparently no mention of how or why the data was modified the way it was leaves it completely unverifiable and might as well be entirely fabricated.
Not that I disagree, but there are conceivable cases were it seems natural to not keep the "base data" around. For example consider a classical thermometer: temperature data is collected by reading the height of quicksilver in a pipe and transforming it into a temperature according to a device specific scale. I don't think many people keep the actual height readings around, most would probably just keep the resulting temperature.
Not that I know that the same applies to the data in question here, just saying. I still wait for the full story.
there are conceivable cases were it seems natural to not keep the "base data" around.
Your example does not demonstrate this point. By taking the temperature readings in degrees C, you, in fact, are taking the "base data". Of course, you also should record the equipment that you are using to take that data. The height of mercury in your thermometer can be determined by the the brand of thermometer (including the Manufacturer and Model #), and the temperature reading given. Most scientific peer-reviewed research articles are very specific about the equipment being used.
"By taking the temperature readings in degrees C, you, in fact, are taking the "base data"."
I don't see how? It is a matter of how you define "the base of the data". My point is that there is a calculation on the measurement, and only the result of the calculation is recorded. Hence the base data is discarded.
We don't know what calculations were done on the climate date in question here. But it seems at least possible that the calculation is also trivial (for climate researchers anyway, could be some standard procedure), just as to us converting heights to degrees is trivial.
Recording the type of thermometer is another matter - likewise the climate researchers will probably record where their measurements were taken, which implies the equipment used.
Edit: of course if the calculations are reversible is another issue. I see now that is the point you made with the thermometer (calculate height from make and degrees). Well, it was just an example, and we don't even know yet if the calculations in question here are not reversible. I suppose if they calculated an average they are not. But I also think there are probably many cases where a measurement is just taking an average (probably even the height of quicksilver is an average in a way...).
we don't even know yet if the calculations in question here are not reversible.
??? If the calculations are reversible, there would be no argument. Everyone would know the base data.
I think you are kind of pushing it with your quicksilver example; I don't see how taking a single measurement is "averaging". If you mean to say that there is some built-in variance - well, that's true. But the fact is the variance itself can be established by other scientists if they know the model of thermometer that is being used. This is the essence of repeatability!
Now, imagine that a scientist had many temperature readings, and then they did some kind of complex calculation which involved regression analysis, statistical modeling, etc. These are the types of calculations that could easily have errors. Without the initial recordings, the results are unverifiable. That doesn't make them necessarily useless, but I wouldn't lean on those kinds of results.
I tend to think good faith is a nice assumption to have, but it should be rebuttable. For example, if someone were to say "I'd rather destroy my data than let it fall to a Freedom of Infomation Act request", and then they were served with a Freedom of Information Act request, and then the data were to be suddenly missing...
Professor Jones: Just sent loads of station data to Scott. Make sure he documents everything better this time ! And don’t leave stuff lying around on ftp sites - you never know who is trawling them. The two MMs [Patrick notes: he is referring to McIntyre and McKitrick] have been after the CRU station data for years. If they ever hear there is a Freedom of Information Act now in the UK, I think I’ll delete the file rather than send to anyone.
Does your similar act in the US force you to respond to enquiries within 20 days? - our does ! The UK works on precedents, so the first request will test it. We also have a data protection act, which I will hide behind.
Well, if a group has your original source data, they can try to reproduce your results and/or challenge your process. I gather that the results are not likely to stand up to scrutiny.
I concur, the only reason to destroy the original source data is if you know your work won't stand up to scrutiny and will discredit your work. Irony here is that now it isn't a matter of if there work would or wouldn't stand up to scrutiny, now it's a matter of ethics and they're clearly discrediting themselves by their severe lack of ethical behaviour.
It would be harder for the scientists to justify making it up to themselves. I assume none of them set out to do bad science when they started investigating climate.
Sorry I don't think this makes sense. Personally I'll just wait to see how the story unfolds.
This makes me glad again that there is some kind of legal system in most civilized countries. So many people seem to be ready to form a lynch mob at short notice.
If that were the case, there would be no need to delete it; the lawyers could decide whether the FOI request or the license won.
Of course, if it turns out that it was illegal for them to accept the data under that license in the first place, then it might be more understandable.
What if they simply felt they didn't have the time to deal with "the MMs", who would most likely not care about the law (also being from a completely different country). Newspapers tend to simply publish stuff they acquire, no matter if they came by it by legal means or not.
Not wanting to deal with the MMs does not imply guilty, just as not wanting to discuss moot points with creationists does not imply evolution theory is wrong. At some point maybe you decide your energy is better spent elsewhere.
Just playing devils advocate... I still feel there is too much context missing.
If a creationist wants access to your data you give it to them; you don't have to debate them, but there is no reason not to keep this data on your ftp...
...and if the FOI won you wouldn't be able to get data anymore.
Under that license, right.
I don't understand what you're getting at with your second paragraph.
What I mean is that if government documents are subject to FOI requests, then it seems possible that there's a law about accepting data under a license which would conflict with that. I don't know that there is, but it seems possible. That would be an understandable reason having no bearing on the climate debate to delete rather than provide the data.
Normally, I'd agree with you. There seems to be more incompetence than malice, in general.
With technology as it is and science as it is and has been for hundreds of years, this seems like an extraordinary goof. I could see a goof being losing a block of data, but all of it? Why was destruction of this data even an option or consideration? How expensive and difficult is it to keep it? I suspect there are numerous other organizations that would store it if they were given copies of it.
"Purposely" might be a bit extreme but it was more than a simple goof.
The thought that someone couldn't walk down to the local store and buy an external hard drive(s) to store this data is unlikely. Heck, a general call out to the community to find storage would work (ask google, they would do it and index it).
No, I believe malice is really the only explanation. Raw data allows reproducibility and intelligent debate.
The data in question dates from the 1980s. You couldn't "walk down to the local store and buy an external hard drive", and if you could a gigabyte of storage would cost you tens of thousands of pounds. Google didn't exist.
FTA: "Jones was not in charge of the CRU when the data were thrown away in the 1980s". What part of that makes it sound like this data was lost more recently than the 1980s?
"Their findings are one of the main pieces of evidence used by the Intergovernmental Panel on Climate Change, which says global warming is a threat to humanity."
So basically there is no evidence the earth is warming? Because evidence based on unreleased data is worthless.
If I can't reproduce your work, you are a crackpot (isn't that the definition of junk science?) I can't reproduce their work, so they are crackpots. (Or deliberate frauds.)
Did they even record their methodology on how they adjusted the numbers? Maybe we can undo that methodology?
If they didn't record even that, then they are incompetent. And I see no reason to pay attention to incompetents.
What if someone made a simple arithmetic error? Without the raw data, and the methodology of adjustment how could we ever know?
Their conclusions are worthless, and unless there are others (are there?), then there is no evidence the earth is warming.
The earth is definitely warming. We are exiting an ice-age. There has been a conscious effort, I believe, to equate global warming with man-made global warming. They are not the same thing, even if used interchangeably by the media.
The "hockey stick" graph appears to shows that the earth is warming faster now than it was earlier. It's this "steeper slope" that we're debating, and that may have been manipulated. The general public, however, has had the debate obfuscated for them into "is it warming, or isn't it?"
"So basically there is no evidence the earth is warming? Because evidence based on unreleased data is worthless."
Unfortunately this is true. As someone who has never doubted that global warming was man made (yes, those last leaked emails were private conversations and held no smoking gun whatsoever. People were reading into one term used ten years ago to sex up a graph), this is embarrassing. This, more than any other thing has made me reconsider my views.
The problem is that glaciers and polar ice are melting at unprecedented rates, this is unarguable. The earth is getting warming in those areas and it is effecting the sea levels and biodiversity of the planet. How do we tell if it's man made? What can we do to stop it? Remember this doesn't prove that global warming isn't real.
What happened here? They threw away their copy of the raw data. That data still exists (they gathered it from public repositories), and the methods of the paper describe what they did. You could reconstruct the results.
Essentially, the "skeptics" are getting worked up over what amounts to throwing away the lab notebooks for a paper published two decades ago. Even if you assume malfeasance on the part of these authors (an assumption that does not appear to be justified), you'd have to disregard the results of thousands of other papers to leap to the (wild) conclusion that "there is no evidence the earth is warming."
But between the raw data and the set of data is a correction. Without being able to show what that correction is, all the models based off the second set of values is not valid science.
So you have to go back and recollect the info from all these sources (not an insignificant undertaking, though mostly do-able), actually give a reason for the "corrections", and then create models that fit that data, and then issue new predictions.
Without starting over from raw data and following all the way to the conclusions reached about man made global warming there is no scientific proof that man causes global warming. None of it is valid until you can show how that "correction" was made. And the loss of this data means they will never 100% be able to show that.
Out of curiosity, why do you put the word skeptic in quotes? Do you imply that my skepticism is not genuine? Believe me, it's genuine. I'm no "skeptic", I'm a skeptic, proud and true.
If what you say is true, that's really good. It's probably not true, judging from the comments in the source code, where the guy laments on his inability to verify the very existence of some of these stations. That's not to say the stations he wanted, didn't, at some point, probably exist, it's just that time passes and the world moves on, people retire, and filing cabinets go to the salvation army.
I put the word skeptic in quotes to denote the difference between scientists (who are already quite skeptical), and the people who employ fallacious "skeptical" arguments to advance a political agenda. I don't know to which group you belong (but then, I wasn't referring to you in my comment).
Anyone who has spent any time around scientists knows them to be a conservative and naturally skeptical group of people. The "scientific establishment" -- if there is such a thing -- is powerfully opposed to new ideas. The climate change debate was a controversial new idea thirty years ago, but after three decades of research and debate, it's been as close to universally accepted as anything ever is in the world of science. The people who would have you believe that climate change is a conspiracy amongst rogue scientists either do not understand how science works, or are purposefully dissembling to advance their cause.
You're being a bit kind to yourself, don't you think? You're skeptical, as are those you agree with, but those you don't agree with are "skeptical".
The climate change debate was a controversial new idea thirty years ago
See, this revisionist stuff drives me crazy. 30 years ago, it was called "global cooling". Now, sophisticated punters call it "climate change", instead of the less sophisticated and more embarrassing flip-flop of "global warming". http://www.denisdutton.com/cooling_world.htm
Just call it global warming and admit that 30 years ago, climatologists were wrong. Don't try to obfuscate their lack of perfection (nobody expects perfection anyway) with this "climate change" nonsense of "we knew it all along, see, we predicted climate change 30 years ago!"
"You're being a bit kind to yourself, don't you think? You're skeptical, as are those you agree with, but those you don't agree with are 'skeptical'."
No, not really. There's a difference between being skeptical and using fallacies and rhetoric to cause confusion. I disagree with people who do the latter in the guise of "skepticism".
"See, this revisionist stuff drives me crazy. 30 years ago, it was called "global cooling". Now, sophisticated punters call it "climate change", instead of the less sophisticated and more embarrassing flip-flop of 'global warming'."
What part of "the climate change debate was a controversial new idea" don't you understand? When presented the theory, scientists didn't believe it. They debated many different theories -- global cooling, global warming, temperature cycles, etc. -- and eventually, the data led them to a theory. It's not "revisionist" when new data leads you to change your hypothesis. It's science.
"So basically there is no evidence the earth is warming?"
If Darwin's original notebooks were lost, there would not cease to be evidence for evolution. There are many thousands of papers on climate change. A loss of the raw data sets for one paper does not change the conclusions drawn by the others.
No, this is the set of raw data beneath the corrected data all the papers are based on. It isn't Darwin's notebook, it is the set of species that Darwin studied. If you removed all the fossils and a lot of the species and tried to prove evolution you'd be in the same ball park as what is going on here.
"No, this is the set of raw data beneath the corrected data all the papers are based on."
No, it's not. First, no reputable journal allows re-publication of the same data set -- after thousands of publications, it's just silly to suggest that it's all based on the same data. Second, even if you don't believe that other scientists would prevent the re-publication of the same data, one only has to peruse the IPCC report to see that there's a lot more than just the "hockey stick" graph and computer models.
This is turning into the biggest science story of the year, and for weeks most of the MSM refused to cover it.
I'm not trying to allege bias, just making the observation that stories that don't fit into pre-existing media narratives take a lot of push to get out there. Usually it's the fringe media that keeps kicking and kicking until somebody finally picks it up. If I remember correctly, the National Enquirer was running stories of John Edwards' love child for months before anybody else would pick it up. (And love or hate Edwards, it was a real story that deserved national attention)
I wonder if blogging has changed any of this? Sure would be great to read a study on how this type of anti-narrative story breaks today versus 20 years ago.
I'll allege bias for you, then. There is no conceivable way that you could produce documentary evidence of a Republican saying, in as many words, "Destroy this so that it can't get used for a Freedom of Information (Act) request" and not have that be the front page of the NYT the next day. I don't care if he was the Deputy Undersecretary of the Department of Who Gives A Care, that sort of thing is normally blood in the water. It attracts front page attention, massive follow-up coverage, and Pulitzer prizes. When it targets the right people.
You mean like the George W. Bush White house email scandal? It got MSM some coverage, not front page, and certainly no awards. In fact, it was only noteworthy because it was a meta-scandal that lived inside the far bigger one of the firing of US attorneys, a story that was basically set ablaze not by the media but by the moronic testimony of Alberto Gonzales and Monica Goodling before congress. And it was never wrapped-up or followed-up on beyond diggings from the progressive blogosphere because after AGAG was hung out of dry the media deemed its job complete.
Anyway, your comparison is apples and oranges. Political scandals will always carry more weight in the public eye than science ones, if for no other reason than the fact that their hands are directly on the levers of power, whereas scientists just use a (relatively) small amount of taxpayer money compared to what is doled out in the halls of political power. And their impact on our daily lives is significantly less direct than even a mid-level staffer.
Do you also not see this as a political scandal given the policy measures that are being proposed in response to the conclusions some of these people make? These political measures being proposed have a real impact (financial and social) that far exceeds any war in recent years.
Personally, I suspect that the AGW thesis probably stands given that it's more than this handful of apparently corrupt scientists. That being said, where does the science stand if we take out these suspect models and the ensuing research that was based on them? Given these scandals, why is there not more of a movement to make these climate models and data in effect open source?
The level of secrecy, defensiveness and arrogance as shown by example in the Scientific American is especially troubling when you realize that practically all this research was publicly funded. Why are there non disclosure agreements on the data to begin with? (A real question - does anyone know?)
Personally, I suspect that the AGW thesis probably stands given that it's more than this handful of apparently corrupt scientists.
I think there is a risk there. Much of the work in the field is reworking and resplicing old results -- for a graphic illustration of this, see McIntyre's graph of which tree-ring series are used in which papers.
Primary data are hard and expensive to collect. Computer munging on existing data is, relatively, cheap to produce -- you only need grad students, pfft, they work for ramen. So the papers tend to gradually build up more series, more stats, more regressions on the same existing data sources. Then you get into computer modeling, which takes the assumptions gleaned from the 43rd reworking of the source data and combines it with the 27th reworking of other source data to predict that in scenario N where X assumption holds...
What if we're building a skyscraper on a foundation of rotten timber?
P.S. That being said, where does the science stand if we take out these suspect models and the ensuing research that was based on them?
I think the scientists are very, very reluctant to do that. Look at the amount of remixing that has to be done for these papers. If you were to draw a graph of all the results in this field, connect all papers via citations (or a less promiscuous method of your choice, such as "Did I use their numbers for a calculation of importance?"), color one early result in the field red, and then floodfill -- I think it is quite likely that the whole discipline goes red. This is presumably why, when McIntyre et al have demonstrated problems in earlier research, they always get fobbed off with "Pfft, yeah, maybe, but we had 200 data sources and that was only two or three of them."
P.P.S. There is no incentive for a scientist to go back and check their own papers for errors introduced by a paper or data source they relied on suddenly going bad. Heads they spent a lot of work and their paper survives scrutiny: good news, but defending results from 1993 didn't get them a grant now did it. Tails, they find an error, can't correct it, and admit so. BAM, loss of professional reputation, funding, etc. And then they get to the really sticky business: calling up all their friends who cited them for the "Bob, got a bit of bad news for you..." conversation.
If what you're saying is true, this is truly frightening. It's surreal looking at Google News and what hits the headlines is the drumbeat towards some type of deal at Copenhagen with this scandal hardly being discussed at all. It's difficult to reconcile the almost dismissive attitude of the general media and this scandal.
What perhaps frightens/annoys me the most is thinking that perhaps politically, some of these people are so invested in AGW that any new data is irrelevant so we will end up spending hundreds of billions if not trillions trying to deal with a problem that may end up not even to exist. Personally I think a lot of the changes in technology and energy usage favor reductions in greenhouse emissions and this should happen regardless, but some of these other massive costs...
I guess we'll see in coming days which narrative wins out - that nothing is done and is as irrelevant as the media thinks or the mainstream media begins covering the story grudgingly only because competing forces in governments (ie congressional investigations / investigations) make it impossible not to.
There are several reasons why the media is reluctant to cover it.
1) The emails were acquired through an illegal hack, which makes them a highly tainted source for journalists who aren't pushing an agenda. Naturally journalists who are pushing the opposite agenda will ignore it, as those who who are skeptics will readily jump on them.
2) This leak was initially trumpeted as a the end of AGW theory, and has since turned out to be nothing more than scientists being douchey and unprofessional wrt their critics. Smoke, but no fire. There's an important lesson in marketing here: Make sure you set appropriate consumer expectations.
3) (US only) It's technically foreign news, and the American media generally doesn't do foreign news unless it pertains to our various wars or some 3rd world dictator says something bad about America.
Long story short, this would be covered in the MSM if: 1) The emails were acquired through a whistleblower with legal protection 2) it wasn't over-sold in the blogosphere the moment it broke 3) US Media actually gave a crap about what happens outside of America's borders that doesn't involve a military escapade.
The emails were acquired through an illegal hack...
Yes, a crime was committed, but you can't put the genie back in the bottle. There's precedent, too. From Wikipedia: The Pentagon Papers... were a top-secret United States Department of Defense history of the United States' involvement in Vietnam... [They] first surfaced on the front page of the New York Times in 1971.
...
This leak was initially trumpeted as the end of AGW theory, and has since turned out to be scientists being douchey and unprofessional wrt their critics...
But the facts remain: The data are gone. The wingnuts knew it. The media didn't report on it, in large part because the only people talking about it were wingnuts. Doesn't that make you feel a little uneasy?
I agree - even if the emails were illegally obtained, one need only look back at all those national security leaks over the past few years to see the glaring double standard.
That being said, I think there's a questionable assumption being made that these in fact are hacked illegally obtained emails. From CBS: http://www.cbsnews.com/blogs/2009/11/24/taking_liberties/ent... - "It's not clear how the files were leaked. One theory says that a malicious hacker slipped into East Anglia's network and snatched thousands of documents. Another says that the files had already been assembled in response to a Freedom of Information request and, immediately after it was denied, a whistleblower decided to disclose them. (Lending credence to that theory is the fact that no personal e-mail messages unrelated to climate change appear to have been leaked.)"
One reason that the media are reluctant to cover it is that they do not know what is normal in science. I remember data crunches in speech recognition research in the late 80's. We would run out of disk space and have to delete stuff. Usually it would be the value added data. If we wanted it back we would re-run the processing scripts to recreate it from the raw data. Sometimes the raw data would be archived on tape and deleted from the disks. If you needed to regenerate the value added data you had to get one of the system administrators to retrieve the raw data from the tape archive before you could start.
Two take home messages. One: we never lost raw data. Two: we never lost the scripts for applying corrections and were always in the position of being able to replicate our work. Shortage of disk space meant that we sometimes had to do this, so it had to work.
I suspect that journalists are slow to realise that deleting the raw data is a very odd thing to do. You delete your data when you have no students, have lost faith in your research and are committed to giving up and leaving the field. Do you delete your raw data if you are battling to save the planet? No.
In answer to your point 3, hasn't Obama just set energy policy, which will affect the living expenses of every American, on promises he made in Denmark? This is a story of great relevance to the domestic American news consumer.
> This is a story of great relevance to the domestic American news consumer.
Which means it probably is not hacker news.
(Also, consider whether we want to see both sides of this issue posting their stories - if the anti-warming folks post a lot of these, the other guys are sure to want to see their side all over the site as well. I think it belongs elsewhere)
I posted this article precisely because I'm not one of "the anti-warming folks". The tribal aspect of this debate somehow manages to be both boring and corrosive, so I ignore it when I can. This article is special because its facts aren't in dispute, and yet still tend to cast doubt on the consensus view that I, at least, thought was the truth.
Have you always taken the position that global warming is not hacker news? Or has something changed so that is was hacker news before and isn't any longer?
The fact that a topic is treated by some people as a religious dispute doesn't mean articles about it can't be interesting. By flagging stuff without even reading it, you're behaving as mindlessly as the partisans on each side.
I agree that the articles are sometimes interesting, but they're frequently just re-hashes, posted as platforms for ranting comment threads by the same handful of people.
It would be nice if there was a flag-like mechanism for the community to turn off comments on articles like this one. It seems that if the karma incentive was reduced on political articles, people would have less incentive to post the thoughtless ones.
It seems that if the karma incentive was reduced on political articles, people would have less incentive to post the thoughtless ones.
This is backwards. Usenet had political flamewars despite having no karma.
People post to political threads because it makes them feel good to stick up for their particular causes. Emotions that reward us for signaling our allegiances and general loyalty are one of the cognitive adaptations that allow our species to solve the Prisoner's Dilemma to benefit from cooperation, specialization, and trade.
I think 'mindless' is unfair. I flag them because I have seen that these articles/topics regularly degrade into junk and so I do not want to see them on this site. So I have a strategy based on my own observations of how sites like this work.
Bravo. HN would be a better site if more people did that. Particularly on Sundays.
I have a few bête noir topics I try to moderate away, too, but I tend to draw the line at explicit endorsements of one side over another in a tribal conflict. I can see how it would make sense to just excise tribal issues and have done, though.
From a tree-shaded plateau facing Mt. Kenya, the worshipers gaze anxiously at its melting ice cap and wonder: Is God dead?...The scientific community is divided over the causes of melting ice caps in Africa. But many experts believe the retreating snow on Mt. Kenya is one of the continent's clearest examples of climate change and global warming.
There are two layers of bias here, one obvious and one subtle. Obviously, "The scientific community is divided over the causes of melting ice caps in Africa" means that some experts think the melting is due to global warming, and some experts think it's not. Long-time observers have come to expect the Times (and the MSM generally) to choose the pro-global warming interpretation every time.
The subtle bias is that the Times wrote this article at all.
finally, a reasonable discussion of this issue. For over two years I've heard rumblings from my sience-y friends that GW was being overhyped. I don't know if anyone had any specific data, but there is some sort of pattern-recognition thing where you just smell that something is fishy.
None of this means that human-caused GW isn't a fact -- what it means is that proving it and quantifying it are turning out to be very difficult. What the email scam shows is that people on the 'right' side of the debate (ie proponents of human-caused change) at some point went over a bit to the dark side, and decided that "failure is not an option" -- ends justify means, etc.
This saddens me because I don't want 'my' side (liberal, rational, forward-thinking) to be guilty of the same anti-science bullshit that GWB and crew were. If science becomes completely politicized, it will cease to be science. Where will be then?
It would actually be very interesting to get a graph of all the papers that are based on the altered data. I get the feeling it would be a significant percentage, and not having the raw data would make things painful.
humm..... citeseer with evidence / result tracking....
So all the weather stations in the world sent their data to CRU, which then deleted it. Because the data dates 100s of years back, it was not in digital form, so they never copied it. All the little field researchers just trusted the big momma CRU to keep their data for them, and now it is all lost. Climate science is dead, anti-warming campaigners have won.
This article is mere bullshit. CRU never dumped their own data. They just dumped their own copy of the data they got from other sources, because anyway they could not give those around. The original are safe where they belong.
Also, this is hardly news related to the hack considered that this information was put online on August 11 2009.
If you adjust data to normalise it due to the differing global climatic effects of El Nino or La Nina then that's perfectly reasonable, if someone can corroborate what you did and if you did it right. However with no original data set, and apparently no mention of how or why the data was modified the way it was leaves it completely unverifiable and might as well be entirely fabricated.