Typed and scraped into Arcturus
From what I see so far Climate Change Research involves a lot of analysis of data sets. I don’t what the ratio of actual measurement to analysis is. I don’t know how often models are tested against experiment or against observed values.
Here’s a scientist concerned about an area of data analysis where there is a great flexibility in choosing models, choosing parameters, choosing methods and with little check against reality. I’ll leave it hidden for a little while where this was published. It’s in a closed access publication which costs about 30 USD for 2 days access, so I’m going to copy the abstract (which I may) and some sentences from the body for which I will claim fair-use. I’ll promise to give the reference later to be fair to the publisher (maybe their sales will increase as a result of my promotion). I’ll hide some key terms (XYZ is a common approach/philosophy) to add to the mystery
A general feeling of disillusionment with XYZ has settled across the modeling community in recent years. Most practitioners seem to agree that XYZ has not fulfilled the expectations set for its ability to predict […]. Among the possible reasons that have been proposed recently for this disappointment are chance correlation, rough response surfaces, incorrect functional forms, and overtraining. Undoubtedly, each of these plays an important role in the lack of predictivity seen in most XYZ models. Likely to be just as important is the role of the fallacy cum hoc ergo propter hoc in the poor prediction seen with many XYZ models. By embracing fallacy along with an over reliance on statistical inference, it may well be that the manner in which XYZ is practiced is more responsible for its lack of success than any other innate cause.
Sound familiar? Here are some more sentences excerpted from the text…
However, not much has truly changed, and most in the field continue to be frustrated and disappointed 
why do XYZ models continue to yield significant prediction errors?
How could it be that we consistently arrive at wrong models? With the near infinite number of [parameters] coupled with incredibly flexible machine learning algorithms, perhaps the question really should be why do we expect anything else. XYZ has devolved into a perfectly practiced art of logical fallacy. Cum hoc ergo propter hoc (with this, therefore because of this) is the logical fallacy in which we assign causality to correlated variables. …
Rarely, if ever, are any designed experiments presented to test or challenge the interpretation of the [parameters]. Occasionally, the model will be tested against a set of [data] unmeasured during the development of the model. …
In short, XYZ disappoints because we have largely exchanged the tools of the scientific method in favor of a statistical sledgehammer. Statistical methodologies should be a tool of XYZ but instead have often replaced the craftsman tools of our trade
rational thought, controlled experiments, and personal observation.
With such an infinite array of descriptions possible, each of which can be coupled with any of a myriad of statistical methods, the number of equivalent solutions is typically fairly substantial. Each of these equivalent solutions, however, represents a hypothesis regarding the underlying [scientific] phenomenon. It may be that each largely encodes the same basic hypothesis but only in subtly different ways. Alternatively, it may be that many of the hypotheses are distinctly different from one another in a meaningful, perhaps unclear, physical way. …
XYZ suffers from the number and complexity of hypotheses that modern computing can generate. The lack of interpretability of many [parameters] only further confounds XYZ. We can generate so many hypotheses, … that the process of careful hypothesis testing so critical to scientific understanding has been circumvented in favor of blind validation tests with low resulting information content. XYZ disappoints so often, not only because the response surface is not smooth but because we have embraced the fallacy that correlation begets causation. By not following through with careful, designed, hypothesis testing we have allowed scientific thinking to be co-opted by statistics and arbitrarily defined fitness functions. Statistics must serve science as a tool; statistics cannot replace scientific rationality, experimental design, and personal observation.