ROBUSTNESS LÀ GÌ

  -  

I am currently a doctoral student in economics in France, I’ve been reading your blog for awhile và I have this question that’s bugging me.

Bạn đang xem: Robustness là gì

I often go lớn seminars where speakers present their viviancosmetics.vnistical evidence for various theses. I was wondering if you could shed light on robustness checks, what is their liên kết with replicability? I ask this because robustness checks are always just mentioned as a side note to presentations (yes we did a robustness check và it still works!). Is there any theory on what percent of results should pass the robustness check? Is it not suspicious that I’ve never heard anybody toàn thân say that their results bởi NOT pass a check? Is this selection bias? is there something shady going on? or is there no reason to lớn think that a proportion of the checks will fail?

Good question. Robustness checks can serve sầu different goals:

1. The official reason, as it were, for a robustness kiểm tra, is to lớn see how your conclusions change when your assumptions change. From a Bayesian perspective there’s not a huge need for this—to lớn the extent that you have sầu important uncertainty in your assumptions you should incorporate this inkhổng lồ your model—but, sure, at the kết thúc of the day there are always some data-analysis choices so it can make sense to consider other branches of the multiverse.

2. But the usual reason for a robustness kiểm tra, I think, is to lớn demonstrate that your main analysis is OK. This sort of robustness check—& I’ve done it too—has some real problems. It’s typically performed under the assumption that whatever you’re doing is just fine, and the audience for the robustness kiểm tra includes the journal editor, referees, và anyone else out there who might be skeptical of your claims.

Sometimes this makes sense. For example, maybe you have discrete data with many categories, you fit using a continuous regression mã sản phẩm which makes your analysis easier to lớn persize, more flexible, và also easier to lớn understvà and explain—& then it makes sense khổng lồ vày a robustness check, re-fitting using ordered logit, just to check that nothing changes much.

Other times, though, I suspect that robustness checks lull people into lớn a false sense of you-know-what. It’s a bit of the Armstrong principle, actually: You vì the robustness kiểm tra lớn shut up the damn reviewers, you have sầu every motivation for the robustness check khổng lồ show that your result persists . . . & so, guess what? You bởi the robustness kiểm tra và you find that your result persists. Not much is really learned from such an exercise.

As Uri Simonson wrote:

Robustness checks involve reporting alternative specifications that demo the same hypothesis. Because the problem is with the hypothesis, the problem is not addressed with robustness checks.

True story: A colleague and I used khổng lồ joke that our findings were “robust khổng lồ coding errors” because often we’d find bugs in the little programs we’d written—hey, it happens!—but when we fixed things it just about never changed our main conclusions.


*
*
*
*
*
*
*
*
*
*
*
Andrew says:

This may be a valuable insight inlớn how lớn giảm giá with p-hacking, forking paths, & the other viviancosmetics.vnistical problems in modern research.

Xem thêm: " Right After Là Gì (2021)

This website tends to focus on useful viviancosmetics.vnistical solutions lớn these problems. And that is well & good.

But, there are other, less formal, social mechanisms that might be useful in addressing the problem. Discussion of robustness is one way that dispersed wisdom is brought khổng lồ bear on a paper’s analysis.

Another social mechanism is bringing the wisdom of “gray hairs” lớn bear on an issue. It can be useful to lớn have sầu someone with deep knowledge of the field cốt truyện their wisdom about what is real & what is bogus in a given field. Such honest judgments could be very helpful. Unfortunately, a field’s “gray hairs” often have the strongest incentives to lớn render bogus judgments because they are so invested in maintaining the structure they built.

Another social mechanism is calling on the energy of upstarts in a field khổng lồ challenge existing structures. This seems to lớn be more effective. Unfortunately, upstarts can be co-opted by the currency of prestige into lớn shoring up a flawed structure.

Maybe what is needed are cranky iconoclasts who derive sầu pleasure from smashing idols and are not co-opted by prestige.

Xem thêm: Chỉ Số Sức Mạnh Dragon Ball Super: Nếu Chỉ Số Sức Mạnh Của, Dragon Ball Super: Nếu Chỉ Số Sức Mạnh Của

I don’t know. There is probably a Nobel Prize in it if you can shed some which social mechanisms work & when they work and don’t work.


My pet peeve sầu here is that the robustness checks almost invariably lead khổng lồ results termed “qualitatively similar.” That in turn is of course code for “not nearly as striking as the result I’m pushing, but with the same sign on the important variable.” Then the *really* “qualitatively similar” results don’t even have sầu the results published in a table — the academic equivalent of “Don’t look over there. I did, and there’s nothing really interesting.” Of course when the robustness kiểm tra leads to lớn a sign change, the analysis is no longer a robustness check. It’s now the cause for an extended couple of paragraphs of why that isn’t the right way lớn vì chưng the problem, và it moves from the robustness checks at the end of the paper lớn the introduction where it can be safely called the “naive sầu method.”