“To p or not to p” –my thoughts on the ASA Symposium on Statistical Inference.

Ron S. Kenett, KPA Group, Samuel Neaman Institute, Technion and Institute for Drug Research, Hebrew University, Jerusalem, Israel. (ron@kpa-group.com).

An eminent statistician labeled the American Statistical Association (ASA) statement on p values with the title of this blog.

This post is about the ASA SSI held in Bethesda Maryland on October 11-12th, 2017, and my own take away, which I hope would be of interest and relevance to my ISBIS colleagues and friends.

The symposium web site is https://ww2.amstat.org/meetings/ssi/2017. 

The symposium could be succinctly described by the above title. An impressive attendance of 400 participants contributed to two days of presentations and discussions with an apparent divide between frequentists and Bayesians.

The opening presentations were by Stanford Professors Steve Goodman and John Ioannidis. Steve presented his perspective as journal editor and emphasized the need for clear policies regarding the publication of scientific findings. Steve also noted that the policy of not publishing papers with an analysis of null hypothesis significance testing (NHST), enforced by Basic and Applied Social Psychology (BASP), was not replaced by an alternative structured approach. John presented a big data analysis of gaps between results in papers with very low p-values presented in abstracts, versus the content of the main part of the article, thus implying a finding’s presentation bias. He also discussed very eloquently the difference between terminology and emphasis within various disciplines where tools such as False Discovery Rate (FDR), hypothesis testing, and confidence intervals are used with varying levels of popularity. John is apparently advocating the application of Bayes Factors (BFs) to compare models. He is also a cosignatory of a recent paper in Nature advocating that a reduced cut off point of p=0.005 be applied as a provisionally band aid solution. His big data studies created an impression that science is approaching a problem of cataclysmic dimensions, with many of the published scientific publications being difficult to replicate.

The impression that this might be perceived as a general phenomenon of apparently fraudulent science was vehemently opposed by Dr. Shai Silberberg from NIH. His comments reflected a general sense that focusing on apparent problems, without an in-depth discussion of solutions, can be detrimental to science. On the other hand, Statistics as a discipline is now in a reactive mode facing criticism from several directions. One of the questions that came from the audience, after the first two keynote speakers, was a basic “what are we doing?” Such existentialist comment by a statistician requires serious consideration. This was accentuated by a survey presented by Blake McShane of Northwestern University regarding the interpretation of statistical outcomes and p-values by various groups of statisticians in editorial positions and of students at varying levels. The overall impression was a disconnect between practitioners and statisticians in how data analysis is interpreted with respect to domain driven questions.

The end of conference keynote speakers were Andrew Gelman, Columbia University; Marcia McNutt, National Academy of Sciences and Xiao-Li Meng, Harvard University. They were asked to present controversial views in a session titled: The Radical Prescription for Change. All three speakers made contributions towards this objective but a clear plan for change was not voiced or presented.

The sessions were designed to combine relatively short presentations, with a discussion with the audience lasting over 30 minutes. This format facilitated consideration of varying ideas and presentation of different perspectives. Particularly useful was the involvement of domain experts in conjunction with their statistician colleagues (e.g., Madhu Mazumdar and Keren Osman from the Icahn School of Medicine at Mount Sinai). The plan is to make the presentations available on the symposium website. An extensive blog with over 90 discussions related to the symposium (that requires registration to access) is available at

http://community.amstat.org/communities/community-home/digestviewer?communitykey=455d8623-185f-4e27-a33a-f2883ec8edbf&tab=digestviewer

The reader interested in the topic will find the blogs of great interest. A special issue of The American Statistician, with articles based on the symposium, is also being planned. Clearly the topic is much wider than simply addressing the role of p-values. The issue seems much related to how findings are generalized with domain specific knowledge, and this is beyond the traditional perspective of statistics. There are implications to research in statistical methodology, to applications of statistics to other disciplines including business and industry, and to statistical education. Overall, I think ASA is to be commended for taking such an important initiative where a wide range of statisticians and practitioners were able to listen to each other present both critiques and proposals of a topic that interests science as a whole.

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s