2010
09.14

Letter From Criag Bennell

Craig Bennell

Associate Professor, Department of Psychology

Faculty of Arts and social Sciences

B5550 Loeb Building

1125 Colonel By Drive,

Ottawa, ON K1S 5B6

Carleton University

Director of Police Research Lab

May 26, 2010

Dear Dr. and Mrs. Connelly,

It was a pleasure chatting with you both today, though it obviously would have been nicer to meet under different circumstances.

I have now had the chance to carefully read the portion of the Coroner’s report that you provided me with, which dealt with the note found on your son’s desk. As you know, this portion of the report refers to some research I published with a previous graduate student of mine, Natalie Jones, and discusses how Dr. McCallum used that research to assist him in determining whether the note in question was in fact a genuine suicide note. With all due respect to Dr. McCallum, upon reviewing this portion of the report I was concerned with a variety of things that were stated by him, specifically about our research and how it was used (or misused) in this case to arrive at the conclusion “that this note written by your son was a genuine suicide note”.

The paper that was cited in Dr. McCallum’s report is entitled “The Development and Validation of Statistical Prediction Rules for Discriminating between Genuine and Simulated Suicide Notes”. It was published in 2007 in the journal Archives of Suicide Research. In that article, Natalie and I conducted an analysis of 33 genuine and 33 simulated suicide notes that were collected for a previous study by Shneidman and Farberow (1957). We analyzed these notes in order to identify variables that might serve as predictors of suicide note authenticity and to develop statistical prediction rules (SPRs) that could potentially be used in the future to estimate the likelihood that any given note is genuine vs. simulated. In our analysis, we identified several variables that differed significantly across genuine and simulated suicide notes. However, when developing the SPRs, two variables appeared to be particularly useful – the expression of positive affect (more positive affect in genuine notes) and average sentence length (shorter average sentences in genuine notes). By appropriately weighting and combining these variables, and then setting an appropriate threshold for rendering decisions, an SPR was developed (what we refer to as the “optimal” model in our paper) that produced a reasonable level of predictive accuracy.

Ignoring for a moment some of the serious concerns raised in our paper (and discussed below) about using our research in actual police investigations until additional research has been conducted to replicate our results, it is important to highlight that there are generally two ways in which SPRs can legitimately be used to assist in making difficult diagnostic decisions – neither of which seem to have been used by Dr. McCallum when examining your son’s note, or at least not in a way that I would define as appropriate.

The typical procedure would be to use the SPR developed in our research and apply it directly to a particular note (in this case, the note written by your son). This would involve a process whereby relevant variables in the note are quantified (e.g., the degree of positive affect and average sentence length) and incorporated into the SPR. The result of doing this would be an estimated probability, produced by the SPR, that the note under examination is genuine (the higher the probability, the more likely the note is genuine). Using some pre-determined cut-off score, the user could then determine whether the predicted probability value produced by the SPR was high enough to warrant a decision that the note is likely to be genuine. While in a technical sense this procedure could have been used in this case, it was clearly not used by Dr. McCallum because the SPRs we developed were never presented in our published paper (although some potential cut-off scores for the predicted probabilities were, along with the outcome data relating to these cut-off scores; see Table 3, p. 227).

Another way of using our research would be to rely on one’s general knowledge of the SPR we developed to make a more subjective, but empirically informed determination of suicide note authenticity. While existing research indicates that this approach is less preferable than applying SPRs in a purely mechanical fashion (as above), such an approach may result in reasonably accurate decisions if implemented appropriately. From the information included in his report, it appears that Dr. McCallum used an approach similar to this one – he identified the two variables included in our optimal SPR and reasoned that because there is some evidence of positive affect in John’s note and the sentences happen to be quite short, the note is likely to be genuine. However, there are problems with this approach, as it was implemented by Dr. McCallum.

One of the problems with the approach that was adopted is that additional information is needed (beyond knowledge of what variables to focus on) to draw sensible conclusions about a specific note. Most importantly there is the need to know what cut-off scores to use for each of the variables. As we state in our paper “…the identification of optimal predictors of suicide note authenticity is futile without determining the degree to which these must be present to render an accurate decision” (p. 231; italics added). In other words, it is not enough for Dr. McCallum to know that positive affect may distinguish genuine from simulated suicide notes or that “there was clear affection for his family expressed by your son in this note”; he needs to know how much positive affect should be present in a particular note before rendering a decision that it is genuine. The same is true of average sentence length. It is important to know that, on average, genuine notes are characterized by shorter sentences than simulated notes and that “the note [under investigation] has short sentences”, but to make a valid determination of authenticity one really needs to know what is actually meant by “short”. Again, Natalie and I did not provide the variable-specific cut-off scores in our paper, which would be needed to implement this procedure in an appropriate fashion.

[As an aside, I think it is important to also note that even if one of the procedures mentioned above had been used, all an SPR can do, even a well-validated SPR, is provide a predicted probability that a note is genuine (vs. simulated). In other words, the task of determining suicide note authenticity is always going to be probabilistic in nature. No definitive conclusion could ever be reached about a particular note using the SPR we developed or any other type of SPR.]

Perhaps of most concern to me is the fact that one of our SPRs was applied to John’s note in the first place for the purpose of determining its authenticity. In my view, and as suggested in our paper, more research should have been conducted before this was done. Specifically, we discuss in some detail the potential limitations of our study and raise some doubts as to whether the SPRs we developed, or any of our other findings for that matter, can be applied to notes like John’s with any degree of confidence.

While Natalie and I discuss a number of potential problems with the practical application of our SPRs, perhaps the most obvious issue we highlight has to do with the age of the suicide notes included in our sample. As indicated above, the notes that Natalie and I based our analysis on were collected by Shneidman and Farberow. As stated in our paper, “[w]ith the co-operation of the Los Angeles County Coroner’s Office in California, [these] authors randomly obtained 33 genuine suicide notes written between 1945 and 1953. Members of the control group (n = 33) were matched with the authors of the genuine notes based on age and occupational level” (p. 223; italics added). The fact that the SPRs we developed were based on notes that were written 60 to 70 years ago is obviously cause for concern and it is uncertain to what extent our results generalize to modern suicide notes. As Natalie and I suggested, before applying our SPRs to such notes, it would be necessary to replicate our research using a more recent sample of notes. This is standard practice (or at least should be) for anyone attempting to apply an SPR to a new sample, especially if there is a high potential for differences between the original (i.e., SPR development) sample and the new (i.e., currently being investigated) sample.

In summary, while I appreciate the very difficult task faced by the Chief Coroner in this case and don’t want to show any disrespect towards him, it is unclear to me how our published paper could be used to reach a conclusion that John’s note was in fact a genuine suicide note. Natalie and I simply do not provide the necessary information in that specific paper to carry out any sort of objective or systematic analysis. In addition, even if we had provided the relevant information, I am not convinced, given the many limitations of our study, that any of the SPRs we developed should be used for this purpose until additional research has addressed some of these limitations and determined that our research can be successfully replicated.

I hope that my comments here help clarify some of the concerns I had with the report prepared by Dr. McCallum. If you have any questions about what I’ve written please don’t hesitate to get in touch with me.

Best of luck,

Craig Bennell

Associate Professor, Department of Psychology, Carleton University

Director, Police Research Lab

write my essay