I arrived a bit late to this party, which apparently began with Vanessa Otero’s December 19, 2016 post on her wonderfully titled “All Generalizations are False” blog. See here (quoting Mark Twain: “All generalizations are false, including this one,” which Otero supports, asserting that most, not all, generalizations are false). The post is the colorful two-dimensional chart (the updated version shown here), which has gone viral, with over 2 million views world-wide. It depicts myriad publications and media outlets assessed along a north-south qualitative dimension and an east-west partisan dimension. Although the chart depicts these media along two dimensions, Otero initially employed upwards of 17 criteria, listed in her initial post, and then modified in later revisions, leading up to this version of the chart. The revisions respond to helpful feedback of professionals in a variety of disciplines.
On methodology:
In a recent post and article, I explained the importance of distinguishing dimensions from issues or other factors. See here. In Ms. Otero’s methodological description for her initial chart, see here, she identifies the 17 criteria she employed. I will distill those into five categories, four associated with the north-south dimension (1 through 4), capturing media credibility, and two associated with the east-west dimension (parts of 3 and 5), capturing media ideology. These categories and subcategories combine into a two-dimensional space. The categories:
1. Commitment bonding: Has the news source made a sufficient resource investment—or in economics-speak, has it issued a credible bond—to ensure an ongoing commitment to journalistic integrity? Ms. Otero identifies two criteria that correlate to the issuance of such a bond, thereby diminishing incentives to cheapen a costly reputation simply to draw in an audience to a sensationalistic story. These include, first, when the source employs the costly print medium, and second, when prior to pursuing a lower cost means of disseminating news, the outlet incurred the higher earlier cost of doing so. For example, before becoming an on line medium, running on cable, or using satellite radio, did the source appear in print, operate on network television, or broadcast on spectrum radio, the latter of which were subject to more expensive and restrictive licensing procedures? These are reasonable proxies for the issuance of a media bond, and together they account for Otero’s first three factors.
2. General reputation: This also accounts for three factors, duration in the business, breadth of audience or circulation, and perception of partisanship among reputable sources, and each appears to correlate to media credibility.
3. Coverage criteria: Here, Otero identifies five factors: (1) whether the medium distinguishes opinion from news, (2) the proportion of opinion to other coverage, (3) news story repetitions, (4) reputation for partisanship on social media, and (5) party affiliation of contributors and interviewees. These combine reasonable credibility proxies with ideology proxies. Indeed, the first two could have been listed under category one. Having a distinct news and opinion department, and having a low proportion of opinion to news, requires a substantial resource commitment. The most serious news organizations place a figurative (and sometimes literal) wall between news and opinion. That’s not to say that opinion sources can’t be credible, but it is to say that sources limited primarily to opinion haven’t incurred the cost to maintain this form of bonded commitment to maintaining durable credibility. The third factor, repetition of coverage, is potentially problematic given the nature of 24-hour coverage cycles. Otero acknowledges on her blog that this might have excessively discounted CNN’s initial credibility ranking despite what she regards as its serious journalistic commitment. Even on the third graphic, shown here, CNN appears further south than it probably should on the credibility dimension, although its ranking on the ideology dimension seems about right. The final two factors are intuitively correlated both to credibility and to the east-west ideology dimension and are used accordingly.
4. Production quality: This too could be treated in the category of commitment bonding. Otero focuses on hyperbolic titles, excessive adjectives, and overall writing quality. This all relates to bonding because ensuring consistently well-written stories demands (1) well trained journalists who abide by professional journalistic standards, and (2) proper review and editorial structures that reduce the risk that unsupported stories sometimes sneak through and make it to print, on the air, or on line. Again, these factors correlate with source credibility.
5. Ideological valence: Ms. Otero identifies two factors here: First, some media self-identify as liberal or conservative, and when they do, she follows their lead, and second, no one, including Ms. Otero (and me), is a blank slate. Both considerations are relevant to placing sources along the ideological dimension, and the second potentially affects both dimensions. Ms. Otero acknowledges that her own self-identity as moderate-left might have encouraged tilting the graphic in favor of elevating some conservative media and discounting some liberal media to compensate, at least in the initial graphic.
My Analysis:
The graphic is notable for at least two reasons. It evinces an identifiable pattern, or shape. On reflection, the shape, a bell curve, is intuitive. The slope of the curve is dependent on the arbitrarily determined distancing between her north-south and east-west demarcations. Despite this, the shape itself, which informs an overall theory, retains its integrity whichever adjustments one might choose to make. Clustered at the top center are credible traditional news media, including major newspapers, magazines, wire services, and network news sources. Descending to the left and right are credible, if increasingly ideological, sources, liberal and conservative respectively. Concentrated at the bottom, and fanning out left or right, are more purely ideologically driven sources. These media do not abide journalistic standards, and they tend to advance unsubstantiated and bias-confirming stories.
Assuming the chart’s methodological validity, the bell shape, as opposed to randomly dispersed media in the two-dimensional space, is intuitive. Indeed, this is why I found Ms. Otero’s project compelling. Because of the bonding costs for investigative journalism, especially for foreign news stories, but even for domestic ones, and for traditional print media, or limited spectrum media (at least historically), those incurring such costs have a strong incentive to ensure and maintain a reputation for a high level of overall accuracy. Such publications respond to these incentives even at the expense of disallowing a salacious or otherwise enticing story that would draw in a large audience short term, but harm its reputation in the long term.
This suggests that such sources, even if they tilt somewhat left or right (compare The Washington Post or The New York Times with The Wall Street Journal, for example) will tend to concentrate toward the center of the ideological spectrum. That is because the most credible news media will, over time, convey news that is neither systematically beneficial to the left or right of the political spectrum. The gradual left and right descents are also intuitive. There are numerous credible sources that lack the resources for original investigative journalism but that offer important analysis beyond what general news media typically offer, for example, on the editorial and op-ed pages of major newspapers. It is not surprising that such sources will tend to have a more cohesive ideological valence. Examples include Slate and MSNBC, on the left, and The Weekly Standard or The National Review, on the right.
The more interesting feature is the fanning out at the bottom left and bottom right. Depending on the distancing employed between the east-west axis categories, this might be more or less visually pronounced. This phenomenon also is intuitive. Those sources that attract a strong ideological following are apt to highlight selective tidbits from traditional news sources and to couple those with what, thanks to Kelly Ann Conway, we now call “alternative facts.” See here. These presentations draw in and retain committed readers who, consciously or not, seek to have their world views reaffirmed. Not surprisingly, these sources are typically devoid of careful reflection and independent screening, and even of consistently sound writing quality.
The great tragedy for American democracy that this graphic helps to demonstrate is how and why an increasing percentage of our electorate receives its news from the final category of sources. The emerging distance between those on the left and right, distant from the ideological center associated with main stream news media, corresponds to two increasingly dominant segments of the population. These groups, also left and right, risk routinely being fed largely dubious, prior-reinforcing, rather than assumption-challenging, fare. As a consequence of concentrating on these sources, which often claim all others not in agreement are lying, readers tend to assume such sources are valid.
Some final reflections:
Ms. Otero concedes that her impressions of various media, which affect her placements on the chart, are inevitably subjective. To her great credit, Otero has invited, and responded to, considerable critical commentary, resulting in two subsequent iterations of her chart. One might imagine two methods of falsification (making the chart more scientific). First, one could devise a more systematic and verifiable method of coding entries to supplant Ms. Otero’s admittedly subjective impressions. The resulting graphic would then be subject to the test of replicability as a basis for falsification. Alternatively, one might draw upon the Wisdom of Crowds literature, see here, which itself follows in the tradition of the Condorcet Jury Theorem, see here. With survey data sent to a representative cross section of informed consumers of the media, left and right, one could aggregate and normalize the distributions for the various media sources. Either approach would be credible, and personally, I think that the latter might be especially interesting and worthwhile.
My strong hunch is that something very close to what Ms. Otero has already produced would emerge from either of these processes. That is to say that even if her individual placements would necessarily be adjusted, the overall shape of the curve and the relationships among media categories would likely remain. It is possible that the result would show a longer tail on the right than on the left side, consistent with Ms. Otero’s concession about seeking to compensate for her own left-moderate bias, and perhaps having overcompensated. Although my evidence is anecdotal, I believe that the right tends to discredit the traditional media as subject to liberal bias more so than the left making the reverse claim. (Indeed, I have even witnessed some on the left claim liberal media bias.) On the right, Donald Trump, for example, routinely describes traditional source media as perpetrating “fake news,” even while uncritically accepting non-sourced and vetted far right media, including Sean Hannity. If I am right that those on the left are generally less prone to seeking to discredit entirely the conventional media, this implies that the tail on the left side of the ideology dimension should be more truncated, thereby raising the question whether the ultimate curve actually is Gaussian, see here.
I have long feared that young adults today have lost something by not being raised on traditional print media. I personally believe that those of us who grew up that way developed stronger instincts for spotting fake news. And I also believe that that honed instinct proves vital to effective and engaged citizenship. This is no trivial matter, and it also relates to concerns about campus speech. See here. I’m not sure how to fix that, but here’s a modest proposal. Imagine a scoring system, with the most credit worthy media receiving a high score, and those that are increasingly distant, receiving lower scores, such as starting at 10 (highest) and descending to 1 (lowest), while also noting the ideological valence, Left or Right. If this study were replicated or the product of aggregated data, as suggested above, it could be incorporated into Facebook feeds. (According to Ms. Otero’s blog, 40% of the public receive their primary news source that way). Any news media that is linked to a story could then have a ranking score attached, with a link to the most current version of the table and data. If a story that reinforces a reader’s view is accompanied by a 2R or 2L ranking, as opposed to an 8R or 8L ranking, this might give the reader some pause as to credibility, including whether to share it with friends. This would not substitute for the education of actually reading traditional news media, but it could be a helpful start.
As noted above, I discovered this chart a bit late. When I did, I followed up with my own investigation. I had assumed the chart was produced by a full-time academic, or perhaps as part of a doctoral dissertation. It was not. Vanessa Otero has a B.A. in English from UCLA, with a concentration in three hard sciences, biology, chemistry, and physics (yes, you did read that correctly). She also holds a surprisingly recent J.D. from the University of Denver, and she practices Intellectual Property law at a firm in a Colorado. She is an avid reader, she likes to write, and she, like me, is seriously worried about the problem of growing news illiteracy. And so, she put this together entirely at her own initiative. To me this is remarkable, and based on my correspondence with Ms. Otero, including her kindness in commenting on an earlier draft of this post prior to publication, so is she. Ms. Otero has provided a valuable public service, and her chart is a perfect illustration of a public good.
[Please note that I have more recently posted an analysis of the newest version of the graphic, Version 4.0, available here.]
Thank you, and I welcome your comments.