The debate over passive versus active investing is akin to the NFL’s Eagles versus Cowboys or Coke versus Pepsi. In short, once our preference for one style over the other is established, it becomes a proven fact or incontrovertible reality in our minds.

This post is not meant to convert a passive investor into an active investor; however, we do explain why we believe some active investing approaches can logically beat passive strategies over a reasonably long time horizon (clearly, it won’t work forever). Our framework also helps investors decipher the quantitative “factor zoo” to determine if data-mining computers have actually identified a sustainable active strategy or a pipe dream.

We cannot overemphasize that identifying sustainable alpha in the market is no cakewalk. More importantly, being smart, having superior stock-picking skills or amassing an army of Ph.D.s to crunch data is only half of the equation. Even with those tools, you are still only one shark in a tank filled with other sharks. All sharks are smart, all sharks have a MBA or Ph.D. from a fancy school, and all the sharks know how to analyze a company. Maintaining an edge in these shark infested waters is no small feat, and one that only a handful of investors has accomplished.

In order to achieve sustainable success as an active investor, one needs skill, an understanding of human psychology and an appreciation of market incentives (behavioral finance). We start our journey where mine began: as an aspiring Ph.D. student studying at the University of Chicago.

Pitching Market Inefficiency in the Land of Efficient Markets

I entered the University of Chicago Finance Ph.D. program 13 years ago (Fall 2002). It was the beginning of a painful, but highly enlightening journey into the world of advanced finance. For context, the Chicago finance department maintains a rich legacy associated with having established, and successfully defended, the Efficient Market Hypothesis (EMH). Ph.D. students in the department spend their first two years in grueling, graduate-level finance courses infused with highly technical mathematics and statistics. The final two to four years are dedicated to dissertation research. I would describe these conditions as: “sweatshop factory meets international mathematics competition.” The program was tough.

After surviving my first two years of intellectual waterboarding, I needed a break. I took a unique “sabbatical” and decided to join the United States Marine Corps for four years. Long story short: I wanted to serve, and I wasn’t getting any younger.

I returned to the Ph.D. program in 2008 to finish my dissertation. My time in the Marines taught me a lot of things, but one lesson stood out from the rest: “Make bold moves.” And of course, what is the boldest move one can do at the University of Chicago?

Write research that claims markets aren’t perfectly efficient.

Inefficient Market Mavericks: Value Investors

I wanted to determine if traditional value-investors can “beat the market.” I had been following a value investing strategy with my own account for over 10 years. I was a tried-and-true believer in the Ben Graham mantra: margin of safety. The story that long-term value investing beats the market was compelling, but much of the rhetoric in academic circles, and the research published in top-tier academic journals, suggested otherwise.

The “value” debate was reinvigorated by the famous Fama and French 1992 paper, “The Cross-Section of Expected Stock Returns.” The paper sparked a debate over whether or not the so-called “value premium,” or the large spread in historical returns between cheap stocks and expensive stocks, was due to extra risk or to mispricing. The risk-based argument for the value premium didn’t sit well with me as a Ben Graham aficionado. Graham and Buffett were famous for beating the market over long periods of time by buying cheap stocks. I began digging.

I started collecting data on nearly 4,000 stock picks submitted by top fund experts, asset managers and value enthusiasts to Joel Greenblatt’s website, ValueInvestorsClub.com. And this wasn’t just any club. Highly selective, screened for quality and regarded as one of the best sites on the web for market ideas, these members were true heavy hitters in the value investing arena.

After a year of toil and anguish, I compiled all 4,000+ recommendations into a database so I could conduct a thorough analysis. The results were extremely compelling—there was strong evidence that these “varsity value investors” exhibited significant stock picking skills.

Excited to share my new findings, I eagerly drafted a paper, which included the following abstract:Wes Gray on active investing and poker hacks

I immediately sent my draft dissertation to my advisor, who happened to be the godfather of the Efficient Market Hypothesis. Eugene Fama, a strong—if not the strongest—supporter of EMH reviewed my results. The response I received was less than ideal:WesGrayActive2

I sped down to Fama’s office to get some clarification. The last thing I wanted was a year’s worth of blood, sweat and tears to get tossed out the window. I had to know exactly why Dr. Fama disagreed. Sweating profusely, with the prospect of the Ph.D. title slowly slipping away, I asked one of the world’s most famous financial economists for clarification. Fama responded:

“Listen, the data and analysis are sound, you simply can’t say that, ‘value investors have stock picking skills,’ but instead you need to qualify that statement with, ‘the sample of value investors we investigated,’ have stock picking skills.”

I sat back, relieved. Sounds like I might graduate after all. Fama was correct: My findings did not suggest that all value investors have skill, merely the sample I was investigating had skill. Crisis averted. I graduated the following year, with my research affirming, at least for me, that markets were not perfectly efficient.

But nagging questions abounded: What gives a certain investor “skill”? What characteristics drive alpha? Why can one active manager (winner) systematically take money from another active manager (loser)?

Enter behavioral finance and “rational” investors

As I plowed through thousands of stock-picking proposals, one key takeaway was present. These analysts were good. They all had skill. They all were smart. They all made compelling cases that, statistically, outperformed in aggregate. But that couldn’t be the only reason why they outperformed. As I mentioned above, everyone in this market is smart and capable; intellect alone can’t be the driver of superior returns. What enabled these varsity stock pickers to buy low and sell high and why was the Efficient Market Hypothesis not stopping them? Behavioral finance.

John Maynard Keynes, a shrewd observer of financial markets and a successful investor, highlights the paradox that behavioral finance represents. At one point, Keynes was nearly wiped out while speculating on leveraged currencies (despite being a highly successful investor). This downfall led him to share one of the greatest investing mantras of all time: “Markets can remain irrational longer than you can remain solvent.”

Keynes’ quip highlights two key elements of real world markets that the efficient market hypothesis doesn’t consider: investors can be irrational, and arbitrage is risky. In academic parlance, “investors can be irrational” boils down to an understanding of psychology. “Arbitrage is risky” boils down to what academics call “limits to arbitrage” or market frictions. These two elements—psychology and market frictions—are the building blocks for behavioral finance (depicted in Figure 2, below).WesGrayActive3

First, let’s discuss limits to arbitrage, more commonly referred to as market frictions. The efficient market hypothesis predicts that prices reflect fundamental value. Why? People are greedy and any mispricings are immediately corrected by those smart, savvy investors that can make a quick profit. But in today’s world of instant information, supercomputers and interconnected markets, true arbitrage—profits earned with zero risk after all possible costs—rarely, if ever, exists. Most arbitrage involves some form of cost or risk (risk of buying at the wrong price, risk of paying high transaction costs, liquidity, etc). Let’s look at a simple example:

Arbitraging oranges:

  • Oranges in Florida cost $1 each.
  • Oranges in California cost $2 each.
  • The fundamental value of an orange is $1 (Assumption for the example).
  • The EMH suggests arbitrageurs will buy in Florida and sell in California until California oranges cost $1.

But what if it costs $1 to ship oranges from Florida to California? Prices are decidedly not correct—the fundamental value of an orange is $1. But there is no free lunch since the frictional costs are a limit to arbitrage. In short, the smart, savvy arbitrageurs are prevented from exploiting the opportunity (in this case, due to frictional costs).

Next, let’s discuss psychology. Newsflash: Humans beings are not rational 100 percent of the time. To anyone who has been married, driven without wearing a seat belt or hit the snooze button on their alarm clock, this should be pretty clear. And the literature from top psychologists is overwhelming for remaining naysayers. Daniel Kahneman, the Nobel-prize winning psychologist and author of New York Times Bestseller Thinking, Fast and Slow, tells a story of two modes of thinking: System 1 and System 2. System 1 is the “think fast, survive in the jungle” portion of the human brain. When we start to run away from a poisonous snake (even if later on, it turns out to be a stick), you are relying on your trusty System 1. System 2 is the analytic and calculating portion of the brain that is slower, but 100 percent rational. When you are comparing the costs benefits of refinancing your mortgage, you are likely using System 2.

Both systems serve their purpose; however, sometimes one system can muscle onto the turf of the other. When System 1 starts making System 2 decisions, we can get in a lot of trouble. Do any of these sound familiar?

  • “That diamond bracelet was so beautiful, I just had to buy it.”
  • “Dessert comes free with dinner, so of course I had to have some.”
  • “Home prices never seem to go down. We gotta buy!”

Unfortunately, the efficiency of System 1 comes with drawbacks—what keeps us alive in the jungle isn’t necessarily what saves us from ourselves in the markets.

Now, let’s combine our irrational investors (System 1 types) with the limits of arbitrage that we discussed above (smart people that simply can’t take advantage of the System 1 types for some reason). Combining the bad behaviors with the limits that smart people run into could be a very compelling strategy indeed.

For example, consider the concept of “noise traders” (think day traders that ignore fundamentals and trade on “gut”—classic System 1 types). These irrational noise traders dislocate prices, but because they are irrational, arbitragers have a hard time pinning down the timing when these System 1 types will make their trades. An element of risk arises when an arbitrageur tries to exploit an irrational noise trader. Brad De Long, Andrei Shleifer, Larry Summers and Robert Waldmann describe this phenomenon, “Noise Trader Risk in Financial Markets,” in the Journal of Political Economy in 1990.

Here is the abstract from the paper:

We present a simple overlapping generations model of an asset market in which irrational noise traders with erroneous stochastic beliefs both affect prices and earn higher expected returns. The unpredictability of noise traders’ beliefs creates a risk in the price of the asset that deters rational arbitragers from aggressively betting against them. As a result, prices can diverge significantly from fundamental values even in the absence of fundamental risk. Moreover, bearing a disproportionate amount of risk that they themselves create enables noise traders to earn a higher expected return than rational investors do. The model sheds light on a number of financial anomalies, including the excess volatility of asset prices, the mean reversion of stock returns, the underpricing of closed-end mutual funds, and the Mehra-Prescott equity premium puzzle.

Let’s translate this into English:

Day traders mess up prices, but these people are idiots, and you can’t really time the strategy of an idiot, so most smart people don’t even try take advantage of them. Consequently, prices move around a lot more than they should because no one is stopping the idiots. Moreover, since prices move around a lot more, the returns can be higher, so idiots think they are actually good at timing markets, which incentivizes more idiots to do more idiotic things.

This combination of bad behavior and market frictions describes what behavioral finance is all about:

Behavioral bias + market frictions = interesting impacts on market prices.

And while this working definition of behavioral finance may seem simple, clearly, the debate surrounding behavioral finance is far from settled. In one corner, the efficient market clergy claims that behavioral finance is heresy, reserved for those economists who have lost their way from the “truth.” They point to the evidence that active managers can’t beat the market and incorrectly conclude that prices are always efficient as a result.

In the other corner, practitioners that leverage “behavioral bias” suggest that they have an edge because they exploit investors with behavioral bias. Yet practitioners who make this claim often have terrible performance.

So what is the disconnect?

The disconnect lies in the fact that both sides of the argument fail to assess both the bad behaviors of the market and the limits to arbitrage. Passive investors assume there are no limits to arbitrage and markets are perfectly efficient. Practitioners who claim to exploit behavioral bias believe they are the smartest poker players at the table, but they ignore the fact there are other smart poker players who have also identified behavioral bias in the marketplace.

Good Investing is like Good Poker: Pick the Right Table

Behavioral finance outlines a framework for being a successful active investor:

  1. Identify market situations where behavioral bias is driving prices from fundamentals (e.g., identify market opportunity).
  2. Identify the actions/incentives of the smartest market participants (e.g., identify market competition).
  3. Find situations where mispricing is high and competition is low.

In the context of poker, a similar strategy is critical for success:

  1. Know the fish at the table (opportunity is high).
  2. Know the sharks at the table (competition is low).
  3. Find a table with a lot of fish and few sharks.

To be successful over the long-haul, an active investor needs to be good at identifying market opportunities created by poor investors but also skilled at identifying situations where savvy market participants are unable or unwilling to act.

Editor’s note: The original version of this article first appeared on AlphaArchitect.com on Aug. 17, 2015.