My daughter returned home from college a few weeks ago. She's a business major and just completed a challenging semester that included classes in statistics, accounting and finance. Of the three, she professed to finding finance the most interesting.

In the stocks and bonds portion of the finance class, the professor spent quite a bit of time teaching from the acclaimed book “A Random Walk Down Wall Street: The Time-Tested Strategy for Successful Investing” by Princeton academic, Burton Malkiel.

RWDWS's fundamental premise is that it's a fool's errand for investors to attempt to outperform the stock market as a whole. Because “the market prices stocks so efficiently that a blindfolded chimpanzee throwing darts at the Wall Street Journal” can select a portfolio that performs as “well as those managed by the experts”. The average Joe is “far better off buying and holding an index fund than attempting to buy and sell individual securities or actively managed mutual funds.”

The stock market chimpanzee behaves rather like a drunk returning home from a bar: her next step has no relationship to the current one. On any given day, the market has a likelihood of advancing and a likelihood for retreating, but information on today's movements isn't helpful for tomorrow.

Malkiel's simple random walk is one in which each observation has a 50% chance of assuming a value of -1 and a 50% chance of being 1. The “problem” with an RW like this for the analyst is that its properties change over time – an RW isn't a statistically well-behaved “stationary” process. The RW's variance increases with the number of time points, potentially generating “patterns” that are in fact just noise.

And indeed, realized RW's often can be confused with series having apparent patterns. Malkiel argues that market “chartists” – technical analysts who purport to time the market for profits by projecting future movement in stocks based on recent patterns -- are in reality being fooled by random walks. To test this hypothesis, he generated a computer-simulated random walk series he then gave to a chartist for advice. The chartist immediately urged he buy the stock.

My daughter found all this quite amusing, but confided she didn't appreciate how one could confuse long term random noise with a true signal. Either the randomness or pattern just had to be evident, right?

Au contraire I argued, and set out to show her just how randomness can appear as a pattern. Using the sampling functions and graphics packages in R, I simulated random walk returns for 100 hypothetical stocks over three years of trading days. Each of the stocks “started” at a $100 price, moving up or down $1 every trading day driven by a random walk process. I then graphed the resulting RW journeys. Of the 100, I finally chose the four that displayed the strongest “signal” for review (Figure 1). Even though all sequences were in fact random, I wouldn't wish to be the CEO of companies that performed like those in the first row!

My daughter was pretty amazed by the simulation results, as are most when initially shown what appears to be a strong pattern driven by randomness. A statistics professor I know assigns his intro students the task of flipping a coin 500 times, noting the sequence of heads and tails. He says he can always distinguish students who actually do the assignment from those who simply fabricate by the results. A “too random” pattern without runs of heads and tails almost certainly points to slackers.

An important lesson for the analytics world: Don't be too credulous with your data. A pattern in randomness is often confused with signal. Make sure the “signal” you see is unlikely the result of chance by splitting your data into train-test or performing cross validation. And always replicate your findings.

Register or login for access to this item and much more

All Information Management content is archived after seven days.

Community members receive:
  • All recent and archived articles
  • Conference offers and updates
  • A full menu of enewsletter options
  • Web seminars, white papers, ebooks

Don't have an account? Register for Free Unlimited Access