It's never too early to start planning your summer vacation...especially if you work on Wall Street.

The stock market is about to enter the time of year when, historically, not much business gets done.

The period between May and October is considered the worst six months for stocks since returns during that period are typically lower than the half-year span from November to April.

Over the past 50 years, the S&P 500 has gained just 1.3% on average in the May to October period, compared with an average gain of 7.1% in the other six months of a given year, according to Jeff Hirsch, editor of the Stock Trader's Almanac.

There's a saying on Wall Street: Sell in May and go away. It's a bit of a cliché, but the idea is that wealthy investors and professional money managers tend to close up shop and head to their summer homes in May.

The saying dates back to the British aristocracy of the 19th century, according to Hirsch.

Related: Americans still don't trust the stock market

The full phrase is "sell in May and go away; don't come back till St. Leger Day."

It's a reference fans of the show Downton Abbey would appreciate, said Hirsch. Saint Leger Day, the final leg of the British Triple Crown horse racing competition in September, was when British aristocrats would return to London from the countryside.

"We still have that summer oriented society," he said. "That's the cultural pattern that creates the best-six-months worst-six-months phenomena."

Related: Wealthy investors flock to fine art funds

The worst six months this year could be even worse than normal.

That's because it's a mid-term election year, notes Sam Stovall, chief equity strategist at S&P Capital IQ.

Since 1945, the worst six month stretch of any four year presidential term has come in the second and third quarters of a mid-term election year, Stovall wrote in a March 31 report.

In other words, those investors who don't leave town this summer will be faced with political uncertainty as lawmakers hit the campaign trail.

"Theres a lot of mud slinging and distraction in mid-term years," said Hirsch.

Related: Is capitalism driving itself out of business?

With all this in mind, Hirsch says it makes sense for investors to "take a defensive posture" over the next six months.

While now may be a good time to play defense, "just remember not to be in cash when the [mid-term election] tide finally turns," said Stovall.

Put differently, don't miss the boat when the best six months kicks off in November.

To illustrate his point, Stovall leaves us with a quote from one of the great market strategists of all time.

"Charlie Brown once said 'I have a feeling that when my ship comes in I'll be at the airport.'"