NEW YORK (CNNMoney.com) -- As if it hasn't been tough enough already, markets are about to move into what is typically the worst time of the year for stock investing: late summer and early fall.

Wall Street is also in the midst of what is typically the worst quarter of the year: the third. And those who follow the four-year cycle of the presidency will tell you that the second year of a presidential term of office, like 2006, is the worst of the four.

Crude futures (blue) have surged this year, while the S&P 500 index (yellow) has dragged.

OK, so the market's seasonal patterns don't hold true every year.

But consider the reasons this year is likely to follow suit. Investors are nervously eyeing the worst violence in years in the Mideast; oil prices are near record highs; worries about a slowing economy and rising interest rates abound; a possible slowdown in corporate profit growth is looming, and a 3-1/2 year old bull market may have very well peaked last spring.

Add to that a cooling housing market, an expected slowdown in consumer spending, and less money going into stocks than there has been in some time, and you've got quite a challenging environment for investors, said Jim Melcher, president of Balestra Capital, a New York-based hedge fund.

"You've almost never had a confluence of factors like this where it didn't lead to a recession or a bear market," said Melcher. "It's just not a great time to be in stocks."

For the last 18 years, August has been the worst month of the year for the S&P 500, and the second worst month for the Dow industrials and the Nasdaq composite, according to the Stock Trader's Almanac.

Go back further and it's not much better: since 1950, August has been the third worst month of the year for both the Dow and S&P 500 and September has been the worst. That's been true for the Nasdaq as well, since its inception in 1971, according to the Almanac.

"Summer is typically not a buyer's market," said Tom Schrader, managing director at Legg Mason. "I think you're going to see continued weakness through the early fall."

That weakness has rocked stocks recently: last week, the market's three-session selloff sent the Dow industrials down nearly 400 points. The months ahead could be equally trying.

"My gut tells me it's a little overdone short term, but beyond that, the outlook isn't great," said Jack Ablin, chief investment officer at Harris Private Bank. "We may get a few pops, but the overall direction is likely down through August and September.

He said headlines on the geopolitical and energy fronts aren't likely to improve soon. Additionally, investors will have to contend with all the finger-pointing and ugly politics that are bound to accompany the fall congressional elections.

The third quarter is typically the worst because of what might be called "the Hamptons effect," meaning, hey, it's summer, and bulls would rather be on the beach than making big changes to their portfolios. Summertime also often is a time many companies reassess the earnings outlook for the rest of the year, which also can pressure stocks.

And 2006 also marks the second year of the presidential cycle, and that's not a good thing. Over the last 60 years, the third quarter of year two has been good for an average decline of 2.2 percent on the S&P 500.

Earnings may not be much help, either. The analysts consulted for this story seem to agree that overall S&P 500 earnings growth in the second quarter through the rest of 2006 is likely to either meet or miss forecasts.

That would mark a change from recent quarters, in which overall growth tended to surpass forecasts.

Currently, the S&P 500 is on track to see second-quarter earnings growth of around 12.4 percent versus a year ago, according to Thomson Financial. That would mark the 12th straight quarter of earnings growth of more than 10 percent. Third- and fourth-quarter earnings are expected to grow about 15 percent.

Earnings that don't surpass forecasts as often won't help stocks heading forward. But they may not hurt much either, since investors have probably taken it into account, said Subodh Kumar, chief U.S. investment strategist at CIBC World Markets.

"Even though analysts' forecasts haven't come down, there are indications that the market is factoring in a slowdown," Kumar said.

He said that the market, as represented by the S&P 500, is likely to find support even as earnings growth slows. Rather than slowing earnings, a greater threat to stocks going forward would be a pickup in inflation that causes the Federal Reserve to keep raising interest rates.

Kumar said that he sees the current rough patch for the market as reflecting a reassessment of how risky stocks are, rather than the start of a bear market.

Some other analysts aren't so sure, though, arguing that when the stock market peaked in April and May, that may well have marked the end of the 3-1/2 year bull market, at least for now.

They agree that a sign from the Fed that it's done after a two-year rate-hiking campaign would help stem the selling. But an end to rate hikes alone would not be enough to lift stocks.

For stocks to really recharge, there would need to be an indication that the Fed is going to start cutting rates, said Harris Bank's Ablin, and should that happen at all, it isn't likely to be until early next year.

As for the next few months, the going is likely to be rough. Consider again the presidential cycle theory. According to the Almanac, since 1913, the Dow has fallen 22.2 percent on average between the high it hit in the first year of the four-year cycle - such as 2005 - and the low it hit in the second year - in this case 2006.

Even after the recent selloff, the Dow is only trading 1.8 percent below that 2005 high - suggesting that a much bigger drop is yet to come.

Related: Tech troubles in the spotlight

Related: The worst time for stocks

Related: Playing the presidential cycle