NEW YORK (CNNMoney.com) - You don't need to be in the market to buy or sell a home to be affected by the cooling housing market.

Economists, investors and the Federal Reserve are watching home building and home sales carefully because the sector has reached so far throughout the economy in recent years, lifting all manner of consumer spending and economic activity.

Mortgage Rates 30 yr fixed 3.80% 15 yr fixed 3.20% 5/1 ARM 3.84% 30 yr refi 3.82% 15 yr refi 3.20% Find personalized rates:   Rates provided by Bankrate.com.

Real Estate: 19 markets 7 'dead' zones 5 'danger' zones 7 'safe havens'

This week has brought several new readings indicating that the super-heated real estate market of 2004 and 2005 is rapidly fading in the rearview mirror.

Housing starts and building permits were down sharply in April, and builder confidence is at its worst level in almost 12 years, according to an industry survey.

And the median price of existing home sales fell in the first quarter compared to the fourth quarter, and while prices were still up year-over-year, the pace of growth slowed.

Even if prices don't fall, even if there is no housing bubble about to pop, the cooling real estate market will be felt by many of those who may not be thinking about real estate prices.

"It's going to be very similar to the stock bubble, but even more so. Many who didn't own stock lost their job when the market plummeted," said Dean Baker, co-director of the Center for Economic and Policy Research, and one of those who believes housing prices have resulted in an asset bubble ripe for a correction.

"They might not be directly connected to the housing industry, but they could find themselves affected, even if they work at a factory, or at a local mall. A lot of people have been sustaining their consumption by borrowing against the rise in their home's equity. If that stops, it certainly has to slow consumption considerably."

Even those who believe that real estate prices are not in any danger of collapsing agree with Baker that the reach of real estate extends far beyond those actually building, buying or selling homes.

"Housing accounts for between a fourth and a fifth of the GDP (gross domestic product)," said Walter Molony, spokesman for the National Association of Realtors, referring to the broad measure of the nation's economic activity. "So many other industries see sales tied to the purchase of a home. We get calls from Singer sewing machines about our home sales statistics."

The Realtors expect that there will be 5.7 percent year-over-year growth in home prices in 2006, but that would be less than half of the 12.2 percent rise seen in 2005. If the forecast is correct, the slowdown in price growth would be the steepest on record.

David Seiders, chief economist for the National Association of Home Builders, agrees with Baker that it's been the run-up in home values that has helped fuel consumer spending in recent years, even if he also believes that home prices will continue to climb, albeit at a slower pace.

"It's pretty clear that equity growth is what's fueling spending and allowing a negative savings rate," said Seiders.

Beyond the impact that home prices have had on spending, the white-hot real estate market has poured billions into the economy.

Jeoff Hall, the chief U.S. economist for Thomson Financial, said that for every 1,000 single-family homes built, there are about 2,500 full-time jobs created, $80 million in payroll and $45 million in local, state and federal taxes paid.

"As we try to clear this inventory of unsold homes, there's going to be a lot of layoffs in construction, and less revenue from the creation of housing," said Hall.

Residential contractors added nearly 200,000 jobs last year during the white-hot building boom, or about one job out of every 10 created in the broader economy.

That means that if the National Association of Homes Builders is correct and there are 150,000 fewer new home sales this year compared to the 2006 record, that's 375,000 fewer construction jobs, or the equivalent of about three companies the size of General Motors.

In its statement issued after its May meeting, the Federal Reserve policymakers pointed to what it saw as a "gradual cooling of the housing market" as one of the factors leading to more modest economic growth later in 2006. Economists like Hall and Baker say that if the Fed does not raise interest rates at its June meeting, it will be because of concerns about the impact of housing on the economy.

"If you're looking for evidence the economy is slowing, it's housing," said Baker. "Everything we've seen the last four to five months shows pretty clearly that housing is slowing."

For more about the impact of housing on the labor market, click here.

For more on the real estate market and how it affects you and the economy, click here.