Look out, mortgage rates are going up!

That's the fear mongering that some are telling homeowners and homebuyers after the Federal Reserve raised interest rates -- a tad -- off their historic lows Wednesday.

But when a realtor or well-meaning relative tells you to buy a house ASAP, remind them that the Fed rate isn't the mortgage rate.

The current rate on a 30-year mortgage is 3.97%. That's incredibly low by historical standards. Most experts don't think mortgages will go much higher than 4% anytime soon.

The early indications are that rates barely budged after the big Fed announcement (and they may even go down).

"I don't think [mortgage rates] are going up," says Ed Yardeni, president and chief investment strategist at Yardeni Research. "Mortgage rates are really tied more to the bond market than the Fed funds rate."

Translation: There probably won't be much difference between buying a home now or next year.

Even if mortgage rates go up to 4.5% this summer, that would only add about $700 a year to the mortgage payments for a $200,000 home.

Home prices are likely to come down

The other key thing to keep in mind is that as mortgage rates go up, home prices usually come down.

"As interest rates go up, we expect home prices will come down eventually. It doesn't happen overnight," says Karen Stone, a real estate broker for TOWN Residential in New York City.

That could be good for buyers. Right now many cities have been "seller's markets." There aren't many homes for sale but there are a lot of people looking.

"In New York City, I'm still seeing bidding wars," says Stone, who admits she's even been surprised at the price some homes have sold for. "The last few years have been a 'no rules' environment."

Now she's starting to see a shift as buyers are saying, "Wait a minute. I'm not quite ready to spend that much more."

A lot of people aren't even aware of the rates

People looking for homes say they would be "anxious" about mortgage rates going up, according to a recent survey from Berkshire Hathaway Home Services. Nearly 4 in 10 say they would be discouraged from even starting the process of looking for a home.

One CNNMoney reader named Parker, who is in her 20s, worries that her dream of owning a home is over.

But the same Berkshire survey also found that many buyers don't have a clue what the mortgage rates are. It's why real estate agents spend time educating home buyers about rates and what their monthly costs will be.

Potential buyers like Parker who are paying attention to the Fed should feel some ease because rates are unlikely to go up much, even by December 2016.

The Fed has stressed that it plans to move slowly and gradually to raise rates further. Some investors even think the Fed will end up having to cut rates again.

Mortgage rates are tied the closest to the yield on the 10-year U.S. Treasury bond. It fell on Thursday.

Many investors are concerned about a global economic slowdown and they continue to buy U.S. government bonds, which helps keep rates low. The Fed also continues to buy a ton of mortgage-backed securities. Until the Fed stops gobbling up mortgages entirely, rates will stay cheap.

It is a good time to refinance

The key figure to watch is when mortgage rates hit 5%. That could be a turning point.

"Some markets will be more impacted than others, especially places where housing is unaffordable like Denver, San Francisco, Los Angeles and Miami," says Svenja Gudell, chief economist at the housing site Zillow.

When the typical home price is near $1 million in San Francisco, even small increases in mortgage costs are much harder to absorb.

For now, experts say the only people who should act soon are homeowners trying to refinance, although the window of opportunity won't close immediately.

"Personally, I'm refinancing myself to capture the lower rate," says real estate broker Stone.