There are transaction costs and fees involved in any refinancing that must be either paid out of pocket or added to the mortgage principal. Some of those costs can be considerable. Title insurance can easily run into four figures and broker fees can be expensive as well.
Like many things in life, timing is everything here. Is your job likely to relocate soon? Will you need a bigger house in the next couple of years? Unless you're planning to stay in the home for a while, the benefits of a lower monthly bill may not be worth the additional expenses that refinancing generates.