Hi Guys, Refinance the house, but couldn't (penalty is huge) please help and advise...?

The bigger problem that you may have is that you do not have 20% equity in your house. When I tried refinancing recently, the banks told me that they would only lend 80% of the _current appraised value_, and any difference between that number and the amount of the previous mortgage would have to be paid out of funds that I have. Which would entirely negate the advantage of the lower interest rate.