Thoughts On Paying Off Mortgage

For most of us, when we buy our first house, having a mortgage or not isn’t really a decision we have to make.  Few Americans, even high-income professionals, can afford to pay cash for a house.  In fact, for many high-income, low-net-worth professionals like doctors just out of residency, it’s tough just to come up with a 20% down payment, leading us to look very carefully at “doctor loans.”  Later in life, however, almost all of us will have to wrestle with the question of when/if to pay down the mortgage.  There truly is no right answer to this question and it isn’t all that dissimilar to the “student loans vs invest” question most deal with shortly out of residency.  However, I’ve been thinking about some new ways to look at this decision.

Pre-pay the Mortgage When Your Risk Is Higher Than The Bank’s Risk


When you first take out a mortgage, especially a 0% down mortgage, the bank is the one taking most of the risk.  All you’re risking is your down payment (if any) and your credit score.  The bank could be left holding the bag if you become unable to make your payments or choose not to make them in the event of the house declining in value.  As time goes on, the bank takes on less and less risk.  For example, when you only owe $100K on a $500K home, the bank only has $100K at risk, and if you default, stands to make $400K on the deal.  Meanwhile, a foreclosure costs you $400K and only saves you $100K.  Why put $400K of home equity at risk for a relatively small benefit of carrying a $100K mortgage?

Mortgage Interest Deduction Becomes Less Valuable

One of the reasons many high-income professionals buy a home is so their monthly payment can become partially deductible.  As long as you itemize, mortgage interest and property taxes may be deductible on your taxes (although the Pease law reinstated in 2013 limits the deduction for those with an AGI greater than $250K ($300K married)).  However, as you pay down the mortgage, more and more of each payment goes toward principle.  This decreases the interest you have to pay, but also decreases the interest you get to deduct.  At a certain point, it may not be worth it to you.  In fact, it may even be possible that without a big mortgage interest payment that your itemized deductions are close to or less than the standard deduction, effectively raising your after-tax mortgage interest rate.  If you end up cutting back and decreasing your income (or retire completely), putting yourself into a lower marginal tax bracket, that interest deduction also becomes less valuable.   $10K of mortgage interest at a 40% marginal rate is worth $4K.  At 25%, it’s only worth $2500.

Paying Off The Mortgage Is Like Disability Insurance

One of my partners considers paying off his mortgage the equivalent of buying a disability insurance policy that would produce the same amount of income.  For example, to cover a mortgage payment of $3K a month with disability insurance would cost $1000-1500 per year in disability insurance premiums.  This is similar to raising the interest rate on your mortgage.

It’s Like Life Insurance Too

If you owe $400K on your mortgage, you probably need to have an extra $400K in life insurance to protect your family in the event of your death.  A 40 year old healthy male buying 30 year level term insurance would pay at least $500 a year for a $400K policy.  Let’s say you have 15 years left on a 5% mortgage and owe $400K on it.  Your payments are $3059, of which $1497 is principal and $1562 is interest.  If you also add in the cost of disability insurance ($1500 per year) and life insurance ($500 per year) that’s the same as increasing your interest rate from 5% to 5.53%.

Skeptics would argue that if you invest that money instead of paying down the mortgage then you should have that $400K (or possibly more) on the side in an investing account.  That may or may not be true.  First, you need the discipline to actually invest that money instead of spending it.  Second, you actually need to earn a return equivalent to or better than your after-tax mortgage interest rate.  Third, you have to pay investment expenses and taxes in order to liquidate the investment.  Paying down the mortgage is a guaranteed return equivalent to your after-tax interest rate plus the costs of the disability insurance and life insurance equivalent to that amount.  That guarantee is worth something.

Paying Off Your Mortgage Allows For A Smaller Nest Egg


Paying off your mortgage early is even more attractive while in or near retirement.  If a retiree can count on spending ~4% of his nest egg each year, then a $3000/month mortgage payment requires a nest egg of $900K, just to pay the mortgage!  Far better to use $400K to pay off the mortgage, and then use the remaining $500K to produce an income of $20K per year.  It’s easy to see that carrying a mortgage into retirement isn’t particularly wise, no matter how you feel about it during your working years.

Asset Protection Considerations

Asset protection laws are state-specific.  Most states protect your 401K and IRAs from creditors.  No states protect a taxable investing account.  Some states protect a significant amount of your home equity from creditors.  In Texas, where up to $1M of home equity is protected, there is an additional benefit to paying down your mortgage and “hiding your money in your house.”  In Virginia, where only $5K of home equity is protected, that might not be the best plan.

Behavior Vs Math

As you can see, there is more to the pre-pay mortgage decision than just weighing a psychological benefit of being debt-free with the ability to invest using low-interest, non-callable margin debt like a mortgage.  It’s an individual decision, but recognize that it is likely that there is a certain point in the term of your mortgage and in your life when it probably makes sense for you to pre-pay your mortgage, especially when the alternative is investing in a taxable account.

My Recommendations

I recommend you have your mortgage paid off by the time you wish to cut back on your workload or retire.  People with debt of any kind aren’t really ready for retirement in my opinion.  One of my financial goals is to be able to cut back on my clinical workload by age 50, so I plan to have my mortgage paid off by then.  Rather than just amortizing it over the next 12-13 years, at some point I imagine I’ll just pay off the last $20K, $50K, or even $100K very rapidly to reduce my risk and improve my cash flow.  In general, I think using money that would otherwise be invested in a taxable account to pay down student loan or mortgage debt is a fine move, but I would hesitate to give up significant tax breaks to reduce low-interest debt.  In other words, max out your 401K, Stealth IRA and Backdoor Roth IRAs before paying down your mortgage.

What do you think?  How do you plan to pay your mortgage off?  Comment below!

Enter Your Mail Address

image_pdfimage_print


Comments

Thoughts On Paying Off Mortgage — 30 Comments

  1. I’ve had my mortgage for almost a year now and I’ve been paying it down somewhat aggressively. Looking at what housing prices were a few years ago and how the drop affected my area, I’m pretty scared of what that looks like. I don’t want to be stuck in a place because I’m really underwater, which is one of the many reasons I’ve been paying it down aggressively. I think that once I hit 50% equity within the next 6-8 months, I’ll slow down a bit.

    I’m still maxing out my 401(k), Backdoor Roth IRA, and I bought $10k of i-bonds this year. So I’m not giving up on tax savings. No student loans, just the mortgage. I’m not scrimping that much either, still enjoying life – just throwing what I would otherwise invest in a taxable account at the mortgage for now.

    How do you find out what the asset protection laws are in your state? I’m not a doctor, so not as worried about liability, but still curious.

    • It would be pretty hard to lose that much equity in such a short time span. The worst hit areas lost 50% during the last crisis but that was over several years. Unless you way overpaid (which is a less talked about major factor of the housing crisis) and put zero down you probably don’t have to worry about being so aggressive.

  2. My goal is similar to WCI in that when I retire from the military around age 50 I intend to work only as much as I want. My plan to do this is based primarily on two things: the military pension I will have and paying cash for a house. If I don’t have the cash for the house I want then I will work full time for another year or so as a civilian to get it paid off. With no debt and a pension that will cover the basic living expenses I can work solely to increase my retirement assets and to cover the costs of extravagant fun like travel and toys.

    I still contribute the max to my TSP and Roth IRA now so that when I hit 59.5 I can maintain the same standard of living while cutting work back even further should I choose. At any time I could just stop and just limit the extravagance in my life without having to worry about paying the mortgage, which is typically one of the largest expenses a person has.

    The freedom having do debt buys you is one that is difficult to calculate because it allows you, at pretty much any point, to just quit if you don’t like the situation you are in. Debt is a slave driver. I have seen debt be the primary reason why people who want to leave the military feel that they can’t or people who hate their job feel that they can’t quit – how would they pay the bills? Debt severely limits your options.

  3. Unless you live in a non recourse state, i dont agree that banks are taking on most of the risk early on especially for physicians. Given physicians almost always have income, the banks arent really taking on a bunch of risk. If you live in a non recourse state then that might change things.

    • Similarly, regardless of whether or not you live in a non-recourse state, the bank does not just get to keep $400k if they foreclose on your $500k house with a $100k mortgage. They are legally required to give you any difference between what you owe and what they get for the house.

      Now that is NOT to disagree with your point that risk shifts from bank to borrower as equity rises. But only up (down?) to where there is sufficient equity to cover the mortgage, the costs of foreclosing and auctioning, and the difference between the price fetched at auction and the open market value. That is the max the homeowner has at risk when they have e.g. 80% equity.

      In fact, I would venture to guess that the sum of those amounts is something around 20%, and that’s why 20% is the traditionally required down payment.

      • Good point. I don’t know if I ever really thought that through before. Although the idea of the bank “fire-selling” my house because it has plenty of equity in it is not an appealing one. If it’s a $500K house, and I owe $100K, and they foreclose, and they sell it for $300K when it’s all said and done I still lose $200K.

  4. Wci and others
    I am currently 36 with asset allocation of 90 Percent stocks and 10% bonds ( all index funds).
    I have now reached 25 Percent of my retirement Goal. This was supposed be a trigger point to move to 80% stocks and 20% bonds.
    My bond holding is Vanguard total bond market index. This has a yield of 1.6 Percent
    Should I
    A) adjust my asset allocation according to my above plan. Or
    B) leave my asset allocation the same and pay off my highest interest rate debt which is non deductible student loan interest which is at a fixed 2.625 Percent. This would done with new money that would have gone into a taxable account. I maxed out all my tax advantaged accounts and hold most of my stock and bonds funds in 401k
    Therefore with bond yield being so low, can paying off fixed rate debt be considered Equivalent to increasing my bond exposure ? What do you suggest I do?

    • So, are you asking if you should deviate from your previously well-thought-out written investing plan because the market has done something you didn’t expect (i.e. very low interest rates)? My thought is no. Go to 80/20 as planned. That’s still a very aggressive investment portfolio and won’t perform that differently from 90/10.

      Given that most asset classes seem richly valued, I think it’s a great time to pay down debt, even low interest debt, instead of investing in a taxable account. Technically, yes, a loan is like a negative bond, so paying it off is the equivalent of investing in bonds, but I don’t think you took that into account when you made your written investing plan. If there’s anything I’ve learned as an investor, it’s that in the end I’m glad I followed my written investing plan, no matter what the market does.

      Keep in mind that even if your bonds are only yielding 1.6%, you’ll be very glad you are holding them when stocks drop 20-30% as they will at some point in the next few years. We know it will happen, we just don’t know when. Plan for it.

      • Holding a fixed rate mortgage over an extended period is an excellent inflation hedge. The Bank is obligated to take your dollars at diminished value.
        “Profiting” from your mortgage requires discipline,however,as you must invest the difference and then take out the returns at a capital gains rate. It is a risk-adjusted investment decision more than anything.
        A contingency HELOC may fulfill all requirements for asset protection. It can also be thought of as a low cost emergency fund.
        I don’t understand the improved cash flow from paying down a mortgage at retirement. Didn’t you just use up the money you would have had?
        Money is fungible. Easy to say, difficult to practice.

  5. WCI
    your statement to my above question is well taken. I will stick to my investment plan.
    But it does raise the question of if rebalancing provides you an oppurtunity to buy undervalued assets ?
    At this point which is overvalued , is it stocks or bonds ? Or is it both ? I guess we will only know retrospectively
    With the rise in the stock market, it would seem that a lot of people should be rebalancing their portfolio into bonds. But Total bond fund yield being at 1.6% does not seem that great a proposition .
    Personally,thinking of my Bond holdings as taking some winnings of the table or as dry powder to purchase stocks when a correction/crash does happen rather than a investment that grows makes the rebalancing process easier

    • Rebalancing automatically causes you to buy “undervalued” assets in that it forces you to sell high and buy low. Now, undervalued is one of those things that can really only be known in retrospect of course. Trust me that when stocks drop 20-30% you’ll be perfectly content to be holding some bonds that “only yield 1.6%.”

  6. 1. One downside of paying off your mortgage early is that it may be hard to access that cash if you need it (disability, job loss). Even with significant equity, you won’t be able to refinance and get cash out without documented income. You could get a HELOC but I thinks banks can cut those off at will. Most docs should be at low risk of this but I keep reading about those who get in legal and ethical trouble and wonder what they are going to do.
    2. If you are going to pay down your mortgage early, you might be better off with an ARM. You pay more for a fixed-rate mortgage because of the risk the bank takes that rates will go up. This is probably worth it in most cases, but if you are going to pay off quickly you might as well get the lower rate.

    I am (finally!) facing this dilemma of whether to pay down my mortgage or invest post-tax. I basically follow the asset allocation David Swenson advocates, and with stocks having gone way up, I’d mostly be investing in bonds at rates much lower than my mortgage, so I’m leaning to splitting up the money 2/3 and 1/3, using the first part to pay down the mortgage and the rest to invest.

  7. Via email:

    I was stunned that your mortgage post overlooked two huge negatives of paying off a mortgage.
    1. Liquidity: Two people have $250K mortgages and $250K in cash. One pays the mortgage off, the other just makes regular payments. One year later both individuals are laid off. It will be very difficult for the person who paid off his mortgage to access the $250K he sent his mortgage company. The one who didn’t pay it off still has in the bank $250K minus a year’s payments.
    2. ROI: Your savvy readers will look at a mortgage as an opportunity cost – will they get a better return paying off the mortgage or investing elsewhere. With mortgage rates (especially the after-tax rate) so low, your readers would derive a greater benefit in the long-term by taking their mortgage-payoff money and investing it in a diversified portfolio. Considering that after-tax rates are probably less than 3%, it is reasonable to expect (though not guaranteed) that they will net more money by not paying off the mortgage.

    • I felt like I addressed the ROI in the section on Behavior vs Math. It has also been addressed in the post I linked to earlier called student loans vs investing.

      The liquidity is overrated when you have hundreds of thousands of dollars in liquid investments. Not irrelevant, but overrated. Yes, someone with only $100K in retirement assets shouldn’t pay off a $400K mortgage early. But someone with $2M? The liquidity isn’t that big a deal later in life, even with some retirement account restrictions.

      Good points though. I’m going to add your email and my reply as a comment on the post.

  8. Pay off mortgage early with the help of perfect guide and steer through your financial crisis by getting rid of debt collectors. It is important to take guidance from an experienced person to get knowledge about the ways to pay off the mortgage as fast as you can.

  9. A simple way to pay down your mortgage, if only by a relatively small amount, is to pay it a month early, every month. Your amortization schedule is based on the bank receiving your payment ON the due date, every month. Here’s how to do it. Pay your mortgage on the due date. Then make the next month’s payment on the following day, and so on. All you have spent is ONE EXTRA PAYMENT. Here’s how it works. If you pay the mortgage 30 days in advance, your principle decreases by the scheduled amount ON THE DAY THE BANK RECEIVES YOUR PAYMENT. That means the bank is charging you interest on a smaller amount for those 30 days. And then the same thing happens each subsequent month. Over a few years, you will be thousands of dollars ahead of your amortization schedule on your original loan paperwork. Naturally, the process moves faster as a higher percentage of your monthly payment goes to principle. And all you had to spend was one extra payment at the beginning.

    • Show me the numbers! I calculate that such a move would shave a whopping 3.3 months off your total payoff time. And you paid an extra payment up front, so actually it’s only 2.3 months.

      • It would still be 3.3 months less time, but only 2.3 fewer payments (assuming your calculations are right.) It’s better than a kick in the teeth. It’s a bit like the paying a half-payment every 2 weeks theory. You end up making an additional payment every year, so of course you pay off the mortgage faster. But it really isn’t significantly different from just throwing some extra principal into each payment, and it certainly isn’t worth paying an additional fee to do. They’re just mental tricks to pay extra principal on the loan.

  10. Interesting arguments for and against paying off a mortgage. I just bought a house in Florida as a second home and decided to finance it with a 5/5 ARM. The first 5 years are 2.62% interest and the next 5 can only go up 2% max, so 4.62% worst case. After deductions it is much less than 2.62%.
    I am 62 years old. The fixed rate was about 4.4% at the time. So for ten years the ARM is better, and if it goes up after that, depending upon inflation, I can always pay it off. I also got the mortgage with zero closing costs. Now, earlier WCI said if you have $2m liquidity is not as important. I have over $2M but I just did not want to tie it up in what I view as a lower appreciating asset, at least as compared to other assets. I would rather invest the money in a safe investment, even if it only the same yield as the mortgage rate. I am not a doctor. I work in the investment field.

    • Sounds like you’ve got enough money that you can do either option and still be fine. If I were in my sixties buying a second home I’d probably pay cash because I’d probably be retired and I don’t like having to service any debt in retirement. You’re clearly more comfortable with debt than I am. For instance, I see no reason to invest in an asset with the same expected return as my mortgage, at least if both of those are after-tax numbers. If you’re going to borrow cheap and invest in something with a higher expected return, I guess I could understand wanting to use some leverage. I hope to be in a position where I don’t need any leverage at all when I’m in my 60s.

      • The money invested in a return equal to my mortgage would be greater than the mortgage rate because the after tax rate of the mortgage will be below 2%. As you said, either way works here. You don’t like to service debt, but you could service the debt with the money you invest rather than put into the house, and you would have more money at your disposal. You always have the option of paying it off if you want. I have the money in the market getting much higher returns, but my example was to show that even if all you got was the same return, you have more options.
        I did the same with a new car purchase. I got a new car loan at 1.99% for 4 years and decided to invest the money in the market instead. The cost of interest over four years was so low, and of course because the market did so well the earnings last year more than covered the cost of interest over four years.

        • I dunno. It seems to me you have more options if you DON’T have to make a debt payment each month. I could put that money toward my next car, put it into various investments etc. It really comes down to your comfort with debt. I see little reason to take on debt I don’t need in order to earn more money I don’t need. But there’s no doubt that leverage works. Just remember that it works both ways.

  11. A wise senior colleague told me, when I first joined the practice, that one can survive a lot of financial hardship if you do not have to make a mortgage payment. Those words stuck with me. I paid off my house in under 10 years and am happy that I have done so. Perhaps I left some potential investment gains on the table, but in the course of my post-residency career, 1996-present, there have been two major market meltdowns–and during the last one, I lived in a house that I wholly owned.

    I see physician friends, colleagues, and junior members of my group make the same mistake, time and again, buying too much house and/or buying an expensive second home. Perhaps they are aiming for the status and the trappings of the “rich doctor” lifestyle, but they end up being a slave to their debt and the profession and are visibly shaken with each downtick in reimbursements, patient volume, the stock market, or whatever financial calamity is around the next corner.

    The real “rich doctor” has control of his financial situation is not nearly so beholden to these external issues over which he/she has little or no control.

  12. Here is my REAL LIFE EXAMPLE with numbers!! I purchased a retirement home in sunny San Diego, in 9/2009 for $1100000. I could have paid cash but my son (a CFP) had a different opinion. He felt that there was almost no chance that we couldn’t do better with that money over 15 years, given the interest rate of the loan. I financed $400000, 15 year fixed @3.25%, no cost loan (no lender fees, credit for 3rd party fees). He opened a dedicated Schwab acccount and placed the $400000 in DFA’s 60/40 (DGSIX) with dividend/cap gains reinvested. He chose this fund primarily because it was the cheapest, simplest and most appropriate option for the experiment. I pay my mortgage and escrow (tax, insurance) directly from this account ($3119/month). Currently, the principal balance on the loan is $287628. The account balance at Schwab is ~$405000. Admittedly, this doesn’t take into acccount the tax issues but I’d be surprised they make a huge difference . . . .so far, I am really liking the math!!

    • Investing on margin (which is what you are doing) can really boost returns, especially in a bull market. However, that is because you are taking on additional risk. Is the risk reasonable? Probably, but don’t pretend it doesn’t exist. If we were experiencing a 5 year bear instead of a 5 year bull, you’d be singing a different tune.

      I agree that the tax issues are fairly minor since the mortgage interest is probably deductible and the dividends/LTCG distributions are relatively tax-advantaged. You may even have slightly lower taxes as a result of your experiment.

      At any rate, glad it’s working out well for you so far. I agree with your CFP son that the risk of a reasonably invested 60/40 portfolio having a 15 year return under 3.25% is pretty low.

  13. showed your response to my son. seems like you and he are on the same page. here’s what he said:

    Dad,

    He is absolutely correct in that it is a margin purchase using your bank’s money. He might also be correct in stating that we might be singing a different tune but I doubt it. As we discussed when we opened your account, there has not been a 15 year interval since 1930 where a portfolio matching yours has not averaged over 3.5%. If we had started in 2000 or 2008, you would still be ok. I think the “risk” of your decision is a lot better than reasonable.

    The decision to pay off a mortgage in or near retirement, for most people, is based on emotions not mathematics. In fact, for the great majority of my clients, I recommend paying it off. They don’t need the extra cash and I aim for stability/simplicity in retiree recommendations. Conversely, at today’s interest rates, nothing short of another major depression and market crash in the early years of a 15+ yr mortgage will result in a loss. So, mathematically, given today’s rates and assuming we can avoid another Great Depression, it makes more sense to take out a long term, fixed rate mortgage.

    It is a unique time we live in. Historically, this little exercise would not have been a reasonable option due to higher rates.

  14. Well, I paid of my mortgage two years ago – actually it was a loan for an extensive remodel/redo, more than the original mortgage of 27 years prior (which had been paid off earlier). I had adequate assets invested and cash flow was no problem. I was ambivalent about paying it off. My fee-only financial adviser said all his clients who paid off their mortgage were glad they did, and advised it. And so was I – it was like a huge weight was lifted.
    Also, since I was making significant extra payments monthly, it was like getting a nice raise. Now that some health issues have forced me to cut back on my schedule I can tolerate the decrease in income. Approaching retirement with no mortgage is a good feeling.

Leave a Reply

Your email address will not be published. Required fields are marked *


*

You may use these HTML tags and attributes: <a href="" title=""> <abbr title=""> <acronym title=""> <b> <blockquote cite=""> <cite> <code> <del datetime=""> <em> <i> <q cite=""> <strike> <strong>