Is your car a daily driver? If so, I would repair the damage to prevent any rusting from further deteriorating your doors. Take the time to properly deaden your doors (I mean OVERKILL), and keep the volumes down until you do so. It's pretty simple, but it takes work and hard work pays off.