My two cents (probably only worth about 1, but whatever):
You haven't had any real contact with him since 2004. He sends you an e-mail or a text or a voice mail now and again. But other than that, nothing, right? For four(ish) years?
It's not like he's hanging around outside of your office, he's not driving past your parents' home on weekends to see if you're coming for dinner. In that case, he's probably not a threat.
The phone calls and voice mails are probably triggered by problems with his current girlfriends or something. He gets lonely/bored/drunk and figures he'll see if you will bite.
He doesn't sound like he's going to do you or your FI any harm. Frankly, if he was the type to do harm, he probably already would have done something. If he was paying that much attention, he would have known you were dating for a length of time, etc.
It's been four years. It's your life, move on. Don't live in fear of the what ifs that sound like probably nots. Announce your engagement!
...but I don't recommend including the phrase "nyah, nyah [insert ExBFs name here]... you lose!"