A few suggestions.
First, check out AVG, a free antivirus program available at http://www.grisoft.com/. I've used it on Win98, 98SE, 2000, and XP on a variety of computers, with no significant problems (unless you're having it check all Outlook Express email, and using OE to read Hotmail accounts).
Second, there's a good chance the problem is your DirectX drivers. Try downloading the newest drivers for your video card, especially if you've upgraded DirectX, especially if you upgraded to a beta. If this is the problem, it's easily possible that all kinds of other programs might work fine, including non-DirectX games or even DirectX games that don't use the one particular feature that's buggy on your system.
Third, you could have a bad memory chip, or just a badly-seated memory chip. This can cause intermittent problems that usually show up only when you stress the computer in certain ways (by running a memory-intensive game, for example), and won't show up during the Power-On Self-Test. There are about a billion free memory test programs out there; do some research and find a good one. I have a linux box that would usually fail when recompiling the kernel, but would almost never fail when doing anything else, and I eventually tracked the problem down to a very-intermittently-bad memory chip.
Finally, if you really think the modem is the issue, completely disable it, uninstall the drivers, and disconnect it from your computer, and see if GeneForge still causes problems for you. If uninstalling your modem fixes everything, and you can't afford the $12 for a new modem (or $5 for a used one), then try reinstalling the drivers (and check to see if there are specific drivers for your modem, although most don't have them). Also, see if there's anything funky in your BIOS settings relating to the modem or the com ports.
Hope this helps.