They appear to be joined at the hip...you could uninstall 4.3 and install 4.2 SP1...no Roxio media manager, just the simpler one that used to be in DM.
You can download desktop manager without meda manager.
With that said, I don't think it's that bad. I use it only when I need to access files on the device itself. Otherwise, I just use windows explorer to drag and drop the files onto the card.
I haven't run into problems at all with Media Manager. It is a little slow copying mp3's from my hard drive to the phone, but eh.....it works. Haven't caused any crashes whatsoever on my computer.