This is a song with a tango sensibility.  The tango is a dance/culture which is cerebrally sensuous, or sensuously cerebral;  full of controlled wantonness and ritualized abandon.  The cuckolded narrator is ruthlessly pursuing his unfaithful lover and her new man.  I like how the scene is set from the first line, with noir  elements interspersed with goofy images like the milk truck and the jolly roger.

Mr. Stewart (GB, 6/27/98):  Bob and I were out at a great cabin in the Alaskan wilderness, battery powered boom box playing "My Rival" as we scrubbed the last of the bear shit off dog Eunice (she'd rolled in it, joyfully), and Bob had this interpretation to offer:
        The Narrator:  A new daddy
        The Rival:  His baby boy
        The Milk Truck:  His partner's breasts
        The detective on the case, scar across face, hearing aide, filming the whole charade:  His partner's father with trusty video camera.
        Think about it.  His boy is going to answer for his crime (taking his father away from his partner during the months and months of breast feeding) when he himself is a dad someday.