I refused to switch seats on a plane so a couple could sit together

I paid extra for a window seat months in advance. A couple asked me to swap for a middle seat at the back so they could sit together. I said no politely. They sighed loudly for the whole seven-hour flight and the woman called me selfish when we landed.