http://www.tomshardware.com/answers/id-1720725/gtx-770-pci-x16-bottleneck.html
Depending on the resolution you game at, no reduction to slight reduction in FPS because you are on PCI-e 2.0 vs 3.0.
nVidia cards handle the backward PCI-e compatibility a little better, but it would only matter if you were running 1.0 or 1.1 haha
Not knowing your price range, I will give you a response by tier (eliminating all double GPU cards, and crazy expensive like Titan and 780ti):
(sales or bundled games could break ties for you)
$550ish With the recent price drops, you can get GTX780 non-reference premium brands (Asus DCUII, MSI Gaming, Gigabyte Windforce) for the same price as reference R9 290x, so the recommendation must be the GTX780. *IF* you have good ventilation and don't mind the extra noise, and are comfortable overclocking your GPU, the R9 290 is a cheaper alternative. That said, with those same conditions, all premium brand 780s overclock very well too.
GTX780 has 3GB vram vs the 290 or 290x's 4GB
$450ish R9 290
$300-350 - GTX770 or 280x, premium brands, the performance once overclocked should be comparable, the GTX770 is generally more, but is also generally quieter and cooler basically a team red or team green preference and the 280x has 3GB vram to the 770's 2GB
$220-290 - GTX760 or 270X - same as above except both have 2GB vram
In a nutshell, if you ignore power consumption, heat and noise as considerations, the new R9 cards are generally slightly cheaper than the comparable GTX700 series equivalent. If you factor them in, nVidia generally use less power, and produce less heat and noise.
So you need to figure if paying x% more $ for y% more performance is worth it to you. Yes = nVidia, No = AMD
If your case / cooling solutions suck, whichever you choose, look for an exhaust style card to get the heat out of your case, they are available in team red or team green.
If in your new system in 2? years you are going to replace everything, then go towards the lower end of GPU now, if you plan to migrate this card over, go towards the higher end and wait on a sale to xfire/sli in the new rig.