The pair, who completed a career Grand Slam in mixed doubles at the French Open earlier this year, led 8-4 in the match tiebreaker but lost 7-6 (1), 3-6, 13-11 to American duo CoCo Vandeweghe and Rajeev Ram on Court 5.
This marks the end of Paes' US Open campaign after he exited the men's doubles tournament partnering Andre Begemann on Saturday.
There was progress though for Rohan Bopanna and his partner Gabriela Dabrowski, who beat Lukasz Kubot and Andrea Hlavackova 5-7, 6-3, 10-7.
The Indo-Canadian pair will now meet unseeded duo of Robert Farah and Anna-Lena Grönefeld in the quarterfinals.
Elsewhere, Sania Mirza and Barbora Strycova moved on to the third round of the women's doubles after a 6-2, 7-6(5) win over Viktorija Golubic and Nicole Melichar.
The 7th seeded pair will now meet unseeded Nicole Gibbs and Nao Hibino next. If they win, a potential clash with top seeds Caroline Garcia and Kristina Mladenovic await them in the quarterfinals.