American Express (NYSE: AXP) has seen its stock do well lately, with gains of more than a third over the past year. After suffering dramatic losses when the company lost out on its past card partnership with big-box warehouse retailer Costco (NASDAQ: COST), AmEx shares have climbed as the company has worked on moving forward with growth initiatives. Yet even with the gains that shareholders have seen, American Express still faces some challenges ahead. Any three of the following things could lead to AmEx giving up some of its recent advances and losing ground. 1. Stronger competition in payments American Express has done a good job of carving out a niche in the lucrative high-end card market, attracting customers with cards that capture the attention of status-seeking cardholders with rich benefits and the promise of exclusivity. Yet one issuer of rival Visa (NYSE: V) cards has answered with a black card of its own, and some believe that the rewards are superior to what AmEx Centurion offers. Any erosion of American Express' reputation in serving high-end customers could take away one of the company's main competitive edges. Meanwhile, broader competition in the payments space could eat into AmEx's core business. With companies like PayPal creating their own payment networks, American Express faces the prospect of getting cut out of the equation. Admittedly, PayPal and similar services have made partnerships with AmEx and other card giants in a way that could prevent full replacement. But having to share profits with more parties will almost certainly eat into margin figures and hurt results at American Express and its peers. Image source: American Express. 2. Weakening creditworthiness Another benefit that American Express gets from serving high-end customers is that credit risk tends to be somewhat less for its clientele than for more mainstream card issuers. That offsets some of the added risks involved in issuing cards directly in the first place, which is a practice that Visa and peer MasterCard (NYSE: MA) have avoided. The U.S. economy has been expanding at a good clip for years, and default rates have been relatively low. However, with the prospect for higher interest rates ahead, borrowers could start to see more pressure on their personal balance sheets. That, in turn, could lead to an increase in late payments and defaults, and any resulting need to boost credit loss reserves would eat into American Express' profits. 3. Adverse litigation results could hurt the company Finally, American Express has had to deal with litigation for years concerning its relationship with merchants. Earlier this month, the U.S. Department of Justice (DOJ) decided to drop its antitrust case against the card company, which had questioned whether American Express could ban merchants from encouraging their own customers to use cards from rival networks like Visa and MasterCard. After winning a lower-court decision, the government had its victory reversed at the appellate court level. The DOJ decision resulted from its choice not to petition the Supreme Court for review of the appellate court judgment. However, several state governments had been parties to the litigation against American Express, and they have moved forward with filing for certiorari in asking the Supreme Court to hear the case. What if the Supreme Court hears the case and decides that the appellate court was incorrect in letting American Express off the hook? According to the card company, an adverse ruling could have a material downward impact on its business going forward. American Express has done well lately, and its stock has reflected its recent success. Yet with so many issues outstanding, there's always the possibility that the rebound in AmEx shares could come to an abrupt end. Investors need to keep their eyes on these concerns to see if any of them grow into larger threats to the card company. 10 stocks we like better than VisaWhen investing geniuses David and Tom Gardner have a stock tip, it can pay to listen. After all, the newsletter they have run for over a decade, Motley Fool Stock Advisor, has tripled the market.* David and Tom just revealed what they believe are the 10 best stocks for investors to buy right now... and Visa wasn't one of them! That's right -- they think these 10 stocks are even better buys. Click here to learn about these picks! *Stock Advisor returns as of June 5, 2017Dan Caplinger has no position in any stocks mentioned. The Motley Fool owns shares of and recommends Costco Wholesale, Mastercard, and Visa. The Motley Fool recommends American Express. The Motley Fool has a disclosure policy.