Kapsis turned in a number of impressive performances for Aris Nikaia in the early 1990s as well as Anagennisis Arta FC before helping Ethnikos Piraeus achieve promotion to the Greek top flight in his first season at the club, 1993/94. The step up in class did not have a negative effect on the player's game, Kapsis amassing 30 appearances - his highest tally ever. In total he scored just one goal in 123 games for Ethnikos prior to joining AEK during the 1999 season.
He finished that campaign with 15 AEK appearances under his belt in 1999 and was a regular over the next two seasons before, in 2001/02, he was fined for his involvement in a training ground row with goalkeeper Elias Atmatzidis, as AEK surrendered the championship to Olympiakos CFP in the closing weeks of the season.
In 2002/03, he featured in AEK's six UEFA Champions League draws and played 20 Alpha Ethniki games. Kapsis was his usual consistent self in what was a disappointing 2004 campaign for AEK, who finished fifth.
Fine performances for AEK Athens FC earned Kapsis a call-up for the critical Euro 2004 qualifying matches against Spain and Ukraine; he marked Raúl González and Andriy Shevchenko superbly in famous victories. He was also ever-present in the wonderful run in Euro 2004 that gave Greece the European title, and remained a regular in 2006 World Cup qualifying.
After Euro 2004, he left AEK and signed for FC Girondins de Bordeaux. He played 29 league matches in France, but then returned to Greece on a two-year deal with champions Olympiakos. In January, 2007, he signed for Cypriot club APOEL. On November 15 of that same year, Kapsis announced his retirement from football, however in July 2008, he signed for Levadiakos FC. After two seasons he returned to Ethnikos Piraeus.