Koligiannis was born in Ellopia, Boeotia prefecture in 1909. He studied Law at the University of Athens and became a member of the Communist Youth movement (OKNE) in the 1930s. During the rule of Ioannis Metaxas, he was arrested and jailed, managing to escape in 1943. He joined the Communist-dominated ELAS under the nom-de-guerre Pavlos Arvanitis.
By 1945 Koligiannis was a member of the Central Committee of the Communist Party of Greece (KKE). He took part in the Greek Civil War as political commissar of the Democratic Army of Greece in charge of Epirus. In this capacity, according to Nicholas Gage, Koligiannis ordered the execution of certain people in villages under his command, in order to intimidate the rest into submission.
After the defeat of the Democratic Army of Greece, Koligiannis fled to Eastern Europe. In 1956, he took over the duties of Nikos Zachariadis when the latter - a staunch Stalinist - was deposed as Secretary General of the KKE. Officially, he was Secretary General from 1961 until 1972.
In 1965, Koligiannis fell out with Mitsos Partsalidis and others over the matter of the composition of the party's Central Committee by members of the Communusts who had remained in Greece as opposed to the Eastern European exiles. The division was widened after the Soviet invasion of Czechoslovakia which the exiles supported. At the 12th Plenum of Bucharest, in February 1968, Koligiannis' Moscow-line exiles prevailed which led to the breakup of the KKE and the creation of the Interior KKE by those who opposed him.
On December 20, 1972, Koligiannis resigned his post, officially due to illness. In actuality, his replacement had been decided by the Communist Party of the Soviet Union. He was replaced by Charilaos Florakis.