Osim played for NK Željezničar in Sarajevo from 1959 to 1968. He subsequently played in France with: Racing Paris, Valenciennes, Sedan, RC Strasbourg. He also played in the Yugoslav national team many times.
In 1978 he ended his career to become a coach. First he coached NK Željezničar until 1986, his team finished second in the Yugoslav championship twice, reached the national cup finals and the UEFA Cup semifinals once. In addition, he coached the Yugoslav Olympic team in the 1984 Olympics, winning a bronze medal.
In 1986, he took over the Yugoslav national team, which reached the quarterfinals in the World Cup 1990. In 1992, Yugoslavia qualified for the European Championship, but were not allowed to play because of the civil war in Yugoslavia. Osim discovered several world-class players such as Davor Šuker, Dejan Savićević, Robert Prosinečki and Zvonimir Boban.
He later coached FK Partizan (June 1991 to May 1992) winning the Cup in 1992) and Panathinaikos Athens (July 1992 to March 1994) winning the Greek Football Cup in 1993 and 1994, and second place in the Alpha Ethniki in 1993).
From 1994 until September 2002 he was the coach of SK Sturm Graz, which he hed to the Austrian Championship in 1998 and 1999, and to the cup victory and the Supercup victory three times each. The team played in the UEFA Champions League three times from 1998 to 2000. In 2000, sensationally, Sturm Graz won their Champions League Group D (against Galatasaray, Rangers and AS Monaco FC), reaching the second round for the first time.
During the last years, Ivica Osim has worked in Japan with JEF United and led this team to the second place in the Japanese championship twice, even though JEF have the second smallest budget in the J. League. In 2005, the team won the J. League Yamazaki Nabisco Cup.