The UN and Nato must heed the lessons of Kosovo and Bosnia: that diplomacy and force are effective only in alliance

Kofi Annan, the UN special envoy to Syria, did not mince his words when the security council met yesterday. "If things do not change, the future is likely to be one of brutal repression, massacres, sectarian violence and even all-out civil war," he said.

The UN's current international impotence is devastating to behold, with Russia and China holding out against the US, Britain and France over the issue of foreign intervention in Syria. But this was the situation the world faced with Libya, as Gaddafi's forces closed on Benghazi; the crucial change came when the Arab League demanded the security council intervene. We are at that point again, with the Arab League head, Nabil Elaraby, asking Ban Ki-moon to submit the fighting in Syria to the security council under chapter VII of the UN charter, as a threat to world peace and security.

The danger of fighting spilling over to Syria's neighbours – Iran, Iraq, Turkey, Jordan, Lebanon and Israel – is very real. Indeed, by any objective reading of the charter, the security council should pass a resolution authorising the council to take measures including military force.

However, this is where the problem in the security council is at its most acute: to use force against the Syrian army means Nato or US forces in a multinational grouping. Russia and China do not trust either arrangement. Both feel that when they abstained to allow the invocation of chapter VII in the case of Libya, the wording was thereafter deliberately misinterpreted to force the Gaddafi regime out of power and create the chaos that resulted in his summary killing. There is some substance to this complaint, and not just in relation to Libya but also to the experience of the UN's involvement in the 2003 Iraq war and Nato's bombing of Kosovo in 1999.

However, another model offers a positive precedent. In December 1992 Cyrus Vance, the UN special envoy to Bosnia, and I, asked Boutros Boutros-Ghali, the then UN secretary general, to write to the Nato secretary general and ask that they start joint planning for an intervention in Bosnia-Herzegovina. This approach ultimately paved the way for a no-fly zone and a much delayed but successful military intervention by Nato in August 1995. We need something similar now: a UN-Nato plan supported by Russia, in which Nato, led by Turkey, would provide the threat of force needed to support Annan's diplomacy.

We have learned much from past conflicts about Nato-UN and Nato-Russian joint activity, and those lessons could be drawn on now with regard to Syria. In Bosnia Russian troops worked very well with Nato on the ground to implement the Dayton accords. In the case of Kosovo, the experience was not so good – in part because Nato did not give Boris Yeltsin and his peace envoy, Viktor Chernomyrdin, sufficient credit for their diplomacy. Nato falsely claimed that the withdrawal of Serbian forces from Kosovo was purely the result of their 78 days of bombing. Quite apart from the basic misunderstanding of the important role that Moscow played in twisting President Milosevic's arm, this led to a mistaken belief in the strength of air power alone. Air power can tilt the balance of fighting on the ground, but only if it is used as an adjunct to diplomacy; equally, successful diplomacy very often depends on the ultimate threat of force.

In the case of Libya, we saw what happens when there is too little UN diplomatic involvement. Early on, a representative of the secretary general was appointed, but with little scope for working effectively with Nato. Here in the UK we regard that intervention as a great success, but Libyan government forces were only able to retake control of Tripoli airport from militia groups this week, and we have still much to do to achieve a stable government there.

In Syria, the situation in this regard is much better. Annan is a credible figure capable of working with Nato and Russia, and having good relations with the security council and some political leaders within Syria. In this way the threat, and if need be the reality, of military force can more easily be put behind working out a strategy for a transitional administration in Syria, if Russian interests can be accommodated.

Many will ask: why bother with Russia? The answer lies in Kosovo, a state whose independent status is still not recognised by many countries – including five in the EU – because the intervention lacked UN authority.

After US and British mistakes over the handling of the war in Iraq, and now Afghanistan, we are living in a new era of what I have called "constrained intervention". The days of going it alone, without UN authority, are over. Russia can't be pushed aside – its interests must be taken into account. It is encouraging that the Russian Middle East peace envoy has now said that the Annan peace plan can be adjusted.

Russia's main naval port in the Mediterranean is in Syria. Russia supplies the Syrian government with arms and has close intelligence links. This gives Russia's leaders leverage if they can be persuaded to use it; they could ease out the Assads and create a transitional administration. The international community must recognise, however, that Russia is not about to abandon its political interests in that country any more than the US would abandon its vital interests and military bases in south-east Asia, Diego Garcia and Guantánamo, to name but three areas.

At the same time, Russia knows that it can't run for too long in the UN against the tide of Arab feeling in the Middle East. Under President Putin, Russia has already been helpful over Afghanistan: using its influence in surrounding countries to guarantee safe exit routes for the 2014 Nato withdrawal. In nuclear weapons talks with Iran, too, China and Russia are working constructively with the US, Britain and France – the UN permanent five – joined for this negotiation by Germany.

The immediate challenge in Syria is to hammer out guidelines for a Nato military threat, led by Turkey, to intervene – not to replace, but to strengthen Annan's diplomacy. Both Russia and the Arab League will have key roles, as will the US, UK and France.

The best mechanism for breaking the impasse and drawing up such guidelines is the Nato-Russia council, which has formally existed since 2002. Sadly, to date it has not been very effective, but over Syria it could start to fulfil its potential for providing a forum for discussions. The Russians know that the prelude to any no-fly zone is cruise missile and air attacks, co-ordinated by Nato, on ground-to-air missile sites in Syria. To start detailed discussions about this now in the forum of the Nato-Russia council, prior to agreeing a specific UN resolution on force, might ease Russian concerns. The scale of the humanitarian tragedy in Syria demands that we try.

• Follow Comment is free on Twitter @commentisfree

• The first Russian motherboard based on ARM-processor Baikal-M