There are two puzzles that for many years have provoked speculation. Why, throughout the world, are so many people fascinated by the fiction and reality of espionage? And why of all people are the British so good at both?

The first is wound up in the fact that people are perennially intrigued by the seemingly unpleasant art of deception. It has been so since Gideon carried out his night raid against the Midianites, the first special forces attack in history. The SAS would have loved him. It goes on through Sir Francis Walsingham and up to the present day, via the X-X System of the Second World War.

As for the British – well, our deception is not only of foreigners but also of ourselves. The self-image of the Britisher is of a genial, bluff fellow who plays with a straight bat and would never consign his opponent to a sticky wicket (the cricketing similes abound). Of course, the reverse is true.

I was once told by an African scholar that the Lord awarded the British an empire on which the sun never set because He could not trust the bastards in the dark. Hence ‘Perfidious Albion’ – a title in which any red-blooded Englishman will exult. And hence a long-standing supremacy in what James Jesus Angleton, the disastrous head of counterintelligence at the CIA from 1954 to 1975, called the ‘world of smoke and mirrors’.

For over a hundred years the British have excelled at the fiction and the fact of espionage. There have been failures – Philby, Burgess, Maclean, Blake – and those have been publicised to the rafters. But there have also been many extraordinary successes of which the public has never learned anything.

It takes an investigator of consummate talent and a narrator of equal skill to unearth one of these triumphs and explain it clearly. Ben Macintyre, who is both, has done exactly that. The tale concerns the second of the two Olegs, Penkovsky and Gordievsky, whose espionage against the USSR changed history by helping the West to win the Cold War.

Penkovsky was a colonel in the GRU (now much in the news) who, disgusted by the immorality of the government he served, chose in the very early 1960s to switch sides and spy for the British. During the Cuban Missile Crisis of October 1962, his contribution was vital. I have heard that stand-off described as two deadly enemies, Kennedy and Khrushchev, playing poker while behind the latter stood a man with a large mirror. Penkovsky was able to reveal to the Americans exactly what the Soviet leader could concede and what he could not. Thus Kennedy never went beyond breaking point and Khrushchev withdrew his missiles. The worldwide praise flowed to Kennedy.

But that story has a grisly ending, which the tale of the second Oleg does not. Penkovsky had taken crazy risks to serve the British. He was unmasked and executed by being strapped to a stretcher and fed, slowly and feet first, into a furnace. Gordievsky survived and now lives quietly and anonymously in the Home Counties.

Like Penkovsky, Gordievsky was born and raised a patriotic Russian. He excelled at school and was accepted into Moscow’s elite Institute of International Affairs, often the precursor of entry into the cream of the cream, the Committee for State Security (or KGB). That meant also being a dedicated communist. And like Penkovsky, he witnessed the sheer brutality of Soviet behaviour and became disgusted by it. While never hating his native land, he too felt he could no longer serve its regime and should try to bring it down.

Rising through the ranks, he was posted to the KGB unit inside the Soviet embassy in Copenhagen. That was where the British spotted and recruited him. The big break came when he was posted to London. There followed a patient, subtle conspiracy to engineer his promotion to rezident (station head) of the large KGB operation in Britain.

The first problem was how to dislodge the existing rezident, Colonel Guk. Fortunately, Guk was an absolute fool. An MI5 officer, deciding he wished to betray the UK and spy for Russia, stole a secret document and thrust it, together with a letter offering his services, through Guk’s letter box. Guk made a dog’s breakfast of the chance to recruit a mole inside MI5, complaining noisily of ‘provocation’. The British government was alerted and the renegade officer was arrested. But word got out. Thoroughly embarrassed, Moscow recalled Guk and he has not been seen or heard of since. His number two, a very astute intelligence veteran, took over. But good things come to those who wait and espionage is so often a game of patience. The number two was recalled and given a job in Moscow.

There was a nail-biting further wait. Would Moscow send a replacement or promote Gordievsky? Careful titbits were provided to prove that Gordievsky was doing a brilliant job. It worked. He was promoted to station chief. This was the mother lode, a huge reef of 24-carat precious metal.

As rezident, Gordievsky at last got access to the complete list of every Soviet ‘asset’ in the UK: all the traitors, all the sleepers, all their cover names, all the ‘agents of influence’. With care the British could – and did – render them useless. In many cases they were granted some apparent gems that were actually of little importance, while also being fed a mass of disinformation. This lasted for years.

But the KGB were not all fools. Slowly it became clear they had a leak, and a bad one. The ‘possibles’ were whittled down to three. Gordievsky was recalled to Moscow.

MI6 advised him not to go but instead to break cover and complete his defection. But he had family in Moscow and wanted to see them. Shortly after arriving, he realised that returning was a mistake. He was followed everywhere. The game was up – bar arrest and terrible interrogation in the deep, silent cellars of the Lubyanka, where only the screams break the calm.

Before Gordievsky left London, some contingency plans were put in place. But even so, how the hell did the British get him out of Moscow under the very noses of the KGB? It helped that the reformist Mikhail Gorbachev was by then in charge, and he wanted more proof before arresting Gordievsky, which provided a few more days. And Margaret Thatcher was in Downing Street. On being informed of the situation, she came out from behind her desk, the baby blues snapping with anger. ‘Of course we must get him out of there,’ she told the head of MI6. ‘He’s one of us.’

So we did, but I will not reveal how. If any spy writer were to put it in a novel, it would not be believed. But, blow by blow, trick by trick, it is all in Ben Macintyre’s book.