The historical profession lives by gloomy maxims – not surprisingly, since its business is human nature – and one of the gloomiest is that the road to Hell is paved with good intentions. That could form the epigraph of John Cornwell’s passionate and engrossing study of the rite of sacramental confession in the history of the Catholic Church. At the centre of it all is one of the most powerful ideas in the Christian tradition, that a priest has the capacity to pronounce God’s forgiveness of sin, around which the industry of confession revolves. Forgiveness, surely, is a good idea. But then comes another: a papal call to holiness for Catholic clergy, by forcing all of them, monks and non-monks alike, into a profession of celibacy. That project began in the 11th century and while it never eliminated de facto clerical marriage in the medieval period, it certainly degraded such unions; one strong impulse in the Protestant Reformation was to reverse this landmark policy. A third good intention was the binding of the laity to a regime of individual confession to a priest, at least once a year. This was one requirement demanded by a very busy council of the Church held at the Lateran Palace in Rome in 1215. Again, it took centuries to get that particular ocean liner going in the right direction and, like celibacy, the practice of individual confession only became universally effective after the 16th-century Counter-Reformation. One influential bishop of great probity and personal austerity, Cardinal Carlo Borromeo of Milan, introduced a new piece of church furniture, the confessional box, to make confessions more private and anonymous. Some 17th-century Protestants, the Scandinavian Lutherans, were sufficiently taken by this innovation to install confessionals in their own churches, a piquant imitation that many Protestant historians have overlooked, while in England, Anglo-Catholics followed suit in the Victorian age.

Cornwell’s contention, documented convincingly and particularly with evidence from the last hundred years, is that these various good intentions have been the road to the innumerable private hells fuelling the travails of Roman Catholicism in recent years: they have lain behind the scandals of clerical child abuse and clerical cover-up that recently threatened to bring the Church’s collective morale to the point of collapse and contributed to the startling decision of Pope Benedict XVI to resign. More good intentions fuel the mix. Take the Counter-Reformation’s admirable efforts to minister more systematically to the poor, especially by providing basic education for those who could not afford it. Such responsibilities fell on clergy who now were forced into real celibacy; the frustrations raised for many by this compulsory discipline frequently played out in power games in which the vulnerable fell victim to clerical anguish.

The bulk of Cornwell’s story concerns the 20th century, for he spans nearly 1,900 years in his first 79 pages. The imbalance is deliberate: first, he can add his own poignant personal recollections to the sorry tale he is telling, but also, from the end of the 19th century, new historical circumstances made matters worse and tied the confessional life of the Western Church even more closely to clerical child abuse. The Vatican restructured and tightened up clerical training, in a campaign orchestrated against an intellectual tendency that successive popes labelled ‘modernism’. Modernism was not exactly a movement, except in the minds of these pontiffs and their advisers, but rather an honest attempt by many Catholic academics to find a Catholic way of expressing the insights of science, scientific history and literary criticism. Rome condemned this pursuit as misconceived and made all ordinands swear an ‘anti-Modernist oath’ – only one symptom of an embattled, defensive attitude to cultures beyond the Vatican’s intellectual frontiers. Seminaries made conformity the prime virtue in clerical formation; that formation began before teenage seminarians had the chance to understand their sexuality or that of those around them, and cut them off from female company as far as possible. From an emphasis on the exalted nature of priesthood, it was very easy to slide to an assumption that rules for the rest of the human race did not apply to priests.

There was worse to come, and this is where Cornwell’s analysis of the crisis of confession breaks new ground. The emotionally anorexic, inward-looking priestly culture of anti-modernism found itself coping with a new pastoral situation created by Pope Pius X in 1910, again with the best of intentions. He overturned the practice of centuries in bringing young people to their first confessions between the ages of 12 and 14, decreeing that from that point on, universally, the age should be seven. Now priests in the confessional were confronted with a much younger range of penitents, with little training in how to deal with them. Already paranoiac attitudes to sexual transgression were woven into the questions asked of frequently bewildered youngsters. Some clergy let their sad sexual obsessions extend to physical contact, and some hid behind centuries of exalted notions of clerical difference as a way of ignoring or compartmentalising their own descent into the very specific sin of destroying childhood innocence.

What a terrible mess has resulted from this long history. From 1962 the Second Vatican Council began trying to clear it up, thwarted by a long rearguard action from senior Church leaders including Popes John Paul II and Benedict. Both tried to reverse a Church-wide movement to replace individual by general confession (which had become the norm in Protestantism during the Reformation), but the laity voted with their feet and did not flock back to advertised times of confession, still less to confessionals. Now the fightback by the hierarchy on confession is probably over, amid the astonishingly rapid change of atmosphere brought about by the accession of Pope Francis. One should never expect perfection, and some of the new pope’s statements about women suggest that he has not moved beyond a rather sentimental Latin American chivalry towards the fair sex, but there is a sudden and invigorating breeze of realism in an episcopal hierarchy which the previous two popes had groomed to be as conformist and compliant as possible to Rome’s wishes. As I write this, I marvel at new forthright statements from German Catholic bishops, reflecting on the surveys of lay opinion which Francis himself initiated: they say bluntly and urgently that devout Catholic lay folk now simply reject vast swaths of official teaching on morality, and they do not pretend that this can ever be reversed. It’s stating the obvious, but after years in which the Catholic hierarchy have made a fine art out of ignoring the obvious, it resounds like a thunderclap: it is, indeed, a powerful form of confession. The reward for realism, and for a sense of real penitence among Church leaders, is already a hesitant, cautious return to the pews, after years of emptying churches. These are interesting and distinctly happier times for the Western Church of the Latin Rite. John Cornwell, who has himself chosen cautiously to return to the Church which did him some very bad turns in his youth, has much to tell it, as it seeks to find a new way forward.