Barack Obama joined the long line of US presidents in 2008, sandwiched between a yahoo and a buffoon. What voters saw when they elected him was not just an African American, but a politician who was humane, rational, honest, progressive and genuinely devoted to his family. He was the best-educated man to occupy the White House since Woodrow Wilson and the most articulate since Franklin D Roosevelt. Indeed, he promised to be, as the Chicago Sun-Times put it, ‘the most considerable political figure Illinois has sent to Washington since Abraham Lincoln’.

Unlike Bush and Trump, Obama favoured gun control and national healthcare, condemned torture and consistently denounced the invasion of Iraq. That was America’s most disastrous foreign adventure since the Bay of Pigs Invasion, carried out by the worst president since Warren Harding. History affords no precedent for the current incumbent of 1600 Pennsylvania Avenue, a compulsive liar with the morals of a sparrow and the attention span of a tweet. How on earth, in the interval between the knave and the fool, did Americans come to choose a figure like Obama as their leader?

This is the question the Pulitzer Prize-winning scholar David Garrow attempts to answer, at intimidating length. His book, which deals almost exclusively with Obama’s pre-presidential career, seems to be modelled on Robert Caro’s encyclopedic biography of Lyndon Johnson. It contains such a vast collection of minutiae – detailing, for example, the exact (and very high) approval ratings Obama scored in every class he taught at the University of Chicago Law School – that one feels inclined to suggest that inside this enormous book a little one is struggling to get out.

Yet as Dr Johnson told James Boswell, when urging him to record the tiniest circumstances of his life, ‘Sir, there is nothing too little for so little a creature as man.’ And after a while Garrow’s relentless accumulation of detail engages, absorbs and mesmerises the reader. His references, over 300 pages of them, are equally exhaustive. They document, among other things, over a thousand interviews, which throw up some fascinating discoveries. Furthermore, Garrow’s writing is pretty good by American (and British) academic standards, though he is not above ending a sentence couched in the future tense with that idiotic redundancy ‘going forward’.

His central argument is that Obama, who never really knew his clever, drunken and promiscuous Kenyan father and whose white mother left him to be brought up largely by his grandparents in Honolulu, was a deracinated and self-invented creature. He was a quiet, amiable, intellectual boy, personally somewhat adrift but unworried by his racial heritage in Hawaii, a place less colour-conscious than any other American state. But he later represented himself as a tough guy, and in his autobiography Dreams from My Father, published in 1995 when Obama was thirty-four, he described a troubled youth unrecognisable to his early friends.

In fact the book was a fiction, owing a lot to Ralph Ellison’s novel Invisible Man. It was an attempt to fashion a coherent identity, to present the author not as an African and an American but as an African American. This was essential, since Obama was just starting to climb the greasy pole, having felt from a young age that he was destined to become the first black president of the USA – a premonition others shared. Moreover, after excelling at Harvard Law School, he was open to nasty charges of being a white elitist with a black face, ‘more vanilla than chocolate’. He protested furiously: ‘I’m certainly black enough to have trouble catching a cab in New York.’

Garrow catalogues Obama’s few premarital affairs with unabashed candour. He quotes one girlfriend as saying that Obama was sexually unimaginative but not shrinking. ‘It means so much more than lust, after all, all this fucking we do,’ she reflected. Another, a budding anthropologist named Sheila Jager, said that ‘sex was a big part of our relationship’. Obama was extremely attractive to women: one of his students exclaimed, ‘He’s so beautiful. He looks like food.’ But he was invariably guarded, controlled, calculating and impenetrable. Garrow even suggests that one reason for his marriage to Michelle Robinson was that, politically speaking, he needed a black wife. At the wedding Obama was embarrassed, not for the last time, by the controversial pastor Jeremiah Wright, who hailed him in his address as a fine pillar of African manhood.

Obama admitted that he accomplished little as a community worker in Chicago, but he used this experience as the basis for a successful campaign to enter the Illinois Senate in 1996. Here too he could do little, since the Illinois General Assembly was a sink of corruption controlled by the Republicans. But he proved an eloquent champion of enlightened policies on education, employment, healthcare and childcare. He also resisted the blandishments of predatory lobbyists, standing out from fellow Democrats, some of whom ragged him for arrogance. Only once did he lose his cool, brawling with a tormentor on the Senate floor. Effortlessly charismatic, Obama inspired what one ally called ‘a vision of an honest and uplifting politics and … hope for real change’.

However, his long, poorly paid absences from home apparently shook his marriage and he only persuaded Michelle to let him run for the US Senate in 2004 by promising to give up politics if he lost. He won, of course. Two crucial factors in his victory were his capacity to transcend race and his opposition to the Iraq War – he warned that attacking Saddam Hussein, who posed no threat to America, would stimulate terrorism. But he also exploited the Democratic machine. He played the faith card for all it was worth, despite being, according to Jager, spiritual rather than religious. And he seemed increasingly prone to elevating pragmatism over principle, telling audiences what they wanted to hear and showing disquieting signs of becoming a consensus politician.

Doubts were assuaged by his brilliant keynote speech in support of John Kerry at the Democratic National Convention in 2004. In it, echoing the cadences of his hero Martin Luther King, Obama cited his own story, the story of ‘a skinny kid with a funny name’, to demonstrate that a tolerant, generous, inclusive America was ‘a beacon of freedom and opportunity’. The speech had an electrifying effect and countless journals (though Garrow has counted them) acclaimed him as a ‘rising star’. Changing the metaphor, Time magazine described Obama as ‘the political equivalent of a rainbow – a sudden preternatural event inspiring both awe and ecstasy’. Such was the adulation he generated that one black columnist felt bound to insist ‘Obama is not Jesus.’

This became evident in the Senate, where, far from scourging the money changers, Obama made friends with them. He reversed his previous support for the public funding of political campaigns, in effect accepting that elections should be financed by private donors, who subsequently gorged on handouts from the pork barrel. Other opportunistic shifts impaired but did not destroy the aura, which also survived revelations about his youthful use of cocaine. Garrow says little about Obama’s race for the White House and less about his presidency. This, summarised in a chapter entitled ‘The President did not Attend, as he was Golfing’, forms an uneasy coda to the book.

He did indeed play more rounds of golf than any president since Eisenhower (306 to Ike’s 800), thus helping to disappoint the impossibly high expectations he had raised. ‘Yes We Can’ was Obama’s slogan, but the reality was ‘No We Can’t’. His foreign policy was feeble and other world leaders, finding him prickly and cerebral, actually preferred Bush. He oversaw the assassination of Osama bin Laden but did not close Guantanamo Bay or stop Israel building more illegal settlements. At home his one great achievement, Obamacare, was deeply flawed.

Otherwise, hating ugly infighting and lacking the ruthlessness of, say, Lyndon Johnson, he made little progress in solving social problems. He increased the national debt from $10.6 to $16.4 trillion and deferred to the CIA, about which Garrow is suitably scathing. The journalist Richard Cohen said that the president ‘seems to stand foursquare for nothing much’. When in doubt, he made a speech, all too often a ‘ringing call to do as little as possible’. Tragically, Obama’s political failure was the seedbed of Trump’s electoral success.

Garrow’s important book is a quarry, not a sculpture. He rounds it off by suggesting that Obama’s career was a triumph of style over substance, and that his self-created persona is hollow at the core. This is a somewhat trite conclusion. It seems that Obama, always opaque and inaccessible, has eluded this biographer, despite his heroic labours, just as he has eluded everyone else. Still, it is clear that for all Obama’s ability and high-mindedness, he finally succumbed to the allure of power and wealth. This appeared during his inaugural love-in with Bush and his post-presidential holiday with, of all people, Richard Branson. There were terrible shades of Tony Blair in Obama’s insistence that, after he left the White House, ‘I want a plane and I want a valet.’