“Jim was a guy you either loved or hated, there was no in-between,” says Amelia Davis, Jim Marshall’s erstwhile assistant and now archivist. “If he loved you, he would lie down in front of a truck for you. If he hated you, he would happily drive the truck over you.”

Marshall, who died in 2010, is known as the father of modern music photography – Jimi Hendrix setting light to his guitar, a young Bob Dylan rolling a car tyre down a New York street, Johnny Cash “flipping the bird” on stage in San Quentin prison. While managers and minders may have bridled at his combative approach – he demanded total access before accepting an assignment – performers seemed to have sensed, in his upfront attitude and commitment to his craft, a kindred spirit. “They loved him. He was one of the guys,” says Davis. “But more importantly, they trusted him because he was so good.”

Marshall’s first love, though, was jazz, and a new book testifies to his more quietly intimate approach to photographing musical titans such as John Coltrane, Duke Ellington, Miles Davis and Nina Simone. Simply titled Jazz Festival, it is not just a collection of great, mostly unseen fly-on-the-wall portraits, but a document of what the veteran jazz writer Nat Hentoff, describes as “one of the most important periods in the cultural history of the United States”. From 1960 to 1966, Marshall photographed the performers and the audiences at the Monterey and Newport jazz and folk festivals, the precursors of the huge hippy gatherings of the late 60s – Woodstock, Isle of Wight, Glastonbury. His images attest to the surprisingly intimate and formal nature of these early events: everyone is dressed smartly in the preppie-meets-boho style of the time and the audience sits attentive in rows of seats as if attending a classical recital.

More intriguingly, when Marshall’s democratic camera captured the young, relaxed, supremely stylish and racially mixed audiences that the jazz giants were attracting, at least on the east and west coasts, he was chronicling a brief utopian moment amid the gathering storm of protest and often violent reprisal that would define the civil rights era of the early-to-mid 1960s. “When Jim Marshall photographed integrated crowds peacefully digging each other’s company at jazz festivals,” writes Hentoff, “news photographs ofpolice dogs attacking black men, women and childrencaptured a very different reality in the south.”

The integrated audiences at Newport and Monterey were, as Davis puts it, “very much the exception in America at that time. Jazz music was the catalyst for change in a very real way. It was universal. People came for the music, whatever their age, beliefs or skin colour. The music united people.”

For Hentoff, the early jazz festivals were an augury of what was to come. In 1963, he reported on the March on Washington that culminated with a concert at the Lincoln Memorial, where Mahalia Jackson sang and Rev Martin Luther King made his electrifying “I have a dream” speech. “Outside of the Newport and Monterey festivals, I had never seen such a large, integrated crowd coming together for a common purpose,” writes Hentoff. “As jazz reached deeply into more white Americans, America began to change.”

In 1966, Marshall also trained his camera on Jefferson Airplane and the Paul Butterfield Blues Band, who were among the first wave of rock groups to play at the Monterey jazz festival. Change was in the air and the following year Marshall would capture Hendrix setting light to his guitar at the Monterey pop festival, a moment that, in retrospect, announced the start of the second wave of – bigger, brasher, louder – music festivals in the late 1960s. Marshall rode that wave too, chronicling the three-day Woodstock festival in 1969 – which he later described as “organised chaos” – and making his name as one of the great rock photographers of the time, despite gaining a reputation for excess that matched that of even his most dissolute subjects.

“He got pigeonholed as a rock photographer, but jazz was his first great subject,” says Davis. “For him it was very personal work and, though he was right there with great jazz photographers like Bill Claxton, he has never really got the respect for it he deserved.”

Jazz Festival should go some way to re-establishing Marshall’s reputation as a photographer who, despite his tough image, made some of the most tenderly observed, quietly powerful studies of a now long gone, but still resonant, cultural moment when jazz was both a unifying force for change and the soundtrack to a turbulent nation whose discontents reverberate to this day.