Stanley Kubrick was the great gnome of modern cinema. Hidden away in his St. Albans bolthole, he would labour agonisingly for years on each film, each one finally emerging as a polished and wholly unique jewel. It is the enormous diversity of Kubrick’s painstakingly fashioned gems that continues to fascinate.

How can the trippy wonderment of 2001: A Space Odyssey be understood alongside the jet-black satire of Dr Strangelove? What has the tricksy pathos of Lolita got to do with the visceral shock of Full Metal Jacket? There is no modern director who has produced such wildly different films. "Kubrick," as Steven Spielberg has said, "reinvented himself with every motion picture".

At the same time, the Kubrick myth speaks of a fanatical director who imposed his vision uncompromisingly across each of his movies, personally scrutinising the entire production down to the tiniest detail. After all, to the ill-disguised envy of most of his colleagues, Kubrick could take as long as he wanted to make the films he wanted. Such a situation simply isn’t even a possibility for most directors.

It’s no surprise, then, that the perception of Stanley Kubrick as a director at work is shrouded in speculation and rumour. His later films, all made in Britain, were made under conditions of such privacy that the feeling grew amongst those left outside the circle that there must be some dark secret, some black magic, at work.

A new documentary made by some of Kubrick’s closest collaborators attempts to break that spell and show him functioning as a director. Titled Stanley Kubrick: A Life in Pictures, the epic film was produced and directed by his long-time producer (and brother-in-law) Jan Harlan. We are unlikely to see anything closer to a definitive portrait of Kubrick’s position within the film world.

One of the documentary’s main aims becomes clear the moment the title sequence rolls. It flashes up newspaper headline after headline about Kubrick’s ‘paranoia’, his ‘secrecy’, his ‘obsessive’ and ‘crazy’ behaviour – all the familiar accusations levelled by the film press.

The caricaturing of Kubrick as an ogreish recluse became particularly bad in Britain after he withdrew A Clockwork Orange from cinemas here, something that clearly rankles with Harlan, as it did with the man himself. In the documentary, Kubrick’s wife Christiane confirms that "Stanley didn’t want to be someone who shot tourists on his lawn, then gave them money when they bled".

The truth of the matter is that Kubrick aggravated the press by refusing to play the usual publicity games. Feature directors are expected to invite camera crews onto set, permit the presence of a second crew filming the inevitable "Making of…" documentary, and then, when the movie is finally wrapped, tour the main territories giving their finished product the hard sell.

Kubrick not only refused to co-operate in the shameful back-scratching requirements of press junkets, he actively loathed explaining what his movies were ‘really’ about. As a result, the increasingly wild and elaborate rumours went unchallenged. Of more significance, though, his directorial method and thoughts on film-making were left ill explained. It’s here that Harlan’s documentary proves itself to be a treasure-trove of first-hand evidence of Kubrick at work.

The key is Kubrick’s refusal to explain his films. According to those who knew him best, this wasn’t high-handed arrogance or the auteur’s ego being affronted. It was simply a reflection of his belief that in making films he was exploring questions to which he didn’t already have pat answers. A large part of his film art was based on the unexpected.

The Kubrick myth is so heavily bound up in the idea of his perfectionism that this reliance on surprise seems to be a contradiction in terms. It is, of course, the case that the man invested himself to an astonishing degree in the finest details of his films, working 18-hours a day for interminable months on post-production.

Every one of Kubrick’s friends and collaborators seems to have their own story of his microscopic attention to detail. This usually takes the form of finding Stanley scanning foreign press ads to check that they are the right size or leaving 15 pages of instructions on the care of his cats. It was also the case that, uniquely for a major film-maker, every print of every film would be viewed for quality before being released. Douglas Trumbull, who worked on 2001, says that he experienced a "level of quality control astronomically near perfection".

These stories all confirm Kubrick’s fastidious attention to detail, and it is easy to see how this care extended to the major processes of movie-making, from the script, set design, sound & music, lighting and camerawork, all the way through to the editing and packaging of the film. But attention to detail is not the same thing as control. The legends of Kubrick’s capacity for infinite pains all tend to the same mistaken conclusion: that the films were produced according to some pre-conceived master plan.

In reality, Kubrick was the antithesis of those directors who, like Hitchcock, claim to have storyboarded their film so comprehensively that they can ‘edit in the camera’. And, crucially, he would never subscribe to the disdainful director’s maxim that ‘actors are cattle’.

One of the party-pieces of Harlan’s documentary is the story of Sydney Pollack’s arrival on the set of Eyes Wide Shut. Pollack, an old friend, was well aware of Kubrick’s reputation for exhausting shoots. When he was able to wrap his first scene within a couple of hours, however, Pollack cockily predicted he would be able to return to the US within the week.

For his next scene, Pollack made the mistake of asking how Kubrick wanted him to walk across the room and answer the door. "I don’t know," the director replied, "you decide". So Pollack tried, and kept trying different ways for the next two days, waiting for Kubrick to give him the nod. Eventually, exhausted, Pollack declared himself satisfied, only to be told, "I wondered how much longer it would take you".

The point of the anecdote is not the casual cruelty of the insatiable director. Instead, it illustrates Kubrick’s astonishing patience and openness when it came to the one aspect of the process he knew he couldn’t control – the performances. His favourite metaphors for directing a film were the game of chess and the general in battle. The latter especially gives a sense of the chaos and unpredictability that Kubrick was trying to indicate as his experience on set.

Just as the general has to constantly adapt his strategy to meet the changing circumstances on the field, not least the uncertain behaviour of his own troops, so the director is always monitoring the performances being delivered, forever leading off in unexpected directions. And there is no question that Kubrick always gave his actors the space to head in whatever direction they felt best.

The process, as indicated by Pollack’s experience, could be both testing and exhausting. Shelley Duvall, who seemed to find the process particularly traumatic, confirms that during the filming of The Shining there would be between 30 and 50 videotaped rehearsals of each scene before it was shot on film.

Tom Cruise was another who found the limitless experimentation gruelling. Nicole Kidman told Rolling Stone magazine in 1999 that, while Kubrick often told her to ad-lib her scenes, he would work with Cruise on dozens of takes before reaching a satisfactory conclusion. Cruise, in frustration, finally asked "What are you looking for, Stanley?" Kubrick replied simply, "I want the magic".

It’s comments like these that betray the origins of Kubrick’s special attitude towards actors. In every set-up, in every shot, he awaited the never-predictable arrival of that special moment.

Harlan’s documentary pinpoints the very first occasion of Kubrick capturing the magic. It came when, as a 16 year-old, he took a picture of a newsvendor weeping at the death of President Roosevelt. The iconic photo was snapped up by Look magazine, and set young Stanley on his way.

The precocious young Kubrick kept producing photojournalism for Look while preparing his first ventures into film-making. Even his first short film, The Day of the Fight, originated with an assignment to photograph the boxer Walter Cartier. The limited control of the photojournalist is quite strikingly similar to that exhibited by Kubrick when on set.

If we equate the multiple exposures of the photographer with the multiple takes of the film director, the logic of Kubrick’s method becomes clear. The photographer is able to choose lenses, filters and sometimes arrange lighting with the utmost care. In fact, Kubrick’s extraordinary use of natural or near-natural lighting is a subject worth examining in its own right, but at the very least it demonstrates a preference for photographing in realistic light that can be related to his early work on photo essays for Look.

What is remarkable, though, is the degree to which the mature Kubrick retained the respect for the autonomy of his photographed subjects that would seem to derive from his early work as a photojournalist. Even the carte blanche handed him by Warner Brothers for his later films could never grant him the capability to control his actors’ performances. It could, however, afford him the ability to manage the equivalent of a stills photographer shooting a whole reel for each pose. Each scene in a Kubrick film could be permutated until, by luck or perseverance, the right performances emerged.

There is one more telling moment in Harlan’s documentary, this time from the subject himself. "Every scene has already been done," according to Kubrick. "Our job is just to do it that little bit better". That assertive perfectionism may neatly fit into the Kubrick myth, but the key word here is actually ‘our’.

‘Our job’ is the grand collective drive towards perfection upon which Kubrick insisted. Stanley Kubrick: A Life in Pictures provides abundant testimony that he was only able to achieve this by inspiring fantastic levels of trust and loyalty amongst his many collaborators.

Stanley Kubrick: A Life in Pictures is included on the Stanley Kubrick Box Set to be released by Warner Bros on 10 September 2001.

2001: A Space Odyssey, Lolita, Barry Lyndon, A Clockwork Orange, Full Metal Jacket, The Shining, Eyes Wide Shut.

DVD and VHS box sets include the above 7 films plus two bonus features:

Dr Strangelove or How I Learned to Stop Worrying & Love the Bomb and the acclaimed BBC2 documentarystanley Stanley Kubrick: A Life in Pictures