Following the British retreat at Dunkirk in 1940, Captain Pat Grant is one of many Allied soldiers captured by the Germans and sent to the prisoner of war camp at Laufen Castle. He is determined to escape and tunnels his way out, disguising himself as a townswoman. However he is recaptured after several days and sent to Colditz Castle, another prisoner of war camp from which escape is allegedly impossible.

In May 1940, Flight Lieutenant Simon Carter, recently married, argues with his wife Cathy after his influential father-in-law has offered to get him a safe wartime job. He gets into his Wellington bomber but has to bail out when it is hit and, on the ground, befriends a young priest before being caught by the Germans. After attempting to escape from the first camp in which he is imprisoned, he is moved to Colditz.

Dick Player, a Royal Navy lieutenant, is washed up on a beach bleeding and in civilian clothes. He is taken to hospital and escapes but, upon recapture, he is found to have no military identity on him and, given that he speaks perfect German, it is assumed he is a spy. In fact he spent his boyhood in Germany before the war as the son of a diplomat and he is sent to see an old friend of his, Count Paul Von Eissinger, who has a proposition to make, the outcome leading to a spell in Colditz.

American Phil Carrington arrives at Colditz as Lieutenant-Colonel Preston takes over from Major Donaldson as the senior British officer at the castle. Preston makes it his business to get on well with the Kommandant and this makes him unpopular with several of the other prisoners, who see him as a collaborationist. However his methods have their uses when a Polish soldier attempts to escape.

The new security officer Hauptmann Franz Ullman comes to Colditz to replace the drafted Oberleutnant Lehr. He is appalled at the laxity of the regime, including the excessive drinking of the Kommandant's friend Willi. The British officers eavesdrop on a meeting held by the Germans, only to learn that the Gestapo are arriving to take overall control of Colditz.

Phil Carrington is put in with the British prisoners, who are resentful and suspicious of him, especially Simon Carter, as Carrington is supposedly writing a book, explaining the war from the German point of view. After Carter assaults him Carrington is put into solitary confinement to complete his book, which he is sending to be published in America, so far not a protagonist in the war. However the Germans veto the book before it is dispatched and discover that it contains a secret code, warning of Hitler's invasion plans.

When a lorry drives into the courtyard at Colditz, Player is chosen to be smuggled on board to make an escape bid. However once outside of Colditz he is hampered by torrential rain, causing him to catch a chill and become poorly, whilst the American consulate, to whom he turns to for help, are no use and he ends up back inside the castle.

When several seemingly fool-proof escape attempts are thwarted by Ullmann, it seems likely that there is a traitor amongst the prisoners tipping the Germans off. After the various contingents are interrogated, the traitor is identified as a Pole who is court martialed and sentenced to death by his compatriots. Learning that he had been blackmailed by the Gestapo who would otherwise have taken reprisals on his family, Preston and the padre attempt to intervene for mercy but to no avail and ultimately Ullmann has to step in.

On a visit to the dentist in the town, Player learns that one of the German guards, a young private named Heller, is in need of a thousand marks in order to pay for an abortion for his pregnant mistress and the escape committee seize on the fact, bribing him to turn a blind eye to their latest attempt to free themselves. Colonel Preston receives both good and bad news, learning that he has been awarded the Distinguished Service Order but also that his wife has died.

RAF officer George Marsh, who had worked in a psychiatric hospital prior to the war and is now a medical orderly, works out that if he pretends that he is mentally ill, he will be repatriated and returned to England. Ullmann, suspecting a deceit, assigns sympathetic Private Hartwig to befriend him, since Hartwig has a brother with a mental illness and is as aware of the genuine signs as Marsh is. Ultimately Marsh is indeed sent home but a letter received by Preston reveals that he got to live the part rather too well for his own sanity.

Martinet Dr. Starb arrives at the castle, a stickler for discipline who insists that the prisoners salute their captors. Aware that if he rebels against Starb he may be court-martialled in Leipzig, Carter deliberately disobeys, hoping that he can escape from the train taking him to the court house. As ever though the escape is doomed, as is that of Carrington, who accompanies him and tries to pose as an American journalist.

A German corporal is found dead in the camp's parcels room. The German soldiers insist that he committed suicide and want the matter to go no further, to avoid Gestapo intervention. However Preston is convinced that the man was murdered and orders an investigation to be carried out among the camp's various contingents.

On hearing that his wife is leaving him for another man by whom she is pregnant, Lieutenant Ian McDonald is frantic to escape but there are no British escape plans in the pipeline. However, Grant persuades the French escape officer to let McDonald participate in their next attempt but it is a delicate operation needing split second timing and, unfortunately for Mcdonald, it ends in disaster.

When Ullmann complains that the prisoners' possessions are cluttering up their living space, Carter tests an escape plan using the tea chests in which their belongings are to be moved out, but falls and breaks his ankle. However his time in solitary confinement with his leg in plaster allows him to pass on information to his colleagues. Player has also devised an escape plan which, although initially not approved by the committee, is considered workable and will allow four prisoners the chance to escape.

With the prisoners' band playing to provide noise cover and warn of any danger, the four men begin their escape through the kitchen window and across the courtyard, into the sewers and then on to the other side of the wall where they descend the cliff with a rope. Ullmann is aware that something is wrong but by this time the quartet are clear. Player and Peter Muir head for Nuremburg whilst Pat Grant and Carrington board a train which ultimately carries them to the Swiss border and freedom.