Everyone knows that Daniel Defoe invented the modern novel. But when, why and how? The textbooks tell us that in Robinson Crusoe (1719) and his other work he took the old forms of romance and rogue literature and injected them with empirical fact. What this leaves out is something that surrounds us today, in the ferment of debate and literary innovation generated by nationalism in Scotland, Ireland and Wales, and the countervailing impulse to Britishness. Just as writers such as Seamus Heaney and Alasdair Gray have been energised by the cultural forces that are breaking up the UK, so Defoe was geared up as a novelist by the agitation around the Anglo-Scottish union of 1707.
The claim might sound unlikely. Defoe is usually thought of as a London Dissenter, but his formation as a writer owed much to his travels in Catholic Europe and to the adventures in Wales and Scotland that go into his Tour Thro' the Whole Island of Great Britain (1724-27). His association with Scotland was especially strong. Troubled by the pro-French, Jacobite ethos that persisted north of the border, he was also attracted to a country in which his fellow Presbyterians were not oppressed, but were members of an established kirk. He came to feel that union would not just help Scotland economically, but would be good for Protestant England facing threats from the continent.
When he went to Edinburgh in 1706, it was as a secret agent working for the crown. Given the unpopularity of the proposed union in Scotland, this was a risky undertaking. A contemporary later reported that Defoe was "a Spy amongst us, but not known to be such, otherways the Mob of Edinburgh had pulled him to pieces". Defoe denied in print that he was a spy, but he so liked to cut a dash in coffee houses that he couldn't resist hinting at his role. This mixture of concealment and showing off is typical of the man, and it gives a sense of how his experiences in Scotland fed into his creativity. Like John le Carré, his practice as a writer was shaped by his activities as a spy. How deeply shaped would become apparent over a decade later, in such novels as Colonel Jack (1722), which deals explicitly with Anglo-Scottish relations, Jacobitism and duplicity.
The idea of the novelist as a spy has often been applied to writers of omniscient, third-person narrative, but it chimes even more closely with Defoe, whose novels tend to be written from the point of view of protagonists who are vigilant because they have something to hide. Briefly an outlaw in England after the publication of an early satire, Defoe had already discovered the pleasures of living undercover, but his mission to Scotland licensed him in deceptions he found congenial. As he gleefully told his spymaster, the government minister Robert Harley, he conversed with "Everybody in Their Own way", airing legal issues with lawyers, business deals with businessmen. These are the ventriloquistic, protean gifts of a novelist in the making.
Defoe was clearly successful in adapting to Scottish conditions. After the Act of Union was passed, he built a career on the contacts he had made. He shipped ale and wine to Scotland, and, in an almost symbolic act, took advantage of the favourable terms that the union created for the Scottish linen industry to commission a weaver to produce tablecloths decorated with the new arms of Great Britain. He put down roots in Edinburgh and was based there for about six years. As a solid citizen, he joined the Society for the Reformation of Manners. Members patrolled the streets rooting out lewd and criminal behaviour - just the sort of grounding in low life that the future author of Moll Flanders needed.
Defoe was so busy networking that it is hard to think of him having any time left to write, yet he wrote so much in Scotland that it is impossible to think of him doing anything but sit at his desk. He churned out hundreds of pages of pro-union propaganda. Behind the scenes, he ghost-wrote speeches. He testified to a parliamentary committee, and made himself so indispensable that (as he later boasted) his proposals on taxing beer "stand in the Treaty of Union in his very Words". In short, he became a player in the history that he recorded in his climactic Scottish production, The History of the Union of Great Britain (1709-10). Though researching this work was initially a front for spying, it ripened into a book that influenced generations of historians and helped Defoe achieve the style of eyewitness immediacy that would be the hallmark of his novels.
Politically, his biggest problem was how to discount street-level hostility to union. The task was particularly awkward because, in England, he had argued for the legitimacy of the popular will. After vainly seeking to deny the strength of feeling in Scotland, Defoe changed tack and, in his History, set out to discredit anti-union protesters. With unprecedented energy and concreteness, he depicted them as a raging mob. There is a powerful story-telling drive, but his account is clinched with everyday details - as when we are told how the Provost of Glasgow, chased by a crowd into a tenement, escaped being murdered by hiding in a fold-up bed.
A respect for material facts had been encouraged by Defoe's education in a Dissenting Academy. Not for him the diet of Latin, Greek and theology that was dished out at Oxford and Cambridge. By 1704, in his preface to The Storm, he was elevating "matter of fact" into a literary principle by doing without subjective, authorial testimony. The Storm validated its account of a disastrous tempest by gathering a mass of letters and statistics, with Defoe acting as compiler. So although it anticipates by a few years the documentary procedures of The History of the Union, it also highlights how innovative the History is in its use of point of view. Take the moment in 1706 when the Edinburgh crowds attacked the house of Sir Patrick Johnston, one of the treaty negotiators:
His Lady, in the utmost Despair with this Fright, comes to the Window, with two Candles in her Hand, that she might be known; and cryed out, for GODs Sake, to call the Guards: . . . one Captain Richardson, who Commanded, taking about thirty Men with him, March'd bravely up to them; and making his way with great Resolution thro' the Croud, they Flying, but Throwing Stones, and Hallowing at him, and his Men, he seized the Foot of the Stair Case; and then boldly went up, clear'd the Stair, and took six of the Rabble in the very Act; and so delivered the Gentleman and his Family.
It was Sir Walter Scott who noticed that Defoe creates "an appearance of REALITY" as a novelist by presenting himself as "a man of plain sense" and by including "some point which ascertains the eyewitness". In the assault on Johnston's house, we are persuaded that Defoe was there by the two candles in the lady's hands, by the mixing of honest-sounding approximation ("about thirty Men") with persuasive exactness ("six of the Rabble in the very Act"), and by Defoe's location as an observer: "the Author of this had one great Stone thrown at him, for but looking out of a Window". All this is designed to convince us that Defoe knows enough to sustain his charge that the protesters were a rabble led astray by Jacobites, yet the construction of the scene recalls the milling crowds in Moll Flanders. The motive is propagandist, but the fruit is a breakthrough in realism.
These techniques are used everywhere in the great, pioneering novels that Defoe wrote a few years later. The first of them, Robinson Crusoe, is an anatomy of early capitalism wrapped up in a Boy's Own adventure. Crusoe on his island, stockpiling goods, keeping records and setting Man Friday to work, is the eyewitness reporter of his own life. He is also, James Joyce put it, "the true prototype of the British colonist". The key word there is British. Though the title page of the novel says that Crusoe came from York, his character was partly based - as early readers knew - on the Scottish castaway Alexander Selkirk. The rekindling of Crusoe's faith when he falls sick and reads a Bible owes much to the Presbyterianism that was bred into Selkirk in Fife. The Scottish strand in Crusoe is a tacit but telling reminder of how the empire became British after 1707. It was the prospect of participating in colonial adventures from which they had been excluded by the English that persuaded many members of the Scottish elite to accept the union.
The novel that most brilliantly exploits eyewitness realism is also the one that deals most extensively with union-related issues. Colonel Jack stands out in Defoe's output for its vivid account of public events. In Moll Flanders (1722) and Roxana (1724), sexually lively women get under the surface of society and tell us luridly what goes on. Colonel Jack is similarly an insider who does not quite belong, an unreliable narrator with a grip on the reader. He rises from a deprived childhood in London, through warfare in Europe and Jacobite rebellion, to affluence in the American colonies. He is always in the frame when a historical crisis flares up, yet, as a witness or minor participant, he never dominates the action. His role is to help the reader see what is wrong in British politics.
The link between crime and Anglo-Scottish disunion is one of the obsessions of Colonel Jack. It appears when Jack heads for Scotland with a boyhood companion. To help them on their way, his friend steals a horse. They are followed by a hue and cry, but, once they cross the Tweed, their pursuers have to give up because the jurisdiction changes. If the border fosters crime, Scotland itself is a muddle of good influences and bad. From the Presbyterians Jack learns the rudiments of morality, but from followers of the exiled Stuarts he picks up destructive politics. He joins the Jacobites in 1708 when they try to stir up a rising in protest against the union, and is with them again in 1715 as the clans march south into England only to be defeated at the battle of Preston.
That the Scottish elite hoped to make money from the union by getting access to the empire is one reason why Jack moves from the cold banks of the Tweed to exotic locations in America. After their defeat at Preston, many of the Scottish rebels were sent as labourers to the colonies. Jack, already a planter in Virginia, is afraid that he will be recognised and becomes a prisoner in his own house. Only when he hears that George I has granted a general pardon do his fears resolve, and he then declares at such length his devotion to the crown that his conversion seems to owe more to relief than conviction. The unease created by this passage is consistent with Defoe's belief that the key thing was to neutralise the Jacobites, not hope to improve their character. The efficacy of the pardon demonstrates the soundness of the post-union policy of making the empire British by exporting troublesome Scots.
As the plots and duplicities unfold, it becomes clear that Colonel Jack is not simply a fable of Jacobite folly and the merits of union. Like other novels by Defoe, it generates sympathy for its dodgy, self-exculpating protagonist. Defoe so empathised with what he opposed that what might have been Hanoverian propaganda came to incorporate convincingly Jacobite thought processes - webs of evasion and denial that were, from his perspective, integral to the Jacobite mentality, but also indispensable to the art of fiction. Like any good spy, and like the generations of novelists who followed him, he understood the importance of internalising the psychology of what he deplored.
· John Kerrigan's Archipelagic English: Literature, History, and Politics 1603-1707 is published by Oxford University Press (£25). To order a copy for £23 with free UK p&p call 0870 836 0875 or go to guardian.co.uk/bookshop