The parish of Swanton Novers lies in the northern portion of central Norfolk about 6 miles southwest of the market town of Holt. Swanton Novers is a small and compact village lying just west of the B1110 road which connects Holt with East Dereham. Swanton Novers is a relatively secluded place as it is hemmed in to the west by a large area of woodland which is not crossed by lanes, Swanton Great Wood. Roads lead from both B1110 and B1354 but only to the village giving it a quiet tranquillity. Much of the area outwith the wood is devoted to arable agriculture like much of Norfolk and Swanton Novers sits on a plateau of land around 100 metres above sea level.

St Edmund's church sits in splendid isolation around a half mile west of the main part of the village at the end of a narrow lane heading for the Great Wood. The church has been much rebuilt over the centuries, the chancel, northern aisle and southern porch in 1881 and the tower in 1960/1. The nave shows features that suggest a very early building lies within as a former southern doorway carries a round-headed arch which Pevsner suggests could have been Norman. The lane leading to the church is rather narrow, Church Lane, and ends as footpaths after the church. St Edmund's is well screened by tall hedges until you are almost upon it and is a peaceful spot indeed.