When the Domesday Survey was being conducted the manor was held by Henry the Treasurer who was an important office of the king. During Henry III's time it was acquired by the famous Adam Gurdon, and it was his daughter Joan who sold the land to the Norton family when Edward I was on the throne. The Nortons continued to hold the manor for almost four hundred years.

In the 18th century Elizabeth Norton married Frances Paulet from Amport and it was their son who sold Nutley to Thomas Hall in 1745. Later the property passed by marriage to George  Purefoy Jervoise who descendents were lords of the manor up until 1905.

There have been several finds that indicate that the area was settled during prehistoric times and Neolithic flints, and axes have been found, also an Iron Age filed system. The 'Gobey Holes' as they are called by the locals are of an unknown time but burnt earth and a Neolithic axe have been found in them. There are also signs of the Romans having at least one building here.

The 12th Century church of Our Lady  was rebuilt in 1845 and it incorporated a few of the parts of the older building, The Victorian structure itself was demolished in 1955 but the graveyard remains.