Most farmsteads and hamlets are sited in the sheltered valleys, often terraced into the hillslope. Of the main settlements nine are conservation areas.
Villages are often at river crossings and crossroads in the valley floors and are generally clustered around the parish church. This can be seen at Pitminster, Churchinford and Stockland. The topography of the landscape often influences the form of the settlement, for example, at Membury where the village meanders along the valley, and at Blackborough where the hamlet follows the escarpment.
The exploitation of stone, marl, sand, gravel and clay for agriculture local building and hedging and pottery production have left a legacy of pits across the Blackdown Hills AONB.
A Medieval pottery industry based on chert-tempered local clays once existed in the Membury and Axminster area and a pottery production site of the post-medieval period has been found in Hemyock.
There are significant numbers of largely 18th and 19th century lime kilns, particularly on the Lias Limestone of the Bishopswood area and adjacent to the chalklands of Wambrook parish. Chert from the upper greensand is extensively used in buildings and walls across the Blackdown Hills AONB. Limestone was quarried for building stone at Tolcis, near Axminster, from the Medieval period onwards.