Morley was a family man. He married Rebekah Maria Hope in 1841, and they had eight children, five of which were girls and three were boys. Their eldest son Samuel Hope Morley became the Governor of the Bank of England and was elevated to the peerage as Baron Hollenden in 1912. Their younger son was the Liberal politician Arnold Morley. Samuel Morley died in 1886 leaving behind a remarkable legacy.