Before the Rock Cemetery, this land was once a working sandstone mine, where large quantities of sandstone rock and fine grains of sand were dug out and used for building material and to sell at markets. Sandstone has long been layered underneath the city of Nottingham for millions of years, and the fine sand still found in the caves today originated from modern day Leicestershire, some 300+ million years ago when the area was a desert. Wind blew these fine grains of sand towards Nottingham over time, which back then the Nottingham area was completely underwater.

Fast forward to the late 17th Century, and this site was famously known in Nottingham as 'Gallows Hill'. It was at Gallows Hill that many men and some women were hanged. The criminal such as a murderer, a thief, a poacher and even a tax avoider, would be tried at the Shire Hall on those charges, and if found guilty were sentenced to their death. They were forced to walk up what is now Mansfield Road, accompanied by guards and the executioner. A custom was made by the landlord of the nearby Nag's Head Pub where the Gallows came into view, in which the landlord would stand outside and offer the unfortunates a pint of Nottingham ale, their final drink before death awaited them. There was even one account that a man was offered a drink before making his way to the Gallows, but he promptly refused and continued his way up the hill. Sadly for him, a messenger followed soon after with a letter reprieving him of his execution, but it was too late. Had the man taken the offer of an ale, he would've lived to see another day and many more.

Crowds gathered at Gallows Hill to witness the hangings take place, with many collecting bits of cut up rope afterwards, which is where the saying 'money for old rope' comes from. The guilty would step up to the gallows and have the noose tightened around their neck. The executioner would then denounce their heinous crime to the public, before pulling the lever and having the criminal drop down through a trap door, dying of slow strangulation until their final breath. It wouldn't be until the late 1700s when breaking of the neck was introduced. This continued at Gallows Hill until 1832, when hangings moved to the Shire Hall. 2nd April 1827 was the final time someone was hanged at Gallows Hill; his name was William Wells and he was sentenced to death for highway robbery.

The site was left untouched for many years, but was eventually founded as Church Cemetery (later becoming Rock Cemetery) in 1848 and designed by Edwin Patchitt, a clerk to the County Magistrates and whose body is buried on the site. A lot of landscaping and work was needed to be done, as it was a sand mine for many centuries and then a execution site. Over 20,000 tons of earth had to be dug up and relocated. Fundraising began in 1851 and 181 shareholders represented a contribution of 959 £5 shares. The tenancies for the land was arranged in 1853, and in 1856, Church Cemetery officially opened. The site was consecrated and sanctified by the Reverend John Jackson, the Bishop of Lincoln. Despite the name having 'Church' in it, a Church wasn't built on this site until a short time after the place had opened. A mortuary chapel was built and opened on 14th August 1879. It costed £2,600 to construct, roughly £300,000 in today's economy. Some of Nottingham's most notable people like James Shipstone and Watson Fothergill are buried here.

Located on this site is Robin Hood's Cave, a place discovered in the 19th Century during landscaping by local clergyman George Oliver. He believed that the set of caves he discovered were part of an ancient druid temple connected to the classic ballads and tales of Robin Hood. The likelihood is that they were just caves leftover as a by-product of the sand mining the couple of centuries before then. This area was originally known as 'Ropewalk', because rope makers worked there. In 1892, the cave became known as Robin Hood's Mammoth Cave, but over the years it was just Robin Hood's Cave.
 
A war memorial was added in 1920 following the First World War, and is Grade II listed, where many soldiers who sacrificed their lives now lie in rest.

Currently the site is owned and operated by Nottingham City Council.