The shipwreck of a whaleboat at the mouth of the Hopkins river in 1836 remains Warrnambool's earliest recorded shipping disaster. It was this event, which may well have gone unnoticed, which caused the three survivors to walk back to the whaling station on Griffith Island, at Port Fairy, along the shore line.

It was during this return journey that they discovered an old shipwreck at the mouth of the Merri River, just south of Tower Hill. This shipwreck has since become internationally famous as The Mahogany Ship, believed to be a Portuguese caravel lost in 1522. The Portuguese, under the command of Cristovao de Mendonca, had sent and exploratory expedition down the east coast of Australia.

Warrnambool was officially declared a town site 1847. Warrnambool's Lady Bay was never a particularly suitable location for a major port as its entrance cannot be negotiated in heavy seas and it is exposed to weather from the south-east. The movement of shipping into Lady Bay was indeed an extremely hazardous procedure. At least 28 wrecks are known to have occurred in the bay prior to 1905, not counting many stranding and minor incidents. As adequate roads to Melbourne and Ballarat were not yet developed, the demands of a developing hinterland made Lady Bay, along with Portland, an important port in the west. Regular coastal shipping trade was conducted through the Port of Warrnambool, despite the obvious dangerous nature of the Port.

Construction of the breakwater ultimately led to extensive siltatation of Lady Bay and as a result many wrecks now lie buried under sand dunes of comparatively recent origin.

At least two wrecks Edinburgh Castle and La Bella, remain exposed on the sea floor and can be explored by scuba divers during favourable sea conditions.