The first building on the site of Aubrey House was attached to a medicinal spring called Kensington Wells. This was built in 1698 by John Wright, a ’Doctor in Physick’, and by 1705 had become ’much esteem’d and resorted to for its Medicinal Virtues’. From 1744 Sir Edward Lloyd owned the lease on the house and purchased the freehold in 1750.

Lloyd was largely responsible for transforming the house into its current form. John Rocque’s map of London indicates that the wings were added to the house between 1745 and 1754, with the north front appearing to date from the same period. By 1767 Aubrey House was occupied by the politician and art collector Richard Grosvenor, 1st Earl Grosvenor. It had been named Aubrey House, after Aubrey de Vere who held the manor of Kensington at the time of the Domesday Book.

From June 1767 to 1788 the house was occupied by Lady Mary Coke, the daughter of the second Duke of Argyll. Lady Mary made alterations to the interior, with commissions believed to have been undertaken by the carpenter John Phillips and the architect James Wyatt. Little is understood to have survived of these alternations.

Following Lady Mary, the house was occupied by a succession of tenants and was used for a time as a school. Aubrey House stood empty from 1819 to 1823, when it was purchased by developer and builder Joshua Flesher Hanson. The house was known as Notting Hill House by this time and was sold in 1827 by Hanson to Thomas Williams, a former coachmaker. Williams did not live there himself but let it as a boarding-school for young ladies from 1830 until 1854. Williams built a house, Wycombe Lodge, on the site of the kitchen-garden.

After Williams death the house was sold in 1859 to James Malcolmson, the occupier of Moray Lodge (now demolished), which was to the south of Aubrey House. Malcolmson added part of the garden of Aubrey House to that of Moray Lodge and shortly afterwards let the house, with its now smaller grounds to the Taylors. In 1863, after Malcolmson’s death, Peter Taylor purchased the house from his estate with the garden restored.

The house became a centre for radical thought and a haunt for political exiles in the 1860s under Clementia and Peter Alfred Taylor; Giuseppe Garibaldi stayed at the house in 1864 and meetings of the nascent British women’s suffrage campaign were held at Aubrey House.

The house served as a hospital during World War I and later became the most expensive property ever sold in London upon its 1997 sale to the publisher and philanthropist Sigrid Rausing.