John Foote, Francis's eldest son acquired the final shares in the estate, and so became the sole owner.
John died in 1800, and his son Robert owned the house until 1828. At some time while in Robert's ownership, the Georgian frontage and end elevations, including the magnificent ballroom, were added, creating the house as it is today. Robert Foote was High Sheriff of Kent in 1815.
Between 1819 and his death in 1830, the house provided one of the locations for the affair between George 1V and Elizabeth,
Countess (later Marchioness) Conyngham, whose country home was Bifrons in Patrixbourne. Elizabeth, who was 50 in 1819, was three years older than George. She was his last mistress, and had her own suite of rooms at Windsor Castle. His devotion to her was widely commented on.
Verbal tradition asserts that, as a result of a bet at a party at which George was present, a white horse was jumped out of one of the ballroom windows, resulting in its death, and that this led to the planting of the extensive snowdrop beds to the west of the house.
The royal connection may well have arisen through Robert Foote's uncle, John Foote's younger brother , Vice-Admiral Sir Edward Foote, who was born at Charlton Park in 1767. Edward Foote was a favourite of George III, and was captain of successive Royal Yachts, the Princess Augusta and the Royal Charlotte, from 1802 until 1812.