James Bruce (1811–1863), 8th Earl of Elgin and 12th Earl of Kincardine, Governor General of India, Lord Lieutenant of Fife