The below chart lays out the cost of each hat and how much experience each item gives.
Baseball Cap – 100g/40xp – 2.5
Snorkel – 120g/50xp – 2.4
Cowboy Hat – 900g/250xp – 3.6
Feather Headband – 4000g/800xp – 5
Football Helmet – 20000g/2000xp – 10
Jester Cap – 250g/90xp – 2.77
Aviator Hat – 1500g/400xp – 3.75
Winter Hat – 6000g/1000xp – 6
Party Hat – 2500g/600xp – 4.16
Pirate Hat – 300000g/2500xp – 12
Graduation Hat – 12000g/1500xp 8
Alpine Hat – 500g/150xp – 3.33
Top Hat – 8500g/1200xp – 7.03
Magic Hat – 50000g/3000xp – 16.66
The Magic Hat looks to be the best cost vs XP at 16.66 gold per XP.