Does anyone know if there is a Leisure Centre in Charlbury or nearby?

Not in Charlbury there isn't.

But the nearest ones are:

Carterton Leisure Centre
Chipping Norton Leisure Centre
Windrush Leisure Centre (Witney)

And if you like swimming there is an outdoor heated Swimming Pool in Woodstock which is open in the summer.