The main hall of Scotch Horn Leisure Centre, in Nailsea, was sealed off after drilling work in the walls revealed the mystery substance.
A decision was taken to close the main hall while air quality checks were carried out. The rest of the building, which is home to a gym, café and squash court, remained open.
The building is owned by North Somerset Council but operated by contractors Parkwood Leisure.
Council spokeswoman Zoe Briffitt said:
“The auditorium at Scotch Horn Leisure Centre was closed as a precaution following the possible discovery of a substance that looks like asbestos.
Air checks were carried out this afternoon and we will act accordingly. The rest of the building remains open as normal.”