Recently repotted some plants and now they’re growing mushrooms

Mushrooms are sprouting up from a few of the plants I recently repotted.

I keep pulling them out and disposing of them but I'm assuming the spores must be in the new soil as they keep coming back.

I have cats so I'm worried that they might be toxic if they accidentally ingested one.

Any advice?