First of all, let's talk about the type of cake you're dealing with. If it's a dry cake, like a pound cake or a sponge cake, you can probably get away with leaving it out for a day or two. But if it's a moist cake, like a cheesecake or a cake with custard or cream, you'll want to refrigerate it ASAP.
And then there's the matter of temperature and humidity. If it's hot and humid outside, you'll want to keep your cake in the fridge to prevent it from growing bacteria or mold. But if it's cool and dry, you can probably leave it out for a bit. Just use your best judgment, okay?