It really depends on the environment. If the winters aren't too bad in that area, then they can stay outside just fine! But if it's too cold, then they have to be brought inside, or else they might not make it. Some flowers can't be brought in though, since they grow along the walls -- but those types usually thrive without much of a problem anyway!

It's a little complicated, hehe.