Most of my story ideas begin with a character, but occasionally something else sparks my imagination: music (Blade Dancer), dreams (If Angels Burn) and art (Nightborn.) Regarding the latter, every year I go to a particular art show to see John Galbo, one of my favorite artists. His work is a constant source of inspiration for mine, and it was one of his gorgeous photos of the French countryside, Le Mistral, that initially gave me the idea for Nightborn.
The three elements in Le Mistral are very simple but striking: a field of lavender ready to be harvested, a lovely old manor house, and a storm brewing over the mountains in the distance. Whenever I've looked at this print on the wall in my office, I've always wondered who lived in that house, and how many storms they've weathered. One day I decided to answer those questions, and wrote the first outline for Nightborn.
Finding inspiration with which you can build an entire world isn't difficult as long as you pay close attention to your reactions to it. Does the source invoke a strong emotional response in you? This is important, and it should be passionate enough to keep you from being distracted by other bright, shiny ideas. Also, your passion will translate onto the page and communicate itself to the reader; not something you ever want to be tepid. Are you curious about the source? For you to base a world on something, there should a significant storytelling opportunity already there, looking you in the eye, and you should really want to jump on it. And finally, are you willing to take the time to explore it? You can't build a world in one day or with one glance, so you have to make a commitment. Outlining, researching, drafting -- you're looking at months, even years of work here.
Revisiting the point about emotional response: you can build your world based on anything -- I once wrote a 100K+ novel inspired solely by the words carnival geek -- but if you're not passionate about the source of your inspiration, you're probably going to lose interest in it. If you think of world-building as a love affair between your imagination and the object of its affections, then you can better judge whether or not to dive in. You don't fall in love based on a nice, lukewarm, ho-hum response to something, nor should you world-build that way.