My idea of prog is pretty much an attitude where you don't let anything restrict you. You don't stick to formulaic procedures such as writing all your songs in 4/4 time. You don't stick to a basic verse/chorus structure, nor to the traditional qualities of a radio hit. But as much as that affects the sound, there's also the important attitude component, that you want to take your music somewhere, or in other words, progress. Why stay bogged down in the same territory when you can push on into other places?
I don't think you can have one without the other. Definitely not the attitude without the music. You can have a prog desire as much as you like, but it's not prog if you're just playing three chords and sound like a Blink 182 cover band!