Flat-out is an adjective, while flat out is an adverb, both meaning absolutely, unambiguously, completely, plainly, clearly, or simply:
He flat-out lied.
Sky-diving is flat-out dangerous.
That was flat-out funny.
In a physical sense, flat-out describes a person lying flat on the ground, with an open posture. Or a map or book lying open on a table:
We spread the map flat out on the table.
His opponent’s punch laid him flat out on the ground.
The out or openness is a metaphor for sincerity and truth in English—if something is open, one can know its truth. And visibility is a metaphor for clarity; i.e., something visible can be known clearly.
Flat-out vs Flat out: Usage Stats
And for an interesting comparison showing the frequency of flat-out vs flat out, see the Google Ngram Viewer chart below. As you can see, both are used with similar frequency—even though they are completely different grammatical functions. (As noted at the beginning of the article, Flat-out is an adjective, and flat out is an adverb).