Not knowing what he said is sometimes rendered (NIV) he did not know what he was saying. This again is taken to mean, he did not know what to say, so he said something inappropriate. The HCSB links the phrase to Mark 14:40 which says, after being caught sleeping, they did not know what to say. Mark 9:6, part of the parallel account in Mark, indicates Peter did not what to say because they were terrified. A few verses later, only Jesus was present, suggesting Peter was confused, thinking the vision was real, and thus the aberrations would need physical lodging. Matthew (17:4-5) indicates they became terrified when the cloud and voice occurred, after Peter had started speaking and while he was still speaking.
Gill's commentary includes this understanding: [Peter]" being ignorant of the design of this appearance; which was, not that this glory should continue, only that he should be an emblem and pledge of what was future; and besides, he was wrong in putting these two men upon an equal foot with Christ, each of them being to have a separate tabernacle as he; and he appeared to be quite out of the way, in proposing earthly tabernacles for glorified persons to dwell in, who had an house not made with hands, eternal in the heavens: moreover, as to the mystical sense, Moses and Elias, the law and the prophets, were not to be considered as in distinct apartments, and separate from Christ, but as agreeing with him, and fulfilled and swallowed up in him; who only, according to the voice that followed, was to be heard and attended to, and not they, as distinct from him. "