Yes, he was talking about fellowship. But the emphasis was on YOU, in other word that HE had to maintain fellowship without the help of God. God keeps my salvation even if I sin. God keeps my salvation, though he chastise me. I remain a child of his even if I don't repent. I will never fall away from God, even if I live a carnal life. I will always remain a child of God. Like the prodigal son, if I go and live a life a sin for a period of time, I remain his child. That is eternal security.
This is an area where hopefully BOTH cal and non cal would agree...
Regardless of just HOW you view the concept of Election, once you get daved, than we are indeed 'predestined" to make it to heaven, and to be continually confirmed while here into image of Jesus!