The priest's office peculiarly belonged to the sons of Aaron (Num 3:10 "And thou shalt appoint Aaron and his sons, and they shall wait on their priest's office"). The Levites as a whole were for the overall service of the tabernacle (Num 3:6,7 "Bring the tribe of Levi near, and present them before Aaron the priest, that they may minister unto him. And they shall keep his charge, and the charge of the whole congregation before the tabernacle of the congregation, to do the service of the tabernacle"). The tribe of Levi was substituted for the firstborn of all Israel (Num 3:12 "And I, behold, I have taken the Levites from among the children of Israel instead of all the firstborn that openeth the matrix among the children of Israel: therefore the Levites shall be mine"). It does not follow that every single descendant of Levi actually served, or that every descendant of Aaron was a priest. Males from ages 25 to 50 served (Num 8:24,25), and it appears they lived in their cities and different ones came to the tabernacle when it was their turn to work (e.g. I Chron 25,26). Some were disqualified (at least from the priesthood, Lev 21:17-23), but all, young & old, male & female, benefitted from their portion or inheritance.
Yes, God was their inheritance, but this is why they had no land grant (or at least one reason; Num 18:20,24 "And the LORD spake unto Aaron, Thou shalt have no inheritance in their land, neither shalt thou have any part among them: I am thy part and thine inheritance among the children of Israel...But the tithes of the children of Israel, which they offer as an heave offering unto the LORD, I have given to the Levites to inherit: therefore I have said unto them, Among the children of Israel they shall have no inheritance").