The animals also die, and that is because of Adam. But, they were not sinners. Adam was God's "image-bearer" to rule over his creation. Hence, death came upon animals also.
Thus, there is no reason (based upon that) that death should not also come upon Adam's human progeny even if they were too young to know good from evil and were not personally partakers in Adam's sin.
Rom 5:14
Nevertheless death reigned from Adam to Moses, even over them that had not sinned after the similitude of Adam's transgression, who is the figure of him that was to come.
It simply isn't required that they "sin in Adam" to explain the Universality of death.
If, there is another Biblical reason to insist that they were guilty of Adam's sin, then fair enough...but, the mere fact of their dying does not demonstrate that.
No one sins in a vacuum. Others always suffer because of sin.