Any time?
That may be what the letter of your Constitution says, but it is clearly not the spirit.
Suppose covid was 100% fatal - by your reasoning, we should defend this pastor's right to effectively slaughter his congregation by having them gather and expose each other to Covid.
No one here is opposed to the "right of religious assembly" - some of us simply recognize that this right sometimes need to be temporarily suspended for dire public health emergenciesé
If your church had 150 members - and if the Chinese Virus was 100% fatal - your church is not demanding that you must attend. I'm sure that you would choose NOT to attend. But why should those who wish to attend, be prohibited from doing so? Americans take their freedom very seriously.
Last year our church did close for a few weeks - since there were so many unknowns - and many in our church had physical conditions that could be cause additional medical conditions. Once things started to pan out - Our County Executive came out and stated that he did not care what the Governor said - but he had no problem with churches meeting - provided that take reasonable measures. (usually the best govt is local givt) So we re-opened.