As “gray is all of the theory”, line of sight calculations might be interesting, anyhow one should not spend too much efforts respectively, as many factors will influence the validity of such a prediction, while disapointment might be preprogrammed if compared to the reality: antenna characteristics, cable & connector losses, propagation variing with weather condition, athmosperical refraction, signal reflections resulting in multipath propagation, (adding to or substracting from the “direct way” signal strength) seasonal effects due to the trees leaves, attenuating a signal more or less. Even the aircrafts bodyshape influencing a transponders signal radiation. There’s even more to the above.
Why not just stick the antenna out “at its best”, use a “plotter” to create range vs altitude plots and then “see what you get”. It might be more, it might be less as compared to a theoretical calculation, but will tell the truth.
Regards
Karl