Let us start with a bit of Latin grammar. In Latin the end of the word gives the word its context and meaning. The ending of the word is called "declension". The Latin has several declensions and thats what makes Ave Satanas wrong.
-as & -i declensions are wrong. That makes Ave Satani also wrong. Ave Satani was made popular in the Omen movie.
There is no such word as "Satan" in Latin although the declension of -a is more of a correct one so that makes Ave Satana correct.