Hilarious and hysterical have essentially the same meaning: “very, very funny.” However, hysterical is more complicated—and is an interesting example of the way meanings change over time.
Originally, hysterical described a person suffering from the psychiatric (psychological, medical) condition hysteria, and can still be used with this meaning.
Hysteria, both in medicine and in common speech, is the condition of a person temporarily overwhelmed by emotion. Hysteria is most often associated with anxiety, but can also describe other emotions that cause dramatic behaviors such as crying and yelling. It is not used for depression or other less dramatic emotions.
Earlier in history, hysteria was a large category of psychiatric illness including any kind of disordered, dramatic behavior. It has a long history in literature, with a lot of sexism—because people used to think hysteria was a particularly female problem, and women who were upset, even for good reasons, were often dismissed as “hysterical.”
So, hysterical originally meant “emotionally out of control” and can still express this meaning, today. However, one kind of hysteria that is often shown in movies is the uncontrollable laughter of an insane person, such as the character Renfrew in Bram Stoker’s Dracula. And this is how hysterical came to mean “very, very funny”; it described an insane person, laughing uncontrollably!
Today, hysterical can be used in the two ways explained above, while hilarious has only the one meaning, “very funny.”