The phrasal verb act out has two main meanings, which are connected: 1) to perform, like in the theater, and 2) to express emotions—especially anger—dramatically (physically, and vocally):
1) Let us act out the argument between Hamlet and Ophelia in Act III of the play.
2) Children act out frustration by screaming and hitting but adults should not!
The original meaning of act out was (1) to perform, like in a play—but not only in a play:
The karate instructor acted out how to steal an opponent’s sword.
Meaning (2) to demonstrate emotions, is based on an idea, in psychological science, that people often behave inappropriately in order to to let out bad feelings which they have been holding inside:
The patient was acting out anger at her mother by yelling at her friend.
Act out does not always imply anger; but people often do act out anger. The phrase act out emphasizes expression, as indicated by the preposition out; compare spell out, speak out, cry out, write out, etc.
Act up may seem similar to act out, but its meanings are very different from act out. Act up itself has two main meanings: 1) to behave rebelliously or improperly, and 2) to become active—for a health problem:
1) Two children were acting up in class and they got in trouble.
2) My arthritis tends to act up in February.
The first meaning also applies to machines, when they behave incorrectly:
My computer is acting up; I need to take it in for repair.
The second meaning can apply to many conditions, including mental conditions such as depression or schizophrenia, and even injuries, large and small. For many conditions, to act up means to cause pain (temporarily):
My headaches are acting up again.
This second meaning also seems to be a metaphor, based on the first meaning, to act rebelliously or inappropriately—as if your body is rebelling against you by causing you pain.