Some people do have allergic reactions and irritation to their skin when using olive oil especially if you use the olive oil too much.
Use of olive oil too much can sometimes clog pores which can also irritate your skin as well.
So if you're getting irritation of your skin from the olive oil you might either have to dilute the olive oil with some other oil or cease use of the olive oil.
Or it could be simply that you're using too much olive oil too often on your skin.
The irritation should go away soon though if you do get rashes or other skin irritation from the olive oil.
Olive oil is used safely though for many people around the world on their skin but some people may have an allergic reaction to the olive oil.