A website that gets 20,000 daily unique would require at least 5 GB of ram minimum and one CPU should do.
But on the other hand if your website isn't bogged down or loaded with a lot of scripts then even 2 GB of RAM should handle 20,000 daily unique website visitors.
But it can depend on the website and how many plugins you use as some plugins on Wordpress blogs require more resources and can overload a server.
What I would do instead of going for a dedicated server is to go with Cloud Servers where your website can be on many different servers and have better up time and as your website grows in traffic you can easily order a new hosting package which ups your RAM and CPU etc.
All you have to do is order a new plan, pay for the new hosting plan and your account and server resources are automatically upgraded without any downtime to your website so your website can cope with the traffic.