First, you need to find the RPM package name, just run something like this:

rpm -qa | grep -i virtualbox

Remove the package

sudo rpm -e <package-name> # <- only the name! no rpm extension

Remove kmod related files

yum remove kmod-Virtual*

If you plan to install a new version of VirtualBox, checkout this link:

http://www.tecmint.com/install-virtualbox-on-redhat-centos-fedora/