Yes you can use some vinegar in a Rug Doctor carpet cleaner machine.
The Rug Doctor cleaners that you can buy along with renting the Rug Doctor do work good though at cleaning carpets really good.
But if you want to you can add a cup of vinegar along with some water in the tank where you'd fill the tank with carpet cleaner.
The vinegar can help get your carpet clean as well as help keep your carpet smelling good.
You can also add a couple drops of lavender essential oil to the vinegar and water in the Rug Doctor which will also help get your carpets smelling good.
I would highly recommend though that if you're renting the Rug Doctor machine anyway you should also get the Rug Doctor carpet cleaner to get the best benefits out of the Rug Doctor.