Most health insurance providers do cover the cost of a visit to the urgent care if you really needed to go to urgent care for the problem.
Usually you'll need to pay a Copay or a deductible when having health insurance pay for the urgent care visit.
Urgent care centers do take health insurance though so you can present your health insurance information to the urgent care center when you visit them.