I would say that nothing bad would happen to you if you didn't clean your belly button out.
I've never really paid any attention or had any thought to clean my belly button specifically and never had a problem with hygiene or anything.
I would think that when I take a bath sometimes or when I take a shower that the water and soap that I use to scrub my belly with would clean the belly button anyway.
But I guess you could use a rag or cotton swab to clean out your belly button but I've never done so and probably never will.
Usually the shower or bath is enough to keep my belly button clean without needing to do any additional cleaning of the belly button.
I guess the worse that could happen if you fail to clean your belly button is bacteria and dirt build up in the belly button leading to possible fungus or just a bad smell.
But as long as you're bathing or showering regularly then your belly button should actually be at least somewhat clean.