How likely is it that I'm infertile?

Its a possibility but it might not be you. He could be the one with fertility problems or you might not ovulate very often making it more difficult to get pregnant but not impossible. See a doctor if you are worried.