Why my boyfriend go "soft" during intercourse?

Me and my boyfriend are both virgins and have been dating a few months now. Twice we have tried to have actual sex and it just hasn't worked out. We've done everything before just haven't been all the way yet. My boyfriend has no problems getting an erection and can keep it for a good amount of time, but as soon as we decide to go all the way he'll put the condom on and go soft a few seconds later. The other night we actually got closer thand we did before but as soon as he was going to go inside me it went soft and we gave up. Even after that though we can still make each other orgasm but as soon as he goes to go inside me it goes soft and we can't. It stinks for both of us because it's the second time and we were so close. Does anyone know what the problem could be? I'd say the condom and I have even tried putting it on him before. Other than that I have no idea.