99 Toyota Solara- Battery dying

I have a 99 Toyota Solara V6 SLE in pretty good condition. The only ongoing problem I have with it is an Oxygen Sensor Malfunction that I haven't fixed so my Check Engine Light is on. Recently though, the battery has been draining while sitting in the vehicle without it running. It did it once with the key on accessory, I jumped it let it run, turned it off, started it again, turned it off and sat in the vehicle without keys in the ignition and 45 min later it was dead again. We jumped it, got it home, checked the battery which was fine. It doesn't seem to be the alternator either. I drove it for about two more weeks, then one day while waiting in the car with the keys on the dash, not even in the ignition, in 1 hour the battery was dead... We jumped it again and haven't had any problems for a few weeks since. I just don't understand what could cause the battery to die while sitting in the vehicle, but not even having the keys in the ignition. It never seems to be an issue unless one of us is waiting in the vehicle...