My lemon tree had multiple buds but some of them turned yellow quickly and fell off. Pictures below. Does anyone know why this is happening? Note, I take the plant outside for hours for direct sunlight and bring it in at night or extreme heat. I also use Neem oil for pests who I believe were the original culprits. The pests are gone now. I fertilize with Osmosis.

