WHY DOES THIS HAPPEN
There are a few different reasons why your plant's flower buds may start to fall off - sometimes this is totally normal, and other times it may be a sign of a wider problem.
For instance, one example of this where there's no need to worry is if your plant is an orchid. Orchids have a natural flowering cycle and will lose all their flowers (including buds) at the end of this cycle. Sometimes they may not flower again for a long time, even though the plant is still healthy and happy! Note that some other types of plants also go through a natural dormancy period, so be sure to check if this applies in the case of your plant.
However, some problems that may cause this symptom include: if your plant has been exposed to a cold draft or recently moved (some plants are more sensitive to this than others), overfertilizing or, conversely, underfertilizing, likewise both overwatering and underwatering may lead to this symptom.
If you're unsure of which particular problem may be causing this symptom in your plant, you should take a look at its overall health and see if it's showing any additional symptoms. For example, if the cause is overwatering, it's likely that the soil will seem overly wet, the foliage may be yellow and drooping and, in particularly bad cases, the roots may have started to go soft and mushy - this is a sign of root rot.
On the other hand, shriveled up leaves and excessively dry soil may be a sign that your plant isn't getting enough water, and so this may be why your plant has lost its buds.
If your plant seems otherwise healthy and happy, you should check to see if that particular plant type has a dormancy cycle.