How to know how many calories to consume?

So I weighed myself again and I am indeed down 3 pounds. Not so worried I guess now as long as I keep losing weight. Part of me wonders if I was just that out of shape and was building muscle at the same rate of burning fat so the weight was the same although I was losing fat.