In the Old Testament, and to a lesser extent in Paul's letters, we can find criticism of the so-called Pagan (Pagan back them meaning "anyone who wasn't Jewish) practice of household gods. These were most often statues believed to protect the household in which they were set up. When you moved, your household god went with you and protected you in your new house. It probably seems silly to most of us today that you would have a god who would protect only a few square meters, but that's what they believed.
Sometimes I wonder how far we have come from those days. It's as if we have given our understanding of God a bit more territory, but everything else remains the same. Our God protects Christians, and perhaps Jewish people and Muslims - but some would disagree, biblical evidence notwithstanding. It's as if God didn't exist prior to us becoming aware of God as Abraham encountered him. Clearly, though, God existed before that as God is held to be the Creator (however we understand the mechanics of that creation notwithstanding).
The question we need to address is "how did we come to be aware of God?" Now, if God is infinitely vast and the Source of all that is, and since God existed before anything else did - including the ability to write things down - I believe that different people in different parts of the world had an experience of God. They then set about trying to communicate that experience of God using the language and cultural images of their part of the world and their time in history. I believe this accounts for the different explanations and images in the different religions of the world. God gave the Asian countries Hinduism and Buddhism as a way of understanding their world. The people of the Middle East received the Abrahamic faiths. Indigenous peoples received a different way of understanding God that made sense to their life and location. The story of Moses leading the people wandering in the desert for forty years wouldn't have made any sense to the people of southeast Asia, to cite but one example, because they don't have deserts!
In the end, the question is whether we believe our religion is bigger than God or God is bigger than our religion. If our religion is bigger, then our religion becomes our God. We have seen the results of that throughout history, as people have killed each other over religious differences in numbers small and large. However, if we claim to be monotheists, then there aren't other competing gods over which to go to war! We are actually going to war in times and places such as the crusades because our religion has become our God, a huge example of idolatry, all the while claiming God created everything that is - including the very people we are killing in the name of God! We have promoted our household gods and given them a bit more territory, but we really don't understand the message, meaning, and example of Jesus at all!
There is only one God, and that God is behind all of the world's religions. They may call God by different names, or as in the case of Buddhism not really address God at all (the Buddha refused to answer questions about God because he recognized the Brahmin priests were trying to trap him), but God is still there and not demanding to be identified. God is a humble God, which is more than I can say for many of those claiming to follow God.
I don't think there is any harm in your spouse or children going to a Buddhist Temple - or in your going, either. It may be a wonderful place for you to love and honor one another.