I have to disagree with you just a bit. While pointing out differences can certainly cause racism, it isn't the entire reason.
You are fortunate that your children didn't notice a difference, or seemed not to anyway. Neither of my children attended preschool, and so was not acquainted with other peoples as much as they might have otherwise been, so when my eldest son started kindergarten, he was telling me a story about "the dark man", a man who was I found out later a substitute teacher. He genuinely liked this gentleman, but when he explained that he was greeting the man by calling "hey dark man", I was mortified. I had to explain that saying that it might be offensive. Then I had to explain what offensive meant, then I had to explain why it could be offensive. So, to keep my son from being seen as racist (or being raised by racists) I had to teach him to be overly cautious about choosing his words.
Additionally, I once overheard two ladies mocking a white mother for not knowing how to care for her biracial child's hair. I was curious to learn, because my youngest son was dating a black girl, and I didn't want to be in the same ignorant position. They seemed to get angry over my curiosity...but how am I supposed to learn how to properly care for my potential future grand babies..?
Instead of denying differences, we should be teaching our children to celebrate and understand our differences. That being different isn't wrong! It would certainly be a boring world if we all came from the same mold!