I don't think that thalidomide is a good analogy here - it's not a normal part of metabolism, and it's not naturally present in foods.

"The dose is the poison" is not true for everything, but it is true for formaldehyde. Unless you also rigorously avoid apples, pears, banana, carrot, spinach, cauliflower, meats, and lots of other foods? Here's a fairly brief but sensible writeup, and some source data.