In his 1886 "An Address to All Believers in Christ" David Whitmer said Joseph Smith translated The Book of Mormon by placing his seer stone in a hat.  1886 is a very late recollection, The Book of Mormon translation was completed nearly 60 years earlier, so this description may not be accurate.  But if it is, why would this be seen as a problem by critics?  Physical objects, for example, Moses' and Aaron's rods, the burning bush, brass snake on a pole, pillar of fire and pillar of cloud, the tassels on Jesus' robe, Paul's handkerchiefs and aprons, and other items, were used in Old and New Testament times to aid in spiritual endeavors.  No one should therefore see Joseph Smith's use of a seer stone as in any way improper, even if it was placed in a hat.