Fires and the smoke they emit impact air quality, health, and climate, but the abundance and properties of smoke remain uncertain and poorly constrained. To explore this, we compare model simulations driven by four commonly-used fire emission inventories with surface, aloft, and satellite observations. We show that across inventories smoke emissions differ by factors of 4 to 7 over North America, challenging our ability to accurately characterize the impact of smoke on air quality and climate.