I think emissions is most of it. The push for 4 cylinders was a result of two things: emissions and economy. This was the decade after the oil crisis, and the only thing on the consumer’s mind at the time was economy, so they pushed them for better mpg. Of course, the government pushed for emissions at the same time, which introduced a learning curve for American auto makers, as before 1974 the requirements for emissions were practically nil. Catalytic converters really choked the engine, and it took a while to learn how to cope, which could also be a product of lazy American engineering. haha In the 80’s numbers started to rise, with the ECU becoming commonplace in nearly every American automobile. 4 cylinders were still tuned for efficiency and low cost, however, so they didn’t see much gain in hp. Making efficient engines was never really an American ideal, so it was never really successful anyway. Americans like big engines with big sounds and big numbers. Leave efficiency to Honda and Toyota (which is how they cornered the American auto market in the first place)! My first car was a 69 Camaro with a 427 getting a paltry 9 mpg back in 1985.
All that said though, American cars are a LOT better than they used to be. Still a bit on the cheap side cuz of the unions, but the mechanicals are getting really good.
They are much better than they were 25 years ago. The world of small engines with big power has finally arrived in the US, circa 2000 and later. Before that time, the only motors which garnered any real attention for performance was the V6/8/10. Anything in an inline configuration was considered an eco box or used for low-end torque (trucks) and treated as such. Now we have factory I4’s making power north of 200 hp and when F/I is added can exceed 300 hp, while still slaughtering the mpg of a comparable V6 or V8. Interior and electrical are still extremely substandard to most foreign models though. 
I say that… and then today I find out that the new Mercedes CLA45 AMG produces 355HP and 332TQ with a turbocharged 2.0L engine.
Since I can’t capitalize numbers, let me put it this way:
THREE HUNDRED AND FIFTY FIVE HORSEPOWER OUT OF A TWO POINT ZERO LITER ENGINE!!!1!!!1!!!
And for less than $50,000.
Just when American automotive manufacturers pull their heads out of their derriere, the bar is raised significantly.
I used to own a Sentra Se-R in the B13 body.
To all not familiar with the car, it’s basically a Nissan Sunny with a Slilvia engine, minus the turbo. (SR20DE)
Mine was tuned to 165hp, which doesn’t sound like much, but the car was properly quick - 140mph in the wet on bald tires. It could do more, but I couldn’t 
We have to remember that for a long time in U.S. fuel economy standards were over the entire range, and large "utility " vehicles didn’t count, hence the move to trucks and SUVs from station wagons.
Now the lack of power was party due to the need to have the car keep to the set level of emissions for years and with carbs that means that you have to have a fairly weak state of tune.
Weight wise a lot of American cars where a lot bigger than there day to day use needed them to be.