E.T.'s two biggest failings were:
[list type=decimal]
[*]The gameplay wasn't self-evident, and you actually had to read the instruction booklet to understand how to play. This wasn't as big a deal then as it is today, now that games almost never even have instructions because they rely solely on in-game tutorials, but considering the amount of players 5 and under who probably couldn't read yet, that is still a legitimate concern.
[*]The collision detection was unintuitive. As the page Cupcake linked explains this pretty well, I'm not gonna re-explain it fully, but it's a big enough deal that it is worth repeating. Game design is for the player more than for the hardware, and it's important to build your mechanics in a way that makes sense to the human mind. Falling in a hole because your head touched it does not work for human sensibilities.
[/list]
Either of these two things would be a moderately big deal alone, but together they spelled doom for the game's sales potential and reception. Certainly enough that I could hardly blame someone for not enjoying it at the time.