What you're referring to is the case fatality rate, which is the reported deaths divided by the reported cases. Reported deaths is likely underestimated by some amount if you look at the excess mortality data and the number of cases is dramatically underreported given the number of asymptomatic/mild symptom people who never even know they have gotten it. Infection mortality rate is the number which I think you're trying to estimate, which is how many people actually die who actually get infected.
The infection mortality rate (IFR) is highly dependent on age and comorbidities. Changes in the age distribution of who gets sick can have a big impact on the death rate. For that reason, I don't like talking about the mortality rate as a single number, because it can be misleading.
Below is the best graph I found that showed the death rate by age, given multiple different estimates. It overlays it with the flu death rate, for comparison. So yes, it would appear that for many younger people, there is nothing more to worry about tis than the flu. However, for older people it's some significant magnitude more deadly.

As for whether the death rate is very small, I guess you could argue that. I would argue that we just had the deadliest year in modern American history by a significant margin, despite turning our lives upside down to try to avoid just that. Each person's individual risk factor is some equation of their age, comorbidities, etc. and for most people will be very small. However, the more community spread is going on, the more likely it's going to get to the people that wont' survive. Then that seemingly small percentage of people dying add up to truly staggering numbers.