you're missing your assumption. denying that "authority comes from the people" doesn't mean people lack authority. no roman, spartan, athenian, theban would have agreed that the authority of their government, or the legitimacy of their laws derived "from the consent of the governed." they derived it from divine law, the gods, the ancestral constitution, their customs and way of life, or the common good as determined by the qualified portion of the citizen body (which itself was derived from the former). none of these reduce to a democratic popular mandate or revolutionary sentiment.
that's why our modern "rights" are just sleight of hand. the founders did not say that rights from from "consensus" or "the people" as a collective act of will. they said - explicitly and repeatedly - that they came from God -this was Self Evident to anyone who practiced Reason, and governments were instituted to secure these rights, deriving their just powers from the consent of the govern. this is conditional because the rights pre-exist the government and come from God, or if you get as deist as possible, from a natural law from the creator that can be derived via reason.
and yet some people don't agree with this today. or, more confusingly, people who don't believe in God at think these same rights are super serious. but no longer because they're Self Evident (how quaint!) but because power is messy and the founders didn't agree so everyone said this is the best we can do. you don't even agree with the founders. this is a mythic framework that is self-defeating and completely unstable. it just means there is no foundation whatever.
secular progressives want to keep the list of rights (speech, religion, whatever) but replace the grounding in a Creator or self-evident natural law (which is NOT subject to change or debate!) with "we all just agree these are good" or "history evolved this way" or "this is the least bad compromise". this turns inalienable rights into a revocable majority preference as soon as consensus shifts.
the flipside to this is people who want to talk about Creator language but act as if the American order is compatible with any metaphysical framing as long as people behave. this is the same error in reverse - pretending rights can float free of their theistic natural law foundation without collapsing.
you treat "power comes from the people and therefore everything comes from consensus" is it is a neutral fact about legitimate governance. that's not correct. that is distinctly a modern, post Hobbes / Locke / Rousseau philosophical commitment. which the founders of the USA only half-bought into - they hedged with divine endowment and self-evident truth specifically to avoid pure voluntarism.
once you sever those foundations you have no barrier (principled or otherwise) against a future consensus that does anything it likes "by the will of the people".
its not that the people need to have a certain amount of power - this is just implying that any government other than democracy is bad, and democracy however functioning is good. the point is that modern democratic faith only looks inevitable if youve swallowed the whole premise that there is no authority higher than the people's will, which itself says there is no authority higher than any person's will.
the older tradition is that there IS a higher authority (the founders still affirmed this!) and that the people's role is to conform to it, not to reinvent it every generation or immigration wave.
without that our "rights and freedoms" are just accidents of history waiting for the next new majority to renegotiate them... which is exactly what we've been watching.