There is a lot to this discussion to unpack but there needs to be a few foundational elements.
Supply and Demand: A shortage is only created when the supply outpaces demand. In many growing urban areas, this is without question the case. In some declining urban areas, this is also the case as housing quality is deteriorating faster than renovations can be made. In other places, there is an abundance of housing, but no one wants to live there so prices are much lower than the cost of construction.
Housing Type: The concept of a "family" being the primary housing need is dying a very fast death. In 1960 only 13% of all homes were classified as "single-person" and now that is almost two-thirds (29%). It is expected to exceed 50% by 2040, and 1 & 2 person households is expected to exceed 50% by 2030. The space we need for these homes is changing, which creates opportunities and challenges.
Building & Fire Code: It is easy to blame zoning and land use regulations, but they are not the only factors in this mix. Both building and fire code dictate how a building needs to be constructed in terms of the number of stairwells, access, and such. Some places like Dallas are expanding the options for up to 8 units with a single stairwell.
Affordable & Workforce Housing: This metric is not locked in stone for the entire country. For example I worked in a community that if you used that community's median family income numbers, you could not afford to build a structure that meets even minimum standards. When you looked at the MSA median income, it was substantially higher, but the wages in the community were often not high enough to cover the cost of rent. Community that I am in now is the top of the MSA.
Expectations: In the 1950's the average size of a single family detached home was 983 square feet and the persons per dwelling was averaged at 3.0 persons. By 2020 the average was 2,272 square feet but the persons per dwelling dropped to 2.5 persons
So yea... we have a crisis. Popular areas have more people moving there than they have dwellings, and their expectations are different. Builders are trying to keep up, but it is not always what the occupants want or can afford. Finally, limitations outside of just downing are also limiting what builders can construct. Come communities are allowing for ADU's by right, which is not a new concept. Look at carriage houses in pre-car communities.
So what do we do about it. First is to promote, or require, mixed product neighborhoods, preferably also mixed use. Having neighborhoods that have a wide range of housing options that include everything from studio apartments and Town Homes to large lot with everything in between creates multigenerational and mixed income neighborhoods. Another is to rethink how we use existing housing stock. Many large mansions in many historic districts were broken into apartments. Perhaps the same should be done with your more typical single family detached homes today. If not apartments, then duplexes that are split once you enter the dwelling. That way visually the have the same appearance as before, and perhaps even the same occupancy in terms of number of persons. Finally, strategic in-fill development on large lot residentials. This goes beyond just ADU's but actually subdividing the land where possible and allowing for additional dwellings to be built.
I think Gray's book Arbitrary Lines is a must read, and while in theory, it makes sense. But when you start implementing variables into the mix it can get messy. In high demand areas, developers will respond with product as fast as they can to fully capitalize on the market boom. This often means lower quality at a higher cost.
Finally, there is Politics. None of anything I typed above this matter if the politics of a community don't support it. That takes time, education, and good plans that can be implemented in a incremental progression.
