While I am a big supporter of moving government decisions making closer to the people (I live in Texas and the government in Austin seems very remote to me, let alone at the national level) but in addition to the earlier comments, limiting the size to around 100 people feels like it would put a lot of work on each person. A lot of people have busy lives already
I saw some research that people who were libertarian tended to share several personality traits - I may forget the details, but they tended to be conscientious, paid attention to details, enjoyed solving problems, etc. While there are a lot of them, that combination of traits is a minority. If you're design a government for the entire nation, you need to be able to find a mechanism that works with the bulk of the population. In one of your replies you mentioned not caring what paint a neighbor used, but... a lot of other people do. I would guess that none of us who take the time to read and write on these topics are actually typical