I am going to analyse what the best villager count to spend on building bank is.
Here is the building time formula that I came up with after testing a lot of times, it's very accurate:
If a building takes y seconds for 1 villager to build, and it takes t seconds for x villagers to build, then we have the following formula:
t = y - (y/3)*ln(x), where ln(x) is the natural logarithm.
A bank takes 30 seconds for 1 villager to build, so t = 30-10*ln(x).
Assume that, in average, a villager has to spend z seconds in total walking toward the foundation of the bank and then walking back to gather resources.
If we drag x villagers to build the bank, then the total villager second cost would be: (30-10*ln(x))*x+z*x
However, when we drag more villagers to build the bank, the bank will start producing gold earlier, we can convert the amount of gold we gain from starting the bank earlier into villager seconds, so the net villager second cost would be: T = (30-10*ln(x))*x+z*x-2.75*10*ln(x)/0.69
Here is the graph of the function T, "average walking time" is our z here, and "villager count" is our x here:

We can immediately see that building bank with only 1 villager is very bad because it takes 30 seconds and the net villager second cost is no less than 30, if we build the bank with 2 villagers, the average walking time of these two villagers has to be longer than 5.8 seconds so that the net villager second cost is also 30 in this case.
If the average walking time is less than 4 seconds, then more villagers means more efficient in terms of net villager second cost.
If the average walking time is more than 4.5 seconds, then more villagers does not always mean more efficient.
The graph also tells us that if the average walking time is smaller than 2 second or so, then we definitely pull all these villagers to build the bank. So in practice, if there are 5 villagers collecting food together, then it's more efficient to pull all these villagers to build a bank near them.
The result is based on the mathematical model that I proposed here, in practice, there should be other factors that I haven't factored in here, like raiding, villager path finding issue, etc.
By the way, here is my Matlab code that I used to output the figure, the variable names that I used in this code don't correspond to the ones that I used in this post:
x=0:0.1:9;
y=1:0.1:8;
[X,Y] = meshgrid(x,y);
z = (30-10*log(Y)).*Y+X.*Y-2.75*10*log(Y)/0.69;
levels = -10:2:60;
[C,h] =contour(X,Y,z,levels);
clabel(C,h,'FontSize',20, 'labelspacing', 500)
xlabel('average walking time (second)')
ylabel('villager count')
title('net villager seconds spent on building bank')
set(findall(gcf,'-property','FontSize'),'FontSize',25)