Insane Disk Usage with every AOE3:DE Update on Steam

No Flag querty
Skirmisher
Posts: 125
Joined: Apr 29, 2015

Insane Disk Usage with every AOE3:DE Update on Steam

Post by querty »

Hello everybody,

Every time an new update drops Steam downloads the update files and then seems to "do" something with the entire set of game files.

I am sitting on 100% disk usage for quite a while with constantly over 100mb/s(!) if writing/reading stuff on my ssd. The system becomes pretty much unusable and updates take forever, at least considering we are talking about 70mb.

All drivers, win10 and steam are on their newest versions.

Does anybody else have the same problem? Any solutions for this?

Thanks in advadnce!
User avatar
Netherlands Mr_Bramboy
Retired Contributor
Donator 01
Posts: 8219
Joined: Feb 26, 2015
ESO: [VOC] Bram
Location: Amsterdam

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Mr_Bramboy »

Yep, I had this problem too. It took about 10 minutes to finish.

The way Eaglemut explained it to me, Steam downloads the entire game and then proceeds to actually 'load' it to the disk, which is tough on your system. I have no idea why aoe3 is installed this way, but it's not ideal.
User avatar
Latvia harcha
Gendarme
Posts: 5136
Joined: Jul 2, 2015
ESO: hatamoto_samurai

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by harcha »

It feels like it repacks the 40gb total installation with every update just to apply the update. It takes a couple of minutes even on an NVME drive.
POC wrote:Also I most likely know a whole lot more than you.
POC wrote:Also as an objective third party, and near 100% accuracy of giving correct information, I would say my opinions are more reliable than yours.
User avatar
Tokelau jesus3
Jaeger
Posts: 2353
Joined: Aug 5, 2016

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by jesus3 »

Same here. Due to not having enough space left for another complete DE installation, updates always terminate mid-way on the first try. After waiting a few minutes and refreshing the display screen of my disc space, the "pseudo-occupied" space goes back to free and it installs the update on the second try.

It's pretty weird.
Image
User avatar
Czech Republic EAGLEMUT
ESOC Dev Team
Donator 05
Posts: 4513
Joined: Mar 31, 2015
ESO: EAGLEMUT
Clan: WPact

Re: Insane Disk Usage with every AOE3:DE Update on Steam

  • Quote

Post by EAGLEMUT »

Yeah there is nothing you can do, that's how the DE update process is implemented; repack everything into the huge bar archives.
It doesn't have to download the entire game from internet though, that is not correct. It only downloads changed files (which are relatively small), then it uses the existing installation's data + changed files to create those big updated bar archives in a temporary folder, then once that's all done the new bars are moved to replace old ones and you finally get to play again. The disadvantages of this update approach are obviously very high disk usage while gigabytes of data are being worked with, and also high disk space requirements in order to essentially store a significant portion of the game twice at once during this process.

I would say there are currently more pressing issues in the game than this, but it's certainly something that would be worthwhile to look into improving.

Looking at how things worked on legacy AoE3, there the devs basically had all the huge bar archives set in stone when the game/xpac initially released and that was it, they would not change them afterwards. Instead, all subsequent game updates used a single new bar archive file which would contain all post-release game changes (but nothing else) and any data placed in this archive would "override" everything else anywhere in the other bar archives, therefore all the baseline game data could just stay there and rot forever.
Now, the legacy approach isn't perfect either, as you may have caught on. Dumping all post-release changes into the single bar means that it would accumulate more and more stuff over time, until we get into a similar situation as we are in now. This was a complete non-issue for legacy, seeing as the devs only ever released just a couple small-sized patches, but on DE this bloat could be significant.

A relatively easy solution could be to just go back to the legacy approach and clean it up (do the DE thing) once a year or something, whenever it starts getting bloated.
No Flag querty
Skirmisher
Posts: 125
Joined: Apr 29, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by querty »

Ok I guess it's nice to hear that I don't have to worry about anything not working properly on my end.

I wonder how updating feels like with an actual hard drive instead of my 970 evo. Must be fun ;/
User avatar
Latvia harcha
Gendarme
Posts: 5136
Joined: Jul 2, 2015
ESO: hatamoto_samurai

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by harcha »

hmm that's weird my system remains usable even on a 970 evo, it just takes a while.
POC wrote:Also I most likely know a whole lot more than you.
POC wrote:Also as an objective third party, and near 100% accuracy of giving correct information, I would say my opinions are more reliable than yours.
User avatar
Norway spanky4ever
Gendarme
iwillspankyou
Posts: 8389
Joined: Apr 13, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by spanky4ever »

solution: Get a desk top ;) it will do da job much faster ;)
Hippocrits are the worst of animals. I love elifants.
User avatar
Latvia harcha
Gendarme
Posts: 5136
Joined: Jul 2, 2015
ESO: hatamoto_samurai

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by harcha »

@spanky4ever wrong suggestion, desktops don't make your disk work faster
POC wrote:Also I most likely know a whole lot more than you.
POC wrote:Also as an objective third party, and near 100% accuracy of giving correct information, I would say my opinions are more reliable than yours.
No Flag helln00
Howdah
Posts: 1410
Joined: Jan 28, 2017
ESO: helln00

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by helln00 »

Yeah I didnt even noticed mine lagging when it update, I think your bottleneck is elsewhere
User avatar
Norway spanky4ever
Gendarme
iwillspankyou
Posts: 8389
Joined: Apr 13, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by spanky4ever »

harcha wrote:@spanky4ever wrong suggestion, desktops don't make your disk work faster
Have you ever thought of the possibility that they might work faster? I think they do, and I have been the owner of a laptop - the most expensive piece of shit I have ever spent tons of money on. :shock: :shock: :shock: Pure garbadge, it turned out. :!:
Hippocrits are the worst of animals. I love elifants.
Australia Hazza54321
Pro Player
Winter Champion 2020 x2Donator 01
Posts: 8049
Joined: May 4, 2015
ESO: PrinceofBabu

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Hazza54321 »

I have to uninstall and reinstall everytime theres a new update because it apparently takes too much memory to install the new update, but if i do the reinstall process turns out i have 3 gb free so idk what thats about
User avatar
Germany Lukas_L99
Pro Player
Donator 01
Posts: 2059
Joined: Nov 15, 2015
ESO: Lukas_L99
Location: Lübeck

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Lukas_L99 »

Hazza54321 wrote:I have to uninstall and reinstall everytime theres a new update because it apparently takes too much memory to install the new update, but if i do the reinstall process turns out i have 3 gb free so idk what thats about
Yeah I had the same on my old PC, even on beta.
No Flag RefluxSemantic
Gendarme
Posts: 5996
Joined: Jun 4, 2019

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by RefluxSemantic »

Hazza54321 wrote:I have to uninstall and reinstall everytime theres a new update because it apparently takes too much memory to install the new update, but if i do the reinstall process turns out i have 3 gb free so idk what thats about
In the beta, when trying to update it seemed to me that it basically needs the full space that the game takes available to do the update. So if the game is 60 GB (I remembered something along those lines), then you need about 60 GB available to install any update. Somehow I doubt that they fixed this.
User avatar
Tokelau jesus3
Jaeger
Posts: 2353
Joined: Aug 5, 2016

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by jesus3 »

Gosh, it's happening again... pain in the arse
Image
User avatar
Argentina Jotunir
Howdah
Posts: 1367
Joined: Mar 31, 2020
Location: Argentina

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Jotunir »

jesus3 wrote:Gosh, it's happening again... pain in the arse
It pains me to see Jesus suffering again.
No Flag querty
Skirmisher
Posts: 125
Joined: Apr 29, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by querty »

Ahh shit here we go again:
genius.JPG
No Flag RefluxSemantic
Gendarme
Posts: 5996
Joined: Jun 4, 2019

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by RefluxSemantic »

EAGLEMUT wrote:Yeah there is nothing you can do, that's how the DE update process is implemented; repack everything into the huge bar archives.
It doesn't have to download the entire game from internet though, that is not correct. It only downloads changed files (which are relatively small), then it uses the existing installation's data + changed files to create those big updated bar archives in a temporary folder, then once that's all done the new bars are moved to replace old ones and you finally get to play again. The disadvantages of this update approach are obviously very high disk usage while gigabytes of data are being worked with, and also high disk space requirements in order to essentially store a significant portion of the game twice at once during this process.

I would say there are currently more pressing issues in the game than this, but it's certainly something that would be worthwhile to look into improving.

Looking at how things worked on legacy AoE3, there the devs basically had all the huge bar archives set in stone when the game/xpac initially released and that was it, they would not change them afterwards. Instead, all subsequent game updates used a single new bar archive file which would contain all post-release game changes (but nothing else) and any data placed in this archive would "override" everything else anywhere in the other bar archives, therefore all the baseline game data could just stay there and rot forever.
Now, the legacy approach isn't perfect either, as you may have caught on. Dumping all post-release changes into the single bar means that it would accumulate more and more stuff over time, until we get into a similar situation as we are in now. This was a complete non-issue for legacy, seeing as the devs only ever released just a couple small-sized patches, but on DE this bloat could be significant.

A relatively easy solution could be to just go back to the legacy approach and clean it up (do the DE thing) once a year or something, whenever it starts getting bloated.
Disclaimer: I do not own the game so me complaining is a bit of a moot point (I should buy the game so that I can complain more convincingly).

This seems like a pretty big deal to me tbh. I know many people have SSD's, but I for example still do not have an SSD and in the beta it would take my computer like 20 minutes to proces the new update. It's kinda ridiculous.
France iNcog
Ninja
Posts: 13236
Joined: Mar 7, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by iNcog »

this bloat is not a joke. just did a complete wipe on my computer and reinstalled AOE3DE. 42 Gb? holy shit
Attachments
Capture.PNG
YouTube: https://www.youtube.com/incog_aoe
Garja wrote:
20 Mar 2020, 21:46
I just hope DE is not going to implement all of the EP changes. Right now it is a big clusterfuck.
User avatar
Latvia harcha
Gendarme
Posts: 5136
Joined: Jul 2, 2015
ESO: hatamoto_samurai

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by harcha »

it was 35gb+ since the betas, that's average for a game in 2021
POC wrote:Also I most likely know a whole lot more than you.
POC wrote:Also as an objective third party, and near 100% accuracy of giving correct information, I would say my opinions are more reliable than yours.
User avatar
Tuvalu gibson
Ninja
ECL Reigning Champs
Posts: 13597
Joined: May 4, 2015
Location: USA

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by gibson »

Yea ~40gb is normal for a game in 2021
France iNcog
Ninja
Posts: 13236
Joined: Mar 7, 2015

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by iNcog »

Just not aware of how much these titles use these days
YouTube: https://www.youtube.com/incog_aoe
Garja wrote:
20 Mar 2020, 21:46
I just hope DE is not going to implement all of the EP changes. Right now it is a big clusterfuck.
User avatar
Argentina Jotunir
Howdah
Posts: 1367
Joined: Mar 31, 2020
Location: Argentina

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Jotunir »

harcha wrote:it was 35gb+ since the betas, that's average for a game in 2021
That's not healthy for any PC.
User avatar
Latvia harcha
Gendarme
Posts: 5136
Joined: Jul 2, 2015
ESO: hatamoto_samurai

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by harcha »

@Jotunir you turning on your PC isn't healthy for anyone on this forum. What does that have to do with anything?
POC wrote:Also I most likely know a whole lot more than you.
POC wrote:Also as an objective third party, and near 100% accuracy of giving correct information, I would say my opinions are more reliable than yours.
User avatar
Argentina Jotunir
Howdah
Posts: 1367
Joined: Mar 31, 2020
Location: Argentina

Re: Insane Disk Usage with every AOE3:DE Update on Steam

Post by Jotunir »

harcha wrote:@Jotunir you turning on your PC isn't healthy for anyone on this forum. What does that have to do with anything?
Your insolence will be your own undoing.

Who is online

Users browsing this forum: No registered users and 3 guests

Which top 10 players do you wish to see listed?

All-time

Active last two weeks

Active last month

Supremacy

Treaty

Official

ESOC Patch

Treaty Patch

1v1 Elo

2v2 Elo

3v3 Elo

Power Rating

Which streams do you wish to see listed?

Twitch

Age of Empires III

Age of Empires IV