Gogrepoc automatically deduplicates files within a particular game, though only if the filenames match ( symbolic links vary across OSes so can't just deduplicate different names without potentially breaking installer sets ), so the most common thing this does is clean up duplicate extras between a game and it's DLC.

Ah, so not when the same files exist across different games (like TW3 with its editions, or games that used to be bundled but are not separated out, and the same extras were tossed into each separate title). Gotcha.

Yeah, the lack of a "shared" symbol link understanding across OS's means that probably the best way to indicate deduplication across games would be a text file, and if things desync then that's no good.

Linux's hard symbolic links are what are needed for something like this to be effective and relatively clean. Though even that would still suck a bit maintenance wise.

GOGRepoC does not seem to be able to fetch the serial keys for x4_foundations_game. I can log this is a ticket on GitHub if you'd like.

Yes, please , please provide all details you can ( don't think I own this game which makes probing it more difficult )

ETA - I do own this game but apparently the version I own isn't any of the ones currently on sale so that's potentially another layer of fun

That is an odd case because I have X4: Foundations + four DLCs (bought separately I think), but behind that link, I don't own any of the editions of that game, including the standard edition (includes the base game + wallpaper)?

Is this a similar case as with Fallout 1-2 games where the earlier "base game" has been replaced in the store with some different kind of base game? (I have pre-Bethesda Fallout 1-2 on GOG, so the current Fallout 1-2 in the store are not marked as owned to me).

Not sure if that is in any way related to that problem you saw, but it seems messy... I sure hope my version of X4: Foundations is still getting all the updates, though.

EDIT: Luckily my version appears to be the latest 6.20 version (the last changelog is for the 6.10 version I think?).

It's not like Fallout. Interplay's and Bethesda's fallouts are different games (by that I mean they have their own library entries).

In the case of X4, they have just remade various packages over and over. So if you want a comparison, Bloodrayne is a better example. There used to be a "BloodRayne Complete Bundle" that was replaced (not renamed) with "BloodRayne Absolute Bundle".

=======================================

Same. I own the game with most DLCs, but none of the 3 current editions. It's just the usual GOG packaging mess.

=======================================

[url=https://github.com/Kalanyr/gogrepoc/issues/108]Done[/url]

(No idea why the link doesn't work LOL. GOG be GOGin').

Okay, I see the problem with X4 foundations, the Serials aren't stored as normal serials (which are attached to the game itself), they are stored as items that have *only* serials, so they never get parsed (since they aren't installers, extras or DLC ). I'll have to parse those and account for the possibility that an installer / extra / DLC can have a serial attached to it as well (not sure if that's actually used but it would be wellformed ).

ETA - After some further poking it actually seems like only DLCs can have serials as sub items, that kinda makes sense, they behave kinda like subgames to start with. Going to guess that the weird format is because the standard setup can't actually support multiple serials

Does anyone have any objections to adding a dependency on dateutil ? To say that Python has been wildly inconsistent about how to parse datetimes and what datetimes it can parse natively between 2.7 and present is something of an understatement. Up until 3.11 they more or less recommended dateutl.

(As of 3.11 I can parse both formats GOG uses natively but that's no real help for 2.7 and is only a partial solution before 3.7 )

Thank you for investigating.

=======================================

I don't really understand Python, so my opinion may be pointless, but if this works natively in 3.11, why is making 3.11 a requirement not a valid option? Is there a specific reason why anyone would choose to stay with 2.7? I think I use 3.10 at the moment.

Versions <= 3.7 I'm not particularly worried about since those are mostly updatable, however some NAS don't support Python 3, so I try and support 2.7 as much as possible.

So a little progress report:
Can now get serials from DLC
Can now get background images from DLC

Using last modified data seems to be going well, I use the last modified from the MD5 if it's available, otherwise I use the header data from the 0 byte call used to get the file (I'd prefer to use HEAD but for some reason it returns completely different data))

For last modified the current default behaviour is that
Installers are treated harshly, if the current update date doesn't match the old one and there's not a positive MD5 match then the installer will be marked as changed ( you can overwrite this with a flag )

Extras are treated loosely, update data is tracked (with the old update data being the oldest non-updated one) but they won't get marked as changed soley because of updated (you can overwrite this with a flag)

Clean will remove the needs to be updated tag when it orphans files
Download will remove the needs to be updated tag when a file is successfully downloaded.

Am still considering using the files Last Modified date rather than using manifest heuristics.

ETA - Since I used BGIII to test this I now get to wait ~10 hours while the maaaaybe updated Mac installers download. but it'll be nice to not have to manually track that.

ETA2 - Think I'm going to adjust how this works, new additions will be presumptively marked as "changed" so that old files with the same name get removed, verification that passes MD5s will remove the change note as will a successful import of a file, MD5 matches from the manifest will inherit the changed state of the previous file, as will a successful download. I'll provide a flag to let verify mark anything that passes as unchanged. Clean probably shouldn't mark things as clear with this change, though it should still orphan things that are marked as changed. That'll mean that only successful downloads , successful imports, verification that passes MD5 (or size and name checks with the looser flag) will clear out a changed marker.

ETA3 - The subtests went well, time to try a full manifest update (though I won't turn on strict extra checking yet, I will probably add an option that lets you set that from current manifest state while offline so that you only need to do the initial full update ( to get the update times) rather than having to do another one just to compare the updated to the last_updated )

ETA4 - Seems promising so far, of the 100 games checked so far only had 2 installer items that got marked as changed (the vast majority are new entries or have MD5s ), so the cost for that was low as I anticipated. The largest pain is probably going to be the Mac game versions that have gamedata files without MD5s that are huge, which will probably cause pain once but unless you've been managing those manually the chances they are out of date is actually ~100% .

ETA5 - Seems good from a functionality stand space. I probably need to implement a debug manifest vs standard manifest though, there's quite a bit of extra data now that's super useful for debugging but that can make for a very large manifest (my ~2600 games makes for a ~450 MB manifest , which takes ~3 minutes to load or save ).

ETA6 - Yes, I definitely need to either slim the manifest down or alter how download works, currently i have to update the manifest after each download to prevent desyching between the real state and the manifest state if things get interupted and that's far far to slow at 3 minutes per write, especially since writing to the manifest and moving files requires total locking. Hmmm, maybe I need a !provisional folder where completed but unchecked files can be temporarily moved too, then I can check that at the end of a download, and for recovery I can check it at the beginning and treat those files as if they'd been moved there. That would allow me to do only one manifest write.

ETA7 - Okay, so got the installers all synched up and verified ( Baldur's Gate III, Tropico 6 and Bard's Tale IV on Mac are by far the largest issues since these all involve huge MD5 less files, either missing MD5s for packages, Zips without MD5s or GameData files that lack MD5s and will require downloading ~300-400 GB if you have all of them, Windows lhas about 40 GB of files and Linux about 15 GB of similar MD5less installers ). Functionality works but I definitely have to fix the performance issues before this is ready for prime time. Probably won't have a chance this week thoiugh, so maybe next week before I do a public release.

Looking at my extra's that's also another potential issue I have a bit over half a TB of extra across games (though a *lot* of that is the pre 2.0 Cyberpunk installer ) and so little of that has MD5s that I don't even check for MD5s on extra so it was right call to treat these loosely.

Ooh, in some mostly unrelated news (found out due to the full update), Imperator Rome has been updated to 2.0.4 and this seems to have solved the caching issue that's plagued it for approximately 10 000 years.

Wow, the game, "Imperator Rome" must be at least 7,000 years older than Rome itself!

So as a summary:

I implemented the functionality for:

Using the server provided Last Modified date to handle stuff like Baldur's Gate III's GameData files on Mac

Scanning DLC for additional background images

Scanning DLC for additional serial keys

However
the performance hit it too high to be practical.

I need to refactor things a bit ( to reduce manifest writes) and look at slimming the manifest down for use mode (as opposed to debugging).

Outcome
Probably looking at next week sometime before I get a chance to get back to this and work on that