Pivity

[FastDL] resource.AddFile() and resource.AddSingleFile() for FastDL'ing addons?

Last read Never
#1

I've been told from a member of another forum to post here, since they told me I can find better help here.

Also FYI. Yes, I tried searching it up and yes, I tried "fixing" it by myself.


Ok, so a bit of context

I got my FastDL setup, the structure is correct, compressed maps are getting downloaded.


First occurrence of my problem appeared when I tried downloading a map ttt_glacier.

(https://steamcommunity.com/sharedfiles/filedetails/?id=228041314)

(I remember we had problems with this map before, so we ditched it)


I noticed this map contains materials, models and sounds in separate folders, not compiled into the map (I think you can do something like that). So I compressed them and put them into the FastDL.

Then I noticed that the map is getting downloaded, but it's materials, models and sounds doesn't.


Later I noticed that when I compress (some) addons' "resources" get downloaded. That isn't true for all addons, though. This is thanks due to them having resource.AddFile() a/o resource.AddSingleFile() and adding their own files.

(Some addons used resource.AddWorkshop(), but I don't want that, because supposedly it's slow and that's the reason FastDL exists for Garry's Mod)


What I tried?

Okay, so ttt_glacier HAS A BIT OF FILES, heck even a normal weapon like https://steamcommunity.com/sharedfiles/filedetails/?id=646754302 has a lot.

Adding every weapon, player model, map manually to the resource.AddFile() would take forever.

Also, I'm worried about the 8192 file limit for this function (but when I look at that number now, it might be way more than enough, who knows).


So I thought about automating that process and making a JSON file using bash (I'm using Ubuntu 18.04) containing all paths to the files and put them into resource.AddSingleFile() with a Lua script.


Also I thought about putting only maps on FastDL and using resource.AddWorkshop() (Or a host collection) for other stuff.


But it feels like that's not the right way to do it... I don't know.


Tell me what you think. How do other gmod game servers approach this problem and what would you do or what should I do?

Also if you want more information feel free to ask.

My friend told me that back then when we tried doing it, we served everything through FastDL (not only maps), but I'm not sure.

Thanks in advance! :D

Disagree x 1
#2

First: WorkshopDL is faster than FastDL (because its downloading a whole addon from steam-cdn instead of every file at one time)

so i recommend to use WorkshopDL (FastDL is older than GarrysMod btw)


Second: if you really want to use FastDL you could try https://github.com/JustPlayerDE/FastDL-Generator (should work on linux, i currently dont know anymore if i used any non-core feature) (you can also use this as a base for rewriting it in bash if you want ^^)

#3

Is actually WSDL faster than FastDL? I don't think I would set it up back then if I didn't read that it wasn't worth it in the first place. Or maybe something changed?

Either way, if I need to go with WSDL. I want to ask a few questions about it:

  1. Does it redownload addons that were already installed by the client before? I think that might be an issue if that's the case (I think I had a few situations like that).
  2. Does it have a file limit? I remember a new server had a problem with that when people were trying to download a map (caused a missing map error). Though I don't remember if they had FastDL and then they switched to WSDL or vice-versa that fixed it.
  3. I heard that WSDL had a limited download speed, that's why people were using FastDL. Or maybe it was because of that file limit... I don't remember. Maybe I should utilize both, FastDL for maps and WSDL for addons?
  4. I just noticed (I haven't checked WSDL, because I was sure it was "worse" until now) that people praise it for not consuming your bandwidth, being faster in download speed, more optimized, etc. Is that true? and could you tell what do you think/know?
#4
  1. WSDL redownloads only damaged or missing/updated addons
  2. Everything has a limit. in this case the workshop files can be max 2GB (how many files you have in that big file doesnt count to a global limit, exception are Lua files which count to the Lua file limit and any file you load to the cache (e.g. models))
  3. The download limit is your internet speed or connection to the steam server you are downloading from.
  4. I had a server with ~150 addons (about 5GB) and downloaded in ~10 minutes (my download is 100mbit). FastDL will be slower than that just because of the one-file-at-a-time problem (starting connection to download may take longer than the download itself) and FastDL speed also depends on the Connection to the One webserver (except if you have an own cdn or something like AWS) and so it may take longer the more are downloading at the same time
Informative x 1
#5

and if WSDL should have a limit then it would be 8K too, and if you have 8K addons on your server you simply have too many addons ^^

(update: it actually doesnt have any limit, but the client may crash after too many big downloads)

#6

Alright thanks! But could you answer last questions, that I forgot to include in the last post?

  1. While I was reading about WSDL, someone wrote that you need to set both +host_workshop_collection and then add each addon with resource.AddWorkshop() in a Lua script. Are both necessary? If not, which one should I use? (https://nodecraft.com/support/games/gmod/how-to-add-workshop-collections-to-garrys-mod)
  2. If I used WSDL for maps are ALL maps going to be downloaded at once, or only the one that is currently played? Or maybe I would need to add a check for resource.AddWorkshop(), so only the current map gets downloaded?
  3. Is it going to download addons that don't even have any content? By that I mean addons that only contain server-side Lua scripts.
#7
  1. host_workshop_collection is for your server to download a whole collection so it has them (also loads lua) (which you should use for server-side addons on workshop), resource.AddWorkshop() is used to say the Clients (player's game) to download specific addons from workshop (you only need it for addons you want to download on client)
  2. Yes. Any workshop addon you put into resource.AddWorkshop() will be downloaded as soon as you run it (not if the player is already fully loaded). You may need to use checks for that. (if the server runs a map which is loaded from the workshop already you actually dont need to add it to resource.AddWorkshop)
  3. It downloads any addon you specify (in short: yes)*


*: it cant be private/removed on workshop tho



https://wiki.facepunch.com/gmod/Workshop_for_Dedicated_Servers

https://wiki.facepunch.com/gmod/resource.AddWorkshop

Informative x 1
#8

you can make a whole collection just for the client (only models/etc) and use this hacky way to add them to the download:


function util.GetWorkshopCollectionIDs(collectionid, callback)
    if collectionid == nil or callback == nil then return end


    http.Fetch("http://steamcommunity.com/sharedfiles/filedetails/?id=" .. collectionid, function(source)
        local source_ = string.Explode("<div class=\"workshopItem\">", source)
        local t = {}


        for k, v in pairs(source_) do
            local source__ = string.Explode("\"><div", v)
            local insert = string.Explode("id=", source__[1])[2]
            table.insert(t, insert)
        end


        table.remove(t, 1)
        callback(t)
    end, function() return false end)
end

util.GetWorkshopCollectionIDs(<<COLLECTION ID>>, function(ids)
    for k, v in pairs(ids) do
         resource.AddWorkshop(v)
    end
end)


(this should not be ran before the InitPostEntity hook because of the http stuff)

Informative x 1
#9

Oh, that's really useful! I'll try it and tell you how it goes.

Also, I don't really want to close that thread yet maybe others have some useful information to share!

Agree x 1
#10

I just created a gist with a better version than the one i posted before:

https://gist.github.com/JustPlayerDE/c09e8179653c1d64170694aa8a5560ba

with this version you can call resource.AddWorkshopCollection(collectionid) any time you want to add the whole collection (all calls will be queued until the first Think hook call so it will run as soon as the SteamHTTP api is loaded to prevent errors)

(also supports adding them via the sv_addworkshopcollection <CollectionID> command now)

(edit: i added it to the workshop now: https://steamcommunity.com/sharedfiles/filedetails/?id=2055027639 )

this can be usefull for someone :)

Friendly x 2
#11

Okay, so I need to say it's going pretty good right now.

I made two collections one for maps and the other for clientside stuff.

I added the collection with maps to host_workshop_collection so it will download all maps on the server, update them and make clients download them automatically.

For addons, I extract them to the addons folder and keep only scripts (clientside, shared and serverside scripts) and physics models to keep some free space and add them to clientside collection if they have any materials, models, sounds etc. otherwise no, so players won't need to download scripts when it's unnecessary. (Server automatically serves clientside and shared scripts)

And yeah, works smooooth!

Users viewing this thread
Nobody