Mod Filtering - just say no to hacked files

[BitVenom]

As of the 8/13 patch there's a new way for Mods to control what content a user sees when enabling their mod.

Many files have a new element called 'ExtFilter', which defines the tags those files expose to the initial external filtering code.

External filtering can be achieved based on these tags in 2 ways... One is safe for Mods, the other is for testing and development:

Command Line Filtering

-extPassFilter=xxx,yyy,zzz
-extFailFilter=xxx,yyy,zzz

These define the pass and fail tag filters for the game's execution... adding to any existing values already set or loaded elsewhere.

ConfigFilters.lua

-- Data:\Scripts\ConfigFilters.lua
PassFilter="xxx,yyy,zzz"
FailFilter="xxx,yyy,zzz"

This file can be included in any Mod (though Mods of Mods may want to stay clear), and also defines the pass and fail tag filters - it is additive, so it can work with the Command-Line flags during testing, etc.

How Filtering Works

Filtering by tag is used all over the engine now, not just in loading various content scripts - so this logic applies more than only here. However, this is a reasonable place to explain it.

When we have an asset (script, race, level, levels, etc) with some sort of tags (or none!) defined, and they are filtered, this is what happens:

Asset tags: aaa,bbb,ccc,zzz
Pass tags: aaa,bbb
Fail tags: xxx

So - We test for any pass tags that match. If there is no Pass tags values, all assets match. If an asset has no tags, it fails (it didn't match). In this case, we pass the asset due to aaa & bbb. Then, we apply a test for the Fail tags, if any tag matches, this filtering attempt fails. In this case, the asset doesn't have xxx, so it still passes.

Because the logic is very basic, it can be important to make tags very atomic - basic tags, combinations, tags with detail and tags with general classifications. That way, when something need to say 'only humans for player one' - the data required for that exists. We were very careful with our tags, so they should act as reasonable examples in that case.

Just as Props files are all found and ready from any required folder (so you can add an extra file instead of editing an existing file) - tags can be placed in a Tags subfolder - though this doesn't impact the ExtFilter logic.

In the near future it is likely that all extra files (Props/Tags/etc) will be allowed to have their own ExtFilters values to add extra granularity.

[EatThePath]

External filtering can be achieved based on these tags in 2 ways... One is safe for Mods, the other is for testing and development:

Oh ■■■■ yes.

Hm. What's the ordering/conflicting like? If ConfigFilters.lua says fail xxx, but the command line says pass xxx, what's the outcome?

[BitVenom]

All pass tests, then all fail.

[sastrei]

Hm, so if I create a configfilters.lua with

PassFilter="race_bs"

Which is what's listed in my blacktron and classicspace race lua's, it loads an empty game (which is sorta neat, but obviously not the desired effect). I take it there are some defaults we need to keep in configfilters.lua?

[EatThePath]

If I'm not mistaken gametypes are filtered too, so you'd need to pass a tag that's on the skirmish gametype or make a new copy with the race_bs tag

[sastrei]

That did it, thanks Siber. "rules_dm"

[BitVenom]

Yeah, all things are filtered - hence why I prefixed most ExtFilter tags with what type of content it is...

[Lazer72]

So "fail" tags kinda override everything else if detected.

[EatThePath]

Something that's giving me some conceptual headaches is the presence of ExtFilter=... and Tags = ... lines in races. And to boot, rules only seem to have ExtFilters but no tags... and the singleplayer GR has a burried tag folder with a single file in it. I'm having some trouble making things work how I want at the moment, and I'm suspecting it's due to this confusion.

[sastrei]

@EatThePath perhaps this will be of use? brickspace's configfilters.lua file.

-- BrickSpace

PassFilter="race_bs,rules_dm,levels_dm"
FailFilter="rules_dm_hw1,rules_dm_hw2"

[EatThePath]

Are you just using the deathmatch.lua rule then? Making any changes to it? I'm trying to filter out all the existing mp gamerules and add a nrw one and it's acting funky.

I'll try to be more specific about the behavior I'm seeing later, but I'm not quite at that level of organization right now. Which is probably part of the problem, admittedly.

[sastrei]

Yeah, just using deathmatch rule (rules_dm) and failing out the hw1 and hw2 options so they don't appear in the dropdown menu in-game.

So for a new rule, make a copy of deathmatch.lua, name it something different, go into it and give it a different extfilter value, a different race_paths value, and create an empty folder with that name.

Then in your configfilters.lua, add the new extfilter value you created to your passfilter list.

I think. ^_^

[BitVenom]

Something that's giving me some conceptual headaches is the presence of ExtFilter=... and Tags = ... lines in races. And to boot, rules only seem to have ExtFilters but no tags... and the singleplayer GR has a burried tag folder with a single file in it. I'm having some trouble making things work how I want at the moment, and I'm suspecting it's due to this confusion.

Okay, almost anything can have tags. If they aren't there, they don't happen - so don't be confused, it may simply be that some things don't have tags at the moment... and we didn't just leave empty files around.

So, some notes:

• ExtFilter="xxx,yyy,zzz" are tags that are ONLY used during the initial bootstrap and gather phase of the game. Once. It finds a file (levels, rules, race) it checks that file's ExtFilter tags against the ones specified in the ConfigFilters file, OR the command-line flags I already covered.

• IF the file is able to pass those tests, we load it, for whatever purpose it is designed. At that point, it gathers all of the 'base' Tags and Props we can find. 'base' means that this file base folder is checked for Tags and Props folders, and those as scanned for .lua files with the right properties.

• For some things (Ships/Resources/Races) - when one of those types is needed and a GR has been activated (basically during a level load, or game setup screen) - additional folders specified by Race_Path are checked for additional Tags and Props folders.

Notice, NONE of the Tags/Props folders have jack to do with ExtFilter... those are things we care about if we choose to use that asset, and nothing else.

If you have a specific item, or just want to talk/Skype tomorrow/this weekend, PM me.

[EatThePath]

Okay, that explanation and a night's sleep together helped me figure out the problem I was having. I was expecting the passes to 'casscade' as it were, with the configFilter's filtering including the gamerule, and then the gamerule passing a race that wasn't explicitly passed by the filter. Instead my understanding is now that it has to be passed by the initial filtering to be in the pool of files that GRs can filter from.

The other idea that wasn't really a good one was related to the fact that if there isn't a pass filter then everything passes that isn't explicitly failed. My thought here was that there would be different pools, that if I passed a rule tag but not any race tags then all races would autopass. In retrospect that was a bit misguided as the code likely has no distinction between the categories of tags.

With that out of the way I can move on to tripping over new things!

[EatThePath]

After some time digesting this topic, it seems to me that things would be much more elegant if we had some boolean logic for available for filtering. So for instance instead of the kushan being tagged as "race_gbx,race_hw1,race_kus,race_hwrm,race_dm,race_dm_hw1,race_gbx_hwrmc", they'd be "race,gbx,hw1,kus,hwrm,dm" and the pass rule for hw1dm would be "race & dm & hw1". Seems like it'd get the job done more cleanly, the current approach suffers from an explosion of combinations that makes it a bit harder to notice errors or omissions (incidentally, why is it race_hwrm but race_gbx_hwrmc? I thought it was an error, but it seems to be done consistently across all races.)

I understand this is a bit of added complexity, but if it were feasible to implement it seems like it'd be the better way.

[sastrei]

I think that goes to the "granular nature" of the tags that Bit was mentioning.

[Mikali]

What BIG file is ConfigFilters.lua stored in? I extracted UpdateHomeworld2.big but it's not in there.

[sastrei]

I don't think it's in one, you just have to create it for your own mod if you need to use it. It's only the two lines (see my above post).

[EatThePath]

Checking my merged all files extract shows now filter, which makes some sense. no filter means everything passes. If you want an example of a working filter, here's the current hwat one

FailFilter="races_gbx"
PassFilter="rules_hwat,races_dm,levels_dm_hw1,levels_hwat,campaigns_hwat,rules_sp"

[Mikali]

Does anyone have a full list of vanilla tags and what they mean?