Significantly enhance the safety of metadata manipulation

[benoit74]

Fix #205

This is a full rewrite of #217, so I've opened a new PR since changes since last review made no more sense from my PoV.

• add types for all metadata, one type per metadata name plus some generic ones for non-standard metadata
  • all types are responsible to validate metadata value at initialization time
  • validation checks for adherence to the ZIM specification and conventions are automated
  • cleanup of unwanted control characters and stripping white characters are automated in all text metadata
  • whenever possible, try to automatically clean a "reasonably" bad metadata (e.g. automaticall accept and remove duplicate tags - harmless - but not duplicate language codes - codes are supposed to be ordered, so it is a weird situation) ; this is an alignment of paradigm, because for some metadata the lib was permissive, while for other it was quite restrictive ; this PR tries to align this and make the lib as permissive as possible, avoiding to fail a scraper for something which could be automatically fixed
  • it is now possible to disable ZIM conventions checks with zim.metadata.check_metadata_conventions
• simplify zim.creator.Creator.config_metadata by using these types and been more strict:
  • add new StandardMetadata class for standard metadata, including list of mandatory one
  • by default, all non-standard metadata must start with X- prefix
    • this not yet an openZIM convention / specification, so it is possible to disable this check with fail_on_missing_prefix argument
• simplify add_metadata, use same metadata types
• simplify zim.creator.Creator.start with new types, and drop all metadata from memory after being passed to the libzim
• drop zim.creator.convert_and_check_metadata (not usefull anymore, simply use proper metadata type)
• move MANDATORY_ZIM_METADATA_KEYS and DEFAULT_DEV_ZIM_METADATA from constants to zim.metadata to avoid circular dependencies
• new inputs.unique_values utility function to compute the list of uniques values from a given list, but preserving initial list order
• in __init__ of zim.creator.Creator, rename disable_metadata_checks to check_metadata_conventions for clarity and brevity
  • beware that this manipulate the global zim.metadata.check_metadata_conventions, so if you have many creator running in parallel, they can't have different settings, last one initialized will "win"

Nota:

• I've moved many tests from tests/zim/test_zim_creator.py to tests/zim/test_metadata.py since most checks are now done at metadata initialization instead of when config_metadata or start are called, but coverage is similar

[codecov]

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 100.00%. Comparing base (76a6408) to head (7e2efa1).
Report is 14 commits behind head on main.

Additional details and impacted files

@@            Coverage Diff             @@
##              main      #221    +/-   ##
==========================================
  Coverage   100.00%   100.00%            
==========================================
  Files           38        39     +1     
  Lines         2227      2447   +220     
  Branches       426       335    -91     
==========================================
+ Hits          2227      2447   +220     

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[rgaudin]

Wow that's a lot of change!
See inline comments ; maybe we should discuss it live once you've looked at it

[rgaudin]

Here's my attempt at making those explicit, dedicated metadata easier to apprehend and use. I wrote this quickly at that time then a loong time passed before I looked at it and had to fix the tests to make it work.

• A base, generic yet usable Metadata type.
  • works with bytes-like input
  • takes an optional name
  • defines the standard API: libzim_value, value, meta_name, etc.
• Dedicated subclasses for well-known metadata: LanguageMetadata, TagsMetadata, DateMetadata, etc.
  • Takes proper native type as input
• Dedicated subclasses for CutomMetadata and CustomTextMetadata as you had. I also added XCustomMetadata for auto-prefixing.

I chose an approach in which the flexibility is built into the base Metadata class, using Class variables that gets overriden by the subclasses and can be shadowed by instances should need be.
Given the constraints/features for those metadata can be multiple and combined, I figured decorators are a very readable way to specify those.
I chose to expose each (is_required, join_list_with, etc) so it's clear, can be properly documented and is properly typed.
The main API for this though is the expecting variable which decides which of the input-type to libzim_data convertion function is used. Could be done differently of course.

One thing that's a tad annoying is that name is optionnal in Metadata() for it to use the class variable (of the subclasses). It thus requires passing value first then name which seems odd… but Metadata is not to be used frequently.
In the CustomMetadata ones, given a name is mandatory I reversed it.
Maybe we could solve it by requiring kwargs on Metadata…

Another take was to spot accepting bytes as inputs as well as strings where there's no good reason to. For instance, DateMetadata takes a date or datetime only now. Supporting those extra stuff is an additional burden and there's no real value in our usecase. "2024-11-21" and date(2024, 11,21) are quite close and in general we use computed dates anyway (.today()). Same goes for tags/languages which now require a list of strings
This led to a lot of changes in the tests

Major changes that has to be introduced (we'll work on CHANGELOG if we pursue with this)

• Use of beartype for zimscraperlib.zim moodule.

We need to validate this but this frees us from a lot of meaningless tests and really simplifies dev and maintenance.
We should do it on zimscraperlib module so it applies everywhere but of course we have errors in the codebase so it requires some fixes.
I did not remove all the extra tests (only where it was expecting a different exp)

• Fixed types for CounterMap and MimetypeAndCounter in zim/_libkiwix (revealed by beartype)
• zim.Archive.counters to return a CounterMap
• Introduced new typing module with a new Callback class to normalize callback management (we already know previous way was sloppy and beartype wasnt happy)
• Changed filesystem.delete_callback into a simple callback function which doesnt know about callback chaining
• zim.Creator.add_item_for and zim.creator.add_item to replace callback args with callbacks arg accepting a single Callback or a list of them
• Removed check_metadata_conventions from creator. Keeping the concept of a single toggle in the codebase (renamed it to APPLY_RECOMMENDATIONS) but such a global var should be explicitly set.
• new DefaultIllustrationMetadata type for this special one that's required.
• IllustrationMetadata that automatically takes care of the meta name. Saves a lot of trouble.
• MANDATORY_ZIM_METADATA_KEYS still built dynamically although I'm not sure it's worth it.

I wonder if we should offer a way to create the StandardMetadataSet with values directly (to make it less verbose). Would be quite easy now with something like (assuming the expects_* decorator add that type:

  class StandardMetadataList:

    ...

    @classmethod
    def from_values(
        cls,
        Name: NameMetadata.input_type,
        Language: LanguageMetadata.input_type,
        Title: TitleMetadata.input_type,
        Creator: CreatorMetadata.input_type,
        Publisher: PublisherMetadata.input_type,
        Date: DateMetadata.input_type,
        Illustration_48x48_at_1: DefaultIllustrationMetadata.input_type,
        Description: DescriptionMetadata.input_type,
        LongDescription: LongDescriptionMetadata.input_type | None = None,
        Tags: TagsMetadata.input_type | None = None,
        Scraper: ScraperMetadata.input_type | None = None,
        Flavour: FlavourMetadata.input_type | None = None,
        Source: SourceMetadata.input_type | None = None,
        License: LicenseMetadata.input_type | None = None,
        Relation: RelationMetadata.input_type | None = None,
    ): ...

All tests are passing. I did not write new onesbut I had to update coverage as it was way off (hence the dev-deps update). It still is to some extent. Might have difficulty understanding the decorator thing. We'll have to look into that.

[benoit74]

Thanks a lot, nothing left to add, I like it! Glad we've made this "not-negligible" move. I just force-push to fixup commits and rebase on main.