JSON handling for non application/json content types

[fivesixzero]

Some useful REST endpoints don't return with a content-type of application/json while still returning valid JSON, preventing the use of the library's built-in JSON parsing/processing.

I ran into this while working with the freely available US National Weather Service forecast API, which can only respond with GeoJSON (application/geo+json) or JSON LD (application/ld+json).

As a workaround in my project I've just called directly into the MagTag object's network.fetch_data(url) to get a response string to parse and wrangle but it'd be nice to do it in a more elegant way with these types of endpoints.

[slootsky]

What if there was a parameter added to fetch?
json_content-type : defaults to None
If supplied, this would be compared to content-type and if it matched, it would be considered JSON.

    def fetch_data(
        self,
        url,
        *,
        headers=None,
        json_path=None,
        regexp_path=None,
        timeout=10,
        json_content-type=None,
    ):
        """Fetch data from the specified url and perfom any parsing
        :param str url: The URL to fetch from.
        :param list headers: Extra headers to include in the request.
        :param json_path: The path to drill down into the JSON data.
        :param regexp_path: The path formatted as a regular expression to drill down
                            into the JSON data.
        :param int timeout: The timeout period in seconds.
        :param json_content-type: A known value for content-type when JSON is returned
        """
        json_out = None
        values = []
        content_type = CONTENT_TEXT

        response = self.fetch(url, headers=headers, timeout=timeout)

        headers = {}
        for title, content in response.headers.items():
            headers[title.lower()] = content
        gc.collect()
        if self._debug:
            print("Headers:", headers)
        if response.status_code == 200:
            print("Reply is OK!")
            self.neo_status(STATUS_DATA_RECEIVED)  # green = got data
            if "content-type" in headers:
                if "image/" in headers["content-type"]:
                    content_type = CONTENT_IMAGE
                elif "application/json" in headers["content-type"]:
                    content_type = CONTENT_JSON
                elif "application/javascript" in headers["content-type"]:
                    content_type = CONTENT_JSON
                elif json_content-type and json_content-type in headers["content-type"]:
                    content_type = CONTENT_JSON

[fivesixzero]

That sounds like a good way of handling it. :)

[makermelissa]

Transferred issue here because that's where the applicable code now resides.

[makermelissa]

I did a big refactor with #8 and there is now an add_json_content_type() function.