Image sources

Collect images connected to the following sources:

image_suggestions.entity_images.get_wikidata(spark, hive_db, weekly_snapshot)[source]

Gather image and Commons category Wikidata claims.

This function invokes image_suggestions.shared.load_wikidata() and aggregates claims by image.

Parameters:

  • spark (SparkSession) – an active Spark session

  • hive_db (str) – a Data Lake’s Hive database name

  • weekly_snapshot (str) – a YYYY-MM-DD date

Return type:

DataFrame

Returns:

the dataframe of:

  • item_id (string) - Wikidata QID

  • page_id (bigint) - Commons image page ID

  • tag (array<string>) - image.linked.from.wikidata.p18 and/or image.linked.from.wikidata.p373 tags

image_suggestions.entity_images.get_depicts(spark, hive_db, weekly_snapshot)[source]

Gather Commons depicts statements.

Parameters:

  • spark (SparkSession) – an active Spark session

  • hive_db (str) –

    a Data Lake’s Hive database name

  • weekly_snapshot (str) – a YYYY-MM-DD date

Return type:

DataFrame

Returns:

the dataframe of:

  • item_id (string) - Wikidata QID

  • page_id (bigint) - Commons image page ID

image_suggestions.entity_images.get(spark, hive_db, weekly_snapshot, include_wikidata=True, include_lead_images=True, include_sdc=True, limit_per_qid=0)[source]

Aggregate all sources of image connections.

Compute an image suggestion confidence score between 0 and 100 based on the sources: if an image has one source, it will inherit its score. Otherwise, it will be a combined or probability.

For instance, Commons categories and depicts statements have a score of 80 and 70 respectively. If an image is connected to both of them, then the final score will be:

100 x ( 1 - (1 - 0.8) x (1 - 0.7) ) = 97

More details here.

Parameters:

  • spark (SparkSession) – an active Spark session

  • hive_db (str) –

    a Data Lake’s Hive database name

  • weekly_snapshot (str) – a YYYY-MM-DD date

  • include_wikidata (bool) – whether to include Wikidata claims

  • include_lead_images (bool) – whether to include Wikipedia article lead images

  • include_sdc (bool) – whether to include Commons depicts statements

  • limit_per_qid (int) – an integer that limits the amount of suggestions per Wikidata QID. If > 0, suggestions are ordered by confidence. 0 stands for no limit

Return type:

DataFrame

Returns:

the dataframe of:

  • item_id (string) - Wikidata QID

  • page_title (string) - Commons image page title, in original case and underscored

  • found_on (array<string>) - wikis where the image was found

  • kind (array<string>) - sources of image connections. Values can be istype-depicts, istype-wikidata-image, istype-commons-category, and/or istype-lead-image

  • confidence (int) - suggestion confidence score