Overused images

When images are overused, they are probably placeholders or icons, thus being unsuitable for ALIS. This module computes overusage.

image_suggestions.overused_images.load_wiki_sizes(spark, monthly_snapshot)[source]

Load wikis with their article page counts through the image_suggestions.queries.wiki_sizes Data Lake query.

Parameters:

  • spark (SparkSession) – an active Spark session

  • monthly_snapshot (str) – a YYYY-MM date

Return type:

DataFrame

Returns:

the dataframe of:

  • wiki_db (string) - wiki project

  • size (bigint) - total article pages

Compute per-wiki thresholds that delimit overlinkage.

If the amount of links from articles to a given image is above a threshold, then the image is considered as overused in the corresponding wiki.

Parameters:

  • spark (SparkSession) – an active Spark session

  • monthly_snapshot (str) – a YYYY-MM date

Return type:

DataFrame

Returns:

the dataframe of:

  • wiki_db (string) - wiki project

  • size (bigint) - total article pages

  • threshold (double) - threshold that delimits too many links

image_suggestions.overused_images.get(spark, monthly_snapshot)[source]

Identify overused images.

If an image is overused, then it’s probably a placeholder or an icon and shouldn’t be suggested.

Note

Data is significantly skewed: some partitions have 2 orders of magnitude more rows than others.

Parameters:

  • spark (SparkSession) – an active Spark session

  • monthly_snapshot (str) – a YYYY-MM date

Return type:

DataFrame

Returns:

the dataframe of:

  • wiki_db (string) - wiki project

  • page_id (bigint) - Commons page ID

  • page_title (string) - Commons page title, in original case and underscored