Section alignment

Generate image suggestions for Wikipedia article sections based on aligned section titles.

Inputs:

• Wikipedia article section images as output by section_topics.pipeline

• machine-learned section alignment dataset of (source, target) section title pairs

High-level steps given a target section:

• skip it if it has more images than a threshold

• gather all equivalent source pages with the same Wikidata QID as the target page

• filter out irrelevant images from the source sections, namely icons or those appearing in the target too

• project all source section images to the target

• combine suggestion candidates

Output pyspark.sql.DataFrame row example:

target_id	target_heading	item_id	target_title	target_index	recommended_images	target_wiki_db
1485753	Biografia	Q317475	Ali_Shariati	1	[{frwiki -> [Shariati7.jpg]}, {ruwiki -> [Shariati7.jpg]}, …]	ptwiki

More documentation lives in MediaWiki.

class section_topics.section_alignment_image_suggestions.Page(item_id, page_id, page_title, wiki_db, section_images)

dataclasses.dataclass() that stores a Wikipedia article page and its available images.

Parameters:

• item_id (str) – a page Wikidata QID

• page_id (int) – a page ID

• page_title (str) – a page title

• wiki_db (str) – a page wiki

• section_images (List[SectionImages]) – a list of section images

class section_topics.section_alignment_image_suggestions.Recommendation(item_id, target_id, target_title, target_index, target_heading, target_wiki_db, source_heading, source_wiki_db, recommended_images)

dataclasses.dataclass() that stores image suggestions for a target Wikipedia article page.

Parameters:

• item_id (str) – a page Wikidata QID

• target_id (int) – a page ID

• target_title (str) – a page title

• target_index (int) – a section numerical index

• target_heading (str) – a section heading

• target_wiki_db (str) – a page wiki

• source_heading (str) – a section heading where suggestions come from

• source_wiki_db (str) – a page wiki where suggestions come from

• recommended_images (List[str]) – a list of suggested image file names

section_topics.section_alignment_image_suggestions.load(article_images)

Convert section images from an section_topics.pipeline’s output row to a SectionImages instance.

Parameters:

article_images (str) – a stringified JSON array of section images. Corresponds to a value of the article_images column in section_topics.pipeline’s output pyspark.sql.DataFrame

Return type:

List[SectionImages]

Returns:

the list of converted section images

section_topics.section_alignment_image_suggestions.rows_to_pages(df)

Convert all input pandas.DataFrame rows to Page instances.

Parameters:

df (DataFrame) – an in-memory dataframe loaded from section_topics.pipeline’s output pyspark.sql.DataFrame

Return type:

List[Page]

Returns:

the list of pages

section_topics.section_alignment_image_suggestions.filter_source_images(source_images, target_images)

Filter out source images that either appear in target ones or are icons/indicators.

Icon and indicator file names have a OOjs_UI_icon and OOjs_UI_indicator prefix respectively.

Parameters:

• source_images (Sequence[str]) – a sequence of source image file names

• target_images (Set[str]) – a set of target image file names

Return type:

List[str]

Returns:

the list of filtered source image file names

section_topics.section_alignment_image_suggestions.make_recommendations(target_page, source_page, max_target_images)

Generate all image suggestion candidates for a target page via the Cartesian product of all (target, source) section pairs.

Filter source images through filter_source_images(). Filter target sections with more images than a given threshold.

Parameters:

• target_page (Page) – a target page

• source_page (Page) – a source page

• max_target_images (int) – a maximum amount of images for a target section to be kept

Return type:

List[Recommendation]

Returns:

the list of image suggestion candidates

section_topics.section_alignment_image_suggestions.combine_pages(target_pages, source_pages)

Generate all (target, source) page pairs via their Cartesian product.

Ensure that no pair has the same language.

Parameters:

• target_pages (Sequence[Page]) – a sequence of target pages

• source_pages (Sequence[Page]) – a sequence of source pages

Return type:

List[Tuple[Page, Page]]

Returns:

the list of (target, source) page pairs

section_topics.section_alignment_image_suggestions.generate_image_recommendations(target_wiki_dbs, max_target_images, df)

Generate all possible suggestions for pages in the target wikis.

Filter out empty ones.

Parameters:

• target_wiki_dbs (Sequence[str]) – a sequence of target wikis

• max_target_images (int) – a maximum amount of images for a target section to be kept

• df (DataFrame) – an in-memory dataframe of pages

Return type:

DataFrame

Returns:

the in-memory dataframe of image suggestions

section_topics.section_alignment_image_suggestions.get_image_recommendations(section_images_df, target_wiki_dbs, max_target_images)

Generate the full dataset of all possible suggestions for all sections of all pages in the target wikis.

Corresponding source pages have the same Wikidata QID as a given target page.

Parameters:

• section_images_df (DataFrame) – a distributed dataframe of section images

• target_wiki_dbs (Sequence[str]) – a sequence of target wikis

• max_target_images (int) – a maximum amount of images for a target section to be kept

Return type:

DataFrame

Returns:

the distributed dataframe of image suggestions

section_topics.section_alignment_image_suggestions.normalize_heading_column(column, substitute_pattern='[\\\\s_]', strip_chars='!"#$%&\\' *+, -./:;<=>?@[\\\\]^_`{|}~')

Same as section_topics.pipeline.normalize_heading_column().

Return type:

Column

section_topics.section_alignment_image_suggestions.process_image_recommendations(recommendations_df, alignments_df)

Build the final output dataset of image suggestions.

Combine all suggestion candidates with aligned sections.

Parameters:

• recommendations_df (DataFrame) – a distributed dataframe of all image suggestion candidates

• alignments_df (DataFrame) – a distributed dataframe of section alignments

Return type:

DataFrame

Returns:

the distributed dataframe of final image suggestions