Image suggestions data pipelines

Don’t leave Wikipedia without images: here are the image suggestions data pipelines:

Get your hands dirty

You need access to a Wikimedia Foundation’s stat box, Then:

ssh stat1010.eqiad.wmnet  # Or pick another one
set_proxy

Install mamba:

curl -L -O "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-$(uname)-$(uname -m).sh"
bash Miniforge3-$(uname)-$(uname -m).sh

Set up the development environment. It’s the same for all projects:

git clone https://gitlab.wikimedia.org/repos/structured-data/image-suggestions-data-pipelines.git mono
cd mono
mamba env create --name mono --file dev-conda-environment.yaml
mamba activate mono

Lint

cd image_suggestions  # Or `section_topics`, or `seal`
pre-commit install

At every git commit, pre-commit will run the checks and autofix or tell you what to fix.

Test

cd image_suggestions  # Or `section_topics`
pytest

Docs

sphinx-build docs/ docs/_build/

Trigger an Airflow test run

Follow this walkthrough to test-run a full pipeline at your latest commit. Inspired by this snippet.

Build your artifact

  1. g p or go to Build > Pipelines on the left sidebar

  2. Run publish_conda_env, wait until done

  3. Go to Deploy > Package Registry on the left sidebar

  4. Copy the Asset URL of the first item in the list. It should look like https://gitlab.wikimedia.org/repos/structured-data/image-suggestions-data-pipelines/-/package_files/1321/download

Get your artifact ready

mkdir artifacts
cd artifacts
wget -O artifact.tgz COPY_ASSET_URL_HERE
hdfs dfs -mkdir artifacts
hdfs dfs -copyFromLocal artifact.tgz artifacts
hdfs dfs -chmod -R o+rx artifacts

Spin up an Airflow dev instance

On a stat box

git clone https://gitlab.wikimedia.org/repos/data-engineering/airflow-dags.git dags
cd dags
sudo -u analytics-privatedata rm -fr /tmp/air  # If you've previously run the next command
sudo -u analytics-privatedata ./run_dev_instance.sh -m /tmp/air -p 1984 platform_eng

On your local box:

ssh -t -N stat1010.eqiad.wmnet -L 1984:stat1010.eqiad.wmnet:1984

This instance is owned by the analytics-privatedata user. Any input or output must be readable or writable by that user.

If you need a Hive database:

sudo -u analytics-privatedata hive
create database T123456; exit;

If you need a HDFS folder:

hdfs dfs -mkdir T123456
# ... add relevant stuff
hdfs dfs -chmod -R o+rwx T123456

On Kubernetes

Note

Work in progress, follow this thread.

ssh deployment.eqiad.wmnet
airflow-devenv create --dags-folder platform_eng --branch T123456
airflow-devenv expose dev-$USER

This instance is owned by you.

Remember to destroy it when you’re done:

airflow-devenv destroy dev-$USER

See also here.

Trigger the DAG run

Section topics example:

  1. Go to http://localhost:1984/

  2. On the top bar, go to Admin > Variables

  3. Click on the middle button (Edit record) next to the platform_eng/dags/section_topics_dag.py Key

  4. Update { "conda_env" : "hdfs://analytics-hadoop/user/ME/artifacts/artifact.tgz" }

  5. Add any other relevant DAG properties

  6. Click on the Save button

  7. On the top bar, go to DAGs and click on section_topics slider. This should trigger a DAG run

  8. Click on section_topics & monitor

Release

The main branch must be on a .dev0 version. Remove the .dev0 suffix from pyproject.toml and push a git tag.

Section topics example:

sed --in-place 's/version = "1.0.0.dev0"/version = "1.0.0"/' section_topics/pyproject.toml
git commit --all --message 'release section topics 1.0.0'
git tag --annotate section-topics-1.0.0 --message 'section topics 1.0.0 tag'
git push --follow-tags

Then, build the artifact:

  1. g p or go to Build > Pipelines on the left sidebar

  2. Run publish_conda_env, wait until done

Finally, bump to the next development version:

sed --in-place 's/version = "1.0.0"/version = "1.1.0.dev0"/' section_topics/pyproject.toml
git commit --all --message 'bump section topics to 1.1.0.dev0'
git push

Deploy

  1. g p or go to Build > Pipelines on the left sidebar

  2. Run bump_dag. This will create a merge request at airflow-dags

  3. Review & merge. You need a Maintainer role to merge

  4. Deploy the DAGs:

ssh deployment.eqiad.wmnet
cd /srv/deployment/airflow-dags/platform_eng/
git pull
scap deploy

Note

Step 4 won’t be needed as soon as this ticket is resolved.

API documentation