cheesechaser.datapool.yande

This module provides a data pool implementation for Yande image data.

It extends the IncrementIDDataPool class to specifically handle Yande image data, utilizing a predefined Hugging Face repository for data storage and indexing.

Note

The dataset deepghs/yande_full is gated, you have to get the access of it before using this module.

YandeDataPool

class cheesechaser.datapool.yande.YandeDataPool(revision: str = 'main', hf_token: str | None = None)[source]

A data pool class for managing Yande image data.

This class extends IncrementIDDataPool to provide a specialized implementation for handling Yande image data. It uses a predefined Hugging Face repository for both data storage and indexing.

Parameters:

revision (str) – The revision of the data to use, defaults to ‘main’.
hf_token (Optional[str]) – Optional Hugging Face authentication token.

Variables:

data_repo_id – The Hugging Face repository ID for data storage.
idx_repo_id – The Hugging Face repository ID for indexing.

Usage:

>>> yande_pool = YandeDataPool()
>>> yande_pool_with_token = YandeDataPool(hf_token='your_token_here')

Note

This class assumes that both data and index information are stored in the same repository (_YANDE_REPO).

__init__(revision: str = 'main', hf_token: str | None = None)[source]

Initialize the YandeDataPool.

Parameters:

revision (str) – The revision of the data to use, defaults to ‘main’.
hf_token (Optional[str]) – Optional Hugging Face authentication token for accessing private repositories.

This constructor initializes the YandeDataPool by calling the parent class constructor with specific parameters for the Yande data repository. It sets up both the data and index repositories to use the same Hugging Face repository and revision.