API Reference

---

skeval.classifier

class skeval.classifier.sentence_classifier.BasicTextClassifier(vocab_size: int, embed_dim: int, num_classes: int)

Bases: Module

EmbeddingBag + Linear text classifier.

A lightweight bag-of-words model: token indices are averaged by EmbeddingBag and then projected to class logits by a single linear layer.

embedding

nn.EmbeddingBag that averages token embeddings.

fc

Linear layer that projects the averaged embedding to class logits.

init_weights() → None

Initialise embedding and linear weights with a uniform distribution.

Weights are drawn from Uniform(-0.5, 0.5) and biases are set to zero. This gives a balanced starting point that avoids saturation.

forward(text: Tensor, offsets: Tensor) → Tensor

Compute class logits for a batch of sentences.

Parameters:

• text – Flat 1-D LongTensor of concatenated token indices.

• offsets – 1-D LongTensor of sentence start positions within text, as produced by collate_fn.

Returns:

FloatTensor of shape (batch_size, num_classes) containing raw (pre-softmax) class scores.

class skeval.classifier.sentence_classifier.SentenceClassifier(embed_dim: int = 64, epochs: int = 5, batch_size: int = 32, lr: float = 0.005, random_state: int | None = None, num_workers: int = 0, pin_memory: bool = False, val_split: float = 0.0, patience: int = 0)

Bases: BaseEstimator

sklearn-compatible sentence classifier backed by a bag-of-words neural network.

Implements the full sklearn estimator interface (fit, predict, score, get_params, set_params) so it works directly with GridSearchCV, cross_val_score, and similar utilities.

embed_dim

Embedding dimensionality.

epochs

Number of training epochs.

batch_size

Mini-batch size used during training.

lr

Adam learning rate.

random_state

Seed for reproducibility, or None for non-deterministic runs.

model

The underlying BasicTextClassifier, or None before fitting.

vocab

VocabBuilder instance populated during fit.

label_encoder

LabelEncoder instance populated during fit.

device

Torch device (cuda if available, else cpu).

fit(X: List[str], y: List[str]) → SentenceClassifier

Build the vocabulary and train the model on labelled sentences.

When val_split > 0 a stratified holdout is carved out before training and validation loss is reported each epoch. When patience > 0 training stops as soon as validation loss has not improved for that many consecutive epochs.

Parameters:

• X – Training sentences.

• y – Corresponding class labels aligned with X.

Returns:

The fitted classifier instance (self).

Raises:

ValueError – If X or y fail input validation.

predict(X: List[str]) → List[str]

Predict the class label for each sentence in X.

Sentences are processed in mini-batches of size self.batch_size for efficient GPU utilisation.

Parameters:

X – Sentences to classify.

Returns:

List of predicted label strings in the same order as X.

Raises:

• RuntimeError – If called before fit() or load().

• ValueError – If X fails input validation.

predict_proba(X: List[str]) → ndarray

Return class probabilities for each sample, shape (n_samples, n_classes).

Parameters:

X – Sentences to classify.

Returns:

Float array of shape (n_samples, n_classes) with softmax probabilities.

Raises:

• RuntimeError – If called before fit() or load().

• ValueError – If X fails input validation.

score(X: List[str], y: List[str]) → float

Return mean accuracy over the provided samples.

Parameters:

• X – Sentences to classify.

• y – True labels aligned with X.

Returns:

Fraction of correctly classified samples (0.0 – 1.0).

train(sentences: List[str], labels: List[str], epochs: int | None = None, batch_size: int | None = None, lr: float | None = None) → SentenceClassifier

Train the classifier (deprecated — use fit() instead).

Parameters:

• sentences – Training sentences.

• labels – Corresponding class labels.

• epochs – Override the instance epochs value for this run.

• batch_size – Override the instance batch_size value for this run.

• lr – Override the instance lr value for this run.

Returns:

The fitted classifier instance (self).

Deprecated since version 0.2.0: Use fit() instead. train() will be removed in v0.3.0.

save(save_dir: str) → None

Persist the trained model and vocabulary metadata to disk.

Writes two files into save_dir:

• model.pt — PyTorch state dict.

• metadata.json — vocab, label mapping, and hyper-parameters.

Parameters:

save_dir – Directory path to write artefacts into (created if absent).

Raises:

RuntimeError – If called before fit().

load(save_dir: str) → None

Restore a previously saved model from disk.

Reads model.pt and metadata.json from save_dir and reconstructs the classifier so it is ready for inference.

Parameters:

save_dir – Directory that was passed to a previous save() call.

---

skeval.evaluator

class skeval.evaluator.evaluator.Evaluator

Bases: object

Evaluates classifier predictions against ground-truth labels.

evaluate(predictions: List[str], ground_truth: List[str]) → Dict[str, Any]

Compute a suite of classification metrics for a set of predictions.

Parameters:

• predictions – Predicted labels produced by SentenceClassifier.predict.

• ground_truth – True labels aligned with predictions.

Returns:

• accuracy (float): Overall accuracy.

• per_class (dict): Per-label precision, recall, F1, and support.

• macro_avg (dict): Macro-averaged precision, recall, and F1.

• weighted_avg (dict): Weighted-averaged precision, recall, and F1.

• confusion_matrix (list[list[int]]): Row = true, column = predicted.

• labels (list[str]): Sorted list of unique labels.

Return type:

Dictionary with the following keys

Raises:

ValueError – If either list is empty or the lengths do not match.

---

skeval.metrics

skeval.metrics.metrics.compute_metrics(y_true: List[str], y_pred: List[str]) → Dict[str, Any]

Compute classification metrics given true and predicted label sequences.

Parameters:

• y_true – Ground-truth label strings.

• y_pred – Predicted label strings in the same order as y_true.

Returns:

• accuracy (float): Fraction of correctly classified samples.

• per_class (dict): Per-label dict of precision, recall, f1-score, and support.

• macro_avg (dict): Macro-averaged precision, recall, and F1.

• weighted_avg (dict): Support-weighted precision, recall, and F1.

• confusion_matrix (list[list[int]]): Confusion matrix where rows correspond to true labels and columns to predicted labels, ordered by labels.

• labels (list[str]): Sorted union of all labels seen in either y_true or y_pred.

Return type:

Dictionary with the following keys

---

skeval.dataset

class skeval.dataset.loader.SentenceDataset(sentences: List[str], labels: List[str], vocab: VocabBuilder, label_encoder: LabelEncoder)

Bases: Dataset

PyTorch Dataset that tokenises sentences on the fly.

sentences

Raw sentence strings.

labels

Corresponding label strings.

vocab

Fitted VocabBuilder used to encode sentences.

label_encoder

Fitted LabelEncoder used to encode labels.

skeval.dataset.loader.collate_fn(batch: List[Tuple[Tensor, Tensor]]) → Tuple[Tensor, Tensor, Tensor]

Collate variable-length sentences into a single batch for EmbeddingBag.

EmbeddingBag expects a flat 1-D token tensor together with an offsets tensor that marks where each sentence starts.

Parameters:

batch – List of (text_tensor, label_tensor) pairs returned by SentenceDataset.__getitem__.

Returns:

A 3-tuple (sentences, labels, offsets) where sentences is the concatenated flat token tensor, labels is a 1-D label tensor, and offsets is a 1-D tensor of sentence start positions.

class skeval.dataset.loader.DatasetLoader

Bases: object

Utility for loading raw data from CSV or JSONL into PyTorch DataLoaders.

static load_csv(filepath: str, text_col: str, label_col: str) → Tuple[List[str], List[str]]

Load sentences and labels from a CSV file.

Parameters:

• filepath – Path to the CSV file.

• text_col – Name of the column containing sentence text.

• label_col – Name of the column containing class labels.

Returns:

A 2-tuple (sentences, labels) of equal-length string lists.

static load_json(filepath: str, text_key: str, label_key: str) → Tuple[List[str], List[str]]

Load sentences and labels from a JSON lines file.

Each line must be a JSON object with at least the two specified keys.

Parameters:

• filepath – Path to the .jsonl file.

• text_key – Key whose value is the sentence text.

• label_key – Key whose value is the class label.

Returns:

A 2-tuple (sentences, labels) of equal-length string lists.

static create_dataloader(sentences: List[str], labels: List[str], vocab: VocabBuilder, label_encoder: LabelEncoder, batch_size: int = 32, shuffle: bool = True, num_workers: int = 0, pin_memory: bool = False) → DataLoader

Wrap sentences and labels in a PyTorch DataLoader.

Parameters:

• sentences – Raw sentence strings.

• labels – Corresponding label strings.

• vocab – Fitted VocabBuilder.

• label_encoder – Fitted LabelEncoder.

• batch_size – Number of samples per batch.

• shuffle – Whether to shuffle the data each epoch.

• num_workers – Number of subprocesses for data loading. 0 means data is loaded in the main process.

• pin_memory – If True, the DataLoader copies tensors to CUDA pinned memory before returning them. Only useful when training on a GPU.

Returns:

A DataLoader that yields (sentences, labels, offsets) batches compatible with BasicTextClassifier.

---

skeval.model_selection

skeval.model_selection.model_selection.train_test_split(X: List[str], y: List[str], test_size: float | int = 0.2, random_state: int | None = None, shuffle: bool = True, stratify: bool = False) → Tuple[List[str], List[str], List[str], List[str]]

Split sentences and labels into random train and test subsets.

A thin, type-annotated wrapper around sklearn.model_selection.train_test_split() that preserves List[str] types for sentences and labels.

Parameters:

• X – Sentence strings to split.

• y – Labels aligned with X.

• test_size – If float, the proportion of the dataset to include in the test split. If int, the absolute number of test samples.

• random_state – Seed for the random number generator.

• shuffle – Whether to shuffle the data before splitting.

• stratify – If True, the split is stratified by y so that each split has roughly the same class distribution.

Returns:

A 4-tuple (X_train, X_test, y_train, y_test) of string lists.

Raises:

ValueError – If X and y have different lengths.

Example

>>> X_train, X_test, y_train, y_test = train_test_split(
...     X, y, test_size=0.2, random_state=42
... )

skeval.model_selection.model_selection.cross_val_score(estimator: Any, X: List[str], y: List[str], cv: int = 5, scoring: str = 'accuracy', n_jobs: int | None = None) → ndarray

Evaluate a classifier using k-fold cross-validation.

A convenience wrapper around sklearn.model_selection.cross_val_score() with sensible defaults for SentenceClassifier.

Parameters:

• estimator – A fitted or unfitted sklearn-compatible estimator, e.g. SentenceClassifier.

• X – Sentence strings.

• y – Labels aligned with X.

• cv – Number of cross-validation folds.

• scoring – Scoring metric passed to sklearn (default "accuracy").

• n_jobs – Number of jobs for parallel fold execution. None means 1; -1 uses all available CPUs.

Returns:

Float array of shape (cv,) with the score for each fold.

Example

>>> from skeval import SentenceClassifier
>>> from skeval.model_selection import cross_val_score
>>> scores = cross_val_score(SentenceClassifier(), X, y, cv=5)
>>> print(scores.mean())

---

skeval.utils

skeval.utils.helpers.normalize_text(text: str) → str

Lowercase and strip punctuation from a string.

Parameters:

text – Raw input string.

Returns:

Normalized string with punctuation removed and leading/trailing whitespace stripped.

class skeval.utils.helpers.VocabBuilder(min_freq: int = 1)

Bases: object

Builds a vocabulary from a text corpus and encodes sentences as indices.

Tokens 0 and 1 are reserved for <PAD> and <UNK> respectively. All other tokens are assigned indices starting at 2 in the order they appear in the counter after filtering by min_freq.

min_freq

Minimum number of occurrences for a token to be included.

word2idx

Mapping from token string to integer index.

idx2word

Reverse mapping from integer index to token string.

is_built

Whether build() has been called.

build(sentences: List[str]) → None

Populate word2idx and idx2word from a list of sentences.

Parameters:

sentences – Training corpus. Each element is a single sentence string.

encode(sentence: str) → List[int]

Convert a sentence into a list of vocabulary indices.

Unknown tokens are mapped to the <UNK> index (1).

Parameters:

sentence – Raw input sentence.

Returns:

List of integer indices, one per token after normalisation.

Raises:

ValueError – If build() has not been called yet.

class skeval.utils.helpers.LabelEncoder

Bases: object

Encodes string labels to integers and decodes them back.

Labels are sorted alphabetically before assignment so that the mapping is deterministic across runs.

label2idx

Mapping from label string to integer class index.

idx2label

Reverse mapping from integer class index to label string.

is_built

Whether build() has been called.

build(labels: List[str]) → None

Assign a unique integer index to each unique label.

Parameters:

labels – Full list of training labels (duplicates allowed).

encode(label: str) → int

Convert a label string to its integer index.

Parameters:

label – Label string seen during build().

Returns:

Integer class index.

Raises:

ValueError – If build() has not been called or the label is unknown.

decode(idx: int) → str

Convert an integer class index back to its label string.

Parameters:

idx – Integer index returned by the model.

Returns:

Corresponding label string.

Raises:

ValueError – If build() has not been called.

property num_classes: int

Total number of unique classes seen during build().

---

skeval.cli

skeval.cli.main() → None

Run the skeval command-line interface.