processors

Processors are used to prepare non-textual inputs (e.g., image or audio) for a model.

Example: Using a WhisperProcessor to prepare an audio input for a model.

import { AutoProcessor, read_audio } from '@xenova/transformers';

let processor = await AutoProcessor.from_pretrained('openai/whisper-tiny.en');
let audio = await read_audio('https://huggingface.co/datasets/Narsil/asr_dummy/resolve/main/mlk.flac', 16000);
let { input_features } = await processor(audio);
// Tensor {
//   data: Float32Array(240000) [0.4752984642982483, 0.5597258806228638, 0.56434166431427, ...],
//   dims: [1, 80, 3000],
//   type: 'float32',
//   size: 240000,
// }

• processors
  • static
    • .FeatureExtractor ⇐ Callable
      • new FeatureExtractor(config)
    • .ImageFeatureExtractor ⇐ FeatureExtractor
      • new ImageFeatureExtractor(config)
      • .preprocess(image) ⇒ Promise.<any>
      • ._call(images) ⇒ Promise.<Object>
    • .DetrFeatureExtractor ⇐ ImageFeatureExtractor
      • ._call(urls) ⇒ Promise.<Object>
      • .post_process_object_detection() : post_process_object_detection
      • .remove_low_and_no_objects(class_logits, mask_logits, object_mask_threshold, num_labels) ⇒ *
      • .check_segment_validity(mask_labels, mask_probs, k, mask_threshold, overlap_mask_area_threshold) ⇒ *
      • .compute_segments(mask_probs, pred_scores, pred_labels, mask_threshold, overlap_mask_area_threshold, label_ids_to_fuse, target_size) ⇒ *
      • .post_process_panoptic_segmentation(outputs, [threshold], [mask_threshold], [overlap_mask_area_threshold], [label_ids_to_fuse], [target_sizes]) ⇒ Array.<{segmentation: Tensor, segments_info: Array<{id: number, label_id: number, score: number}>}>
    • .Processor ⇐ Callable
      • new Processor(feature_extractor)
      • ._call(input) ⇒ Promise.<any>
    • .WhisperProcessor ⇐ Processor
      • ._call(audio) ⇒ Promise.<any>
    • .AutoProcessor
      • .from_pretrained(pretrained_model_name_or_path, options) ⇒ Promise.<Processor>
  • inner
    • ~center_to_corners_format(arr) ⇒ Array.<number>
    • ~post_process_object_detection(outputs) ⇒ Array.<Object>
      • ~box : Array.<number>
    • ~PretrainedOptions : *

processors.FeatureExtractor ⇐ Callable

Base class for feature extractors.

Kind: static class of processors
Extends: Callable

new FeatureExtractor(config)

Constructs a new FeatureExtractor instance.

processors.ImageFeatureExtractor ⇐ FeatureExtractor

Feature extractor for image models.

Kind: static class of processors
Extends: FeatureExtractor

• .ImageFeatureExtractor ⇐ FeatureExtractor
  • new ImageFeatureExtractor(config)
  • .preprocess(image) ⇒ Promise.<any>
  • ._call(images) ⇒ Promise.<Object>

new ImageFeatureExtractor(config)

Constructs a new ImageFeatureExtractor instance.

imageFeatureExtractor.preprocess(image) ⇒ Promise.<any>

Preprocesses the given image.

Kind: instance method of ImageFeatureExtractor
Returns: Promise.<any> - The preprocessed image as a Tensor.

imageFeatureExtractor._call(images) ⇒ Promise.<Object>

Calls the feature extraction process on an array of image URLs, preprocesses each image, and concatenates the resulting features into a single Tensor.

Kind: instance method of ImageFeatureExtractor
Returns: Promise.<Object> - An object containing the concatenated pixel values (and other metadata) of the preprocessed images.

processors.DetrFeatureExtractor ⇐ ImageFeatureExtractor

Detr Feature Extractor.

Kind: static class of processors
Extends: ImageFeatureExtractor

• .DetrFeatureExtractor ⇐ ImageFeatureExtractor
  • ._call(urls) ⇒ Promise.<Object>
  • .post_process_object_detection() : post_process_object_detection
  • .remove_low_and_no_objects(class_logits, mask_logits, object_mask_threshold, num_labels) ⇒ *
  • .check_segment_validity(mask_labels, mask_probs, k, mask_threshold, overlap_mask_area_threshold) ⇒ *
  • .compute_segments(mask_probs, pred_scores, pred_labels, mask_threshold, overlap_mask_area_threshold, label_ids_to_fuse, target_size) ⇒ *
  • .post_process_panoptic_segmentation(outputs, [threshold], [mask_threshold], [overlap_mask_area_threshold], [label_ids_to_fuse], [target_sizes]) ⇒ Array.<{segmentation: Tensor, segments_info: Array<{id: number, label_id: number, score: number}>}>

detrFeatureExtractor._call(urls) ⇒ Promise.<Object>

Calls the feature extraction process on an array of image URLs, preprocesses each image, and concatenates the resulting features into a single Tensor.

Kind: instance method of DetrFeatureExtractor
Returns: Promise.<Object> - An object containing the concatenated pixel values of the preprocessed images.

detrFeatureExtractor.post_process_object_detection() : post_process_object_detection

Kind: instance method of DetrFeatureExtractor

detrFeatureExtractor.remove_low_and_no_objects(class_logits, mask_logits, object_mask_threshold, num_labels) ⇒ *

Binarize the given masks using object_mask_threshold, it returns the associated values of masks, scores and labels.

Kind: instance method of DetrFeatureExtractor
Returns: * - The binarized masks, the scores, and the labels.

detrFeatureExtractor.check_segment_validity(mask_labels, mask_probs, k, mask_threshold, overlap_mask_area_threshold) ⇒ *

Checks whether the segment is valid or not.

Kind: instance method of DetrFeatureExtractor
Returns: * - Whether the segment is valid or not, and the indices of the valid labels.

detrFeatureExtractor.compute_segments(mask_probs, pred_scores, pred_labels, mask_threshold, overlap_mask_area_threshold, label_ids_to_fuse, target_size) ⇒ *

Computes the segments.

Kind: instance method of DetrFeatureExtractor
Returns: * - The computed segments.

detrFeatureExtractor.post_process_panoptic_segmentation(outputs, [threshold], [mask_threshold], [overlap_mask_area_threshold], [label_ids_to_fuse], [target_sizes]) ⇒ Array.<{segmentation: Tensor, segments_info: Array<{id: number, label_id: number, score: number}>}>

Post-process the model output to generate the final panoptic segmentation.

Kind: instance method of DetrFeatureExtractor

processors.Processor ⇐ Callable

Represents a Processor that extracts features from an input.

Kind: static class of processors
Extends: Callable

• .Processor ⇐ Callable
  • new Processor(feature_extractor)
  • ._call(input) ⇒ Promise.<any>

new Processor(feature_extractor)

Creates a new Processor with the given feature extractor.

processor._call(input) ⇒ Promise.<any>

Calls the feature_extractor function with the given input.

Kind: instance method of Processor
Returns: Promise.<any> - A Promise that resolves with the extracted features.

processors.WhisperProcessor ⇐ Processor

Represents a WhisperProcessor that extracts features from an audio input.

Kind: static class of processors
Extends: Processor

whisperProcessor._call(audio) ⇒ Promise.<any>

Calls the feature_extractor function with the given audio input.

Kind: instance method of WhisperProcessor
Returns: Promise.<any> - A Promise that resolves with the extracted features.

processors.AutoProcessor

Helper class which is used to instantiate pretrained processors with the from_pretrained function. The chosen processor class is determined by the type specified in the processor config.

Example: Load a processor using from_pretrained.

let processor = await AutoProcessor.from_pretrained('openai/whisper-tiny.en');

Example: Run an image through a processor.

let processor = await AutoProcessor.from_pretrained('Xenova/clip-vit-base-patch16');
let image = await RawImage.read('https://huggingface.co/datasets/Xenova/transformers.js-docs/resolve/main/football-match.jpg');
let image_inputs = await processor(image);
// {
//   "pixel_values": {
//     "dims": [ 1, 3, 224, 224 ],
//     "type": "float32",
//     "data": Float32Array [ -1.558687686920166, -1.558687686920166, -1.5440893173217773, ... ],
//     "size": 150528
//   },
//   "original_sizes": [
//     [ 533, 800 ]
//   ],
//   "reshaped_input_sizes": [
//     [ 224, 224 ]
//   ]
// }

Kind: static class of processors

AutoProcessor.from_pretrained(pretrained_model_name_or_path, options) ⇒ Promise.<Processor>

Instantiate one of the processor classes of the library from a pretrained model.

The processor class to instantiate is selected based on the feature_extractor_type property of the config object (either passed as an argument or loaded from pretrained_model_name_or_path if possible)

Kind: static method of AutoProcessor
Returns: Promise.<Processor> - A new instance of the Processor class.

processors~center_to_corners_format(arr) ⇒ Array.<number>

Converts bounding boxes from center format to corners format.

Kind: inner method of processors
Returns: Array.<number> - The coodinates for the top-left and bottom-right corners of the box (top_left_x, top_left_y, bottom_right_x, bottom_right_y)

processors~post_process_object_detection(outputs) ⇒ Array.<Object>

Post-processes the outputs of the model (for object detection).

Kind: inner method of processors
Returns: Array.<Object> - An array of objects containing the post-processed outputs.

post_process_object_detection~box : Array.<number>

Kind: inner property of post_process_object_detection

processors~PretrainedOptions : *

Kind: inner typedef of processors