AI Transformers usage

Here I will explore how to use transformers framework with pretrained artificial neural networks. Looks like transformers is the best way I know to implemente AI networks. It is way more convenient and reliable then Tensorflow at least.

basics of transformer framework with artificial intelligence and Development|lower

TRANSFORMERS IN ARTIFICIAL INTELLIGENCE

0. Transformer is a neural net model architecture which is pays more attention to specific words and ignore others.

1. To install $ pip install transformers

1.0. To use transformers Jax, PyTorch or TensorFlow needed to be installed.

1.1. Transformer Huggingface library is needed to easy load and use ML models.

PIPELINE

EXAMPLES

2. The core thing is pipeline(). The simplest example where you define a task. Pipeline tasks are here huggingface.co/docs/transformers/main_classes/pipelines

from transformers import pipeline

classifier = pipeline("sentiment-analysis")

classifier(["Some text for example.","Second sentence"])

2.1. Usage with pretrained model without task.

result = pipeline(model="distilbert/distilbert-base-cased-distilled-squad", tokenizer="google-bert/bert-base-cased")

2.2. Another example of usage

model = AutoModelForTokenClassification.from_pretrained("dbmdz/bert-large-cased-finetuned-conll03-english")

tokenizer = AutoTokenizer.from_pretrained("google-bert/bert-base-cased")

recognizer = pipeline("ner", model=model, tokenizer=tokenizer)

2.3. One more example of usage:

music = pipeline(task="text-to-audio", model="facebook/musicgen-small", framework="pt")

generate_kwargs = {

"do_sample": True,

"temperature": 0.7,

"max_new_tokens": 35, }

outputs = music("Techno german funk march with high melodic riffs", generate_kwargs=generate_kwargs) 

2.4. Example of implementing

dataset = load_dataset("ashraq/esc50")

audio = next(iter(dataset["train"]["audio"]))["array"]

classifier = pipeline(task="zero-shot-audio-classification", model="laion/clap-htsat-unfused")

classifier(audio, candidate_labels=["Sound of a dog", "Sound of vaccum cleaner"])

2.5. Pipeline transformer implementation for LLM:

gen = pipeline(model="openai-community/gpt2")

gen("I believe ", do_sample=False) // [{'generated_text': "I believe I can fly"}]

2.6. Simple instance:

generation = pipeline("text-generation")

generation("In this course, we will guide you how to")

3. What tasks for pipelines we have:

  1. "audio-classification"
  2. "automatic-speech-recognition"
  3. "depth-estimation":
  4. "document-question-answering"
  5. "feature-extraction"
  6. "fill-mask" // $ pipeline("fill-mask")("This book will tell you all about <mask> LGBTQ+ history.", top_k=2) Mask word could be different for other models.
  7. "image-classification"
  8. "image-feature-extraction"
  9. "image-segmentation"
  10. "image-to-image"
  11. "image-to-text"
  12. "mask-generation"
  13. "object-detection"
  14. "question-answering" // pipeline("question-answering")(question="Where do I work?", context="My name is Sebasian and I work at Google in Bronx",)
  15. "summarization"
  16. "table-question-answering"
  17. "text2text-generation"
  18. "text-classification" or "sentiment-analysis"
  19. "text-generation"
  20. "text-to-audio" (alias "text-to-speech")
  21. "token-classification" (or "ner") // It could be used to find in text persons, organizations and geographical places.
  22. "translation"
  23. "translation_xx_to_yy"
  24. "video-classification"
  25. "visual-question-answering"
  26. "zero-shot-classification"
  27. "zero-shot-image-classification"
  28. "zero-shot-audio-classification"
  29. "zero-shot-object-detection"

4. Set generated frase length for LLM.

gener= pipeline("text-generation", model="distilgpt2")

gener("In this article, we will ask you", max_length=30, num_return_sequences=2)

5. You can construct transformer from encoder and decoder of differnet models. For example BERT encoder and GPR decoder.

WITHOUT PIPELINE

Tokenizer, Model

6. Usage

import torch

from transformers import AutoTokenizer, AutoModelForSequenceClassification

checkpoint = "distilbert-base-uncased-finetuned-sst-2-english"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)

raw_inputs = [

"I've been waiting for a bathroom my whole life.",

"I hate pizza so much!", ]

inputs = tokenizer(raw_inputs, padding=True, truncation=True, return_tensors="pt") # You get array with numbers.

model = AutoModelForSequenceClassification.from_pretrained(checkpoint)

outputs = model(**inputs) # Here you get raw meaningless values

predictions = torch.nn.functional.softmax(outputs.logits, dim=-1)

7. Use certain model

from transformers import BertModel

model = BertModel.from_pretrained("bert-base-cased")

8. Save model

model.save_pretrained("directory_on_my_computer")

9. Decode string of tokens // decoded = tokenizer.decode([353, 180, 1003, 270, 743, 181, 3014])

FINE-TUNING

10. I have to create Trainer.

from transformers import Trainer, AutoModelForSequenceClassification, TrainingArguments, DataCollatorWithPadding

raw_datasets = load_dataset("glue", "mrpc")

tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")

def tokenize_function(example):

return tokenizer(example["sentence1"], example["sentence2"], truncation=True)

tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

trainer = Trainer(

AutoModelForSequenceClassification.from_pretrained(checkpoint, num_labels=2),

TrainingArguments("test-trainer", evaluation_strategy="epoch"),

train_dataset=tokenized_datasets["train"],

eval_dataset=tokenized_datasets["validation"],

data_collator=DataCollatorWithPadding(tokenizer=tokenizer),

tokenizer=tokenizer, )

trainer.train()

 

 

Problems

1. In Google Colab got error:

RuntimeError: Failed to import transformers.pipelines because of the following error (look up to see its traceback):

partially initialized module 'jax' has no attribute 'version' (most likely due to a circular import)

1.1.Try to $ pip install -U accelerate // Doesn't help

1.2. Create new Colab notebook. // helped