I want to learn modern AI practical approaches from zero to implement it in my CAPCHA solving tasks.
AI BASICS
1. This approaches called as Machine learning or Neural network actually or even artificial neural network (ANN).
2. What solutions AI frameworks we have today. // scikit-learn (python), brain.js, ml5.js, TensorFlow github.com/tensorflow/tensorflow (Google), PyTorch, Theano or CNTK (Microsoft), Keras keras.io (works only on top of tensorflow or theano, MXNET, or CNTK), Chainer, NORB
for proffesional AI implementation: ONNX, OpebVINO, TVM they could be 10 times faster.
2.0. What models for image classification we have kaggle.com/models, transformer models huggingface.co/models, Inception V3 Model, VGG16, ConvNeXt, ResNe(X)t, DenseNet, EfficientNet, MobileNet,
2.0.1 Sound classification models YAMNet
2.1. What language generative models we have PaLM (Google), MakerSuite, LaMBDA, Bard (chatbot), BERT (used in google engine), Vertex AI, Open AI's Chat LGBT, Yandex LGPT.
2.2. Image gen AI. Vertex AI, ResNet, Inception, EfficientNet.
3. What terms could be used for object detection. // HOG and CNN (Convolutional Neural Networks)
4. What datasets we have. kaggle.com/datasets, huggingface.co/datasets, MNIST, CIFAR-10, IMDB, kaggle, movielens, Google Gmail Smartreply, Google Books Ngram, Palmer Pinguins, Movielens, Kaggle Dogs vs Cats, COCO captions, ImageNet (1.4M images). You could look for datasets at datasetsearch.research.google.com, GLUE, SST, horse or humans, Auto MPG, EuroSAT, Flickr8k, Wikimedia datasets (dumps.wikimedia.org), BooksCorpus, MEDLINE and PubMed, COLA (nyu-mll.github.io/CoLA/), tensorflow.org/datasets/catalog/, CelebA (faces), LAION-58 (text-image), Cityscapes, CMP Facades, (labels-photo),
4.1 Audio datasets AudioSet-YouTube corpus, Mozilla’s Common Voice, TIDIGITS, CHiME-5, LibriSpeech, ESC-50, Maestro (Audio RNN)
5. The main thing you have to do in Tensorflow is to make dataset in TF format then feed it to the model.
6.
Implement machine learning in SEO
1. Possibly I could gather articl'es parametrs and article's places in Search Engines to construct linear regression or maybe multiple regression or polynomial regression (not sure which is the best maybe last one) and predict parameters for first place at search, like:
from sklearn import linear_model
d = pandas.read_csv("data.csv")
X = d[['Weight', 'Volume']]
y = d['CO']
r = linear_model.LinearRegression()
r.fit(X, y)
predictCO = regr.predict([[2300, 1300]]) #predict
1.1 To verify if data actually correspond to regression use $ from sklearn.metrics import r2_score
PRETRAINED MODELS USAGE
1. To load pretrained models in Tensorflow use tf.keras.applications."model_name" and you could switch top level off by tf.keras.applications.ConvNeXtBase(include_top=False)
2. To use pretrained models in Keras use keras.applications."model_name"
3. Use models from kaggle.com/models and from github.com/tensorflow/models
4. For GLUE tasks you can use EXPERTS-BERT pretrained models.
TENSORFLOW
1. While importing tensorflow in pyCharm I've got an error: Process finished with exit code 132 (Interrupted by signal 4:SIGILL)
1.1.0 Or illegal instruction (core dumped) mean CPU architecture does not support tensorflow
1.1. SIGILL means illigal assembly instruction probably for another architecture.
1.2. Try to delete second python interpreter. Probably I've downloade package for wrong architecture. // Nothing
1.3. Try to install with pip3 different version of tensorflow // $ pip3 install tensorflow==2.00
1.3.1. To check available versions $ pip3 index versions tensorflow
1.3.2. Install version 2.15.0, 2.14.1, 2.13.0, 2.12.0 // Same
1.3.3. Downgrade to 2.1.0 // Nothing
1.3.4. Try to compile TF from the source it should help.
BAZEL is a piece of crap
1.3.4.1. Bazel is needed to compile tensorflow. It is not part of linux ditribution but Google use it. I don't know why it is so complicated just to build from source. And why I have to install some third party crap like Bazel. Maybe it is good idea just to choose different ML solution instead of TF.
1.3.4.2. Get error while $ basel build asked for basel version 6.5.0 which doesn't exists. // Change version in .baselversion hiddenfile
1.4. To build from source follow bazel.build/install/compile-source // Didn't reached the end
1.5. Try install from .deb // works
1.6. Download TF from $ git clone https://github.com/tensorflow/tensorflow.git && cd tensorflow
1.6.1. $ ./configure
1.6.2 $ basel build // And nothing in bazel-bin folder. I have no idea what to do next and I don't have time for this useless Bazel. Screw it. I going to delete this bazel crap and will try use Docker.
1.7 DOCKER didn't helped me with the problem probably because it uses host machine. Maybe classic VM will help.
1.8. VM also doesn't help.
1.9. Find some solution on Transformer basis. May be I can use TF model with Transformer by TFPreTrainedModel class?
from transformers import TFAutoModel
model = TFAutoModel.from_pretrained("google-bert/bert-base-cased")
2. How to create neural network with keras. Import
from tensorflow import keras
from tensorflow.keras import layers
2.1. Create 3 layers
import tensorflow as tf
m = tf.keras.Sequential()
m.add(tf.keras.layers.Dense(64,activation='relu') # relu adds unlinearity
m.add(tf.keras.layers.Dense(32,activation='relu')
m.add(tf.keras.layers.Dense(10)
or like:
model = tf.keras.models.Sequential([
tf.keras.layers.Flatten(input_shape=(28, 28)),
tf.keras.layers.Dense(128, activation='relu'),
tf.keras.layers.Dropout(0.2),
tf.keras.layers.Dense(10)
])
2.2. Compile network.
m.compile( optimizer='adam', loss=tf.keras.losses.SparseCategoricalCrossentropy(from_logits=True), metrics=['accuracy'] )
2.2.1. Visualisation (optional)
vis = TensorBoard(log_dir="logs")
2.3. Train
m.fit(train_images, train_labels, epochs=10, callbacks=[vis])
2.4. To download MNIST
mnist = tf.keras.datasets.mnist
(x_train, y_train), (x_test, y_test) = mnist.load_data()
2.4.1 How to download MNIST on my machine. // You will see url in terminal
2.4.2. How to plot images from dataset
image = np.array(train_images[0], dtype='uint8')
plt.imshow(image)
plt.show()
2.5. To predict
probability_model = tf.keras.Sequential([model, tf.keras.layers.Softmax()])
predictions = probability_model.predict(list_my_new_images)
2.6. How to make tensor from image. load image file into python's matplotlib.
from matplotlib import image as mpimg
image = mpimg.imread("/home/mytest/5.png")
2.6.1. I have to make numpy.ndarray from image // It is not enough to use in tf
n = np.array(image)
2.6.2. Or try to import with //
f = open('fath/my_image.bmp', "rb")
data = np.fromfile(f, dtype=np.uint8) // Nothing
2.6.3. Try with keras
from keras.preprocessing.image import load_img
from keras.preprocessing.image import img_to_array
from keras.preprocessing.image import array_to_img
img = load_img('/home/mytest/5.png')
img_numpy_array = img_to_array(img) # convert the image into numpy array // Nothing
2.6.4. Try
data_dir = pathlib.Path("/home/AI/mytest/")
val_ds = tf.keras.utils.image_dataset_from_directory(
data_dir,
validation_split=0.2,
subset="validation",
seed=123,
image_size=(img_height, img_width),
batch_size=batch_size)
// I've got error No images found in directory /home/john/Downloads/AI/mytest/. Allowed formats: ('.bmp', '.gif', '.jpeg', '.jpg', '.png')
2.6.5. Convert image to tensor Tensorflow. Create image bmp black&white (8-bit grayscale) vertical images each 28x28
from PIL import Image
import tensorflow as tf
img = Image.open('path/my_image.jpg')
t = tf.convert_to_tensor(img, dtype=tf.float32)
t2 = tf.reshape(t, [1,28,28])
t2 = t2 / 255
t2 = t2 - 1
// But it is half solution i want to use colored images
2.6.6. How to implement mask.
2.6.7. How to import image with target size (like 28x28)
from keras.preprocessing import image
img = image.load_img("/home/picture.bmp", color_mode="grayscale", target_size=(28,28))
2.6.8. Reshape ndarray from shape 28x28x1 to 28x28. // new_arr = np.reshape(old_arr, (28,28))
2.6.9. How to invert colors ndarray // numpy.invert(img)
image = 255 - image
2.6.10. Alternative method to predict image from folder
import PIL.Image as Image
im = Image.open('/home/Datasets/saves/transf/ostrich.jpg').resize((200,200))
im = np.array(im)/255.0
result = model.predict(im[np.newaxis, ...])
2.6.10.1. Import an image with tensorflow only.
path = "/home/path/img_cat.jpg"
image = tf.io.read_file(path)
img = tf.io.decode_jpeg(image, channels=1)
img_r = tf.image.resize(img, size=[333,333])
im = tf.image.convert_image_dtype(img_r, dtype=tf.float32)
im = tf.expand_dims(im, axis=0)
2.6.11. To make grayscale:
gray = tf.image.rgb_to_grayscale(img)
2.7. CNN Convolution neural network may help to find some object in unknown position at image. In keras it Conv2D layer.
2.7.1. Convolution technic somehow helps to detect common patterns of image like shape or edges or background. It uses different filters (kernels).
For example filter below helps to pass sharp edges:
[[0,1,0],
[1,-4,1],
[0,1,0]]
Filter for vertical lines:
[[-1,0,1],
[-2,0,2],
[-1,0,1]]
Filter for horizontal lines:
[[-1,-2,-1],
[0,0,0],
[1,2,1]]
Ridge and edge detection:
[[-1,-1,-1],
[-1,8,-1],
[-1,-1,-1]]
Sharpen:
[[0,-1,0],
[-1,5,-1],
[0,-1,0]]
Blur: // Doesn't work
[[1,1,1],
[1,1,1],
[1,1,1]]
Gaussian blur: // Doesn't work
[[1,2,1],
[2,4,2],
[1,2,1]]
Emboss:
[[-2,-1, 0],
[-1, 1, 1],
[ 0, 1, 2]]
2.7.2. Along with convolutions you have to use pooling technic. It is kind of increase contrast of the convoluted image.
2.7.3. CAM technic, semantic segmentation or R-CNN (region proposal network) also could be helpfull for recognizing oblects at different parts of an image.
2.8. To generate more images (rotate, zoom, flip) for dataset use augmantation.
images = ImageDataGenerator(
rescale = 1./255,
rotation_range = 40,
width_shift_range = 0.2,
height_shift_range = 0.2,
shear_range = 0.2,
horizontal_flip=True,
fill_mode='nearest')
2.9. To use pretrained model you have to freeze old layers and train only last new few layers. It is called fine-tuning feature extraction or transfer learning (fine tuning it is when you little bit retrain old model with a very low learning rate and small amount of epochs. For fast finetuning use LORA technique. To load models.
import tensorflow_hub as hub
hub_model = hub.load(model)
3. To detect objects on image use TensorFlow Object Detection API
4. Datasets you downloaded with tfds.load() are saved in the local folder /home/john/tensorflow_datasets/t
5. Use Active Lerning model principle.
6. Use Committe sampling or commette based paradigm. When you get decision based on many models.
TRANSLATION
4. For translation use NMT approaches (Neural machine translation)
4.1. You need language datasets like Anki.
4.2. For translation best choise is Transformer neural network layer.
5. While running transformer.summary() on tensorflow model I get an error ValueError: This model has not yet been built. Build the model first by calling `build()` or by calling the model on a batch of data.
5.1. Try to run a test on example of data before transformer.summary():
for (pt, en), en_labels in train_batches.take(1):
break
output = transformer((pt, en))
PERFORMANCE ISSUES. MAKE FITTING FASTER
0. You can optimize speed or you can optimize GPU memory usage.
Make faster
1. Usage of model.compile(jit_compile = True) make it faster for tensorflow.
2. I get a problem. While I am trying to fit my model in Tensorflow it takes to mach CPU and never reach the end.
3. Use int8 or float16 data format rather then float32, It is called quantization and executes faster.
quant_model = tfmot.quantization.keras.quantize_model
q_model = quant_model(model)
4. The most simple way to decrease fit time is to decrease number of layers and number of units in layers.
5. Tensorflow has special package. // $ pip install tensorflow-model-optimization
from tensorflow_model_optimization.python.core.keras.compat import keras
7. Python is not the best option for performance issues.
8. Use fast frameworks instead of TF.
GPU RAM
6. Decrease batch size. It optimizes RAM usage. I think decreasing of batch size will increase performance of LLM because of padding (all training examples should have the same length inside of a batch, thats why short texts will be filled with zeroes, to reach the size of the biggest example in batch.)
9. Change optimizer. AdamW use a lot of RAM.
10. Enable attention slicing. For HF pipe.enable_attention_slicing()
11. Reduce size of input.
5. While saving my model got an error NotImplementedError: Learning rate schedule 'CustomSchedule' must override `get_config()` in order to be serializable.
5.1. Try to set optimizer = 'adam' // Still get WARNING:tensorflow:Model's `__init__()` arguments contain non-serializable objects. Please implement a `get_config()` method in the subclassed Model for proper saving and loading. Defaulting to empty config. But model has been saved but when I load it I get ValueError: Unable to restore custom object of class "MeanMetricWrapper" (type _tf_keras_metric). Please make sure that this class is included in the `custom_objects` arg when calling `load_model()`. Also, check that the class implements `get_config` and `from_config`.
5.1.1. Serializable means convertable
5.1.2. Seems to be it is inpossible to save keras model with subclasses. Delete classes or try to save weights.
5.2. Try transformer.save_weights('path/filename.cpkt') then transformer.load_weights('path/filename.cpkt')
6. If you have laptop and want to avoid problems with loading huge dataset into small RAM try to use >>> from datasets import load_dataset
5. While I am trying to fine tune AI model in Colab on GPU I get RAM overflow error:
OutOfMemoryError: CUDA out of memory. Tried to allocate 1.95 GiB. GPU 0 has a total capacity of 14.75 GiB of which 1.74 GiB is free. Process 12641 has 13.00 GiB memory in use. Of the allocated memory 9.26 GiB is allocated by PyTorch, and 3.60 GiB is reserved by PyTorch but unallocated. If reserved but unallocated memory is large try setting PYTORCH_CUDA_ALLOC_CONF=expandable_segments:True to avoid fragmentation. See documentation for Memory Management (pytorch.org/docs/stable/notes/cuda.html#environment-variables)
5.1. Use model.to("cuda") for GPU
model = AutoModelForAudioClassification.from_pretrained("facebook/wav2vec2-base", num_labels=num_labels, label2id=label2id, id2label=id2label).to("cuda") // Doesn't help
5.2. Try model = AutoModelForAudioClassification.from_pretrained("facebook/wav2vec2-base", device_map = 'cuda') // Doesn't help
5.3. Decrease per_device_train_batch_size and per_device_eval_batch_size
training_args = TrainingArguments(
per_device_train_batch_size=16,
per_device_eval_batch_size=16,
)
MY THOUGHTS
1. What is difference between neural net and human brain. In human brain not only weights are changing during learning but the model structure itself.
2. You can even train model on unlabeled data then finetune it on labeled. It could be more effective than same ammount of just labeled data training.