Django automatic article translation

I need to automate django article translation and create image automically.

django autocreate and translate image with джанго and Вебдизайн|lower


AUTOCREATE ARTICLE TRANSLATION

1. CREATE TRANSLATION

1.1. Made Django based translation

1.1.2. Django admin templates is here /home/<user>/django/lib/python3.8/site-packages/django/contrib/admin/templates/admin

1.1.3. It is possible to override the admin template. Set settings.py 'DIRS': [os.path.join(BASE_DIR, 'templates')],

1.1.3.1. Change original template's name /home/<user>/django/lib/python3.8/site-packages/django/contrib/admin/templates/admin/change_form_new.html

1.1.3.2. Add extension file in my app template folder change_form.html -> add {% extends 'admin/change_form.html' %}

1.1.4. To save something into the database use save() method for your django model instance for example:

from cards.models import Article

a10 = Article(title="Biba", text="Boba")

a10.save()

1.1.5. While I am trying in template to use <a href="{% url 'card-list' %}"> I got an error Reverse for 'card-list' not found. 'card-list' is not a valid view function or pattern name.

1.1.5.1. Try <a href="{% url '<app-name>:card-list' %}"> // It works

1.1.6. Template for admin button panels called submit_line.html // I create extension file for this template with a button. You press the button -> It will generatNULLe POST request.

1.1.7. How to deal with POST request? I have to bind some function to it. // View.py functions deal with POST like:

if request.method == "POST":

    data = request.POST

    action = data.get("follow")

    if action == "follow":

    current_user_profile.follows.add(profile)

1.2. Create python script

1.2.1. I will use selenium to create script. I decline idea to make server based script integrated in django.

1.2.2. I could take translated text from webpage or from database. I prefer from database.

1.2.3. How to translate html code? Maybe split tags and corresponding text into a dictionary or substitute text between tags with array elements like <h2>array[0]</h2>. Try to use BeautifulSoap package.

1.2.4. Which transators I could use: Yandex, Sogou, Baidu, Tencent, Youdao, Alibaba, Deepl, Caiyun.

1.2.5. Split html tags from text. Script:

s = "<h2 dir='auto' style='box-sizing: border-box;'> <li style='box-sizing: border-box;'>Extract content of given online website/pages and stored under JSON data <strong>sudo ls -ls&nbsp;</strong></li> "

s_new= []

flag_close = True

flag_start = False

start = 0

end = 0

ar = []

for i in range(len(s)):

#DEBUG

  a0=s[i]

  a1=s[start]

  a2=s[end]

if (s[i]=='>' and flag_start):                           # Find end of the first tag

flag_close = True                                 # You reach end of the tag

flag_start = False                                # You reach end of the tag

end = i                                                  # Remember position of a tag ending

s_new.append(s[start:end+1])          # Write tag into list

if (s[i]=='<' and flag_close):                         # Find the string between tags

flag_close = False

flag_start = True

start=i   

b=s[end+1:start].strip()                     # If wrapped in strong tag do not translate

if b:                                                      # Check if text it is not empty and not wrapped into strong tag     

if s_new[-1]!="<strong>": 

ar.append(s[end+1:start])                      # Write text to dictionary

s_new.append('ar['+str(len(ar)-1)+']')    # put array element instead of text

else:

s_new.append(s[end+1:start])              # If wrapped in strong tag do not translate

1.2.6. If text is more then 5000 symbols I have to split it. To split element I use : ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")

1.2.7. Final script:

from selenium.webdriver import Firefox                # for mozilla browser

from selenium.webdriver.common.by import By

from selenium.webdriver.firefox.options import Options

import time 

import mysql.connector                                # For mysql

from datetime import datetime

import os

from PIL import Image

from PIL import ImageFilter

from PIL import ImageEnhance

from PIL import ImageOps

import random

import pexpect                                        # To use linux command from python

#0. INPUT PAGE ALLIAS YOU WANT TO TANSLATE

ali = "seo-django"                                    # First of all enter ali of article you want to translate

#1. PREPARE DATABASE

db = mysql.connector.connect(                         # Initialize database

    host="localhost",

    user="user", 

    password="12345",

    database="django")   

m = db.cursor()

#3. GET ENGLISH ARTICLE YOU WANT TO TRANSLATE FROM DB

m.execute("SELECT `id`,`al_id` FROM `how_to_article2` WHERE `ali`= '"+ali+"' ;")   # GET id of english row with specified ali from database

r = m.fetchall()

r1 = r[0][0]                                          # Get article's id

m.execute("SELECT `title`, `details`, `text`, `img_alt`, `meta_description` FROM `how_to_article2_translation` WHERE `master_id`= "+str(r[0][0])+";")    

r2 = m.fetchall()                                     # GET data of english article from database

m.execute("SELECT `al` FROM `how_to_cou` WHERE `id`= "+str(r[0][1])+";")    

r3 = m.fetchall()                                     # GET cou name

# IMAGE GENERATOR INTEGRATION

#1.Take user input. Which subcategory new article belongs.

subcategory = r3[0][0]                                # Get cou

nm = r2[0][0].replace(" ","-").strip("!")             # Take name from database. Article name. Get from database

#2.Then I have to get category which subcategory belongs.

category = ""                                         # Is a main category name which we are looking for.

local_dir = "/home/.../eclipse-workspace/django001/outleys/static/image/"   #Set the Base loca directory with images

for ctg in os.listdir(local_dir) :                    # Get list of categories ,ctg is the top level category

if os.path.isdir(local_dir+ctg) and category=='':

for k in os.listdir(local_dir+ctg):           # k is subactegory

if os.path.isdir(local_dir+ctg+"/"+k) and k==subcategory:  # Iterate over all subcategories until match

category = ctg                    # We find top level categoty which correspond subcategory

break

#3.Get first two images 

p1 = Image.open(local_dir+category+"/background.png")

p2 = Image.open(local_dir+category+"/"+subcategory+"/1.png")

#3.1.Get random third image

folder_list = os.listdir(local_dir+"_gallery_png/")   # Get list of folders

file_dir = local_dir+"_gallery_png/"+folder_list[int((len(folder_list)*random.random()))] # Get random folder

files = os.listdir(file_dir)                          # Get list of files inside of random folder

# Choose random file image from a random folder

p3 = Image.open(file_dir+"/"+files[int((len(files)*random.random()))]) # Choose random image from gallery

#3.1.1 Increase or decrease p3 if it is too small or too big

if p3.size[0]<200 and p3.size[1]<200:

krat = 300//p3.size[0]                            # How much times it should be enlarged

w2 = p3.size[0]*krat

h2 = p3.size[1]*krat

p3=p3.resize((w2,h2), Image.Resampling.NEAREST)   # Increase image

if p3.size[0]>400 or p3.size[1]>400:

krat1 = p3.size[0]//300                           # How much times it should be decreased

krat2 = p3.size[1]//300  

krat = max(krat2, krat1)

w3=p3.size[0]//krat

h3=p3.size[1]//krat

p3=p3.resize((w3,h3), Image.Resampling.LANCZOS)   # Decreas size

#3.2. Construct composite image

w = int(0.7*p1.size[0]*random.random()-p1.size[0]//2) # Generate random position to insert p2

h = int(0.7*p1.size[1]*random.random()-p1.size[1]//2)

p1.paste(p2, (w,h), mask=p2)

w1 = int((p1.size[0]-p3.size[0])*random.random())     # Generate random position to insert p3

h1 = int((p1.size[1]-p3.size[1])*random.random())

p1.paste(p3, (w1,h1), mask=p3)

#3.3. Apply random effects

imdir = local_dir+category+"/"+subcategory+"/"+nm+".png"

rullet = int(150*random.random())                     # Random number from 0 to 150

if 0<= rullet <= 5:                                 

p1 = p1.convert("L")                              # Made black&white

if 5< rullet <= 10:                                  

p1 = p1.convert("1")                              # Made crop

if 10< rullet <= 15:                                  

p1 = p1.filter(ImageFilter.CONTOUR)               # contour

if 15< rullet <= 20:                               

p1 = p1.transpose(Image.FLIP_LEFT_RIGHT)          # Mirror

if 20< rullet <= 25:                                  

p1 = p1.filter(ImageFilter.GaussianBlur(radius = 2*random.random()+1 )) # blur

if 25< rullet <= 30:                                

p1 = p1.filter(ImageFilter.SHARPEN)               # Sharpen

if 30< rullet <= 35:                                 

p1 = ImageOps.expand(p1, border = 10, fill = 100) # border

if 35< rullet <= 40:                                  

p1 = ImageEnhance.Color(p1).enhance(5.0*random.random()+1.0)   # more colors

if 40< rullet <= 45:                                  

p1 = ImageEnhance.Contrast(p1).enhance(5.0*random.random()+1.0)   # more contrast

if 45< rullet <= 50:                                 

p1 = ImageEnhance.Brightness(p1).enhance(5.0*random.random()+1.0)   # more bright

if 50< rullet <= 55:                                  

p1 = p1.filter(ImageFilter.MaxFilter(size=int(3*random.random()+1)))  # fancy

p1.save(imdir,"PNG")                               #Save generated image

#4.   Upload image to the remote server /www/outleys.site/mydiary/how_to/static/image/

c1 = pexpect.spawn('scp '+imdir+' u02/...y/how_to/static/image/'+category+'/')   # Write an image to the server

c1.expect('(?i)password:')                            # Wait for password prompt

c1.sendline('pass')                               # send password

#5.   Write image to database

sql = "UPDATE `how_to_article2` SET `im3` = %s WHERE `how_to_article2`.`id` = '"+str(r1)+"';"  # Write into database

w = [ "image/"+category+"/"+nm+".png",]

m.execute(sql,w) 

db.commit()

# END OF IMAGE GENERATOR

 

#2. START SELENIUM BROWSER

o = Options()

b = Firefox(options=o)                                      # start browser

ur = 'deepl'

b.get(ur)                                                   # open specific page

time.sleep(5)

ls = {                                                      # list of languges for translations. Fill data from database     

"English":{"tit":r2[0][0], "det":{"all":r2[0][1],"html":[],"txt":[]}, "txt":{"all":r2[0][2],"html":[],"txt":[]}, "des":r2[0][3], "alt":r2[0][4]},

"Spanish":{"tit":"", "det":{"all":[],"html":[],"txt":[]}, "txt":{"all":[],"html":[],"txt":[]}, "des":"", "alt":""},

"Russian":{"tit":"", "det":{"all":[],"html":[],"txt":[]}, "txt":{"all":[],"html":[],"txt":[]}, "des":"", "alt":""},

#3.1 HERE WE SPLIT ENGLISH ARTICLE AND PREPARE HTML FOR TRANSLATION

for j in ["det","txt"] :

flag_start = False

start = 0

end = 0

for i in range(len(ls["English"][j]["all"])):

if (ls["English"][j]["all"][i]=='>' and flag_start):       # You reach end of the tag

flag_start = False                                     # You reach end of the tag

end = i                                                # Remember position of a tag ending

ls["English"][j]["html"].append(ls["English"][j]["all"][start:end+1])   # Write tag into list

if (ls["English"][j]["all"][i]=='<' and not flag_start):   # Find the string between tags

flag_start = True

start=i 

strip = ls["English"][j]["all"][end+1:start].strip()     

if strip and strip!="&nbsp;":      # Remove empty spaces. Check if text it is not empty and not wrapped into strong tag                 

if ls["English"][j]["html"][-1]!="<strong>": 

ls["English"][j]["txt"].append(ls["English"][j]["all"][end+1:start])            # WRite text to dictionary

ls["English"][j]["html"].append('ar['+str(len(ls["English"][j]["txt"])-1)+']')  # put array element instead of text

else:

ls["English"][j]["html"].append(ls["English"][j]["all"][end+1:start])           # If wrapped in strong tag do not translate

#4. PREPARE DEEPL AND TRANSLATE

if(len(b.find_elements(By.CSS_SELECTOR, ".lmt__language_container #source_language_label .flex button"))):  # Deepl change interface from time to time thats why I made two options

b.find_element(By.CSS_SELECTOR, ".lmt__language_container #source_language_label .flex button").click()

else:

b.find_element(By.CSS_SELECTOR, ".lmt__language_select--source button").click()

b.find_element(By.XPATH,"//span[contains(text(),'English')]").click()                               # choose english language

inp = b.find_element(By.CSS_SELECTOR, ".lmt__source_textarea")     # Get input text area from deepl

out = b.find_element(By.CSS_SELECTOR, "#target-dummydiv")          # Get output textare for second time

inp.clear()                                                        # Clear input field

for r in ['Russian','Spanish']:

if(len(b.find_elements(By.CSS_SELECTOR, "span #headlessui-popover-button-7"))):                 # Check if element is found

b.find_element(By.CSS_SELECTOR, "span #headlessui-popover-button-7").click()

else:

b.find_element(By.CSS_SELECTOR, ".lmt__language_select--target").click()

b.find_element(By.XPATH,"//span[contains(text(),'"+r+"')]").click()                             # choose language in DEEPL

for k in ["tit","des","alt"]:

inp.send_keys(ls["English"][k])                            # Translate simple fields

time.sleep(9) 

ls[r][k] = out.get_attribute('innerHTML')[:-2]

inp.clear() 

#4.1 DEEPL LIMITED BY 5000 SIMBOLS. HERE WE DIVIDE BIG TEXT INTO SMALL PIESES

for l in ["det","txt"]:                                        # Translate big fields

temp = ", [2]. ".join(ls["English"][l]["txt"])                 # Translate arrays extracted from HTML then pack them back into html

if len(temp)>3500 :                                        # Split if text is more then 5000 simbols

t=len(temp)//3500                                      # How much times it is bigger?

step = len(ls["English"][l]["txt"])//(t+3)             # Calculate step to split array

for y in range(t+3):                                   # Feed DEEPL with small textes

if y==(t+2):   

inp.send_keys(", [2]. ".join(ls["English"][l]["txt"][y*step:]))

time.sleep(12)

ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")

inp.clear() 

else:

inp.send_keys(", [2]. ".join(ls["English"][l]["txt"][y*step:(y+1)*step]))

time.sleep(12)

ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")

inp.clear()

else:

inp.send_keys(temp)                                    # Translate fields

time.sleep(12)

ls[r][l]["txt"] = out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")

inp.clear() 

for n in range(len(ls["English"][l]["html"])):                  # iterate over html list 

for m2 in range(len(ls[r][l]["txt"])):                     # Form html Recompile translated html before write it back to database             

if ls["English"][l]["html"][n]=="ar["+str(m2)+"]":          # iterate over all text element's array        

ls[r][l]["html"].append(ls[r][l]["txt"][m2])               # recompile html

break

if ls["English"][l]["html"][n][:3]!="ar[":                   # skip to not dublicate an array element

ls[r][l]["html"].append(ls["English"][l]["html"][n])

#5. I have a translated html. NOw I have to just write it back to the database

m.execute("SELECT `id` FROM `how_to_article2_translation` ORDER BY `id` DESC LIMIT 1; ")    # Get last id from database

rid = m.fetchall()

lastid = rid[0][0] + 1  

ls["Russian"]["det"]["all"] = "".join(ls["Russian"]["det"]["html"])

ls["Russian"]["txt"]["all"] = "".join(ls["Russian"]["txt"]["html"])

ls["Spanish"]["det"]["all"] = "".join(ls["Spanish"]["det"]["html"])

ls["Spanish"]["txt"]["all"] = "".join(ls["Spanish"]["txt"]["html"])

langs = {"ru":"Russian", "es":"Spanish"}

for k in langs:

sql = "INSERT INTO `how_to_article2_translation` (`id`, `language_code`, `title`, `details`, `text`, `master_id`, `img_alt`, `meta_description`,`date`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)"  # Write into database

w = [ 

lastid,

k,

str(ls[ langs[k] ]["tit"]), 

ls[ langs[k] ]["det"]["all"], 

ls[ langs[k] ]["txt"]["all"], 

r1,

str(ls[ langs[k] ]["alt"]), 

str(ls[ langs[k] ]["des"]), 

str(datetime.now())

]

m.execute(sql,w) 

db.commit()

lastid +=1

 

DJANGO MULTILINGUAL

1. Create locale directory in base path.

2. $ django-admin makemessages -l en // It creates folders and .po files inside of locale folder. Po file will contain translations.

3. $ pip install django-translation-manage

4. In settings.py add in INSTALLED_APPS = [ ... "translation_manager" ... ]

5. $ python manage.py makemigrations -> $ python manage.py migrate

6. Go to admin panel and check. I have no idea how to use this app

4. In template change "Read more" to {{ _("Read more") }}

5. in .PO file add strings:

msgid "Read more"

msgstr "Leer mas"

Doesn't work

6. Try to $ django-admin compilemessages  // That's it