I need to automate django article translation and create image automically.
AUTOCREATE ARTICLE TRANSLATION
1. CREATE TRANSLATION
1.1. Made Django based translation
1.1.2. Django admin templates is here /home/<user>/django/lib/python3.8/site-packages/django/contrib/admin/templates/admin
1.1.3. It is possible to override the admin template. Set settings.py 'DIRS': [os.path.join(BASE_DIR, 'templates')],
1.1.3.1. Change original template's name /home/<user>/django/lib/python3.8/site-packages/django/contrib/admin/templates/admin/change_form_new.html
1.1.3.2. Add extension file in my app template folder change_form.html -> add {% extends 'admin/change_form.html' %}
1.1.4. To save something into the database use save() method for your django model instance for example:
from cards.models import Article
a10 = Article(title="Biba", text="Boba")
a10.save()
1.1.5. While I am trying in template to use <a href="{% url 'card-list' %}"> I got an error Reverse for 'card-list' not found. 'card-list' is not a valid view function or pattern name.
1.1.5.1. Try <a href="{% url '<app-name>:card-list' %}"> // It works
1.1.6. Template for admin button panels called submit_line.html // I create extension file for this template with a button. You press the button -> It will generatNULLe POST request.
1.1.7. How to deal with POST request? I have to bind some function to it. // View.py functions deal with POST like:
if request.method == "POST":
data = request.POST
action = data.get("follow")
if action == "follow":
current_user_profile.follows.add(profile)
1.2. Create python script
1.2.1. I will use selenium to create script. I decline idea to make server based script integrated in django.
1.2.2. I could take translated text from webpage or from database. I prefer from database.
1.2.3. How to translate html code? Maybe split tags and corresponding text into a dictionary or substitute text between tags with array elements like <h2>array[0]</h2>. Try to use BeautifulSoap package.
1.2.4. Which transators I could use: Yandex, Sogou, Baidu, Tencent, Youdao, Alibaba, Deepl, Caiyun.
1.2.5. Split html tags from text. Script:
s = "<h2 dir='auto' style='box-sizing: border-box;'> <li style='box-sizing: border-box;'>Extract content of given online website/pages and stored under JSON data <strong>sudo ls -ls </strong></li> "
s_new= []
flag_close = True
flag_start = False
start = 0
end = 0
ar = []
for i in range(len(s)):
#DEBUG
a0=s[i]
a1=s[start]
a2=s[end]
if (s[i]=='>' and flag_start): # Find end of the first tag
flag_close = True # You reach end of the tag
flag_start = False # You reach end of the tag
end = i # Remember position of a tag ending
s_new.append(s[start:end+1]) # Write tag into list
if (s[i]=='<' and flag_close): # Find the string between tags
flag_close = False
flag_start = True
start=i
b=s[end+1:start].strip() # If wrapped in strong tag do not translate
if b: # Check if text it is not empty and not wrapped into strong tag
if s_new[-1]!="<strong>":
ar.append(s[end+1:start]) # Write text to dictionary
s_new.append('ar['+str(len(ar)-1)+']') # put array element instead of text
else:
s_new.append(s[end+1:start]) # If wrapped in strong tag do not translate
1.2.6. If text is more then 5000 symbols I have to split it. To split element I use : ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")
1.2.7. Final script:
from selenium.webdriver import Firefox # for mozilla browser
from selenium.webdriver.common.by import By
from selenium.webdriver.firefox.options import Options
import time
import mysql.connector # For mysql
from datetime import datetime
import os
from PIL import Image
from PIL import ImageFilter
from PIL import ImageEnhance
from PIL import ImageOps
import random
import pexpect # To use linux command from python
#0. INPUT PAGE ALLIAS YOU WANT TO TANSLATE
ali = "seo-django" # First of all enter ali of article you want to translate
#1. PREPARE DATABASE
db = mysql.connector.connect( # Initialize database
host="localhost",
user="user",
password="12345",
database="django")
m = db.cursor()
#3. GET ENGLISH ARTICLE YOU WANT TO TRANSLATE FROM DB
m.execute("SELECT `id`,`al_id` FROM `how_to_article2` WHERE `ali`= '"+ali+"' ;") # GET id of english row with specified ali from database
r = m.fetchall()
r1 = r[0][0] # Get article's id
m.execute("SELECT `title`, `details`, `text`, `img_alt`, `meta_description` FROM `how_to_article2_translation` WHERE `master_id`= "+str(r[0][0])+";")
r2 = m.fetchall() # GET data of english article from database
m.execute("SELECT `al` FROM `how_to_cou` WHERE `id`= "+str(r[0][1])+";")
r3 = m.fetchall() # GET cou name
# IMAGE GENERATOR INTEGRATION
#1.Take user input. Which subcategory new article belongs.
subcategory = r3[0][0] # Get cou
nm = r2[0][0].replace(" ","-").strip("!") # Take name from database. Article name. Get from database
#2.Then I have to get category which subcategory belongs.
category = "" # Is a main category name which we are looking for.
local_dir = "/home/.../eclipse-workspace/django001/outleys/static/image/" #Set the Base loca directory with images
for ctg in os.listdir(local_dir) : # Get list of categories ,ctg is the top level category
if os.path.isdir(local_dir+ctg) and category=='':
for k in os.listdir(local_dir+ctg): # k is subactegory
if os.path.isdir(local_dir+ctg+"/"+k) and k==subcategory: # Iterate over all subcategories until match
category = ctg # We find top level categoty which correspond subcategory
break
#3.Get first two images
p1 = Image.open(local_dir+category+"/background.png")
p2 = Image.open(local_dir+category+"/"+subcategory+"/1.png")
#3.1.Get random third image
folder_list = os.listdir(local_dir+"_gallery_png/") # Get list of folders
file_dir = local_dir+"_gallery_png/"+folder_list[int((len(folder_list)*random.random()))] # Get random folder
files = os.listdir(file_dir) # Get list of files inside of random folder
# Choose random file image from a random folder
p3 = Image.open(file_dir+"/"+files[int((len(files)*random.random()))]) # Choose random image from gallery
#3.1.1 Increase or decrease p3 if it is too small or too big
if p3.size[0]<200 and p3.size[1]<200:
krat = 300//p3.size[0] # How much times it should be enlarged
w2 = p3.size[0]*krat
h2 = p3.size[1]*krat
p3=p3.resize((w2,h2), Image.Resampling.NEAREST) # Increase image
if p3.size[0]>400 or p3.size[1]>400:
krat1 = p3.size[0]//300 # How much times it should be decreased
krat2 = p3.size[1]//300
krat = max(krat2, krat1)
w3=p3.size[0]//krat
h3=p3.size[1]//krat
p3=p3.resize((w3,h3), Image.Resampling.LANCZOS) # Decreas size
#3.2. Construct composite image
w = int(0.7*p1.size[0]*random.random()-p1.size[0]//2) # Generate random position to insert p2
h = int(0.7*p1.size[1]*random.random()-p1.size[1]//2)
p1.paste(p2, (w,h), mask=p2)
w1 = int((p1.size[0]-p3.size[0])*random.random()) # Generate random position to insert p3
h1 = int((p1.size[1]-p3.size[1])*random.random())
p1.paste(p3, (w1,h1), mask=p3)
#3.3. Apply random effects
imdir = local_dir+category+"/"+subcategory+"/"+nm+".png"
rullet = int(150*random.random()) # Random number from 0 to 150
if 0<= rullet <= 5:
p1 = p1.convert("L") # Made black&white
if 5< rullet <= 10:
p1 = p1.convert("1") # Made crop
if 10< rullet <= 15:
p1 = p1.filter(ImageFilter.CONTOUR) # contour
if 15< rullet <= 20:
p1 = p1.transpose(Image.FLIP_LEFT_RIGHT) # Mirror
if 20< rullet <= 25:
p1 = p1.filter(ImageFilter.GaussianBlur(radius = 2*random.random()+1 )) # blur
if 25< rullet <= 30:
p1 = p1.filter(ImageFilter.SHARPEN) # Sharpen
if 30< rullet <= 35:
p1 = ImageOps.expand(p1, border = 10, fill = 100) # border
if 35< rullet <= 40:
p1 = ImageEnhance.Color(p1).enhance(5.0*random.random()+1.0) # more colors
if 40< rullet <= 45:
p1 = ImageEnhance.Contrast(p1).enhance(5.0*random.random()+1.0) # more contrast
if 45< rullet <= 50:
p1 = ImageEnhance.Brightness(p1).enhance(5.0*random.random()+1.0) # more bright
if 50< rullet <= 55:
p1 = p1.filter(ImageFilter.MaxFilter(size=int(3*random.random()+1))) # fancy
p1.save(imdir,"PNG") #Save generated image
#4. Upload image to the remote server /www/outleys.site/mydiary/how_to/static/image/
c1 = pexpect.spawn('scp '+imdir+' u02/...y/how_to/static/image/'+category+'/') # Write an image to the server
c1.expect('(?i)password:') # Wait for password prompt
c1.sendline('pass') # send password
#5. Write image to database
sql = "UPDATE `how_to_article2` SET `im3` = %s WHERE `how_to_article2`.`id` = '"+str(r1)+"';" # Write into database
w = [ "image/"+category+"/"+nm+".png",]
m.execute(sql,w)
db.commit()
# END OF IMAGE GENERATOR
#2. START SELENIUM BROWSER
o = Options()
b = Firefox(options=o) # start browser
ur = 'deepl'
b.get(ur) # open specific page
time.sleep(5)
ls = { # list of languges for translations. Fill data from database
"English":{"tit":r2[0][0], "det":{"all":r2[0][1],"html":[],"txt":[]}, "txt":{"all":r2[0][2],"html":[],"txt":[]}, "des":r2[0][3], "alt":r2[0][4]},
"Spanish":{"tit":"", "det":{"all":[],"html":[],"txt":[]}, "txt":{"all":[],"html":[],"txt":[]}, "des":"", "alt":""},
"Russian":{"tit":"", "det":{"all":[],"html":[],"txt":[]}, "txt":{"all":[],"html":[],"txt":[]}, "des":"", "alt":""},
}
#3.1 HERE WE SPLIT ENGLISH ARTICLE AND PREPARE HTML FOR TRANSLATION
for j in ["det","txt"] :
flag_start = False
start = 0
end = 0
for i in range(len(ls["English"][j]["all"])):
if (ls["English"][j]["all"][i]=='>' and flag_start): # You reach end of the tag
flag_start = False # You reach end of the tag
end = i # Remember position of a tag ending
ls["English"][j]["html"].append(ls["English"][j]["all"][start:end+1]) # Write tag into list
if (ls["English"][j]["all"][i]=='<' and not flag_start): # Find the string between tags
flag_start = True
start=i
strip = ls["English"][j]["all"][end+1:start].strip()
if strip and strip!=" ": # Remove empty spaces. Check if text it is not empty and not wrapped into strong tag
if ls["English"][j]["html"][-1]!="<strong>":
ls["English"][j]["txt"].append(ls["English"][j]["all"][end+1:start]) # WRite text to dictionary
ls["English"][j]["html"].append('ar['+str(len(ls["English"][j]["txt"])-1)+']') # put array element instead of text
else:
ls["English"][j]["html"].append(ls["English"][j]["all"][end+1:start]) # If wrapped in strong tag do not translate
#4. PREPARE DEEPL AND TRANSLATE
if(len(b.find_elements(By.CSS_SELECTOR, ".lmt__language_container #source_language_label .flex button"))): # Deepl change interface from time to time thats why I made two options
b.find_element(By.CSS_SELECTOR, ".lmt__language_container #source_language_label .flex button").click()
else:
b.find_element(By.CSS_SELECTOR, ".lmt__language_select--source button").click()
b.find_element(By.XPATH,"//span[contains(text(),'English')]").click() # choose english language
inp = b.find_element(By.CSS_SELECTOR, ".lmt__source_textarea") # Get input text area from deepl
out = b.find_element(By.CSS_SELECTOR, "#target-dummydiv") # Get output textare for second time
inp.clear() # Clear input field
for r in ['Russian','Spanish']:
if(len(b.find_elements(By.CSS_SELECTOR, "span #headlessui-popover-button-7"))): # Check if element is found
b.find_element(By.CSS_SELECTOR, "span #headlessui-popover-button-7").click()
else:
b.find_element(By.CSS_SELECTOR, ".lmt__language_select--target").click()
b.find_element(By.XPATH,"//span[contains(text(),'"+r+"')]").click() # choose language in DEEPL
for k in ["tit","des","alt"]:
inp.send_keys(ls["English"][k]) # Translate simple fields
time.sleep(9)
ls[r][k] = out.get_attribute('innerHTML')[:-2]
inp.clear()
#4.1 DEEPL LIMITED BY 5000 SIMBOLS. HERE WE DIVIDE BIG TEXT INTO SMALL PIESES
for l in ["det","txt"]: # Translate big fields
temp = ", [2]. ".join(ls["English"][l]["txt"]) # Translate arrays extracted from HTML then pack them back into html
if len(temp)>3500 : # Split if text is more then 5000 simbols
t=len(temp)//3500 # How much times it is bigger?
step = len(ls["English"][l]["txt"])//(t+3) # Calculate step to split array
for y in range(t+3): # Feed DEEPL with small textes
if y==(t+2):
inp.send_keys(", [2]. ".join(ls["English"][l]["txt"][y*step:]))
time.sleep(12)
ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")
inp.clear()
else:
inp.send_keys(", [2]. ".join(ls["English"][l]["txt"][y*step:(y+1)*step]))
time.sleep(12)
ls[r][l]["txt"]+=out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")
inp.clear()
else:
inp.send_keys(temp) # Translate fields
time.sleep(12)
ls[r][l]["txt"] = out.get_attribute('innerHTML')[:-2].replace(",[2].",", [2]. ").replace("[2] ",", [2]. ").replace("(2). ",", [2]. ").replace("[2], ",", [2]. ").split(", [2]. ")
inp.clear()
for n in range(len(ls["English"][l]["html"])): # iterate over html list
for m2 in range(len(ls[r][l]["txt"])): # Form html Recompile translated html before write it back to database
if ls["English"][l]["html"][n]=="ar["+str(m2)+"]": # iterate over all text element's array
ls[r][l]["html"].append(ls[r][l]["txt"][m2]) # recompile html
break
if ls["English"][l]["html"][n][:3]!="ar[": # skip to not dublicate an array element
ls[r][l]["html"].append(ls["English"][l]["html"][n])
#5. I have a translated html. NOw I have to just write it back to the database
m.execute("SELECT `id` FROM `how_to_article2_translation` ORDER BY `id` DESC LIMIT 1; ") # Get last id from database
rid = m.fetchall()
lastid = rid[0][0] + 1
ls["Russian"]["det"]["all"] = "".join(ls["Russian"]["det"]["html"])
ls["Russian"]["txt"]["all"] = "".join(ls["Russian"]["txt"]["html"])
ls["Spanish"]["det"]["all"] = "".join(ls["Spanish"]["det"]["html"])
ls["Spanish"]["txt"]["all"] = "".join(ls["Spanish"]["txt"]["html"])
langs = {"ru":"Russian", "es":"Spanish"}
for k in langs:
sql = "INSERT INTO `how_to_article2_translation` (`id`, `language_code`, `title`, `details`, `text`, `master_id`, `img_alt`, `meta_description`,`date`) VALUES (%s, %s, %s, %s, %s, %s, %s, %s, %s)" # Write into database
w = [
lastid,
k,
str(ls[ langs[k] ]["tit"]),
ls[ langs[k] ]["det"]["all"],
ls[ langs[k] ]["txt"]["all"],
r1,
str(ls[ langs[k] ]["alt"]),
str(ls[ langs[k] ]["des"]),
str(datetime.now())
]
m.execute(sql,w)
db.commit()
lastid +=1
DJANGO MULTILINGUAL
1. Create locale directory in base path.
2. $ django-admin makemessages -l en // It creates folders and .po files inside of locale folder. Po file will contain translations.
3. $ pip install django-translation-manager
4. In settings.py add in INSTALLED_APPS = [ ... "translation_manager" ... ]
5. $ python manage.py makemigrations -> $ python manage.py migrate
6. Go to admin panel and check. I have no idea how to use this app
4. In template change "Read more" to {{ _("Read more") }}
5. in .PO file add strings:
msgid "Read more"
msgstr "Leer mas"
Doesn't work
6. Try to $ django-admin compilemessages // That's it