I need some help with a programming task: use Python to grab a web page, and email it so the email can be read normally

This is surprisingly much more difficult than the simple description in the subject. That’s because many mail readers don’t like to render CSS; for instance, if you’re sending to Gmail you need to take apart the CSS and decorate every individual element on the page with the styling, as Gmail ignores CSS.

And, that’s not the only issue. I’m trying to email myself a Steemit page, and I’m reading mail via Thunderbird (a Mozilla project, like Firefox but for mail reading). It puts an “X” in a box at the top, then shows a bunch of menu entries, one on each line, then a bunch of whitespace, and then it does show the entire post including the comments, but the formatting still looks wrong, with lots more whitespace than in the web browser.

I’d rather not render it as an image, and take screencaps, as I want there to be searchable text in the emails.

When I load a Steemit page in a browser, I see some things “paint” after it loads — for instance, load my Replies page, and the upvote arrows are all initially white (meaning hasn’t been upvoted), and a few seconds later, the ones that I’ve upvoted turn blue. So that’s an action that is happening deeper down, like with AJAX or Javascript.

I’ve found some Javascript render packages (different form of rendering than the above -- the Javascript ones are supposed to load the page elements that load after the HTML is loaded, so I'd have the "full page"), but I can’t get any of them working.

I’m using Python 3 in Ubuntu, and am using the Python module “requests” to get the page. I think the module “premailer” should be helpful here, but I haven’t managed to make it work yet. Right now, it gives a huge amount of warnings and errors, taking about five minutes before it then throws an exception — which Komodo (an IDE by ActiveState) says it can’t find the source code for, so it stops there.

I’ve found a few other resources, like the module “scrapy”: https://doc.scrapy.org/en/1.5/topics/email.html — however, it isn’t clear whether their email code will parse HTML/CSS correctly, haven’t tried it yet.

Tried a bunch of other things and am rather frustrated right now, so asking for help.

This one, I couldn’t install the prerequisites: https://impythonist.wordpress.com/2015/01/06/ultimate-guide-for-scraping-javascript-rendered-web-pages/ — I did what it stated, “sudo apt-get install python-qt4”, which appeared to succeed; however, when it gets to the second line in the script, it raises an exception:

Traceback (most recent call last):
File “/home/steemitdev/get_page_for_email.py”, line 2, in
from PyQt4.QtGui import *
ModuleNotFoundError: No module named ‘PyQt4’

I’ve tried installing that in various ways, and nothing has worked. This page https://stackoverflow.com/questions/40277382/pyqt4-installed-but-importerror-no-module-named-pyqt4-in-ubuntu-16-04 seems to indicate a way forward, but the “pyvenv” command it says to run gives an error:

$ pyvenv –system-site-packages .
WARNING: the pyenv script is deprecated in favour of python -m venv
Error: Command ‘[‘/home/steemitdev/projects/example.com/steemit/bin/python’, ‘-Im’, ‘ensurepip’, ‘–upgrade’, ‘–default-pip’]’ returned non-zero exit status 1.

Tried Spynner, from https://pypi.python.org/pypi/spynner — but apparently it’s got a Python 2-style “print” statement, as it complains like so:

File “/tmp/easy_install-tw3rrh31/autopy-0.51/setup.py”, line 50
print ‘Updating init.py’
^
SyntaxError: Missing parentheses in call to ‘print’.

I know that one difference between Python 2 and Python 3 was that print was made into a proper function, so in Python 3 it must be called with parentheses; thus, the above line should really be:

print(‘Updating init.py’)

But I can’t change that file because it’s expanded during the operation, and appears to be part of Python module “autopy”. Let’s try installing that separately, via “sudo apt install autopy”: yep, that errors out as well, with the same print issue. So, there’s a bug in autopy, and I can’t move forward with this approach, at least not until they’ve fixed that bug. Wonderful.

Anyway I’m starting to curse and I don’t want to give energy to the dark forces, so I’m writing this post and taking a nap.

Thanks for any assistance you can provide!

---