(Untitled)
SQLite is great, until you try to throw it into a threaded environment (ie, Apache+mod_python). It worked for awhile, then just started crapping out as multiple interpreter instances started hammering it. So I rewrote the scraper and viewer to use a MySQL backend instead, which was suprisingly easy since THE FORCED INDENTATION OF THE CODE lays out a standard database interface in PEP 249.
mysql> SELECT board_sname, COUNT(*) AS `img_count` -> FROM 4s_images i -> JOIN 4s_posts p ON p.img_id=i.img_id -> JOIN 4s_boards b ON b.board_id=p.board_id -> GROUP BY b.board_id; +-------------+-----------+ | board_sname | img_count | +-------------+-----------+ | w | 2751 | | e | 2435 | +-------------+-----------+ 2 rows in set (0.39 sec)
Those counts used to be over 9000, but I cbf’d converting the SQLite data over to MySQL because I restructured the database a bit etc. So I just deleted the 3 days worth of images scraped, not a big deal.
The web frontend uses Apache+mod_python, as mentioned. I took one look at all those heavy frameworks (Django, Pylons, etc) and lol’d. Way too much flashy wank; I ended up writing my own 70-line “web-framework” which gets the job done. I am using Cheetah for a template engine, because I cbf’d writing yet another template engine. I wrote one in PHP once, it was a couple of lines long and doesn’t afraid of anything.
As slow and bloated as Python may be, I’m fucking glad I don’t have to use PHP, and that I don’t have to use it through across a FCGI interface.
Also, I ordered an Atheros wifi PCI NIC, so the porn server which is now also scraping images will probably be serving as my primary router some time in the near future. Which means I’ll be able to configure the goddamn thing the way I want and have it externally accessible. And I’ll get to dick around with bpf hur hur hur.
Comments are off for this post