The random rantings of a concerned programmer.

May 12

So, I lied.

Category: Random

4scrape is alive again. Completely rewritten from scratch in Go, featuring even more stupid bullshit on the frontend. And it’s using a real fulltext indexer this time around. In it for the long haul this time around.

Tagged with: ,
15 comments

15 Comments so far

  1. Eksopl May 12th, 2012 3:12 pm

    Oh, that was you, then! I posted on warosu saying that I liked it, but I got accused of being you. :<

    Hope you're ready for 4chan's HTML changes that are supposedly going to be rolled out today, hah.

  2. Taro May 12th, 2012 4:26 pm

    I’m not ready for the changes, actually. And the scraper is running in the same process as the HTTP server (I’m going to implement real-time scrape viewing and shit with websockets) so we’ll see what happens!

    PROBABLY SIGSEGV.

  3. Eksopl May 12th, 2012 5:15 pm

    WELP.

    Also, sorry for beating you to the punch and totally ninja’ing the name Asagi!

  4. Anonymous May 13th, 2012 10:39 am

    what the fuck. where did this come from.

  5. Anonymous May 13th, 2012 12:03 pm

    Also, do you write any Haskell nowadays?

  6. Taro May 13th, 2012 3:56 pm

    > where did this come from

    Pulled it out of my ass a couple weekends ago. Kinda missed having it around.

    > Haskell

    Not really, no. Haskell is fun and all, but it forces you to spend a lot of time up-front coming up with a good design for the program. I found myself fighting with the compiler more often than not. My language du jour is Go these days.

    Shrug.

  7. Anonymous May 14th, 2012 12:16 am

    Ok. I was reading some old posts about 4scrape/haskell and I was thinking about learning it.

    Will there be a larger write-up of the setup? How are you hosting it, what are the plan for the future, any pitfalls when using Go, and so on?

  8. Taro May 14th, 2012 2:24 am

    Haskell isn’t bad; I’d say it’s a good language to learn, but not to necessarily use in production environments.

    I’ll probably do a write-up at some point; it’s hosted on a dedicated machine leased from 100TB.com. With respect to Go, the only nasty snags I’ve encountered have been semi-shoddy library support:

    • Couple of bugs in the Sphinx bindings I’m using
    • image/jpeg didn’t support progressive JPEGs, so I had to write custom bindings to libjpeg
    • database.sql‘s Rows.Scan leaves a bit to be desired — kinda wish it let me pull out individual rows piecemeal instead of building up a big []interface{} (which consists of a lot of things I don’t care about).

    Other than that, everything has been pretty straightforward. The code’s a fucking nightmare right now (just read main.js — the server is roughly as bad), but it’s mostly just a matter of refactoring. I’m more interested in adding back features than making the code look pretty.

  9. Anonymous May 14th, 2012 2:45 am

    Ok cool.

    I’ve noticed some problem with the filter menu, mainly with dropdowns that don’t work (can’t select stuff || after selection it won’t register clicks) but other than that it’s very nice.

  10. Taro May 14th, 2012 10:50 am

    The filters aren’t applied retroactively — they’re used whenever an API call is made, which means you (currently) have to either scroll past what’s on the screen or click the ‘Home’ button (or do another search). I’m going to add an extra button that wipes what you’re looking and and re-performs the API call sometime this week.

  11. Anonymous May 18th, 2012 1:52 am

    Oh btw, what are your thoughts on the raspberry pi?

  12. Taro May 18th, 2012 9:49 am

    They can’t take my money fast enough. I want to strap an IMU to it, write a shader-based integrator and build a quadcopter frame around the things. I have pretty weak plans to do the same to my pandaboard, but the design may be prohibitively large.

  13. Anonymous June 17th, 2012 12:40 pm

    I dig the older style more, checkout nik.bot.nu the way he adapted the original code for 4scrape just looks simply amazing.

  14. HMA MEME FAN January 1st, 2013 12:24 pm

    Nice job man.

  15. Sussman January 13th, 2013 2:29 am

    4scrape is amazing, you rock.

    I suggest a feature to filter out /mlp/ content.

Leave a comment