YQL (Yahoo Query Language) allows you to do amazing things, and has many feeds for a developer to play with. In Peak Stocks, we use it for Option Chains and it works beautifully. An additional feature that we'd thought of was expanding social media feeds. We've gotten a lot of requests for twitter and thought, why not the yahoo message boards too? This is where YQL comes into play.
YQL has community tables, which aren't actual SQL tables. These "tables" are just XML files that scrape pages or using existing feeds and return a response object that is formatted to your liking. Within YQL console, these XML files are passed off a tables making it seem like your actually accessing a database. Anyway, the community tables allow anyone to create their own custom table, be it through the methods mentioned earlier. The finance message boards dont have an open API so i had to use the page scraping method. Having not a lick of experience in doing any of this, it took me a couple of hours to get it going, with countless attempts of trial and error (view the commit history, you'll see). Once i finally got some data to work with, it got easier and easier. The final XML file can be found on my GitHub here.
<tr> elements within
<tbody> and appends the rows to a
Within the YQL console, you can have two YQL queries to choose from:
As it stands, the file only parses through the a message board for a particular stock, getting all the threads within it. While it does provide you the the thread link, there is no way to get a feed for that individual thread.....yet.
The next step is to create another file that handles an individual thread. This shouldn't be difficult since I have experience from creating this one.