When data-mining, the first step is to obtain the data that you would like to mine. I have decided that I would like to try my hand at playing the stock market so it became necessary for me to obtain historical stock market data. To that end, I have devised a method to obtain end of day results for every listing on NYSE, AMEX and NASDAQ since their inception. The data is in the process of being assembled and I expect it to be complete within a few days. Current estimates expect the data to take up approximately 2GB, making it the largest single dataset that I have ever played with. Just having this much data makes my data hoarding senses tingle.
I’ll probably spend a little bit of time putting the data into an easy to understand and use format and then I’ll start looking for patterns. I’m hoping to throw my modeling background and experience at the stock market to see if I can’t beat the system. If I can beat the stock market and make bajillions of dollars (or euro if the dollar collapses) that would be pretty sweet but if I don’t, at the very least, I expect to have fun playing with lots and lots of numbers.
As a second approach, since it turns out to be rather difficult to get this sort of data in the first place, I’m half considering the idea of cleaning it up a bit and then reselling it myself.
So your hope is that past prices predict future prices (instead of earnings, volume, etc.)? You might save yourself some time (but not money) by looking here: http://www.crsp.com/products/index.html
I don’t hope that past prices predict future prices, I hope that I can use past prices to help correlate prices to something else in a manner that can predict future prices. At this point, there’s no need for me to pay for someone else’s dataset, I’ve got mine built; it turns out to be only 700MB too.
Nice! Well remember us little people when you make it big.
Done; when I make it big, you can be my ridiculously overpaid, personal attorney.