gwax’s rumblings

When data-mining, the first step is to obtain the data that you would like to mine. I have decided that I would like to try my hand at playing the stock market so it became necessary for me to obtain historical stock market data. To that end, I have devised a method to obtain end of day results for every listing on NYSE, AMEX and NASDAQ since their inception. The data is in the process of being assembled and I expect it to be complete within a few days. Current estimates expect the data to take up approximately 2GB, making it the largest single dataset that I have ever played with. Just having this much data makes my data hoarding senses tingle.

I’ll probably spend a little bit of time putting the data into an easy to understand and use format and then I’ll start looking for patterns. I’m hoping to throw my modeling background and experience at the stock market to see if I can’t beat the system. If I can beat the stock market and make bajillions of dollars (or euro if the dollar collapses) that would be pretty sweet but if I don’t, at the very least, I expect to have fun playing with lots and lots of numbers.

As a second approach, since it turns out to be rather difficult to get this sort of data in the first place, I’m half considering the idea of cleaning it up a bit and then reselling it myself.

I am–and have for quite a while been–a huge proponent of GNU Screen for the many-fold improvements that it provides to terminal and SSH sessions. One of the things about screen is that it runs in two processes, one is a headless process that redirects terminal output of applications to it (server process) and the other is a process which connects to the server process allowing interaction with and the viewing of output from applications in the server process (client process). The advantage to this two process approach is that you can run screen from an SSH connection, disconnect the client process, leave and then later reconnect a new client process to the same server process you started earlier, thus allowing session persistence. It’s a wonderful application with many other features that make my life easier.

In addition to being fantastically useful, screen appears to have been written by someone with a sense of humor as it seems to contain rather a few Easter Eggs. One particular Easter Egg that I recently came across is that if the server process dies while the client process is still connected, you meet the error message:

Suddenly the Dungeon collapses!! - You die…

It’s a cryptic and ominous error message that might put you off a little if you don’t know what’s going on. A quick googling will show that some have accidentally misinterpreted the meaning; thankfully I could guess what it was and check google for confirmation.

When I see a number or sequence, I have a tendency to deconstruct it into other numbers or sequences; I do this with license plates, addresses, receipt stubs and all other manner of things. It is receipt stubs, specifically from restaurants that I’d like to take as my discussion nucleus today. Yesterday, for the first time, I got a steak and cheese from Theo’s Cheesesteak at the Rincon Center nearby and I happened to be order number 64. Upon seeing 64, I internally deconstructed it to 2^6 and spent the subsequent few moments thinking generally about powers of two. Today, having enjoyed yesterday’s cheesesteak, I decided to get another and went, again, to Theo’s where I, once again, happened to be order number 64. Performing the same deconstruction to 2^6 immediately reminded me that I had been given 64 yesterday, allowing me to note that I had received the same number from the same restaurant, two days in a row. Coincidence: yes; synchronicity: I found it meaningful, so yes; evidence for numerology: oh come on, no.

This particular synchronicity got me thinking philosophically about the nature of significance and how we attach meaning to things. We humans are pattern finders, we do it very well and we tend to do it unconsciously. Patterns simplify the world and allow us to abstract things into easier pieces. Since we use patterns to better understand tho world, we tend to do our best to fit things into patterns, even if it’s merely a coincidental pattern, which is fine because patterns do arise spontaneously. People, in my experience, tend to conflate the existence of a pattern with the presence of meaning. In the case of numbers, I believe that the relative ease of constructing arbitrary patterns often leads people to attach meaning to things that are random in nature. I know that I’m guilty of attaching meaning where it isn’t due but, at least, I tend to be conscious of and complicit in my misattributions.

The question now is whether or not I should get a cheesesteak tomorrow, for pseudoscience and all.

Yesterday, I wandered over to Best Buy and finally picked up a copy of Rock Band for my Xbox 360. As I have already establish, Rock Band is awesome, so I’m mighty psyched to have a copy at the place now. I wasn’t properly set up for it earlier but now that we’ve got a projector and a proper sound system, we have an amazing Rock Band setup. Rock Band, of course, is far better as a multiplayer experience than a one person game, so anytime that any of you folks want to stop by and jam with me, you’re more than welcome to do so.

Since the recent demise of my poor Subaru, I have been effectively without a car. Being without a car isn’t such a big deal since I live in the city and can use my longboard for most of my transportation needs but occasionally I have business out in Sunset, down the Peninsula or in the South Bay, which are not the most public transportation accessible locations. Up until now, I’ve been abiding limited mobility, inconvenient mobility and borrowing my roommates car but no longer. Friday, I signed up for an account with Zipcar. The whole process from applying to accepted took about 20 minutes and then, rather than waiting for it to come in the mail, I wandered to the local Zipcar office, a few blocks away, to pick up my Zipcard. All in all, it was about 30 minutes between applying and being able to borrow cars. Yesterday, rather than borrowing my roommates car to run an errand, I picked up a Zipcar; it was a painless and easy process. Based on my current automotive needs, Zipcar really does seem like the best approach and, based on my experiences thus far, it seems like they’ve implemented their service correctly.

Arthur C. Clarke died today in Sri Lanka at the age of 90. I won’t even try to summarize the amazing things that Arthur C. Clarke has contributed to the world, I wouldn’t be able to do him justice; if you care, ask wikipedia. Rest in peace Sir Arthur C. Clarke and thank you for all that you’ve given us.

I’ll leave you with Clarke’s three laws:

1. When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.
2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.
3. Any sufficiently advanced technology is indistinguishable from magic.

Saint Patrick’s Day, 2008 is not on March 17th as usual but, rather, on March 15th. This shift is because March 17th falls during the Christian Holy Week. The Roman Catholic Church moves Saint Patrick’s Day whenever it coincides with the Holy Week, a coincidence which last occurred in 1940 and will next occur in 2160.

In quickly polling around, it seems as though very few people are aware of the date shift. In fact, even my wall calendar has the date wrong. Don’t get me wrong, I like a good drinking holiday as much as the next person but if you’re going to hijack a holiday and use it for drinking, you really should do the hijacking correctly.

Drinking on the wrong day makes the as-yet-unressurrected baby Jesus cry. Don’t do it!

In all seriousness, I don’t know why we need daylight savings time. All it does is make me have to change my clocks twice a year and likely miss at least one scheduled event shortly after the transition. Sleep schedules don’t have much to do with daylight so why are we trying to force to two to coincide. We really need to get rid of these pathetic vestiges of days past and make our time get with the time. I hereby, publicly request that Daylight Savings Time tender its resignation, effective immediately.