Data Syndrome
Meanderings of Russell Jurney @rjurney
-
Will my apartment on the ocean cliff next to the San Andreas fault (where it enters the ocean) fall into the water and kill me when the Big
-
Help me become the Greatest Software Engineer in the World
Many of you have read my code, and you are aware that it can best be described as, ‘eventually good.’ It is with great honor then that I announce my bid to be one of the greatest software engineers in the world.
http://www.quora.com/Who-are-some-of-the-best-software-engineers-alive#answers
Scroll down, vote up.
I had a shit year, but things are looking up. I got a brand new pacemaker and bionic powers, and so I encourage you to help promote this truthiness. In return your support, you will all receive boons in the coming successes I am sure to enjoy. Perhaps a case study in my book. In addition, by upvoting me, you will be earning a Jurney number of 1, that Erdos shit having gotten stale.
Thanks for your support,
The Greatest Software Engineer in the World and Medtronic Bionic Man,
Russell Jurney -
The Data Lifecycle Part One: Avroizing the Enron Emails
This is part one of a series of blog posts covering new developments in the Hadoop pantheon that enable productivity throughout the lifecycle of big data. In a series of posts, we’re going to explore the full lifecycle of data in the enterprise: Introducing new data sources to the Hadoop filesystem via ETL, processing this data in data-flows with Pig and Python to expose new and interesting properties, consuming this data as an analyst in HIVE, and discovering and accessing these resources as analysts and application developers using HCatalog and Templeton.
-
Dumb Like Me
What does it feel like to be stupid?
I recently found out I had Bradycardia for a year. Oximetry indicated frequent hypoxia. My heart paused for 5, 6 then 7 seconds during one short window. My brain was starved for oxygen for a year and every day I was exhausted by these ‘near death’ experiences while I slept.
Once doctors put a pacemaker in, I noticed a profound improvement. So did my family. I am myself again. :)
Becoming dumb was like forgetting your lines as an actor. Lines you’ve worked like hell to remember and then stumble over on stage, in front of a live audience. The words don’t come.
Being dumb came with a shocking loss of privilege. This was humbling. Smart people, like the very attractive, get special treatment they do not know they are getting. A slower mind and the normal treatment that comes with it makes the world look cruel, unfair and cold.
It was a shocking blow, a loss of identity. I was the guy that came up with the answers. Not anymore.
It was most frustrating to be tricked, manipulated and used. I could sense what was happening but could not sidestep the tricks. I could see myself being manipulated but was helpless to defend myself as my intellect became overwhelmed.
Worst of all was feeling like people thought I was some kind of fraud, in that I could no longer reproduce or even accurately describe feats from the past. To be challenged in your accomplishments adds insults to injury as you struggle with your limitations.
There was some upside. I learned to cry freely. I can now openly weep at an emotional part of a film and feel it more than I would have before. I learned to control my temper. I learned patience. And I learned what remains if you take away all that makes me unique and special - and I learned to be okay with that person. I’d rather be smart, but I can always find something to do, enjoy my family and spend my spare time painting.
I decided to focus more on writing and teaching than doing, which I found to be easier than the applied work I was doing before. Now that I can finish it, I got a book out of it.
-
Isomorphismes: I feel vindicated in several ways by the Netflix Engineering team’s...
I feel vindicated in several ways by the Netflix Engineering team’s recent blog post explaining what they did with the results of the Netflix Prize. What they wrote confirms what I’ve been saying about recommendations as well as my experience designing recommendation engines for clients, in…
-
Apache Pig 0.10 is out!
Hortonworks has described the new features here. -
How many hours/week of specific tasks should be assigned to members of data science teams?
Data Science: How many hours/week of specific tasks should be assigned to members of data science teams? 1 answer on Quora
How many hours/week of specific tasks should be assigned to members of data science teams? -
Kickstarter - Light Table - A Better IDE
I call on everyone to support the Light Table Kickstarter project. The goal is $200,000 and it is not yet met. Light Table is a huge step forward in Integrated Development Environments.
Light Table is based on a very simple idea: we need a real work surface to code on, not just an editor and a project explorer. We need to be able to move things around, keep clutter down, and bring information to the foreground in the places we need it most.
Please give. -
Where does research end and development begin?
Data Science: Where does research end and development begin? Write an answer on Quora
Where does research end and development begin? -
Apache Pig is now a LinkedIn Skill
Apache Pig is now a LinkedIn Skill: http://www.linkedin.com/skills/skill/Pig. Be sure and add it to your profile, and to network with other Piggies!
-
I predicted the iPhone 4S in August, 2007
In August, 2007 commented on a post on Lance Weatherby’s Blog.
I spoke thus:
And, before there was a dev kit:The iPhone is a toy and a cell phone compared to what it will become: The next dominant computing platform. Finally someone capable of wrapping advances in GPS location, bluetooth personal area networks, wireless communications, artificial intelligence, digital speech/voice recognition and web services has released a computing platform to take advantage of them all.
You are walking down a city street. You have your iPhone 2.0 in your pocket, and a tiny Apple bluetooth headset on. You want coffee and wifi. Your personal digital assistant’s name is Cleo. “Cleo, find me coffee with wifi access.” Cleo queries Google Maps for you and then tells you, in your bluetooth headset, the names of 5 of the nearest coffee shops with wifi, starting with the nearest. You pick the first one. “Cleo, directions to Javaology.” Cleo gives you directions, and tells you when you take a wrong turn.
On the way, you find you love the neighborhood you are in. “Cleo, find me an apartment in this area.” Cleo does. She starts rattling off listings. Pick one and she’ll give you directions.
You glance in a store window and see a new book called ‘Gary Potter Volume 11,’ and you have 1-10. “Cleo, order Kama Sutra Volume 11 from Amazon to my home.” Feeling scared? “Cleo, what is the crime rate in this area?” Cleo tells you. Cleo will announce the names of other people you meet and will exchange cards between your iPhones.
We’ll all start doing things like this in the next few years. The step after that will be glasses which will coordinate with GPS and cell location to overlay 3D imagery on the street, in malls, etc. A la Spook Country, a highly recommended read. I know that we’ll be doing this, because if nobody else does… I’ll start a company to build this. But I suspect apple will do it for us.Heh, fortunately there is a complete dev toolkit for the iPhone and it would require some platform integration. -
Y Combinator Amex
Today Y Combinator announced that they’re funding teams without an idea. This is another step in pushing seed funding towards its ultimate destination: a credit card application. Web form, attach video, check social networks, get back to you soon, ship a card!
-
Dedicated Platform Engineers OR Elastic MapReduce::Pig.execute( S3/Avro -> MongoHQ)
It took a month of platform work, but I just used Amazon’s Elastic MapReduce to read data from Amazon S3 in Avro format, process it in pig on a 3 node Hadoop cluster and store it in MongoHQ.
I did not anticipate such difficulties in doing this, but as I labored to make it work… and as friends of mine labored with similar problems with Hadoop I/O as I did, it reminded me:
Every data science team needs an embedded platform engineer that spends a good deal of her time responding to issues data scientists and developers have. And resolving them.
-
Alchemy
Alchemy was the quest of transmuting lead into gold. Data Science is the journey of minting data into money.
Alchemy was immature chemistry. Data Science is immature…?
-
AvroStorage on S3 from EMR
This makes me very, very happy. I got Pig AvroStorage working on S3 from EMR.
grunt> REGISTER /me/pig/build/ivy/lib/Pig/avro-1.5.3.jar grunt> REGISTER /me/pig/build/ivy/lib/Pig/json-simple-1.1.jar grunt> REGISTER /me/pig/contrib/piggybank/java/piggybank.jar grunt> REGISTER /me/pig/build/ivy/lib/Pig/jackson-core-asl-1.7.3.jar grunt> REGISTER /me/pig/build/ivy/lib/Pig/jackson-mapper-asl-1.7.3.jar grunt> grunt> DEFINE AvroStorage org.apache.pig.piggybank.storage.avro.AvroStorage(); grunt> a = LOAD 's3n://agile.data/again_inbox' USING AvroStorage(); 2012-02-29 09:49:13,022 [main] INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening 's3n://agile.data/again_inbox/part-4-0.avro' for reading grunt> describe a 2012-02-29 09:49:19,951 [main] INFO org.apache.hadoop.fs.s3native.NativeS3FileSystem - Opening 's3n://agile.data/again_inbox/part-4-0.avro' for reading a: {message_id: chararray,from: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},cc: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},bcc: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},reply_to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},in_reply_to: {PIG_WRAPPER: (ARRAY_ELEM: chararray)},subject: chararray,body: chararray,date: chararray}
Page 1 of 4
