Innovation Session: Gathering Data on Myself

If you can measure it, you can improve it.

With that in mind I plan on measuring more things about myself in 2013 as I continue optimizing areas of my life. I currently do a quarterly review of how I spend my time and an annual 50K ft look – but I think there can be benefit to a daily granularity.

In general, I am a strong advocate of data based decision making.  Often this does not require complicated computation or advanced statistical techniques – it simply requires having the right question in mind and some relevant data to look at over time. Trends will show themselves. This is especially true when the area in question is not already highly optimized.

As anyone that works with data will tell you, though, getting clean relevant data is often what most of the effort goes into on a project.

There are a lot of really cool things going on with self measurement right now – it is sometimes called quantified self . The focus on it is making it easier to find tools that seamlessly track aspects of your life. Some that come to mind are:

Tonight, I’m going to put some energy into creating a system that lets me easily gather data about myself. Hopefully I’ll be able to iterate through a few versions before the new year starts.

Requirements:

  1. Daily – I have done some sampling before but want something more consistant and granular
  2. Easy – it is hardly optimizing if the process costs more time & energy than the results
  3. Diverse – it should include structured & unstructured data, qualitative & quantitative about a wide range of topics
  4. Actionable – I want to stick to those things which I can impact
  5. Future-proof – there are questions I will want to ask later that I don’t know yet

First off I need to select a system to use. I was thinking about daytum.com but am hesitant because of the fact that the founders now work at Facebook. I don’t want to rely to heavily on a system that might become unsupported or shut down shortly.

After a bit of looking, it seems like most of the forward thinkers are using custom tools. I’m going to opt for Google Docs & use the form tool which gives me both ease of entry and organized storage. I’m then going to set up a reoccurring calendar event that will pop up a link to the form every day at 8:00 pm on my phone. That should take care of requirements #1 & #2.

questions-event

Now to come up with questions – I am immediately thinking of three categories I want to measure.

  1. Things I want to quantify so I can later try to correlate them
  2. Things I want to codify so I can later look back on them
  3. Things I want to ask so they will stay on my mind

1. Quantifiable info

The first few items will likely include items I just want to get raw number for and which are fairly factual by nature. I’m guessing that most of these will be interesting but more so when used to shed light on other items.

  • How many hours did you sleep?
  • How many hours did you work?
  • How many hours of deep concentration did you have?
  • How many hours did you read for?
  • How many meaningful conversations did you have?
  • Did you exercise?

Then there are items I’m going to put numbers to that are a little more abstract. These ones are starting to drift away from requirement #4 as they are less actionable – but I suspect they will indicate problems with the items above.

Health in each of the following:

  • Physical
  • Mental
  • Social
  • Emotional
  • Spiritual
  • Vocational

2. Situational items

These questions, I’m hoping will provide deeper context for the items listed above. Mini-snapshots of my life.

  • What is the thing you are most excited about today?
  • What is the thing you are most worried about today?
  • What was one thing you accomplished today?
  • What was one way you failed today?

3. Items to keep on my mind

There are some things, that by keeping them on the top of your mind will affect your actions. I’m hoping these questions have that result.

  • What new thing did you do today?
  • How did you serve others today?
  • What did your actions today say about your character?
  • How were you a loving husband today?
  • What time did you spend today that you now wish you hadn’t?

 questions-screen-shot

 Here is the final product on my phone – just in time to wrap the session. It is a bit long at 24 questions, but I like a lot of it. I’m going to consider the next 10 days a test run and make some tweaks before settling on something to consistently use for 2013.


Edit: After a few days of working with the longer list I realized that I needed to drop a few questions. I was able to get it down to 13 without losing too much by dropping the first set of quantifiable questions. I’m not too concerned with time tracking here and that I can get fairly accurate data on that with a sampling method I use elsewhere.

I also realized that for the health questions, 1-4 was not a granular enough scale so I’ve changed that to 1-10.

Innovation Session: Pothole-No-More

I live in San Francisco and bike everywhere. It is the fastest and easiest way to get around the city. There are a few things that make the life of bikers in the city difficult though; hills, rain, cars & road conditions. Tonight I’m going to invest some energy into a creative solution to one of those – potholes.

Problem

There seem to be two steps in the process of a pothole getting fixed. First, the pothole has to be identified by the necessary party, second, a crew needs to show up with materials to fix it.

Part One: Identifying potholes

San Francisco is actually pretty advanced in how it deals with potholes. The public works department lets you call into 311 where you can report a hole and they will add it to a queue to get fixed. Unfortunately like most things the government does, speed is not their strength. On the other side of the state in San Diego, one citizen got so frustrated after repeatedly calling about a pothole that he eventually fixed it himself.

Part Two: Fixing potholes

Apparently pothole fixing presents a few limitations. The first problem – like that of any government funded operation is budget constraints. The second is that some potholes get fixed hastily and then break again (sounds like software). Apparently the city would need to spend upwards of $40 Million per year to keep the roads in even decent condition. We currently don’t have the money and raising taxes doesn’t always go over well.

An individual pothole can cost between $20-40 to be properly fixed which isn’t a ton on its own, but apparently there are over one million potholes in San Francisco. Woah! The causes include traffic, water & damaged sewers – all of which the roads are repeatedly subjected to – so new potholes pop up every year.

Solution

Combining the citizen servant-hood of the aforementioned San Diegan, Primo Vannicelli, with mobile infrastructure and a micro-payment bidding system I’ve come up with ‘Pothole-No-More’.

Potholes are something that affect people personally and specifically. While a resident might have a hard time agreeing to a tax increase for a general repairs project – they would feel closer tied to a specific pothole that they encounter on their everyday commute. You know – that pothole you always have to dodge on your way home.

The system lets you use your mobile phone to capture the location and a picture of the pothole which is then submitted for processing.

Google maps on an iPhone?Instagram filters recommended

We then categorize (read: mechanical turk, image recognition &/or task rabbit) each pothole to come up with a project completion cost. Once the pothole is processed it goes live on the site where anyone can donate money towards it getting fixed.

No that is not VLC player - though VLC does cost $0 - bless them.

 Once the full amount has been donated the funds are routed to the city to be repaired by the appropriate team. ALTERNATE ENDING: we take the money, send a crew to get supplies and fix the pothole ourselves using the cover of night and ninja outfits to avoid arrest!

Thoughts

I can think of a number of ideas to iterate on as this idea gets traction.

1) Potholes get cheaper to fix the more you fix – economies of scale and all. If this gets popular enough we can use our friend math to optimize the best route for repairs and batch the fixes in an appropriate manner.

2) With this system potholes in areas of high traffic will likely get fixed sooner which is good. There might also be a bias however towards potholes in more affluent areas being fixed – as people living there will have more discretionary income to donate. We could explore charging a slight premium on potholes and using the excess money towards fixing potholes in other parts of the city. We might also need to explore using some of those funds for app maintenance.

3) Some cities won’t be on board with a project like this. Tough. If you’re not getting arrested or sued, you’re probably not disrupting hard enough. Working with local governments will certainly be important though – most will likely be fine having citizens provide extra money towards public projects – but the implementation will look different in every city. Having local champions help get the app introduced to the city will have to be a part of the growth hacking plan.

4) Army of robotic pothole fixers. Automation for the win!

5) Map-my-ride feature. Use the app to automatically map your frequent commutes and dedicate a set amount per month towards fixing any potholes on that commute line. Don’t just fix the problem – get proactive and make sure new potholes are addressed quickly before they get out of hand.

Final Note

I am not currently planning on doing this project – tonight was more of a thought exercise. If it sounds interesting to you though – you have my blessing to follow through with it.

If you want to pitch in but are looking for other people to help – contact me using the link above and I’ll connect you to whoever else replies. Safe biking!

 

Innovation Session: Best NFL Team

I am an Atlanta Falcons fan. This is by no obvious reason, I have lived in California for most of my life. A few years ago, though, I had the pleasure of working with Tony Gonzalez, aka. #88 & Tight End of the Falcons. Since then I have been rooting pretty hard for them. I want him to win a Super Bowl ring – he is a good guy and one of the best players the sport has seen. No one deserves it more.

The Falcons are having a good year, they are 11-1 and tied for first place. But, unfortunately they are getting less credit for it than they deserve. Despite their record, most commentators are less than optimistic about their playoff hopes. I hear a lot of that around the office as well. We can’t get no respect.

Tonight I will attempt to use a few of my data skills to visualize the NFL as a network graph to see it lends any strength to my opinion. I am hoping that by setting up the teams as nodes and games played as edges we’ll be able to see that the Falcons are clearly the best team in the NFL. If not – I guess I’ll have to be an honest data guy and trust what I find.

Challenge:

  • Get clean data on the NFL 2012 season
  • Vizualize it as a network graph
  • Tweak the weights and layout to add context
  • BONUS: getting the Gephi image API to work so the nodes are the team logos/helmets

Step 1: Get clean data on the NFL 2012 season

Getting clean data is always a bother – after searching for a short bit my best bet seems like ESPN. This page looks fairly clean right off the bat and I trust it’s accuracy (ESPN NFL scores & schedule).

A bit of cleaning and I have a nice edges csv – ideal for importing.

Step 2: Vizualize it as a network graph

The visualization tool I’ll be using today is Gephi, an open source tool that makes beautiful displays. (Trivia: LinkedIn hired one of the lead devs behind Gephi and he now gets to do cool things with them)

I ran into a bit of an issue with the fact that teams in the same division play each other twice. I counted each game as a directed edge with the weight being the diff in game scores. Unfortunately Gephi doesn’t support multiple edges between two nodes. Bummer. I think we will be ok though – if two teams play each other more than once, the two diffs will just combine into a single edge. A win & loss of equal amounts will cancel each other out.

Now I’ve loaded the data into Gephi – here is what I’m seeing on first pass.

Viewing the NFL as a network graph

Step 3: Tweak the weights and layout to add context

My goal is to find the team that is best by seeing who has won the most games by the most points against the toughest teams. So in theory, more wins is good, but wins against teams with low records isn’t as good as wins against teams with large records. This seems like a perfect application for something like Google’s PageRank – I’m not sure if I’ll get that far tonight though.

The first thing I’ll do is apply a weighted in-degree to the node size. This looks at the amount of points the team beat it’s opponents by. Basically a seasonal point differential.

 

This highlights the teams that have beaten their opponents by the most points. Bigger team names = bigger victory margins. Atlanta isn’t doing too well here – they have been winning by small margins all season. The New England Patriots on the other hand had a few weeks where they made other teams look like youth teams as they ran the score up. That helped my fantasy team a ton on Thanksgiving.

Next up I am going to combine this with the total win-loss record. Using color to distinguish the teams with the most wins.

 

Now things are a bit more interesting. Teams in red have the most wins while blue teams have the fewest. New England might have a large point differential but they are only 9-3 right now, so they are in light purple.

What I want to do now is look at the opponents the team played. Winning is important, but winning against good teams is more important. We have already established a color pattern with blue teams being those with less wins (read: easy teams) and red teams those with more. Lets apply this to all of the edges to see what our strength of schedule looks like.

 

This one is a bit revealing. We can see some bright red lines between a few teams – Houston & Green Bay, San Francisco & The Giants, Denver & New England. These are the games you probably made sure to watch – the clashes of titans. My Falcons aren’t in many of those – the  games against the Saints were great – but New Orleans is a purple team at best this season.

I want to do one more thing – apply a layout algorithm to this to place teams closer to opponents they played more often.

Of the power house teams in red, a few are much closer together, particularly those in the top left of the graph. Meanwhile Atlanta is very far to the bottom right – basically they were shielded from most of the good teams in the NFL and only played blue to purple teams.

Conclusion: Atlanta has done well this year, but their wins have been small and their opponents haven’t been the toughest. They have a shot at the title, but it isn’t an easy road ahead.

I’m running out of time for tonight’s innovation session – but I want to change one last thing. Just for style I’m switching the color scheme to ATL colors; black & red. There are still a few weeks left in the regular season, including our upcoming games with the Giants – but I think like myself most Falcons fans are most concerned with our upcoming playoff games. We can use this to remember which teams it will feel the best when we beat on our road to victory.