• Create a table of numbers from a CSV file
• Control the number of significant figures (significant digits) displayed
• Format numbers with “,” and ” ” like this 123,456.789 123
• Align columns on decimal points–this is best for readability
• Fit the columns automatically based on data and formatting

      483,000. |       471.7 |     0.460 6 | 0.000 449 8
      966,000. |       943.4 |     0.921 3 | 0.000 899 7
    1,449,000. |     1,415.  |     1.382   | 0.001 349
    1,932,000. |     1,887.  |     1.843   | 0.001 799
    2,415,000. |     2,358.  |     2.303   | 0.002 249
    2,898,000. |     2,830.  |     2.764   | 0.002 699
    3,381,000. |     3,302.  |     3.224   | 0.003 149
    3,864,000. |     3,773.  |     3.685   | 0.003 599
    5,796,000. |     5,660.  |     5.528   | 0.005 398
    7,728,000. |     7,547.  |     7.370   | 0.007 197
    9,660,000. |     9,434.  |     9.213   | 0.008 997
   19,320,000. |    18,870.  |    18.43    | 0.017 99
   38,640,000. |    37,730.  |    36.85    | 0.035 99
   57,960,000. |    56,600.  |    55.28    | 0.053 98
   77,280,000. |    75,470.  |    73.70    | 0.071 97
   96,600,000. |    94,340.  |    92.13    | 0.089 97
  144,900,000. |   141,500.  |   138.2     | 0.134 9
  193,200,000. |   188,700.  |   184.3     | 0.179 9
  289,800,000. |   283,000.  |   276.4     | 0.269 9
  386,400,000. |   377,300.  |   368.5     | 0.359 9
  579,600,000. |   566,000.  |   552.8     | 0.539 8
  772,800,000. |   754,700.  |   737.0     | 0.719 7
  966,000,000. |   943,400.  |   921.3     | 0.899 7
1,159,000,000. | 1,132,000.  | 1,106.      | 1.080      

(2) This one seemed obvious, but in case it didn’t quite come out:  The charts shown are of much of the same data–it is the window on the data that may be an explanation for the difference in perceptions.

(3) Something I did not point out, but also probably contributes to a difference in perceptions is that, during the time period represented in my window of investing, the very wealthy got decidedly wealthier.  Clearly, we can be sure that the wealthy were not relying on the stock market to grow their wealth in the way that I was to grow my investments.

Where were the very wealthy investing? I don’t know.

But many of my contemporaries have been encouraged to and have practiced investing their new-style pensions (401Ks) in the stock market, which isn’t likely to go anywhere for at least a few more years.  Among other things, this makes Bush’s proposal to privatize Social Security by “letting” us invest in the public markets seem rather cynical.

What experiences were different for my parent’s generation and mine? Can any of these differences help account for similar values leading to opposing politics? I am not sure this explains much, but below is a pair of charts that illustrate one way in which my parent’s experience and my experience lead us to see things differently. I did the analysis below out of curiosity, but I decided to write it up after I realized a similar analysis could be done with health care costs or income gains, neither of which have evolved in our favor in the last 15 years. (For broader perspective and context on where the S&P is now relative to the past, check out advisorperspectives.)

The scenario I modeled was investing for retirement: Imagine that at 30 years old, both my father and I have resources and get serious about investing. We invest the same amount of money every year until age 65, for my father, this is the period from 1966-2001, for me, the period from 1996-2031.

During the 30 years my father invested, the stock market did this:

He ends up with 7.23 times the money he put into the market. (Simple model: invest and compound gains (losses) on the first day of each year.)

During my investment lifetime, the stock market has done this:

And I am faced with the realisation that I currently have 1.18 times what I have put into the market–I have gained 18% in the last 15 years.

One clear perspective difference is that my father is looking back on the last 20 years of gains as the model for what his and his peer’s individual investments got him. I, on the other had am looking forward to 20 years of investments with uncertainty. My 15% gains are more appropriately compared to my father’s 41% during his first 15 years of investing. Clearly, the bulk of my father’s gains come in the second half of the cycle, the last 20 years. But at the point I am in my life, he was well ahead.

I doubt the important difference is so much how things happened in the past as how things appear to be going forward. What sort of action do I believe will significantly and positively influence the outcome of prosperity for myself and my family and my community? Is this very different from the sorts of actions my father believed were effective at the same point in his life?

From their website:

For the first time ever, see who is browsing on any Web page. Chat, get tips, share links, and meet new people with similar interests. Use the Web like you always do, only now, see what happens when you can see Who is Live!

Well…sort of.

One major difference between Whoislive and me.dium, again, from their website:

Can I also see people who are browsing on other pages of the same website?

No. You can only see who is live with you on one page at a time.

Me.dium had a map of the local neighborhood of sites your friends were surfing. More or less creepy?  You decide.

Anyway, this reminded me that, while the business model of me.dium didn’t get anywhere, there was some interesting technology under the covers. It was just getting off the ground, but there were hints at some possibly fertile directions for research and development.  It also reminded me that I started to write this up a few years ago and still had a draft around.  Most of the ideas are covered under patent applications made by me.dium (and so are public information already, but nearly unreadable) and some are fairly “obvious” (though that seems to be mostly neglected as criteria for patentability (see, e.g. this patent for a wooden dog toy you throw)). The explorations of how the system might work and attempts to create some toy examples to assist with explanations, as well as, the section on proposed new signals is my work. If you are at all interested in the details of how we were trying to create surf-with-friends, take a peak. (Fair warning, this draft hasn’t been edited much, so some of it might be a little cumbersome.)

I find too many skeptics who assume that we know a lot more about what isn’t than it seems possible to know.  Vigilant skepticism adds to the quality of conversation.  As a vehicle for combating dangerous silliness like parents not vaccinating children against smallpox or polio, sorting through the facts carefully and re-evaluating the explanation, insisting on coherent, repeatable experiments makes sense. But knowing what isn’t is much more difficult. Certainty on the right and the left has left us less able to solve difficult problems.

Neuroscientist David Eagleman’s Pop Tech! presentation is worth a watch.

David Eagleman on Possibilianism from PopTech on Vimeo.

(Reposted from KK)

1. Ability to choose what statistical tools to use.  For example, it would be great to have students who could apply significance measures, evaluate probability and its relationship to risk and know when the average is meaningful (Normal Distribution) or nearly useless (Power Laws).
2. Practical skills with data collection and manipulation tools such as text editors and spreadsheets, statistics packages (maybe something like R?) and moving data around. Because bigger data sets are so common and rich, it is not enough to know how to calculate the average of a few numbers with a calculator.
3. Experience and confidence with quantitative analysis as a basis for consistent qualitative reasoning and decision making.

I was delighted to see that the BBC production “The Joy of Stats” with Hans Rosling is now available on YouTube.  This piece does a great job of showing why and what to explore with statistics.  And Hans is entertaining.

Shelf of my notebooks (1/2 dozen more somewhere)

After my last post, I started thinking about all of the pieces of my personal information system.  It turns out that it is surprisingly complicated.  I have tried many social media services, online information services and quite a few desktop personal information products.  I have abandoned a lot of them because they take more time and energy than they are worth.  Some I have abandoned include StumbledUpon.com, digg.com, delicious.com (imported to Evernote), filtrbox.com (now owned by Jive), Jigsaw.com, sparknotes.com,  pluck.com, me.dium ( I was an employee at the time), plaxo…

When I thought about how many I have quit using, I started thinking about how much was left. And of course, I made some notes (yes, in the margin of a notebook).  After a couple of minutes, I decided I needed to start over with two full pages. Below is a scan of the map of my Personal Information System–click on it to embiggen. I use RSS feeds (Google Reader) daily and LinkedIn, but they don’t seem to fit because nothing I use integrates well with LinkedIn and articles in my RSS feed that I want to keep become Evernote notes.  It is still pretty complicated but (I think) I currently value all of the pieces.

Dr. Skippy's personal information system

Recently, I have jumped into using Evernote for nearly everything I want to remember.  I currently use it on the Web, on the Windows desktop at work and at home, on my Android phone and on my iPad.  I have consolidated all my links from Delicious and abandoned that.  I had a library of PDF’s I have downloaded, read and valued on a local hard drive.  I uploaded those to Evernote and deleted the originals.  I dropped all of my writings from work, my dissertation, etc. into Evernote and deleted most of those.  I scanned some receipts and other paper documents into Evernote and shredded those.

I now capture any notes I might need from a whiteboard with my phone and import those.  Evernote is pretty good at recognising hand-written notes and making them searchable. I still like writing with a real pen in paper journals.  But now, when I record something I think I might want to find later, I take a picture with my phone and load into Evernote.

This changed prompted some observations:

Abundance.  Disk space is cheap and loading things into Evernote is available everywhere I am.  It is cheap and easy.  This means that I can upload more than I previously could file, draw, record, etc. and easily find it.  Because of tagging and search, having recorded all of this stuff gets in the way of finding what I want less and less.  In the past, digging through a file cabinet for that one receipt or paper your wrote 10 years ago when 99% of the stuff isn’t what you are looking for was very time-consuming.  Now it doesn’t matter if 99.999% of your notes aren’t relevant to a search–you never see them.  This brings me to another point…

Letting go.  There may be a temptation to manage your Evernote repository to make sure there isn’t redundancy, out of date stuff, or stuff you no longer care about.  Doing this with a file cabinet is probably not too bad an idea if you want to be able to find something fast.  In Evernote, unless you can do this with the original tags or notebooks you created, be sceptical of the value of doing it at all.  Abundance of space and the ease of storing and retrieving notes means “managing” as an activity isn’t worth your time.  Tag as best you can when you put the note into Evernote.  Write a few words that will remind you why this item was important to you and move on.  If you run across a note you want to fix, edit or delete it.  But don’t go looking for them.

The value of forgetting.  One of the best things about paper notebooks is that when you revisit them, you realise how much you forgot that never needs to be relived.  I often highlight or strike out old notes when I am looking for something in an old notebook, noting that something is forgettable (and good riddance!). The value of forgetting is immense for cognition, synthesis and creativity. I have a few details we need to remember very specifically (addresses, passwords, etc.). I have a few creative ideas that I will want to use as a launching pad for a future project.  Occasionally, I may have created a clever poem, story, essay or picture.  But everything else is in the way.

Evernote promises a simple and more effective way of remembering, and, hopefully, of forgetting. I doubt that I will search for Everything I have put into Evernote.  Some things will be forgotten, as they should be.