(2) This one seemed obvious, but in case it didn’t quite come out:  The charts shown are of much of the same data–it is the window on the data that may be an explanation for the difference in perceptions.

(3) Something I did not point out, but also probably contributes to a difference in perceptions is that, during the time period represented in my window of investing, the very wealthy got decidedly wealthier.  Clearly, we can be sure that the wealthy were not relying on the stock market to grow their wealth in the way that I was to grow my investments.

Where were the very wealthy investing? I don’t know.

But many of my contemporaries have been encouraged to and have practiced investing their new-style pensions (401Ks) in the stock market, which isn’t likely to go anywhere for at least a few more years.  Among other things, this makes Bush’s proposal to privatize Social Security by “letting” us invest in the public markets seem rather cynical.

What experiences were different for my parent’s generation and mine? Can any of these differences help account for similar values leading to opposing politics? I am not sure this explains much, but below is a pair of charts that illustrate one way in which my parent’s experience and my experience lead us to see things differently. I did the analysis below out of curiosity, but I decided to write it up after I realized a similar analysis could be done with health care costs or income gains, neither of which have evolved in our favor in the last 15 years. (For broader perspective and context on where the S&P is now relative to the past, check out advisorperspectives.)

The scenario I modeled was investing for retirement: Imagine that at 30 years old, both my father and I have resources and get serious about investing. We invest the same amount of money every year until age 65, for my father, this is the period from 1966-2001, for me, the period from 1996-2031.

During the 30 years my father invested, the stock market did this:

He ends up with 7.23 times the money he put into the market. (Simple model: invest and compound gains (losses) on the first day of each year.)

During my investment lifetime, the stock market has done this:

And I am faced with the realisation that I currently have 1.18 times what I have put into the market–I have gained 18% in the last 15 years.

One clear perspective difference is that my father is looking back on the last 20 years of gains as the model for what his and his peer’s individual investments got him. I, on the other had am looking forward to 20 years of investments with uncertainty. My 15% gains are more appropriately compared to my father’s 41% during his first 15 years of investing. Clearly, the bulk of my father’s gains come in the second half of the cycle, the last 20 years. But at the point I am in my life, he was well ahead.

I doubt the important difference is so much how things happened in the past as how things appear to be going forward. What sort of action do I believe will significantly and positively influence the outcome of prosperity for myself and my family and my community? Is this very different from the sorts of actions my father believed were effective at the same point in his life?

I find too many skeptics who assume that we know a lot more about what isn’t than it seems possible to know.  Vigilant skepticism adds to the quality of conversation.  As a vehicle for combating dangerous silliness like parents not vaccinating children against smallpox or polio, sorting through the facts carefully and re-evaluating the explanation, insisting on coherent, repeatable experiments makes sense. But knowing what isn’t is much more difficult. Certainty on the right and the left has left us less able to solve difficult problems.

Neuroscientist David Eagleman’s Pop Tech! presentation is worth a watch.

David Eagleman on Possibilianism from PopTech on Vimeo.

(Reposted from KK)

Shelf of my notebooks (1/2 dozen more somewhere)

After my last post, I started thinking about all of the pieces of my personal information system.  It turns out that it is surprisingly complicated.  I have tried many social media services, online information services and quite a few desktop personal information products.  I have abandoned a lot of them because they take more time and energy than they are worth.  Some I have abandoned include StumbledUpon.com, digg.com, delicious.com (imported to Evernote), filtrbox.com (now owned by Jive), Jigsaw.com, sparknotes.com,  pluck.com, me.dium ( I was an employee at the time), plaxo…

When I thought about how many I have quit using, I started thinking about how much was left. And of course, I made some notes (yes, in the margin of a notebook).  After a couple of minutes, I decided I needed to start over with two full pages. Below is a scan of the map of my Personal Information System–click on it to embiggen. I use RSS feeds (Google Reader) daily and LinkedIn, but they don’t seem to fit because nothing I use integrates well with LinkedIn and articles in my RSS feed that I want to keep become Evernote notes.  It is still pretty complicated but (I think) I currently value all of the pieces.

Dr. Skippy's personal information system

Recently, I have jumped into using Evernote for nearly everything I want to remember.  I currently use it on the Web, on the Windows desktop at work and at home, on my Android phone and on my iPad.  I have consolidated all my links from Delicious and abandoned that.  I had a library of PDF’s I have downloaded, read and valued on a local hard drive.  I uploaded those to Evernote and deleted the originals.  I dropped all of my writings from work, my dissertation, etc. into Evernote and deleted most of those.  I scanned some receipts and other paper documents into Evernote and shredded those.

I now capture any notes I might need from a whiteboard with my phone and import those.  Evernote is pretty good at recognising hand-written notes and making them searchable. I still like writing with a real pen in paper journals.  But now, when I record something I think I might want to find later, I take a picture with my phone and load into Evernote.

This changed prompted some observations:

Abundance.  Disk space is cheap and loading things into Evernote is available everywhere I am.  It is cheap and easy.  This means that I can upload more than I previously could file, draw, record, etc. and easily find it.  Because of tagging and search, having recorded all of this stuff gets in the way of finding what I want less and less.  In the past, digging through a file cabinet for that one receipt or paper your wrote 10 years ago when 99% of the stuff isn’t what you are looking for was very time-consuming.  Now it doesn’t matter if 99.999% of your notes aren’t relevant to a search–you never see them.  This brings me to another point…

Letting go.  There may be a temptation to manage your Evernote repository to make sure there isn’t redundancy, out of date stuff, or stuff you no longer care about.  Doing this with a file cabinet is probably not too bad an idea if you want to be able to find something fast.  In Evernote, unless you can do this with the original tags or notebooks you created, be sceptical of the value of doing it at all.  Abundance of space and the ease of storing and retrieving notes means “managing” as an activity isn’t worth your time.  Tag as best you can when you put the note into Evernote.  Write a few words that will remind you why this item was important to you and move on.  If you run across a note you want to fix, edit or delete it.  But don’t go looking for them.

The value of forgetting.  One of the best things about paper notebooks is that when you revisit them, you realise how much you forgot that never needs to be relived.  I often highlight or strike out old notes when I am looking for something in an old notebook, noting that something is forgettable (and good riddance!). The value of forgetting is immense for cognition, synthesis and creativity. I have a few details we need to remember very specifically (addresses, passwords, etc.). I have a few creative ideas that I will want to use as a launching pad for a future project.  Occasionally, I may have created a clever poem, story, essay or picture.  But everything else is in the way.

Evernote promises a simple and more effective way of remembering, and, hopefully, of forgetting. I doubt that I will search for Everything I have put into Evernote.  Some things will be forgotten, as they should be.

The presentation contains step-by-step instructions for getting your first simple project running on EMR and a short overview of the map-reduce way of thinking about data problems. The slides also contain many links to tools, papers, and examples.

Download EMR-HadoopMeetup slides and example code and slides (including latex/beamer file).

Second, a beetle-like walker that can easily navigate rough survaces:

Progress on robots has been pretty steady for years, so what is interesting about these particular examples?

The ability to direct a robot’s higher level movements with reliable outcomes.

That is, a shift from directing robots with syntax (manipulating control surfaces, motor speeds, servo positions) to directing with semantics (go to the living room).  Robots like those shown above will soon be able to take commands at a high level and carry them out reliably and safely.

Broad access to robot servants

We have reached a point in sophistication and cost where small, autonomous robots can be controlled with higher level commands.  To learn to fly a quad-copter via remote control of throttle and pitch in a way that can navigate a window with 3 inches of clearance takes hours of practice and crashed copters. As shown above, now we can merely command that it be done.  The cost of these devices has reached a few hundred dollars each and will continue to fall quickly.

Swarms of robots (“many hands make light work”)

These developments mean that small robots can be directed to “got into the living room” or “fly across the park without hitting anything and find little Bobby.”

The possibility of cheap, small autonomous robots performing tasks such as searching for hurricane survivors, mapping oil spills, collecting anti-personnel mines, carrying out surveillance, assisting you in the garage or mowing your lawn is now very real.

With the ability to direct them at a high level, the opportunity to carry out large tasks with many small specialized robots is in reach. With our current knowledge of collective behaviors, swarms of robots could be called on to perform large scale work requiring collaboration and coordination.

Bio-mimicry and fear

There is something unsettling about both of these videos.  They both demonstrate movements that are life-like, and often associated with animals we’d rather not have too big or too close.  Both robots execute robust recovery and control actions much like beetles or house flies. I postulate that this bio-like attribute is core to the technical success demonstrated here.  But it is also clear that this attribute may be more a than a little distasteful to people.

Will we adapt to the creepiness and have a swarm of crawlers clean the house? Or is this technology DOA because it violates some survival instinct?

Another concern is the new, cheap capabilities to do harm that come with any technological advance. Will ethics and countermeasures evolve fast enough to head off a major mis-appropriation of robot swarms?

example of an approach, reinforced by ongoing global financial disasters, that uses media manipulation and confidence boosting as a substitute for real solutions.  It fails to punish bad behavior due to the need for collusion between the government and the offending corporations to construct the information campaign.  It fails to construct real solutions since the facts are not known and the number of people able to address the problem is extremely limited.  Also, since these people are the same people that caused the crisis, real solutions are avoided to prevent adverse publicity.  Most importantly, it is yet another body blow to the nation-state and the global market system as legitimate organizational constructs.

In organizational leadership, there is an inverse relationship between transparency and legitimacy.  As secrecy and obfuscation go up, societies collective support of the legitimate powers and roles of the organization go down.

This happens because (1) people don’t trust their leaders and the organizational systems serve their interests. Trust can break down to a point where people undermine the parts of the system that are beneficial (e.g vaccinations are hurting you more than polio or smallpox) and (2) there are fewer better solutions people lack information or open creative dialog about problems and possible responses.

This has always been the case, but the cause-effect time delay is getting shorter partly because the systems encompass many layers of social, financial and political systems and are more richly connected.