Notice: An Update for your Perlang FPGA Cluster is Available

Ok, I've been bit by Tim Bray's Wide Finder meme.

I noticed the conversation swarm as it bubbled up, but didn't pay too much attention. Mark Masterson's article It's Time to Stop Calling Circuits "Hardware" caught my attention, as I have pondered the plasticity of the boundary between hardware and software in a previous life.

So I've been digesting the conversation swarm. It's one heck of an interesting read.

Tim presents a problem case that frames a fundamental shift occurring in modern CPU/system architectures. The shift is moving us away from ever increasing CPU speeds towards ever increasing CPU counts. Certain classes of problems are extremely well suited for the shift to multcpucore architectures. Other problems gain no direct benefit, particularly if they are migrated without change. Tim uses the problem of summarizing log file data as an example of this latter case.

Without brainpower focused on this aspect of the problem, the techniques being employed to increase aggregate compute capacity will not provide much benefit for many of the common tasks performed in IT shops.

There are three interesting aspects to Tim's conversation swarm. Two are explicit. The third is implicit.

The first aspect consists of all the solutions for the stated goal - how to leverage the latest trend in processor/system architectures for the seemingly mundane task of processing log data.

For what it's worth, here are my first thoughts on the problem of leveraging multiple cpus ofor the task of processing log data. My preference leans towards use of existing technology, most likely to be implemented by the people most likely to feel the pain.

Divide and conquer: (the sysadmin in me)

• Coerce the logging engine(s) to dump into multiple log files (to multiple disks or disk channels if necessary).
• Run a pile of processes to process the log files independently.
• Consolidate the data - either as post processing or incrementally via some form of IPC.
• The choice of language is immaterial, but history would probably vote for perl or shell goop
  

This smacks of the type of solutions Tim sees existing in most IT shops. It's blunt. It's probably sufficient enough to allow us to move on the next point of pain. It's almost completely devoid of any interest from a software engineering perspective.

Streams and Trigger: (mentioned in the conversation comments)

• Hook into the log stream(s)
• Spawn readers for the various data collection functions
• Send events from the log stream(s) to the readers, processing the data as it's received

This is generally a solved problem using any one of a variety of existing programming languages/tools.

Neither of these two solutions are particularly interesting, but I imagine they are the most likely to be implemented in the wild.

My final offering is more of a meta solution.

• Formulate a red herring idea
• Pose it to a bunch of brainy people
• Watch them chew on it
• Gain new insight

Oh wait. I'm getting that deja vu feeling. :-)

The second interesting aspect of the conversation swarm is the rumination over the relationship between computer languages and the shift in cpu/system architectures.

One participant (sorry, can't recall the link) offered the suggestion that it's probably easier to improve a language like Erlang than it is to modify the mainstream languages to provide the capabilities inherent in Erlang.

I don't disagree with this point of view, but Tim's point regarding the widespread use of perl/awk/etc points to a fundamental fact in IT shops - the tool must be wickedly effective at getting the job done. Optimal performance is often optional.

So how to effectively use 64-1024 CPU machines?

First off, who says our currently technologies are effectively using the existing architectures? Follow things from the hardware up the application stack - it staggers the mind.

The reality is we seldom go back and fix. We come up with clever ways to incrementally capitalize on architectural changes. We reframe existing code in ways that take advantage of changes in architectures. I'm overgeneralizing somewhat, but no matter.

At the risk of sounding like a pessimist, I think we'll end up with thousands of little SOA web services engines. Each one handling a single piece. Each one with its own HTTP stack. Each one using PHP/Perl/Ruby/etc to implement the service functions. Each one sitting on top of a tiny little mysql database. Eeeep! I just scared myself - better drop this line of thought. I'll have nightmares for weeks.

The third interesting aspect of the conversation is how it shows some of the most important characteristics of the modern concept of networks vs. groups. It's decentralized, it's unlikely to be swayed by an alpha geek, it creates a variety of unanticipated results, it's a bit messy, and it provides fertile ground for exploring the topic at some point in the future.

Good stuff!

The Laundry List in my Brain

In What an Enterprise Architect needs to know, Adrian Grigoriu lists a plethora of topics an EA team is expected to navigate.

I am conflicted in my opinion of the list.

On one hand, it seems woefully incomplete. It omits several things I use on the regular basis. I'm tempted to enumerate them, but fear making the list even more daunting.

On the other hand, the list seems to focus on the morass of technologies encountered in a modern IT organization. While I'm tempted to rant on this topic, I'm more interested in focusing on what an EA needs to understand. To this point, many technologies can digested based on a several key points.

• Some things never change
• Some things are really just a variation on another thing
  
• Some things are simpler than they appear
• All things came from another thing
  

On the third hand¹, the technologies aren't the hard part. Stephen Downes has an interesting list of Things You Really Need to Learn.

How to predict consequences
How to read
How to distinguish truth from fiction
How to empathize
How to be creative
How to communicate clearly
How to learn
How to stay healthy
How to value yourself
How to live meaningfully

These seem the more difficult topics to master.

1. Maybe that's the true secret to our success.

Quantum Architecture?

Todd Biske works through the question of Service focus or product focus? He references several articles, providing several pieces of interesting reading material.

Something occurred to me while reading through the links.

Is it possible this is another form of the particles vs. waves duality?

Fab@Home - because I need another gadget

This one aligns with an Architect's unnatural attraction to metamodels and whatnot.

Fab@Home

Ions ate us for lunch

CIO Insight posted an interview of Bob Otto , the retiring CIO/CTO of the US Postal Service. I found the interview from a linked posted by Bob Gourley in How is the USPS like your IT enterprise?.

Bob Gourley quotes a segment that identifies Bob Otto's three guiding principles, summarized below in bullet form.

• Standardize everything
  If you find a process you like, standardize it
• Centralize everything you can
  If you have services in five different places and you can centralize them, you will have reliability, predictability
• Simplify
  The computer has taken over your life, so I want it to be intuitive [for people to operate and manage]. I also test my own dog food.

Distilled further, we have standardization, centralization, and simplification.

Sigh...

These seem quite reasonable on the surface, but let's think about them for a moment or two.

Standardizing on a process you like? This presupposes that what you like is actually the best fit for the organization. I suppose one could assume that "like" includes this as an assumption. Hmm...

Centralize everything you can? This presupposes that all services are best delivered centrally. I agree that increased predictability is likely, but I question reliability and a small list of other potentially important attributes. Hmm...

Simplify? Yes - we have one point of agreement. Generally...

Hmm...

Still convinced these three principles are sounds? Perhaps they are true for some environments. A slow growth company in a mature market primarily requiring maintenance activities might benefit from these principles. Very few business, however, can assume these conditions.

Still convinced? Let's combine these principles and see where it might lead us.

We are newly appointed as CIO of example.com.

We discover a variety of seemingly duplicated services spread across the landscape. Hungry to show value to the business, we centralize them to achieve reliability and predictability. We even gain some economies of scale, so we get some cost savings to boot.

As part of the centralization, we decide one of the service implementations has the most efficient and effective processes. This becomes the standard process for all of the newly centralized services.

As we centralize, we notice several portions where some simplification can occur, so we do some process trimming. We've also eliminated some of the tasks performed by the once disparate services, so we're starting to see some dramatic moves towards simpler processes and systems. So far, so good.

We expect dramatic improvements. We might even expect some appreciation for our efforts.

For a time, we might actually receive the kudos for our accomplishments.

Then we notice a curious phenomena. The grumbling continues. Needs are not met. Changes are still required. Costs are still steep.

We investigate the situation.

We are shocked at what we find.

It turns out that our centralization bulldozing exercise cause some key functionality to get pitched overboard. The functionality confused the centralization task force, so they accidentally left it off the analysis spreadsheets.

We discover that our choice for most likable process as in fact the result of decisions made during a golf game. (the golf game immediately following one particularly frustrating day of difficult process discovery discussions).

Alas, the process simplification activities further exacerbated the problem. As it turns out, the process simplification team couldn't fit certain features into their model of a perfect world. (more 'inadvertant' deletions).

We also notice that our competitors have been watching our strategic moves, countering them with a strategy based on more balanced principles. We watch as our competitor eats us for lunch.

Help! HR is Stalking Me!

(Michael) Coté asks Is Google Stalking OK?

Outside of the HR/hiring context? Sure - you bet. That's half the fun of meeting people. :-)

As for potential issues regarding the practice within HR? That's an interesting question. There are limitations to what a potential employer can ask a previous employer, but I'm not sure if this limitation is applicable. There are legions of googleable offenses that don't fall under the purview of discrimination.

I wonder if this is a non-issue if done as part of a preliminary screening process with no interview taking place. Regardless, some organizations will likely want or need to consider updating their policy, if for no other reason than to feed the demons of risk avoidance.

Regardless, I'm sure we'll hear of associated lawsuits in the future. This seems inescapable.

"Yes, we're checking references - all the references to you we can find on Google."

Can IT be lean?

Andrew Clifford has posted The dismantling of IT, in which he ponders the result of a wide scale trimming down of IT architectures.

No arguments on the concept here, but I'm still a bit puzzled by particular article in the series.

The essence, in my opinion, is the following statement,

The most obvious change is that the new architecture would remove technical layers, such as databases and middleware. These capabilities would of course still exist, but they could be standardised and hidden inside the systems. They would not need so much management, and we would need fewer specialists.

This "still exist, they could be standardised and hidden inside the systems" seems no different than the waves of abstraction we have seen in the past.

Perhaps elaboration is in order.

I do not reject the concept because of a dislike for the idea of dismantling of traditional IT. Far from it... I simply do not see an avoidance of the fundamental problems mentioned in previous posts.

Have I missed a step?