Category Archives: Uncategorized

Apache Arrow: Lining up the ducks in a row… or column

As we noted a couple years back, data is getting bigger and fast data is getting faster because of the onward declining cost of infrastructure. And nowhere has that been more apparent than with in-memory and Flash storage. For instance, when SAP HANA yanked the in-memory database from its formerly specialized niche, IBM, Oracle, and Teradata subsequently one-upped with in-memory columnar add-ons to their core platforms. And in the NoSQL world, where Aerospike debuted with its Flash-based operational database, today use of in-memory and Flash storage is no longer unusual. And, while in-memory processing is not the only advantage of the Spark compute engine, the Apache project would not have caught on with the wildfire pace were memory still cost-prohibitive.

But the not so subtle problem with all this is that Silicon-based storage is not simply a faster version of disk. Disk is optimized for retrieval – especially in cases where you have different temperatures of data and want to tier, stripe, or shard it so that the most frequently-accessed data is on the most accessible (typically outer) part of the spindle. Or with Flash, where you want to minimize writes (which are inefficient) and rewrites/updates (which can shorten Flash lifespan), or with memory, where it’s all about lining up similar data types and functions so you can in effect operate it as a form of pipeline so the chip operates at an even pace.

That’s why, as we stated a couple years back, the architecture of storage impacts the architecture of the database and the application(s) running on it.

Now, compound the issue with CPU: there are storage and processing ramifications here. There is chip level cache that gives you an even faster form of storage on-board for highly volatile processes, and then the compute itself, where pipelining techniques can cram multiple actions into a single compute cycle. These factors were not that critical when systems were disk-based, but when you start to level the playing field between storage and CPU performance, the slightest perturbation can add serious speed bumps that could defeat the whole purpose of going to Flash or in-memory storage.

So the attention paid to Spark is a reflection of the importance given to speed in processing Big Data. When you can run what-if scenarios and trend analytics in seconds or minutes instead of hours (or days), machine learning becomes useful. But just as Spark, Flink, and the increasingly endless array of interactive SQL, streaming, and graph engines are emerging, each of them has to solve the problem of literally lining up their data for in-memory processing. This is a thankless task and one that offers zilch added value. Having interfaces for converting data for in memory instantiation is, in effect, running in place.

That’s where a new project, Apache Arrow, comes in. Led by the CTO and co-founder of Dremio (who came from MapR), a stealth startup that will be doing something with Apache Drill, they’ve taken a momentary detour to build a standard interface and protocol for marshaling data and, literally, lining it up in an easily consumable columnar format for processing. So all the 4-byte integers are processed together, and all the join operations are processed together, and so on. It’s built around the data SIMD parallelism engineered into Intel Xeon processors.

Significantly, this project has the potential of going viral in a similar, but quieter way, than Spark. It’s backed by a who’s who list of over 20 Apache committers from Dremio, MapR, Cloudera, Hortonworks, Salesforce, DataStax, Twitter, and AWS. Coming out of the gate, it’s going to get supported by Spark, Storm, Drill, Impala, Pig, Phoenix, Hive, Cassandra, Pandas, Parquet, HBase, and Kudu. And the project, just coming out of stealth today, is not even bothering with incubation. The Apache community has already ratified it as a new top-level project.

Arrow will provide a standard for columnar in-memory processing and interchange of data to an in-memory representation that can be shared by multiple engines (e.g., Spark and Impala) residing on the same node. That means that each database or compute engine does not have to have its own dedicated slice of memory – in-memory columnar storage can now be a common pool, meaning that users can get more use out of the same in-memory footprint, reducing infrastructure costs. They will have implementations for C, C++, Python and Java, with more languages to come.

The project will succeed as long as it keeps its aspirations contained – open source projects get more widely adopted as long as they don’t usurp the unique IP of commercial products. And so, while the Arrow project may develop some sample functions (e.g., joins) for manipulating data within the column, we suggest that they not grow overly ambitious.

Footnotes from Mars Curiosity

It’s natural to look back at the passing of Neil Armstrong and conclude that they just don’t make The Right Stuff like they used to. Or maybe in an era of declining expectations, it’s an unusual feeling to get a sense of pride that the U.S. is still able to muster a major accomplishment.

Yet the shots of people standing at 1:30am on a Sunday night/Monday morning in Times Square appeared a throwback to a more innocent, hopeful time. About a month ago, the Mars Science Laboratory Curiosity (MSL Curiosity) made the most improbable of landings on Mars. An eerie freak of timing that America’s greatest space achievement since the landing on the moon coincided within a few weeks of the passing of the man who uttered those words as he took the footsteps from the lunar lander.

We were reminded of this during a keynote from Doug McCuistion, who heads NASA’s Mars Exploration program, at Siemens PLM’s analyst conference last week. It was a fascinating talk, where he gave us background on why we’ve kept going to Mars (40 times over the past 40 years) and rarely succeeded (only 16 missions have made it there).

What are we doing there? It’s the obsession with familiarity: Mars is the closest relative to Earth, from adjacency and similarity (it’s the only terrestrial planet in the neighborhood). And all the surveillance and experiments points to a truism: there but for fortune Mars lost its atmosphere and most of its water. The evidence of water is both black and white – white as in the patches of silica (beach sand) uncovered by the tire tracks of a recent rover, and dark discolorations of sedimentary rock at the foot of Mt. Sharp. The Phoenix rover that visited the Martian pole back in 2008 discovered ice sheets that are several kilometers thick.

McCustion explained that the series of missions to Mars have followed a logical progression; the Global Surveyor identified old river channels while the Mars Reconnaissance orbiter has been taking high resolution photos of the entire planet, both of which have been used to select landing spots with greater likelihood for evidence of organics and water.

The dramatic landing of MSL Curiosity was just the latest of a series of high risk maneuvers that the mission endured. As for those Seven Minutes of Terror, it was closer to 10 minutes according to McCustion, but who’s counting anyway? That’s where the relevance of speaking at a PLM conference came in; McCustion spoke of the importance of simulation to “buy down” risk to the extent possible (the team used plenty of Siemens modeling tools to optimize component design), because artifacts like the operation of the huge parachute through the Martian atmosphere (which is 10% as dense as Earth’s) could not be physically tested. Simulation helped the team optimize and in some cases completely change the designs or plans for the plutonium power module and guided instruments. As for the unusual descent, it was dismissed as out of hand until all the options were weighed.

While hardly the only game in town, the Mars Exploration Program has replaced manned spaceflight as the public face of NASA. And to its credit, NASA marketed this mission extremely well, having a comprehensive web strategy replete with Twitter and Facebook feeds, partnering arrangements with games providers like Angry Birds, and staging the spectacle of live viewing in Times Square. Just think, if the touchdown had occurred at a more civilized hour, imagine the size of the crowds. It was an all-too-rare moment of feeling of shared accomplishment – and it wasn’t America’s alone. Technology onboard Curiosity had an unmistakable international pedigree, including a neutron detector from Russia.

The good news is that beyond the images of a shuddered manned spaceflight program, that private ventures like SpaceX are starting to fill a void. But SpaceX et al would not be possible had NASA not ventured where no man has gone before (SpaceX didn’t build that, but capitalized on it).

The question is whether, in an era where the national debate is all about cutbacks, that we are willing to invest anew in science, math, and engineering education. The Curiosity landing did not have the same global impact as Apollo 11. But would it be too naïve to hope that those Seven Minutes of terror becomes the early 21st century’s Sputnik moment?

South Korea, The Energizer Country

OK, for a change we’re not going to talk about IT.

It was quite a month, three trips mashed into one. A family trip out to the west coast to celebrate mother’s 94th birthday morphs into a business trip, visiting clients and prospects throughout Asia/Pacific. Seoul, Taipei, Hong Kong, Singapore, Kuala Lumpur, and Australia. Economies in various states of maturity. Asian sense of organization – there are lots of people, for which infrastructure and systems were well designed to handle. You get food And drinks, even on 1-hour commuter flights. No delays – well, except for the Hong Kong – Singapore leg which was on United (the power went out just before we were supposed to push back from the gate).

As for the third trip, when this whole thing was booked last spring, the recession was in its nadir and Quantas direct flights between New York & Sydney were dirt cheap. With the logic of when the heck else would we ever find ourselves out there, we tacked on about 10 -1 12 days and made the last part a family trip.

A few highlights. Arriving in Australia after a red eye from “Asia” was a bit anticlimactic; you go most of the way around the world, traversing exotic cultures and cities, only to find out that you’re practically back “home” in a western nation. Call it Canada with a better climate and a sense of humor. Nothing exotic except for the flora and fauna. A prosperous country that doesn’t realize how good it has. A country with the land mass and natural resources (except water) on par with the states, but with only about 5 – 10% of the population. No wonder they’ve already figured out how to do national health care; Oz (Australia) can easily afford it.

Australia is a prosperous country with the optimism that there are better days ahead, mate. When things are going well and you don’t have to worry about freezing your as off, it’s easy to laugh about life’s absurdities. Australians are happy to be in Australia.

Changdeokgung Palace

Cut to the chase. The most surprising place was our first stop in Asia. We spent nearly 4 days in Seoul. Until now, the closest we got was Little Korea, the block of West 32nd Street between 5th and 6th (just below the Empire State Bldg.).

Of all the places we visited in Asia, Korea was the one where the least English is spoken. That’s in spite of the US historical presence (which made South Korea, and its economic miracle, possible), and the fact that most of the signs are in English as well as the local language. Korea is off the beaten track for westerners, and as a tourist destination, draws Asians to its flea markets. On its present course, however, Korea isn’t going to be such a shopper’s bargain forever.

Korea is A Tale of Two Cities, err… countries. It’s the golden age for the South, while it continues to be the Dark Ages up north; a glance at a nighttime map of the world clearly demarcates the boundaries of North Korea. Seoul (which is promoted as The Soul of Asia, as if anybody could understand what that really means) is barely 30 miles from the DMZ, but you’d never know it. In fact the 8-lane airport freeway ends abruptly in the northern suburbs; where the airport bus got off was barely even half the distance from the frontier. The freeway’s abrupt end also signifies that the country seems like it’s still under construction – sort of the throwback to the US in the 60s. Save that thought.

Back to the golden age, South Korea is booming – they’ve hardly been touched by the global recession. If you’re looking for a better indicator close to home, forget the ma and pa shops of Little Korea, head to the Samsung showroom at Time Warner Center in New York. Inside Manhatttan’s premier upscale shopping mall, Samsung has a store that sells nothing but image. You can’t buy anything at The Samsung Experience, but you can gape at all the cool 100-inch flat screens and multi-function mobile devices. A few years back, Samsung was a second tier consumer electronics supplier whose products were primarily found in off price stores. They made their strategic thrust in LCD, while Sony, the previous high end TV brand, was caught napping. Today, Samsung, not Sony, supplies the panels for Sony’s Bravia flat panels in addition to their own brand. Along with Sharp (Aquos), Samsung has cornered the high end of the flat panel market.

Samsung is a parable of the Korean economy. They are positioning themselves as the higher quality alternative to China in manufacturing. That has fueled a boom that is now manifested in the Seoul metro area, which has become one of the world’s largest construction sites.

Back to the US in the 60s thought; we were building the modern infrastructure that is now, after years of disinvestment, falling apart. Case in point: the DC Metro. When it was built in the 70s, it was considered a state of the art transit system. Decades of disinvestment later, the Metro was the site of the nation’s worst transit accident since the Malbone Street wreck in Brooklyn, dating back to World War I.

Back to Korea, the question is what happens when the construction is finally completed? Korea has a bumper crop of university graduates who are aspiring for more in life beyond an office salary. Like Japan, or India’s offshore developers, their expectations are being inflated as they join the global economy. Salary levels are going to rise. The manufacturing base will get challenged by the next country that introduces new crops of engineers to the global market at an earlier point in their development. Korean needs to learn from its age-old nemesis Japan, which has never fully recovered from the inflation born of rising income from exports, that in turn fueled a real estate spiral that careened out of control. If Korea is to claim its position at the higher end of the value chain, it will have to evolve beyond manufacturing (where there will be new competition) and construction (which will flatten out once the country or metro areas have built themselves out).

Hopefully Korea can also learn from its own history. Go to Changdeokgung Palace, the palace of palaces among Korea’s royalty. The place, designed in harmony with its natural surroundings, was built in the 15th century, was regularly burnt down by invaders and rebuilt roughly every 150 – 200 years, through the present.

The common thread is resiliency; through most of its history, Korea has either been fighting or been conquered by bigger guys in the neighborhood, principally Japan, China, and after the Second World War, the Soviet Union.

So today is truly a golden age in Korean history. They have never been so prosperous or seemingly secure. North Korea could be an exurb of Seoul, but South Koreans don’t take North Korea seriously – a lot of that is denial. With the wealth so conspicuous, it’s hard to think about the what-ifs. But today the world is buying Korean (Koreans are furiously buying American at Costco but that’s another story). Tomorrow South Korea will have to reinvent itself to go higher in the food chain if it is to preserve its newly won wealth. Just like the Japanese could not play the global economy on their own terms forever.

The day of reckoning will come when North Korea implodes. It’s not a question if, but when. And that’s where you’ll see movement in the world’s political geotectonic plates.

At some point it will be in China’s interest to seek a Grand Bargain. China doesn’t care who runs the north, and in fact, the nuclear nonsense is probably not in China’s best interest. China doesn’t want North Korea to disintegrate because of the huge refugee problem and unrest that it might cause in China’s northern provinces, not to mention causing a drain on its economic development. Consequently, North Korea is not the pawn of Russia and China that Iran is. North Korea lacks useful resources (oil) and a unique strategic position that could knock the west off balance.

There a solution to the North Korea problem, but unfortunately it is one that reflects American weakness. U.S. power and influence are waning; China holds most of our debt, and for China, the U.S. is too big to fail. As an export-oriented economy, South Korea is going to get to a crossroads where it must decide where its political and economic security best lie. Given current trends, we wouldn’t be surprised if at some point China offers South Korea a Grand Bargain – acknowledge China’s sphere of influence, and get the North back.

That of course would be a continuation of old history. Korea has never been big enough to survive on its own; it has either been conquered or operated as a protectorate. It’s too small and in too rough a neighborhood to stand alone. South Korea has operated under the U.S. protection umbrella since the Second World War; it’s not inconceivable if at some point its allegiance would shift to China, especially if that would get the Northern regime out of the way.

We’ll make one more prediction. Should the Koreas reunify, it would make the German counterpart look like child’s play. West Germany inherited a nation that had the highest standard of living in the former eastern block. Yet the cost of integrating the former eastern zone into the western economic and political system has drained the country. On the other hand, South Korea would be inheriting a region that ranks with Africa as one of the worlds basket cases. No matter how rich the south is, it will be up to its eyeballs civilizing the north.

Let My Handsets Go!

We’ve always been amazed at how, in North America, mobile carriers perpetuated a captive business model that in the computing world went out of style nearly two decades ago. So it was ironic when release of yet another proprietary system – the iPhone – opened the first crack in the captive turnkey systems environment that was the North American mobile market.

Since then, the iPhone has carved out a powerful niche in the market, Google’s noises have prodded the FCC to reserve new spectrum for open devices, while Microsoft has made limited impact with Windows Mobile. But for the core of the mobile market, it’s still been business as usual. Until now, handset makers have been the odd men out.

However, as mobile devices morph into computing platforms, something has to give if you ever expect to see a critical mass market for mobile apps. It will have to follow the same script as the PC, which provided a critical mass target that led to explosion of what became the consumer software market.

Today’s announcement of the Eclipse Pulsar initiative is the handset maker’s revenge. Led by Motorola, Nokia and Genuitec, with participation from IBM, RIM and Sony Ericsson Mobile Communications, Pulsar is a new Eclipse effort to develop a common IDE that would support different handsets through a series of device profile plug-ins. The idea is that handset makers have better things to do than keep reinventing the wheel with their own unique development tools.

Pulsar is a modest but important first step towards creating a coherent target for mobile app developers. Until now, the market for mobile handset environments has been very fragmented, with no single OS or presentation layer having more than 10% of the market. Because the carriers controlled what went on their market, it took away motivation on the part of handset makers to agree on standards of any sort when it came to development targets. Apple’s iPhone App Store was the shot across the brow that prompted them to act.

Pulsar won’t rationalize the mobile app development market on its own, as there will still be a proliferation of profiles that govern which apps can make it onto which devices. But in the long run it will make it more economically attractive for device makers to rationalize their offerings so that software developers gain critical mass targets. Ideally, device profiles should hide the complexity of writing to different handset models, just as printer drivers for PCs have eliminated the burden for software developers to account for different printers. It will open wide opportunities for players like Adobe, whose Open Screen Project encourages developers to write their own Flash mobile players.

In actuality, as handsets plays such widely different roles to different classes of customers, it will never be that simple. Instead, what is likely is that handsets will divide into different classes, from basic phone to PDA and gaming or entertainment platform, and so on, with each type of device likely fitting within some form of de facto standard. Pressure for rationalization will come from the fact that devoice makers want to draw more application software support, which in turn makes them more attractive to consumers for what becomes the open market.

Oracle to BEA Customers: No Surprise Here

Oracle finally placed its cards down on the table regarding its roadmap for BEA products, and for the most part there’s little surprise. As Gavin Clarke reported several weeks ago, Oracle will be deconstructing AquaLogic. But taking a cue from its Applications Unlimited Strategy, Oracle is telling BEA customers that it will support all existing BEA products, regardless of whether they wind up on maintenance.

Cut to the chase, Oracle’s choices for strategic products going forward are not exactly a shock. WebLogic Server, until a few years ago the leader in the space, is going to be Oracle’s Java middle tier going forward. It wasn’t until Oracle 11g that, in Larry Ellison’s words, it finally got the appserver right. So it’s obviously going with the more established offering. Tools are another story: Oracle claims to be the second largest contributor to Eclipse, yet it continues to spurn the IDE on which the Eclipse Foundation was founded. Instead Oracle focused its efforts on EclipseLink, the implementation of JPA that it donated to Eclipse.

While Oracle has been schizoid in its Eclipse strategy, BEA was schizoid on tools in general. Originally embracing a VB-like approach with the original WebLogic Workshop, BEA later forsook the technology following acquisition of an Eclipse-based successor, throwing its installed base into a confusing migration strategy. No question here as to which way Oracle is going.

In other areas, Oracle’s roadmap is a case of seasoning its Fusion portfolio with BEA pieces where they fill gaps. That includes AquaLogic Service Bus, which now gets fortified with Oracle’s service composition fabric that comprises its implementation of the proposed SCA standard. Ditto for SOA governance, where AquaLogic Repository fills a gap, and where both relied on OEM bundles with Systinet for the UDDI service registry. And the same goes for entitlements, a BEA technology that will be added to Oracle’s Access Management suite. As for BPM, BEA’s AquaLogic offering –the former Fuego product – provides the middle ground for business unit level implementation that complements the top-down of its existing bundled offering from IDS Scheer. While BEA/Fuego customers liked the degree of control that their tool provided, under Oracle they are going to have to give up the run time to Oracle’s BPEL Process manager. Call that a case of realpolitik.

In a deal this size, there are always going to be a myriad of details as to which product features make the cut and which don’t. With Oracle’s previous large acquisitions (read: PeopleSoft, Siebel), there has been little question about which application is the strategic one going forward. In spite of Larry Ellison’s early posturing, it was never about sunsetting a particular ERP or CRM package because customers are not going to make forklift migrations. Middleware is another story because of the predominance of standards; migrations are painful, but nothing compared to dumping your ERP system. And thanks to de facto standards, and more importantly, open source, tooling is fungible.

Unless the viability of a company is in question, most customers don’t like to see their vendors get acquired as they often fear getting lost in a larger fishbowl. It’s even more the case with Oracle, whose past bravado, aggressive sales force, and high pricing have turned customers off. SearchSOA’s Michael Meehan cites results of a BEA customer poll that his group conducted last week, and the results aren’t exactly shocking: most BEA customers were happy with the way things were and aren’t looking forward to life with Oracle. So it’s not surprising that one of the points that Oracle emphasized in its presentation is that BEA pricing would remain grandfathered in for existing customers. The die was cast back in 2003 when Charles Phillips backtracked from Larry Ellison;s posturing about wiping PeopleSoft apps off the planet with what eventually became the Applications Unlimited policy. No matter, Oracle still faces living down its reputation as it cries all the way to the bank.

But back to product roadmaps, Oracle’s strategy for BEA products is a conservative one – not simply from the decision to continue supporting even obsolete products, but rather, that it has not been as fast on the draw with new technologies as was BEA. That explains BEA’s earlier embrace of OSGi, but the flip side of bear hugging the new and trendy explains why it has never had a consistent tooling story while Oracle’s, love it or hate it, has.

IT Forecast is Partly Cloudy (II)

Last week, we opened the Pandora’s Box about the inevitability of the cloud. And we spoke of the tension between SOA and WOA camps as to which is the best means for getting services from or providing services to the cloud.

You can bet with the Web 2.0 Expo this week that there is plenty of noise about the cloud. For some, it’s so much noise that the whole notion of cloud computing, or the cloud itself, has become rather foggy.

One of the arguments over SOA is that the web services standards that are used for implementation have generated intimidating layers of complexity, and that web-oriented alternatives (e.g., WOA) are far more straightforward and far better fits for the web development skills that are already commonplace.

So there will be a flurry of announcements. A few examples: Kapow Technologies for instance, is launching an on demand mashup server, providing black box capabilities like Excel plug-ins for data services out in the cloud. Meanwhile Serena is teaming with Cap Gemini to launch a sandbox enabling business professionals to design and compose a mashup without the need for programming skills.

One of the more interesting announcements from a lineage standpoint is the emergence from its cocoon of SnapLogic, a startup with a WOA-oriented takeoff on RSS that it promotes as “Really Simple Integration.” Started by Informatica founder Gaurav Dhillon, SnapLogic represents a closing of the circle for simplified data access. Just as Informatica was the first to adopt a visual, component-based approach to developing database integrations, SnapLogic is doing same with resources that are accessible over the web.

It’s based on an HTTP server that supports RESTful services; a repository comprised of metadata written in HTML; generic resources for reading, transforming, and writing data; and support of Java and various dynamic scripting languages on the server, and multiple web output formats including HTML pages, RSS or ATOM feeds, and JSON (a JavaScript-based data interchange format).

Using RESTful style, each data source is treated as a resource. In turn, access to those resources can be managed, not through adding tokens or other entitlement technologies, but by making each individual or class of individual’s access a separate URI. Imagine, if you will, table, where the columns are data sources and the rows are specific users. Such tables could be fed by directories and internal access control tools, or the HTML metadata repository, rather than adding a separate layer of complexity for access and authentication.

Providing a clever way for RESTful services to become reusable, SnapLogic helps flesh out the vision of WOA, which is could be nicknamed, technology that is just good enough to get the data you need, wherever it sits out in the cloud. Don’t mistake the elegance of simplicity; although web-oriented approaches essentially take the user friendliness of client/server database apps to the web, the simplicity of the architecture rules out embedding properties or sophisticated capabilities such as federated identity, orchestration, security assertions that come with WS-standards. That’s not necessarily a bad thing if your app doesn’t necessarily involve processes involving high sensitive data or require high performance. But if they do, there’s no reason why they can’t be implemented within secured environments where all the necessary governance and performance are applied extrinsically.

But what’s interesting is that with emergence of the cloud, SnapLogic and StrikeIron offer approachable alternatives that let you have your data services without the reengineering baggage.

Random Notes

Evidently social computing is becoming so last year. A couple pieces in today’s eWeek are providing indicators that virtualization at all levels, from desktop to storage and cloud computing, are drawing critical mass VC interest. Chris Preimesberger reports from an annual VC forum at Microsoft’s Silicon Valley campus, while Rene Boucher Ferguson writes of a startup’s idea about freeing those unused cycles across supercomputing grids that are becoming more ubiquitous. Think of it as the SETI@home screensaver on steroids. And we’ve also looked with interest on another startup’s elastic computing idea to take on demand computing beyond monthly subscription fees to something that more closely replicates paying by the drink.

It reflects the fact that, while we’re not necessarily in a Malthusian world yet (there was an interesting piece in the Journal this morning), we are in an age where rising resource costs, not to mention the green ethic, is changing perceptions fueled by explosion of the Internet that computing is essentially free.

And one last endnote before we put EclipseCon 2008 to rest. Mike Milinkovich provided an excellent response and clarified some of our questions about interoperability. As you’d expect, he also gave a spirited rebuttal on how Eclipse is not losing its focus — it’s a challenge that faces any organization as it grows its mission, and we credit them for keeping the issue on their radar.

And as for all those strange bedfellows that we spoke about, Daryl Taft raised an interesting point today about another consequence about diversity: what would life be at Eclipse beyond IBM?

The Third Wave of Content Management

When we were searching around for a new content system to power the next version of our main website, we were amazed at the selection of open source offerings that has brought the kinds of slick capabilities traditionally associated with the Interwovens and Vignettes of the world down to bite-sized packages that SMBs like us could get our arms around. Although this blog site is on WordPress, will we shortly be debuting our flagship site courtesy of the open source content management system Joomla.

In effect, we were a beneficiary of what we’d term the third wave of content management. Jeff Whatcott, a veteran of Adobe/Macromedia and Allaire, briefed us on Acquia, on a startup founded by the creator of the Drupal open source web content management (WCM) offering. The new company intends to play a Red Hat/Zend type of role, which is to offer a commercially-supported distribution of an open source technology.

Announcing itself this week, Acquia will be opening two open source projects: Carbon, which will be the commercial distro of Drupal, and Spokes, which will be the distribution mechanism for updates. The company expects to release both in the second half.

The first two waves of history are well known – it’s the transformation of forms-driven document management to web content management. Whereas wave 2 adapted workflows to web authoring, the third wave takes advantage of Web 2.0, and more specifically, social computing. That is, the goal is to encompass everyday WCM with the interactive tools such as Wikis, blogs, email, instant messaging, social network services, and social bookmarking.

It’s a tall order, given that content management tends to be a sequential process, whereas social computing interactions are highly dynamic, and more collaborative and real-time than sequential. It also implies the ability to take advantage of mashing up content that tends to happen when you have multiple authors, and content that tends to evolve almost like a chameleon. In place of files, you have streams, which makes the idea of categorization difficult, and navigation nonlinear (search engines become the only practical means of deciphering what appears, where, and when it appeared).

What’s neat about the third wave is that it has made commodity what used to require tens of thousands of dollars just to get to entry level. For the cost of development, an SMB can get a content system that provides the kind of thoroughly professional, slick engine that you used to have to pay through the nose for. Instead, the rough edges of the third wave will be making sense of all the social computing capabilities – there’s certainly plenty of standalone Wiki, forum, and messaging tools out there, but the trick will be to put them together under a content umbrella. For now, what Acquia’s Whatcott terms “social content management” is anything but commodity.

Everything You Know is Wrong

With the Open Group testing the waters to extend its reach beyond enterprise architects, we were part of an inaugural offshoot from their EA conference that discussed the role of what the group terms IT Specialists. It’s part of a kickoff to a new certification program for the people who are considered the unsung heroes of IT: those who work at ground level to get projects done.

Ironically, the name “IT Specialist” is something of a misnomer as the goal of the Open Group’s certification efforts is to create specialists who become, in effect, more like generalists. That is, you retain your technical specialty, but develop an awareness of technology disciplines that you routinely intersect with. For instance, an application developer who regularly deals with architects, DBAs, and QA. And the certifications are also supposed to emphasize cultivation of so-called soft skills through an evaluation process that accounts for actual experience in delivering project or non-project work to fruition.

We had the chance to address the session with a presentation describing why, at the end of the day, soft skills are becoming more critical to IT professionals because of the necessity to position themselves higher in the value chain in a globalizing market where traditional heads-down programming skills are increasingly available offshore at a fraction of the cost. (We’ll post our presentation online in a few days.)

We also had the chance to moderate a spirited discussion involving Phil Stauskas, who runs IBM’s IT Specialist program (which, along with Cap Gemini’s, is what the pen Group’s program is largely shaped after); Ron Tolido, Capgemini’s Northern Europe CTO (and a former UNIX geek who obviously broadened his horizons); Scott Radeztsky, one of Sun’s principal engineers; and ebizQ contributor Beth Gold-Bernstein.

A recurring theme was the inadequacy, if not obsolescence, of university computer science programs that tend to emphasize language and engineering skills at the cost of learning how to work within a business setting. Tolido suggested the notion that Java developers who were once the toast of the dot com world could become obsolete as more powerful, easy to use tools backed by frameworks like Spring that hide complexity transform development into a higher-level, less programmatic task.

Nonetheless, few on the panel bought into the notion that tooling would become so easy to use as to eliminate the need for developers – in spite of the hype surrounding new mashup offerings. Gold-Bernstein maintained that for every simplified front end, you would still need adults to build and maintain the robust, governable back ends that could support all the easy stuff up front.

Later in the afternoon, IBM’s Sheila Thorne delivered a highly personal presentation with the provocative title, “Dealing with People You Can’t Stand,” which provided useful pointers on how IT professionals – who are not exactly known for their people skills – could more successfully win friends and influence people. Almost acting like a plant in the audience, an obnoxious developer peppered her with instant trash psychoanalysis during the Q&A, providing ready proof of the challenge that remains in getting geeks to simply grow up.