Enterprise mashups, end user programming (EUP), and glue. | September 4, 2006
by Byron Binkley
A major shift is under way in the world of computing and at the center are the end user who have proven they can and will use software to create applications, not just use applications. The war between CIOs and end users was fought over Excel and end users won. They proved that those who mastered the ability to solve their own problems worked more effectively and garnered enough power in the enterprise to keep Excel in the trenches. The recent surge of research and dialogue surrounding end user programming, web authoring, mashups, and now the idea of “process mashups” is a response to this trend. There is an emerging opportunity to serve millions of people at work and at home with a better application development environment than Excel.
So who is going to reap the rewards from delivering “application building” to everyone? Or is the question is moot and will Excel just expand to claim the title? I predict that whoever creates the best end user “mashup glue” that can flexibly combine disparate data sources and processes will inherit the earth.
The gluing environment may end up with a name like a “business process spreadsheet” or some other term with the word spreadsheet in it. Not that it will look anything like a spreadsheet, but spreadsheet has semantically taken on the meaning of “programming” in the minds of millions of people who don’t call the spreadsheets they build “applications.” I’ve seen a bunch of different terms in discussions of the topic including: enterprise mashups, process mashups, BPM 2.0, and composite application. But I haven’t consistently seen a noun to describe the authoring and development environment like “mashup spreadsheet.”
There are a couple issues germane to the glue argument that I want to highlight from the current activity both in the academic community and the business community.
First, note that in the recent past there has been a resurgence of research into end user programming (EUP, also called end user development, EUD. Some great resources if you want context here are EUSES and an “End User Programming” survey by Brad Meyers at CMU). The targets of much of this research are (i) increasing the reliability of user built applications (not surprisingly the focus is on Excel. Check out the European Spreadsheet Risks Interest Group for some reasons why), and (ii) the continued search for the holy grail of 4GLs: a programming language simple enough for everyone to use. Research could turn up the programmer’s mentalese, but the bottom line is that some people aren’t that great at analytical thinking and problem solving. So forget the perfect 4GL for my argument because I don’t think everyone can become an expert programmer. However, I do think reliability is an issue that deserves the attention of people considering mashup spreadsheets so that we do not repeat the paradigmatic constraints and subsequent risks inherent in Excel for application development. And I think the largest improvements to reliability in end user developed applications will come from effective glue that increases the liquidity and structure of knowledge transfer among the different strata of programmer competence. Object oriented design, clear APIs, clearly defined integration of new processes and data, documentation of user generated functionality, and an environment where professional programmers and end user programmers can work cooperatively will be make-or-break features of the glue and the mashup spreadsheet.
Secondly, the emergence of a new noun to talk about end users mixing up data and functionality outside of the ubiquitous spreadsheet seems to have focused the buzz. There have been a number of really good posts lately discussing mashups (which only appeared on Wikipedia less than a year ago as a term outside the context of music) and the distinction between “data mashups,” “process mashups,” and “enterprise mashups.” Two posts that touched on this that I really enjoyed were Meme-check – Enterprise & Web 2.0 and Enterprise mashups: More about processes and less about services?. The point to draw from all this talk is that a new class of software will emerge to fill the business demand for enterprise mashups. What’s going to be at the center? For end users to effectively do all this mashing of disparate data sets and existing processes, they will need flexible and powerful glue.
To recap: good glue will be at the center of a new kind of “mashup spreadsheet” that will lead to reliability in end user programming and will make end user built “process mashups” a reality in the enterprise.
In Brad Meyer’s survey of end user programming, there are some encouraging numbers associated with different classes of end users (only for the US in this study) and the kinds of “programming” that they do. Here’s a rough summary: 3 million professional programmers, 12 million people in the workforce who are self-described programmers, 50 million people use spreadsheets and databases, and 90 million end users. I’m using these numbers as the basis form my upside down “end users who mash” pyramid estimates.
The glue is so important because it empowers each subset of people to stand on the shoulders of the group below them. By easily combining data and functionality provided by a more sophisticated user population (below), end users will be able to build more sophisticated applications than would be possible (or reliable) through their individual efforts.
- Programmers will build libraries of core technologies that can be used by the component builders through simple APIs to create useful business processes.
- The component builders will combine, re-wrap, and evolve the core processes into many “glue friendly” objects (services, macros, spreadsheets maybe, access to large sets of data, etc.). From this, a body of redundant objects will emerge with descriptive business process names usable by different domain experts. The component builders essentially create a DSL with the components necessary for each pocket of users residing in their domain to effectively use the core technologies. You might infer from this one aspect of the glue that I think is a requirement. Namely – the glue, besides allowing users to build mashups, should be structured enough so as to allow for the definition of new services, apis, and components derived from gluing together existing components.
- Domain experts, the mashup builders, will create useful applications using the components. These applications will further inform the development of technologies and components by the programmers and component builders who will spot the holes and inefficiencies in applications developed by end users.
- Finally, anyone can take any application and branch it into dozens of new applications by redefining the user interface and basic data constructs. These operations can hardly be considered programming, and are really more like configuration. The glue has to hold the user interface and logic together while allowing for flexibility in both. Excel taught us that: a) end users need the flexibility to change their UIs and b) glue should allow for the separation of UI and logic so as to avoid the many errors due to the intimate ties of data, processing, and presentation in spreadsheets.
Ok – How will this glue lead to the complex, custom and reliable software ‘designed’ by end users? As per the pyramid above, the glue essentially enables collaborative development through the use of shared libraries. This will bring many of the benefits present in professional programming languages to the masses. To name a few:
- Errors are reduced through the reuse of tried and true components.
- Allow for specialization (both domain and ability). Should end user programmers be expected to write an OCR component? Should OCR researchers know about subtleties in the abbreviation of healthcare data that influence the user interface of a doctors billing application? Neither is well suited to solve the other’s problems.
- The steady progress of the collective knowledge will lift the ceiling on how sophisticated end user applications can get.
Is the glue on the market yet? While Excel is the perfect “mashup” environment for single value return functionality and simple data, I don’t think it’s in the running for the mashup spreadsheet. The next Excel will be glue to mashup multidimensional data and business processes together in one place with a more robust software development architecture than spreadsheets.
From where I sit, the three critical failures of Excel in being a candidate for the mashup glue are: 1) not object oriented; 2) Excel does not deal with multidimensional data as single objects (ie- tables don’t fit into cells, and as such it is difficult to define sequences of operations on tabular data in a spreadsheet); and 3) Excel’s paradigm is limited in its ability to support a complex object model.
On the other hand, we have learned from Excel some key features that are requirements, but as of yet I haven’t seen it on the market in other end user application development environments. I think Proto represents a very good start, and with the inclusion of interfaces to additional technologies that allow programmers to extend Proto (VBA? Web services?), we might come close.
