Chapter 1 The New IT Management Paradigm
1.0.1 Looking Ten Years Forward
1.1 IT Urgencies
1.1.1 Shorcomings of present software business models
1.2 Chaos and Paralysis: Proprietary Software in Health Care
1.2.1 Costs and fees
1.3 Control
1.3.0.1 The database, as well as the code, is proprietary
1.3.0.2 Inaccessible code
1.3.1 Communication is layered
1.3.2 The Sun Sets Eventually
1.3.3 Upgrades occur on the Vendor's timetable
1.4 Vendor-centric "community" development
1.5 Efficiency and control
1.6 Exploitation
1.6.1 The Exploitation Model
1.6.1.1 Reprise: Management Challenges to which We've Grown
Accustomed
1.7 The Open Source Paradigm: A Way Out
Chapter 2 Open Source Best Practices
2.1 Competition and Cooperation
2.2 Open Source Paradigms Improve Quality
2.3 Case studies of the successful use of modular techniques
Chapter 3 A CIO's Perspective of Open Source
3.1 The Windows Equation
3.2 The Consultantocracy
3.3 The Functionality Vacuum
3.4 Bringing open source into the healthcare enterprise
Chapter 4 Recommendations
The New IT Management Paradigm
A lot of people have heard about the Linux operating system, and many know about something that's come to be called "open source software." Yet only its aficionados understand it well. I want to tell you something that even the aficionados don't usually realize: the most important strength of this software movement - to a manager - is not the reliable, powerful software it has produced. Its strength is that it brings to us a new paradigm for Information Technology management, one that enables the client firm to regain control of its software costs and its software functionality and to escape the vendor-lock trap.
Most firms use commercial software that's developed and owned by traditional software companies - the "closed source" or "proprietary source" model. Such software can be very good, and relieves the client firm from the complexity and burden of internal software development.
The management cost to a company is that in using standard proprietary commercial software, the client firm gives up control of mission-critical functionality to the vendor firm and opens itself to financial exploitation - "vendor lock."
The commercial model that has evolved around open source is fundamentally different: it parallels the service model of the medical profession: Knowledge is shared openly; but this knowledge must be expertly applied. Professionals compete based on their skill in applying this knowledge to the needs of individuals. Whether any particular professional is chosen by a client is based on a perception of competence, availability and affability.
Any sizable business, especially one in health care, is a complex system. The maintenance of this system is what we call "management." This task requires skill and effort in public and peer relations, personnel management, cost control, and process management.
To simplify, we can divide management responsibilities into various kinds of personal relationship management and function management. Our IT systems mainly are a challenge of function management. To manage functionality well, the manager must be able to control both process and cost.
The main problem with "vendor lock" is that the manager intended - in essence - to delegate IT systems development and maintenance to a contractor (the "vendor") in order to have predictable, contractually specified costs and to take advantage of expertise, but may end up losing control of both process and cost (the "lock"), because if it fires the vendor, the vendor extracts the client company's IT heart - the software. After all, the vendor owns the code and takes it back home when the contract terminates.
In the evolving world of open source commercial software, if the client must separate from the vendor, the client at least can continue operating, as it gets to keep the software, even though it may have to scramble to find adequate support.
But there are more significant benefits to using open source software than escaping vendor lock.
First, competitors can share research and development costs for common components, allowing them to focus their competitive efforts on the customization and service that creates loyal customers.
Second, it brings with it a model of collaborative development. In this scheme, developers and users work together, through debate and cooperation, to forge a system of better quality that could be done working apart. To do this well requires disciplined adherence to a set of procedures and standards for those procedures which together create a productive and high quality collaboration.
Collaborative development is the most urgent need of health care today: we have congruent needs for patient data, technological data, knowledge, and management information. But instead of working toward information systems that have interchangeable basic components, we struggle with an inefficient, expensive, medieval collection of software fiefdoms. This needs to stop. Health care institutions are under intense financial pressure while being expected to maintain world-standard quality. The only way we can make our IT systems more efficient is for institutions and commercial vendors to cooperate in making software tools and modules that share common components wherever there is common need.
Experience has already taught us that it is not enough to create common standards for coding and data interchange. This is the failure of HL-7: we have created an elaborate and all-encompassing system that is manifested chiefly in proprietary, unsharable code. This is a terrible waste of society's capital. Along with creating HL-7 standards, we need to be creating software modules that express those standards, which any client institution or vendor can freely include in their IT systems.
1.0.1 Looking Ten Years Forward
If we start working on ways to incorporate open source tools, systems, and collaborative development into our health care institutions, we are embarking on a project that will see fruition in perhaps a decade. Society cannot afford our negligence of this opportunity. We in health care have the training and understanding of what is necessary to maintain and improve the quality of our systems; it is only we who can make this happen. Congress cannot! Our vendors cannot.
This is not a proposal to sunset health care software vendors! Institutions will always find it wise to purchase IT systems and support commercially - this is, after all, simply commercial collaboration. Vendors see vendor lock as giving them security, but this is false: it ensures that the majority of commercial firms will fail, as has continually occurred, and that only the strongest will survive. It thus ensures oligopoly and encourages financial exploitation; furthermore, it removes control of development from health care institutions, who understand their responsibilities, and development is instead led by technologists, with consequent natural inefficiencies and strictures.
1.1 IT Urgencies
We are used to our present software business model. We accept its frustrations and deficiencies as natural and inevitable. If you put a frog on a stove in a pan full of cool water, and turn on the burner, the frog will not notice the gradual change and sit happily until it's cooked. We, the healthcare frogs, are so used to the current paradigms of the IT marketplace that we aren't able to step back and see how terribly dysfunctional it is - and how readily repaired.
1.1.1 Shorcomings of present software business models
Current paradigms presume, in particular, that the best and only way to manage software within commercial society is to treat it as privately owned property that must remain secret, that the only meaningful social benefit which can be provided to the copyright owner is royalties, and that the user acquires no rights to proprietary code even if the code is critical to the operation of its business.
The social problem with this model is that these standard assumptions promote financial exploitation of clients by vendors and permit them to attain managerial control of their clients. A financially stressed organization cannot long endure this.
The governance problem with this model is that the vendor that provides services to its government (or the military) is in this area in control of its government (or of the military), a reverse-power situation that will not long be endured by the government. It is this reality, that the government dog must not be wagged by the vendor tail, that is behind the move, first in Peru, and later in other countries, to require state-purchased software to be open source. If the vendor is from another country, this mandate is felt even more strongly. Thus, this movement is not specifically anti-Microsoft; it is pro-independence. That Microsoft is the chief protagonist in this debate is simply a side effect of the fact that Microsoft is aggressively exploitive of its monopoly position.
Beyond the need for management (governmental or private) to have necessary control over its own critical IT systems, it is past time for health care professionals and managers to reclaim control of development. When software only was an elaborate and useful accounting device, the control of development and standards by vendors was trivial. But now that IT systems are essential to every nuance of our enterprises, we discover that we control of the development of nothing important; our medical functionality is in the hands of non-medical technicians, whose main interest is exploiting this control financially.
Does our hospital need intensive care physiologic monitoring? We are forced to shop among vendors, each of which has emphasized priorities somewhat differently. We must accept whatever hardware platform they have chosen, whatever operating system they have developed upon. And we must, regardless of the vendor, accept that its output is not designed to be sent to any interface but the vendor's own, if any at all.
The large health care enterprise finds itself, ultimately with 250 to 400 different software products running on a collection of hardware that displays just how amazingly divergent engineering minds can bee from one another, using several operating systems. Each product requires its own maintenance. And none of these hundreds of systems is designed to interchange data with any other.
Is there something admirable about this?
Let's look in more detail at the hurt.
1.2 Chaos and Paralysis: Proprietary Software in Health Care
I'll review the current situation from three aspects:
I.Costs and fees
II.Efficiency and control
III.Exploitation (vendor lock)
1.2.1 Costs and fees. Licensing.
Commercial software carries high license fees that are not proportional to service received from the vendor. This is not an acquisition cost; it's a rental arrangement. License fees are an economic issue only to the extent that they create economic hardship for the client; they are a social issue only if they are so sizable as to represent gouging of an impoverished client by a prosperous software vendor.
Thus whether any license fee is problematic is relative to the economic situation of the vendor and client. In a strictly business sense, licensing fees have long had a rationale in providing a return on R & D costs of the vendor. Some have argued that licensing intellectual property is a monetary reward for the social good represented by the work or invention that it represents. This is a relativistic value only; any fee is determined by current business social norms and the ability of the client to pay it.
Microsoft licensing fees are especially problematic for clients because of their
complexity. Managers of small enterprises often hire a full-time employee just
to keep track of Microsoft installed software and to calculate license fees;
those in large enterprises tell of having to hire legal consultants to help
interpret the requirements and of creating small teams that only ferret out and
keep track of licensure. It is said that it is impossible to maintain 100%
compliance.
Microsoft led the formation of the Business Software Alliance, which is
interested in enforcing compliance of intellectual property laws and license fee
payments. BSA conduct is threatening to clients because it presumes all
unlicensed use to be willful and criminal. The fact is that most unlicensed use
is not with criminal intent, but that installation and licensure are not tightly
bound at the client, and there are no comprehensive tools available by which a
client can assuredly track installation and use of proprietary software. Hence
incomplete tracking is inevitable, no matter how much time and effort (money) is
put into compliance.
That the BSA considers the use of unlicensed software copies to be a criminal
offense, and that complete compliance in a company seems impossible is deeply
threatening to the managers. The risk in being caught is high because of the
severe penalties involved, because the rigorous enforcement disrupts company
functioning, the difficulty and complexity of mounting a defense, and the risk
to the manager's own career.
1.3 Control.
The loss of control over IT systems that is inherent in the proprietary paradigm
hinders management more significantly than does the mere threat of a BSA
enforcement visit. Loss of control occurs in several ways with proprietary
software. Most of this loss of management control is a side effect of this
paradigm, which permits the vendor to keep a dark wall of secrecy around every
aspect of his application.
1.3.0.1 The database, as well as the code, is proprietary.
Although the client owns its own data, the vendor owns both the database design
and the code which stores it and makes it accessible. Any business catastrophe
affecting the vendor may make the client's access to its own data expensive or
impossible. Catastrophe could be business failure of the vendor, but also loss
of key technical personnel.
More often, the client discovers that modifications to the database are
impossible or very expensive, that any change in user ergonomics, new needs for
reports or exporting data is not only expensive, but are accomplished on the
vendor's timetable, under the vendor's priorities - and with sometimes a whole
crew of intermediaries, which practically guarantees misunderstanding.
Thus the proprietary IT management model that calls for a vendor-client
relationship causes data to be inaccessible by the client except with the
permission of the vendor, which leads to interoperability frustrations when data
exchange is needed between software packages from different vendors, and leads
to expensive interfaces. This frustration can be abated with modern middleware
solutions and languages such as Perl, but this involves intercepting or
emulating I/O to or from the vendor software, not access to the vendor database
or code.
These restrictions are absent with open code; an intermediate solution is to
purchase access to vendor code if possible, which is a solution Mayo has used.
1.3.0.2 Inaccessible code
Inaccessible code delays repair of bugs and security holes.
The sine qua non of the proprietary software commercial model is secret code,
unavailable to and withheld from the client. Thus bug fixes and security
patches are completed on the vendor's timetable, within the vendor's priorities
and resources.. This may work well, or it may not; the point for management is
that this is not at under the client's control, and it is impossible to bring in
under client control except by purchasing the vendor. (Such things have been
done.)
1.3.1 Communication is layered.
>All too often, there are several layers of client and vendor staff between the
user who needs a change and the programmer who implements it. On the client
side, the employee who discovers the need for change usually doesn't have
sufficient discretionary authority to initiate a change request. There may be a
committee or a supervisor and a manager who must understand the need, describe
it, and specify the change formally; and as theses things cost money, the
expense must be estimated and approved. The person who is actually responsible
for getting the work done usually loses her place at the table in this process,
so that the change request, after being edited and analyzed managerially, may
not be closely aligned to the actual need.
Even if all goes smoothly in the client firm, they typically talk, not to the
programmer who must actually produce the code, but to a sales representative,
who has firsthand knowledge neither of the work being done nor of the coding
process. No one is less qualified to understand and accurately communicate the
need. But usually the programmers are kept sequestered, protected from contact
with end-users, ostensibly to reduce distractions that would lower programmer
productivity, but perhaps more often simply to maintain the pecking order, to
justify the existence of layers of administrative types.
Open source does not prevent this sort of common stupidity, but at least in
makes it easier to design organizational structures and procedures that avoid it
because the client owns both the database and its design. A support catastrophe
- loss of a key programmer, for example - is still a problem, but it can be
addressed directly by the client firm, which is not faced with the problem of
having no access to the code.
Similarly, byzantine communications channels can just as well exist within as
between companies, and having open source does not mean that anyone in the
client firm can actually read the code or implement change; but it does mean
that the client, if necessary, can continue to hold and use the code no matter
what happens to the vendor, and can seek competitive alternatives for support.
And if some support or development is done within the client firm, there is
always the chance that the user and programmer may meet at the water cooler and
productively shortcut layered misunderstanding.
1.3.2 The Sun Sets Eventually.
In the proprietary world, it is the vendor who decides whether an application is
worth supporting. This is not some mysterious and reprehensible commercial plot
to make client firms suffer! The vendor must survive or thrive in order to
exist. But its strength and prosperity are not closely correlated to the needs
of any particular client. Thus if the needs of the vendor and client are
dysconjugate, then the client suffers. It's the rule of commerce. However, if
the client uses open source, then it is free to recruit support of software
satisfactory to its needs from other sources if its current vendor discovers
that to continue is not profitable. The client controls its fate.
A small hospital, for example, used an accounts-receivable package that found
cost-effective and satisfactory. After a few years, its vendor was acquired by
another firm, which did not intend to continue maintaining the package: its own
strategy had been to acquire clients for its own package by buying the smaller
company, which it intended to "upgrade" to its wholly incompatible system.
The small system was not Y2K compliant, which created the opportunity to migrate
the clients that the acquiring firm wanted. About six months before the end of
1999, it sent notices to the clients using the small system that they would be
upgraded for Y2K compliance at the end of the year - at a cost of a million
dollars, plus maintenance fees that would be twice what they'd been paying; and
of course training would be available, for a fee, in the new system. And by the
way, new hardware would be required.
The clients using the small package in this case were able to temporarily rescue
themselves through legal action against the acquiring firm, forcing it to make
the product Y2K compatible and giving the clients time to bail out in other
directions. But in the long run, all the clients had to make conversions that
had not been part of their long range plans, and incurred all the expenses of
conversion on a short timetable.
All software sunsets eventually. With open source, the client can choose when
this may occur. With proprietary software, the vendor has complete control over
maintenance, upgrades, and access to data.
1.3.3 Upgrades occur on the Vendor's timetable.
In the proprietary model, the vendor determines when software is to be upgraded.
An upgrade may be welcomed; but again, the management point is that the timing
and cost of upgrades is not controlled by the client. In the situation where
the vendor is trusted, this will not be felt to be an "issue." Again, my point
is that with open source software this is never an issue; the client can use its
current software version as long as it's cost effective to do so.
In a healthy vendor-client relationship, upgrades are indeed welcome and
generally provide a welcome improvement in functionality and fix annoying but
non-critical bugs. I do not mean to imply that the proprietary model is useless
or consistently unsatisfying. And I don't mean to imply that open source
software assuredly makes for an inherently more pleasant IT experience. Here,
as everywhere, it is the execution that determines the worth of the
relationship. My point is that using open source software puts control back in
the hands of the management of the client firm.
Microsoft does not have a client-vendor relationship with any particular firm;
their software is mass-produced, not aimed at meeting any particular need. This
is not troublesome when Microsoft software works well "out of the box." The
problem specific to Microsoft is that of mandatory upgrades and the deliberate
absence of backwards compatibility, which forces a company to upgrades on
Microsoft's schedule, and forces not only the company but all its customers or
clients with which it shares documents to upgrade at the same time.
(By "absence of backwards compatibility" I mean, for example, that Word files
created with version 6 cannot be read at all by version 5, nor can version 6
read files created by Word 97. This is what can force an entire industry to
upgrade simultaneously, for example clients of US federal agencies who must read
the Word documents produced by the agencies.)
1.4 Vendor-centric "community" development.
Even when an industry's leaders realize there's a need for common standards or
procedures, in the context of proprietary software solutions, vendor interests
tend to control this process, as each vendor attempts warp the standards to its
benefit. This may have the goal of avoiding the re-writing of its own software,
or of forcing its competitors into a position of having to bear a crippling R &
D burden to meet those standards, or of simply having the prestige and power of
taking the leading role.
But standards development seldom has as its highest priorities simple
effectiveness or client-firm needs.
Health care has tremendous common software needs. We work with the same types
of clinical data, use the same pharmaceuticals, performs common services for
patients - and patients migrate from one provider to another frequently, making
interchange of data important for quality of care.
But the proprietary software model has put no priority whatever on interchanging
data between providers for the patient's benefit. Each has focused on creating
its own solution,and conducting its marketing and development with the goal of
becoming the de facto standard through monopolization.
HL-7 has been somewhat less vendor-driven but has been vendor friendly. HL-7
did not from the beginning produce software tools, and followed the proprietary
development model. As a result we have a giant, continually changing
specification, and almost no common software modules that programmers can
incorporate into their development process.
This is - how can I put this kindly - clueless. We have hundreds of brilliant,
insightful, zealous, well-trained professionals working under a heuristic that
does not have as its main goal solving the problem of connectivity, portability,
and communication that HL-7 should have aimed from the beginning to solve.
Perhaps because health care professionals must be perfectionists in order to
provide the best possible quality of care, our data systems tend to be
overwhelmingly complex. They seem to try to do "everything" rather than
identifying what's crucial and doing this well.
Because health care professionals are intelligent, responsible, and articulate,
because medical practice is - in the care of the individual patient - an art,
each practitioner evolves an work style and work habits that are idiosyncratic.
Because health care institutions tend to be somewhat inbred duchies, each
likewise evolves its own idiosyncrasies.
Unfortunately, both individual practitioners and institutions come to believe
strongly that their own habits are each the best and most effective. This is a
great hindrance to achieving software that meets common needs.
1.6 Exploitation.
"Vendor lock" is a complex and sometimes subtle collection of influences that
operationally ties the client to the vendor, making separation unduly expensive
- to the benefit of the vendor, which may exploit this financially.
1.6.1 The Exploitation Model
The history of computers in business has been largely one of exploitation of
client firms by vendors. The model to which software firms aspire is to achieve
a functional monopoly and to exploit this for maximum financial gain. Pursuit
of this model has led to a few wealthy, dominant firms and many business
failures.
Exploitation is possible because of two factors: 1.6.1.1 Reprise: Management Challenges to which We've Grown Accustomed
The result for the enterprise is --
- high software costs;
Thus the issues for the client enterprise are those of expense, management
control, efficiency, and uncertainty or threat.
1.7 The Open Source Paradigm: A Way Out
The important characteristics of open source for management are not that it has
brought us Linux distributions and usable applications, or that these are
gaining widespread acceptance and use, but that it provides a paradigm of IT
project management and a means of IT business management that is outside the
proprietary-source box - yet without requiring that any vendor close down
business; it's just a different way of doing business; as we said, it's a change
from software as "product" to software as "professional service."
How Collaborative Development has Worked
Open-source collaborative development has already produced high-quality systems
through distributed development. The GNU project showed the way, Linux
development brought into being a reliable, robust operating system to which GNU
tools could be fit, the Apache project was the first large collaboratively
developed large application, and there have been many more since. A culture of
professional support is emerging within this community.
How has this worked? Here is a description by Michael K. Johnson, a
participant in Linux development since the beginning and Red Hat Kernel Group
Manager.
Open Source Best Practices
Written from work of
2.1 Competition and Cooperation
Cooperation with competitors is possible. It is wasteful for competitors to duplicate fundamental efforts. If competitors work together to create the
infrastructure they all need, this frees resources for each of them to focus on
the service and customization that are the most important ingredients of
competitive success. Consider any basic industry, such as automobile or
furniture manufacturing. The basic design of a car or a couch is no secret;
competing companies differ in their aims to satisfy the needs of individuals who
have different needs or tastes. To the extent that competing vendors reject
cooperation, they also lose opportunity to better meet the needs of individual
customers.
When competitors wish to cooperate, a third-party organization can serve as a
forum that defuses conflict and promotes objective debate. Within the UNIX/Linux
community, these organizations include Unix vendors, Usenix, Open Group IETF,
Linux distribution builders, Linux International. Linux Standard Base (LSB). And
Xfree86.
Academics and the medical profession have been doing this for years. They
compete, but share knowledge and tools. This is also how open source functions.
This works for writing software as well as for doing research.
Consider, for example, Red Hat Linux. Competitors use it as base of their
distributions. It may surprise you to know that Red Hat takes advantage of this.
How does this work? The wide use of Red Hat's Linux distribution makes it the
"trusted base." This recognition (and potential market share) allows Red Hat to
provide more interface stability, because Red Hat can re-include competitors'
changes and benefit from their experiments. Red Hat can take advantage of its
competitors' mistakes.
Conversely, Red Hat's competitors take advantage of this situation by trying
some modifications before Red Hat does. When they make a good modification, they
get reputation and market.
End-users are advantaged by this, as competition enhances both Red Hat's
products and its competitors'. The users have choice among distribution
providers with different priorities.
Red Hat and its competitors still have proprietary processes; not all code and
tools are publicly released. Red Hat does not publish schedules ahead of time
(avoiding vaporware); they do internal development when appropriate; they don't
always publish code the instant they write it. But Red Hat follows the
open-source paradigm in that their default policy is not to distribute binaries
without source except where legally constrained.
Red Hat prefers to develop in the open: It is their default policy; otherwise
they would be just one more competitor for Microsoft to crush.
Developers, including vendors, do well to make a default policy of cooperating,
and should choose secrecy only because of well-developed arguments. In planning
open source development ignore vague fears, observe wise caution. Choose secrecy
for modules rather than projects when possible.
Opening code has technical benefits as well .
2.2 Open Source Paradigms Improve Quality.
How long does the average tech employee stay in one job? Open Source coding and
documentation techniques simply produce code that is easier to maintain by the
next employee. How many proprietary shops internally write modular, self-
documenting, clearly versioned code that's up to the open source standard? How
would it aid the task of enterprise IT management to use open source tools and
code, and use its standards and techniques internally by adopting those as well
as exploiting the availability of open source software?
1. Self-documenting code.
2. Concurrent documentation record.
3. Modular software techniques.
This has several intrinsic benefits: To use modular techniques does not mean just having multiple C++ files. It means
having shared libraries, run-time loaded libraries, and separate programs.
Strong separation forces better design. It can help cleanly separate proprietary
from public code, for example.
It also permits the developer to enjoy some of the advantages of Open Source
without giving secret research away early, and without giving away proprietary
components. It improves generalization, facilitates third party contributions,
and facilitates transfer of maintenance.
(For a formal view of open source code quality, see the Reasoning white paper, How Open Source and Commercial Software Compare: A Quantitative Analysis of TCP/IP Implementations in Commercial Software and in the Linux Kernel.)
2.3 Case studies of the successful use of modular techniques.
UNIX text filters have extreme modularization, and have been a historical
success.
GIMP plugins have very high modularization, which contributed strongly to the
GIMP's meteoric success. Simplicity encouraged third-party participation.
In the Linux kernel, loadable modules make it easy to keep personal work
private, and makes it harder to distribute binary-only.
A CIO's Perspective of Open Source
From
Mike McCoy, MD Because I'm responsible for UCLA Healthcare's entire IT structure, ahead of all
else I have to take a systems and enterprise perspective. The medical center -
and therefore its IT structure - must function all day, every day, all year.
This means that reliability and uptime is the primary goal; all else is
secondary, even financial efficiency.
There's a misconception that open source software is costless. The truth is that
open source software brings freedom to the user. Open source software is
royalty-free. But 80% of software costs are not in royalties, but in hassles.
Open source software is without royalties or significant acquisition fees; but
this low financial cost brings a social responsibility to the user: to help
foster improvement in the software, and to return to the community improvements
it makes. This is a small but important cost of free software, software libre.
Open source software liberates the user from expensive hassles and inefficiency
that encrust the vendor-driven software markets:
The traditions of this market begin with expensive "consultants" who are
constricted by their own traditions to recommend only solutions that yield for
them and their business clients rich financial rewards, without really
considering whether the solutions they recommend are in the short run
ergonomically efficient for the client health care center or in the long run
will be malleable to its evolving needs or to improving technology. The
consultant is only the thin wedge of a process of sales - consulting,
evaluation, and contracting - that seems to require a succession of expensive,
minimally functional specialists who have no accountability for the eventual
value of their proposed software "solution." for the center.
An example of contemporary hassle after installation is in regards to Microsoft
licensing: their licensing arrangements are so complex that you need consultants
to merely to figure out what you owe.
Proprietary, closed-source software involves many daily hassles in
implementation and operations. Vendors use dongles, software time bombs and
other nonsense to guarantee that use of purchased software is strictly within
licensing limits; these devices create unreliability, inefficiency, and expense
for the user.
The user is interested in service. This includes whatever helps make the
software run reliably, helps it fulfill the actual work requirements of the
user, permits reasonably-costed modifications for emerging special needs. If
you're paying for service you're paying for what you need. There's no
possibility of copyrighting or patenting service; you can't steal service from
people! You can only ask for it, and you will only continue to receive it if the
compensation for it is fair.
Vendors often waste vast tracts of time fussing about small details of
contracts, details that have nothing to do with service and much to do with
defending territorial rights to use of intellectual property that are of little
or no interest to the user. Abandon fussiness, give me service.
Vendor-driven software is not developed to meet the needs of customers.
"Functionality" is often touted, but without a prior examination of whether the
product's built-in "functionality" matches the user's needs. This would be an
easy challenge if functionality only were a matter of dollars, if the cost of
software were somehow magically linked to its functionality. But the
functionality of commercial software is miserable, and the cost is linked to
what the market will bear, compared to the other miserable "solutions."
At UCLA, we spend $20M on capital IT structure, $20M on licenses, and $20M on
software maintenance and internal development. What does that $20M in license
fees buy us? The right to possess and use software that does not make us more
efficient.
In the vendor market there isn't that much I want to buy! Quality is lacking:
functionality, reliability, malleability, responsiveness all are deficient. On
the other, hand the open source software we've used is of unbelievable
engineering quality: it's humbling simply to look at it. Meanwhile the software
code base of most major vendors is laughable. Vendors have 495 people in
marketing and 5 in research and development. Their programmers are badly
overextended; there isn't a rigorous review process; fix after fix is layered
onto the code.
3.1 The Windows Equation
Into this salad of expensive, badly conceived mediocrity is added the Windows
Equation. Empty-headed prophets got millions of sweet, ignorant people to think
that Windows Is The Future. Sadly, it will be if everyone thinks it should be.
Among vendors there is a herd mentality, all stampeding to Windows as an
"upgrade" on the server. Sadly, there are major software vendors who have
working, scalable systems on Linux who are saying, "The customers want Windows."
They end up with clumsy Windows systems that require much more hardware, more
maintenance, and function less well that Unix or Linux systems.
For example, IDX has converted its radiology information system from Unix to
Windows; on Unix it wasn't beautiful, but it was reliable, solid, inexpensive.
Now only the "Windows upgrade" is supported; at UCLA, one Unix box has been
replaced by a cluster of 20 NT servers - it's a horribly inefficient system.
Furthermore, to generate reports from it, Crystal Reports is required; this
additional package can't be run as added task on the existing 20 NT servers, it
requires a new box.
This is a continuing frustration with Windows platforms: because their operating
system design is deficient, if you deploy Windows you basically need a separate
box for every process, increasing the number of licenses you have to buy from
Microsoft and worsening electrical bills and hardware and network maintenance.
Because of these hassles, UCLA Healthcare avoids vendors that require Windows
only; but the number of vendors offering cross platform applications is
shrinking.
What health care centers need is vendors and software that deliver
functionality. Unfortunately, buying the established vendors instead of open
source software reduces short term personal risk for managers because no one
ever got fired for buying (for example) Cerner, even though it's dogged with
problems, because it's the conventional wisdom to do this.
3.2 The Consultantocracy.
A frustrating challenge is surviving the "consultantocracy:" Nothing is done
without consultants; but there's an incestuous relationship between consultants
and vendors. The consultants are unable to look beyond their dysfunctional
proprietary-software box to realize that in the open software market there are
solutions which are of vastly superior design quality, functionality, and
reliability.
We administrators need to ask ourselves: Why should software cost millions to
install?
Consultants have no interest in anything that can be installed easily;
consultants reward inefficiency because it creates more work for them! - more
billable hours.
The long-term risk to health care centers is Microsoft's P/E ratio. As
Microsoft's sales plateau, they can maintain their P/E ratio only by increasing
revenues from current business - upgrades and licenses. The consequence is that
they are ratcheting up license fees in every way imaginable. This is causing a
great fiscal burden to many companies, not only health care.
Competition among proprietary vendors is fierce. But instead of competing for
customers based on service, they are competing for total control of their
customers. As companies struggle and fail, they are going bankrupt; they are
being bought. But this is not resulting in the commodification of product that
would occur in any other industry with resulting lower costs for their
customers. Instead, the opposite: Because their grail is control and
exploitation of this control by vendors, not service, the products of acquired
companies are "sunsetted," and because the code is secret and cannot be acquired
or maintained by the customer, the new owner forces unwanted software
replacement at a cost of millions in license fees, hardware replacement, and
employee retraining. All to replace systems that were already functioning, whose
replacement was not planned or controlled by the user.
The stampede to Microsoft platforms by medical software vendors exacerbates this
cost inflation and unwittingly protects Microsoft's PE ratio; the result is that
Microsoft, secure in its control of the market, are migrating to a licensing
scheme that allows them to increase revenue seamlessly; they can dial profit up
as desired. As cash flow and new sales flatten out, users will pay dearly to
keep the OS. Microsoft's control of its market and thus its cash flow makes it
reluctant to sign a contract that goes beyond a year, Even the University of
California is not large enough to negotiate with Microsoft; "take it or leave
it" is the rule.
For UCLA Healthcare, the cost of moving to IDX on Windows was $5M. It was
working fine on Unix; UCLA had other, more pressing institutional priorities
than changing versions of medical software. But IDX was convinced that "Windows
is the future;" production and support for its product on Unix was stopped, and
upgrade to the Windows platform was required, regardless of the needs, financial
pressures, and priorities of the institutional user.
3.3 The Functionality Vacuum
There are tremendous challenges and opportunities for any firm willing to use
the resources of the open source software community to deliver quality and value
that are an order of magnitude above what is provided now by closed-source
commercial vendors.
To permit this to evolve, awareness and advocacy is essential.
Managerial leadership is necessary, for consultants not aware of open source,
and will ignorantly lead clients into inefficient and expensive closed-source
solutions.
Within academic medicine the American Association of Medical Colleges (AAMC) may
be a forum in which healthcare leaders can educate each other on the
possibilities for more cost-effective software solutions that will also put
users back in control of their IT budgets. It is possible to conceive of AAMC
forming a software collaboratory to share development of high quality medical
software.
It's important to publish best practices. For example, Clinicomp, which provides
software for bedside charting, is so far the only vendor in US to offer Linux
systems for enterprises. Clinicomp leaders are fanatical about quality and
reliability; they changed from SGI platforms purely because Linux is much more
robust for 24x7 real time ICCU monitoring. Their Linux system supports 1000
users simultaneously and runs for years non-stop on little inexpensive boxes.
Compare this with Windows systems.
3.4 Bringing open source into the healthcare enterprise.
Executives don't know what operating system is underneath their software, and
they don't care as long as it's working. IT professionals are usually allowed
considerable freedom in choosing their software tools, and so you can be a
champion of open source simply by using it.
Where can open source be used in the healthcare enterprise?
First, it is ready for use in commercial support infrastructure. There are many
open source tools that can be used to increase efficiency and reliability
without disturbing contractual relationships.
What development models do we have? Where is the ActiveState
(www.activestate.com) of healthcare applications? We need an independent,
service-based, collaborative enterprise to develop language and modeling tools
for healthcare. This does not seem to be Siemens or HBOC - but each of these
firms has the size and leverage to become this if they could discover the open
source model.
Who has a track record for true innovation? There's never been a Bell Labs of
Medical Informatics; we'd be greatly benefited if there were one, but who will
support it? We can begin by creating a "collaboratory" of open source
development among healthcare institutions.
Innovation is important, but in a pragmatic sense is overrated: Most tools are
copies of an existing design ... It's easier to design a complex system if you
have a working model. But healthcare does not have a well-designed
implementation that can be widely emulated: We don't have an EMR comparable to
UNIX, the model and standard (POSIX - http://www.opengroup.org/onlinepubs/007904975/toc.htm)
to which Linux was built.
If we want to make progress, we also have to avoid boiling the ocean. Medical
care encompasses all aspects of human distress. In providing software
"solutions" to aid our work, we try to tackle everything at once. Grandiose
projects come from this mistaken ambitiousness. The level of abstraction these
projects take on elevates them well above the level of practical effectiveness.
We spend 5 years designing an abstract model; then we discover it's too complex
to be implemented. Consider the HL-7 project and learn from it. If you want to
make a career of drawing blueprints, this is your right, but most of us are
working hard to built a basement and keep it dry and warm.
So if we want to make a difference, we need to make a little piece that does
something nice first.
Another thing that hinders achievement is "religiosity" versus secular
practicality. Being pure is admirable, but my job doesn't require purity; it
requires that the IT infrastructure of UCLA Healthcare be up and running 24 x 7.
Open source people hobble themselves by thinking it has to be all open source.
Using non-open-source, Windows pieces pragmatically is OK. We can't cripple our
enterprise by wishing an open source tool or application were available, we have
to take whatever works, even if it's expensive, mediocre, or impure.
Even though open source is built on collaborative development, frameworks and
methodologies for sharing are lacking in open source healthcare. Yet the
building supplies are out there. Examples are XML - It's extensible; you can do
a little piece first. Or Zope - you can download into an environment and find it
works. Open source solutions are marvelous for Web services.
I keep hammering on vendors to open their source. I spare no vendor. For
example, I told Cerner that UCLA didn't buy them because they don't run on
Linux. We all need to do this because it saves cost and effort (personnel time
and costs). Linux takes little support. OS/390 Linux saved us a bundle because
we can take all these Wintel boxes and put them on the mainframe. All the
disaster recovery stuff is then integrated with the mainframe backup.
A step in the right direction is Clinicomp's WINCIS frontend. Because it is
CORBA-based, you don't need the source if you have standard connectivity. This
should be standard practice.
What can the government do? There is an appropriate role of federal government
in encouraging the dissemination of functionality. Yet right now public funds
are put into the development of closed source programs, public money
disappearing into proprietary systems and subsequently benefiting the public not
at all - and sometimes not even the vendor unless the vendor can find a way to
create and market a commercial application.
As a general principle, government funding of vendors development should add
value to government-funded basic tools and paradigms that are then openly
available to the healthcare community.
The entire University of California means nothing to Microsoft financially, so
we have no bargaining position. We hope to make UCLA independent of Microsoft
so that we can make business decisions. Linux systems run for years non-stop on
nice little boxes. I can't imagine what a 1000 bed hospital would look like on
Windows. Sometimes the CIO's don't know what OS their systems are running on.
How best can we make progress? Medical open source developers should focus on
little useful tools rather than boiling the ocean. Managers can be open source
advocates simply by using open source. It's not necessary to develop open
source to support it. Just use it.
When UCLA Healthcare is all open source, that is going to be a powerful message.
Recommendations
return to contents
What management steps can be taken to make use of the strengths of Open Source?
I: Employ Open Source software wherever it makes practical sense for your enterpise. To do this effectively, it's necessary to learn what resources that are available. This may require consulting your own technical staff or making use of technical consultants who are expert in open source resources.
II: Seek platform and application independence within the enterprise to avoid unexpected vendor entrapment.
III: Work legally toward fail-safe mechanisms against unwanted sunsetting or
business failure.
IV: Solicit support contracts from professional open source firms to encourage
the growth of the professional commercial software model (i.e., firms that are
not tied to specific applications).
V: Support industry-wide basic applications
VI: Migrate to open source platforms and applications as a long-range plan for
your IT system.
VII: Participate as founders in a medical open-source initiative with other
groups.such as AAMC, AAFP, WorldVistA, OSHCA, OpenEHR, EU-Spirit, and others as they emerge and we discover them,with the following goals:
1.The complexity of IT systems in the enterprise makes migration expensive
2.The rigid protections on intellectual property under current law require all-
or-none conversions and give complete control to the vendor, which is permitted
never to disclose its source code.
- more importantly, loss of control of several key management parameters:
* Cost control: vendors dictate license and support fees.
* Ergonomics: vendors dictate user interfaces and connectivity
capabilities.
* Personnel: the vendor of course hires its own developers.
* Planning: vendors dictate
- upgrades,
- sunsetting of products,
- hardware re-capitalization.
- The possibility of vendor business failure threatens basic enterprise
functionality.
- A threat on Microsoft platforms is of BSA (Business Software Alliance)
enforcement actions, a concern because the complexity of Microsoft
licensing schema makes perfect compliance impossible and because
enforcement actions shut down normal functioning of the client enterprise
during "reconciliation" and involve expensive penalties.
Michael K. Johnson
Manager, Kernel Technology
Red Hat, Inc.
return to contents
- Interface stability,
- Debugability, and
- Maintainability.
CIO, UCLA Healthcare
November, 2002
return to contents
A: Infrastructure. Linux is a reliable, robust operating system. GNU/Linux tools are excellent for web pages servers, file and print service, communicatiosn, email, and other IT infrastructure tasks because of their reliability, design excellence, and resistance to security problems.
B: Applications. Work with vendors to port current UNIX applications to Linux, in order to reduce license fees. Study open source application resources to determine whether they are adequate for our needs.
A: Require internal documents and communications to be in non-proprietary or
universally supported formats.
B: Work toward removing our dependency on Windows where licenses are not nnecessary (e.g., smart terminals, email, simple word processing, most spreadsheets, "desktop" software).
C: Pressure vendors to use Linux platforms, and not port applications
working well on other platforms to Windows.
D: Pressure vendors to provide platform-independent or X-windows frontends.
A: Contractually require proprietary code to be provided with compiled
software, under NDA, as part of contracts - with limited ownership rights if the
vendor fails or is absorbed into another firm.
B: Require new software acquisition projects to survey for open source
applications, include the option of contracted or in-house adaptation of extant
software that is not exactly suitable.
A: MPI (master patient index)
B: Seek support models for universally-needed functionality through mechanisms
other than licensing (e.g. CPT codes)
C: Support open source HL-7 modules using software tools that are open.
D: HIPAA security application plugins.
1: Pragmatism: Create a usable software project to implement every cooperative
project undertaken: code should be an integral part and necessary outcome of
every joint project, whether for standards or implementation.
2: Cake before frosting: Identify common essentials through debate; aim at
prioritizing desirable non-essentials for subsequent work
.
3: Follow the discipline of collaborative concurrent documentation
.
4: Develop and respect leadership
a: technical leadership
b: medical leaders
c: maintainers
Copyright (C) 2003, Daniel L. Johnson, MD All rights reserved.