What Software Architects can learn from Farmers

In the old days (or even today in many parts of the world including India) , they used oxen for heavy pulling, and when one oxen will not be able to pull the budge, they would add one more and not worry about creating a bigger and better oxen. Thats the fundamental concept of distributed computing - we shouldn't try for bigger computers but more systems of computers.

Don Knuth and The Art of Computer Programming

"Art of Computer Programming" volumes were named among the best twelve physical-science monographs of the century by American Scientist, along with: Dirac on quantum mechanics, Einstein on relativity, Mandelbrot on fractals, Pauling on the chemical bond, Russell and Whitehead on foundations of mathematics, von Neumann and Morgenstern on game theory, Wiener on cybernetics, Woodward and Hoffmann on orbital symmetry, Feynman on quantum electrodynamics, Smith on the search for structure, and Einstein's collected papers.

Code of Ethics for Software Architects

The ancient Romans had a tradition: whenever one of their engineers constructed an arch, as the capstone was hoisted into place, the engineer assumed accountability for his work in the most profound way possible: he stood under the arch.

As Bjarne Stroustrup (inventor of the C++ language) observed - "Our civilization runs on software". Whether its the mobile phone or the aeroplane, cyberknife or heart monitor, planning a vacation or buying books, connecting with friends or communicating critical information, software has become all pervasive in our life. It took radio 30 years to reach an audience of 50 million whereas Facebook took 2.

In that light, Morality and Code of Ethics for Software Engineers are becoming more and more important. Some commendable work has been done so far by Don Gotterbarn, Keith Miller and Simon Rogerson in "Software Engineering Code of Ethics". But the challenge remains on how we enforce that and make engineers and architects put their 'skin in the game'.

The Human Side of Software Architecture

Intelligent System to Help Autistic Children Recognize Emotions

Computer scientists from Nanyang Technological University in Singapore are working on the development of an efficient and intelligent facial expression recognition system. The system is capable of locating the face region using derivative-based filtering and recognizing facial expressions using boosting classifier. The portable device is being developed to help autistic children understand the emotions of surrounding people. A paper detailing the specifics of the device will be published in the journalIntelligent Decision Technologies (Volume 3:3).

Teik-Toe Teoh, Yok-Yen Nguwi and Siu-Yeung Cho of the Centre for Computational Intelligence of the School of Computer Engineering of Nanyang Technological University state that “emotion is a state of feeling involving thoughts, physiological changes, and an outward expression. In this paper, we propose a system that synergizes the use of derivative filtering and boosting classifier. “

The portable facial expression recognizer locates the edge of the human face through Gaussian derivatives, Laplacian derivatives and filter out non-face images using Adaboost. Secondly, the feature locator finds crucial fiducial points for subsequent feature extraction and selection processing. Finally, the meaningful features are classified into the corresponding classes.

http://iospress.metapress.com/content/h4135763ut15/?p=46a3405b686541e8882bde9186f584e2π=1

'Stay Hungry. Stay Foolish'

This is the text of the Commencement address by Steve Jobs, CEO of Apple Computer and of Pixar Animation Studios, delivered on June 12, 2005.

I am honored to be with you today at your commencement from one of the finest universities in the world. I never graduated from college. Truth be told, this is the closest I've ever gotten to a college graduation. Today I want to tell you three stories from my life. That's it. No big deal. Just three stories.

The first story is about connecting the dots.

I dropped out of Reed College after the first 6 months, but then stayed around as a drop-in for another 18 months or so before I really quit. So why did I drop out?

It started before I was born. My biological mother was a young, unwed college graduate student, and she decided to put me up for adoption. She felt very strongly that I should be adopted by college graduates, so everything was all set for me to be adopted at birth by a lawyer and his wife. Except that when I popped out they decided at the last minute that they really wanted a girl. So my parents, who were on a waiting list, got a call in the middle of the night asking: "We have an unexpected baby boy; do you want him?" They said: "Of course." My biological mother later found out that my mother had never graduated from college and that my father had never graduated from high school. She refused to sign the final adoption papers. She only relented a few months later when my parents promised that I would someday go to college.

And 17 years later I did go to college. But I naively chose a college that was almost as expensive as Stanford, and all of my working-class parents' savings were being spent on my college tuition. After six months, I couldn't see the value in it. I had no idea what I wanted to do with my life and no idea how college was going to help me figure it out. And here I was spending all of the money my parents had saved their entire life. So I decided to drop out and trust that it would all work out OK. It was pretty scary at the time, but looking back it was one of the best decisions I ever made. The minute I dropped out I could stop taking the required classes that didn't interest me, and begin dropping in on the ones that looked interesting.

It wasn't all romantic. I didn't have a dorm room, so I slept on the floor in friends' rooms, I returned coke bottles for the 5¢ deposits to buy food with, and I would walk the 7 miles across town every Sunday night to get one good meal a week at the Hare Krishna temple. I loved it. And much of what I stumbled into by following my curiosity and intuition turned out to be priceless later on. Let me give you one example:

Reed College at that time offered perhaps the best calligraphy instruction in the country. Throughout the campus every poster, every label on every drawer, was beautifully hand calligraphed. Because I had dropped out and didn't have to take the normal classes, I decided to take a calligraphy class to learn how to do this. I learned about serif and san serif typefaces, about varying the amount of space between different letter combinations, about what makes great typography great. It was beautiful, historical, artistically subtle in a way that science can't capture, and I found it fascinating.

None of this had even a hope of any practical application in my life. But ten years later, when we were designing the first Macintosh computer, it all came back to me. And we designed it all into the Mac. It was the first computer with beautiful typography. If I had never dropped in on that single course in college, the Mac would have never had multiple typefaces or proportionally spaced fonts. And since Windows just copied the Mac, its likely that no personal computer would have them. If I had never dropped out, I would have never dropped in on this calligraphy class, and personal computers might not have the wonderful typography that they do. Of course it was impossible to connect the dots looking forward when I was in college. But it was very, very clear looking backwards ten years later.

Again, you can't connect the dots looking forward; you can only connect them looking backwards. So you have to trust that the dots will somehow connect in your future. You have to trust in something — your gut, destiny, life, karma, whatever. This approach has never let me down, and it has made all the difference in my life.

My second story is about love and loss.

I was lucky — I found what I loved to do early in life. Woz and I started Apple in my parents garage when I was 20. We worked hard, and in 10 years Apple had grown from just the two of us in a garage into a $2 billion company with over 4000 employees. We had just released our finest creation — the Macintosh — a year earlier, and I had just turned 30. And then I got fired. How can you get fired from a company you started? Well, as Apple grew we hired someone who I thought was very talented to run the company with me, and for the first year or so things went well. But then our visions of the future began to diverge and eventually we had a falling out. When we did, our Board of Directors sided with him. So at 30 I was out. And very publicly out. What had been the focus of my entire adult life was gone, and it was devastating.

I really didn't know what to do for a few months. I felt that I had let the previous generation of entrepreneurs down - that I had dropped the baton as it was being passed to me. I met with David Packard and Bob Noyce and tried to apologize for screwing up so badly. I was a very public failure, and I even thought about running away from the valley. But something slowly began to dawn on me — I still loved what I did. The turn of events at Apple had not changed that one bit. I had been rejected, but I was still in love. And so I decided to start over.

I didn't see it then, but it turned out that getting fired from Apple was the best thing that could have ever happened to me. The heaviness of being successful was replaced by the lightness of being a beginner again, less sure about everything. It freed me to enter one of the most creative periods of my life.

During the next five years, I started a company named NeXT, another company named Pixar, and fell in love with an amazing woman who would become my wife. Pixar went on to create the worlds first computer animated feature film, Toy Story, and is now the most successful animation studio in the world. In a remarkable turn of events, Apple bought NeXT, I returned to Apple, and the technology we developed at NeXT is at the heart of Apple's current renaissance. And Laurene and I have a wonderful family together.

I'm pretty sure none of this would have happened if I hadn't been fired from Apple. It was awful tasting medicine, but I guess the patient needed it. Sometimes life hits you in the head with a brick. Don't lose faith. I'm convinced that the only thing that kept me going was that I loved what I did. You've got to find what you love. And that is as true for your work as it is for your lovers. Your work is going to fill a large part of your life, and the only way to be truly satisfied is to do what you believe is great work. And the only way to do great work is to love what you do. If you haven't found it yet, keep looking. Don't settle. As with all matters of the heart, you'll know when you find it. And, like any great relationship, it just gets better and better as the years roll on. So keep looking until you find it. Don't settle.

My third story is about death.

When I was 17, I read a quote that went something like: "If you live each day as if it was your last, someday you'll most certainly be right." It made an impression on me, and since then, for the past 33 years, I have looked in the mirror every morning and asked myself: "If today were the last day of my life, would I want to do what I am about to do today?" And whenever the answer has been "No" for too many days in a row, I know I need to change something.

Remembering that I'll be dead soon is the most important tool I've ever encountered to help me make the big choices in life. Because almost everything — all external expectations, all pride, all fear of embarrassment or failure - these things just fall away in the face of death, leaving only what is truly important. Remembering that you are going to die is the best way I know to avoid the trap of thinking you have something to lose. You are already naked. There is no reason not to follow your heart.

About a year ago I was diagnosed with cancer. I had a scan at 7:30 in the morning, and it clearly showed a tumor on my pancreas. I didn't even know what a pancreas was. The doctors told me this was almost certainly a type of cancer that is incurable, and that I should expect to live no longer than three to six months. My doctor advised me to go home and get my affairs in order, which is doctor's code for prepare to die. It means to try to tell your kids everything you thought you'd have the next 10 years to tell them in just a few months. It means to make sure everything is buttoned up so that it will be as easy as possible for your family. It means to say your goodbyes.

I lived with that diagnosis all day. Later that evening I had a biopsy, where they stuck an endoscope down my throat, through my stomach and into my intestines, put a needle into my pancreas and got a few cells from the tumor. I was sedated, but my wife, who was there, told me that when they viewed the cells under a microscope the doctors started crying because it turned out to be a very rare form of pancreatic cancer that is curable with surgery. I had the surgery and I'm fine now.

This was the closest I've been to facing death, and I hope its the closest I get for a few more decades. Having lived through it, I can now say this to you with a bit more certainty than when death was a useful but purely intellectual concept:

No one wants to die. Even people who want to go to heaven don't want to die to get there. And yet death is the destination we all share. No one has ever escaped it. And that is as it should be, because Death is very likely the single best invention of Life. It is Life's change agent. It clears out the old to make way for the new. Right now the new is you, but someday not too long from now, you will gradually become the old and be cleared away. Sorry to be so dramatic, but it is quite true.

Your time is limited, so don't waste it living someone else's life. Don't be trapped by dogma — which is living with the results of other people's thinking. Don't let the noise of others' opinions drown out your own inner voice. And most important, have the courage to follow your heart and intuition. They somehow already know what you truly want to become. Everything else is secondary.

When I was young, there was an amazing publication called The Whole Earth Catalog, which was one of the bibles of my generation. It was created by a fellow named Stewart Brand not far from here in Menlo Park, and he brought it to life with his poetic touch. This was in the late 1960's, before personal computers and desktop publishing, so it was all made with typewriters, scissors, and polaroid cameras. It was sort of like Google in paperback form, 35 years before Google came along: it was idealistic, and overflowing with neat tools and great notions.

Stewart and his team put out several issues of The Whole Earth Catalog, and then when it had run its course, they put out a final issue. It was the mid-1970s, and I was your age. On the back cover of their final issue was a photograph of an early morning country road, the kind you might find yourself hitchhiking on if you were so adventurous. Beneath it were the words: "Stay Hungry. Stay Foolish." It was their farewell message as they signed off. Stay Hungry. Stay Foolish. And I have always wished that for myself. And now, as you graduate to begin anew, I wish that for you.

Stay Hungry. Stay Foolish.

Thank you all very much.

The Laws of Simplicity by John Maeda

Ten Laws of Simplicity

The Next Perfect IT Storm

by Dr Colin Boswell
Cloud computing and SaaS are gaining real market traction, especially with the likes of Google and Amazon pouring so much resource into on-demand or utility computing. But what does this mean for the humble datacentre? IBRS analyst Dr Colin Boswell predicts heavy weather ahead.
A perfect IT storm is looming, driven by merging category four storms such as Utility (or cloud) computing and the Red Shift growth in massive computing. The force of the storm will be exacerbated by rising energy costs and their impact on the datacentre energy budget. As a consequence, in a few years many mid to large organisations have at least all their non-differentiating applications running on remote shared SaaS-like sites. This will have a significant impact on the IT department and it’s CIO.
The first use of the term ‘the perfect IT storm” was in the nineties when Y2K, ERP, the euro transition and the internet hype unleashed almost unconstrained energy in the IT world. We are now entering a phase where at least two major developments could create a new era of turbulence in IT.
Utility computing, where vendors make remote IT resources (CPU power, storage space, etc) available to be shared and used by clients on an as-needed basis and paid for on a unit cost basis. Think SaaS.
Red Shift organisations (those whose use of computing power is growing at a rate greater than Moore’s Law) are making their massive computing resources available to all, potentially obviating the need for organisations to own and control very large and expensive datacentres. Data processing as a utility
At the same time as red-shift organisations (see panel) are emerging, the demands of powering, cooling and maintaining racks of servers in conventional datacentre are stretching the IT resources of most organisations beyond capacity. Sun’s Papadopoulos uses an energy-utility metaphor to solve this problem: just as organisations moved from generating their own electricity to purchasing it off a grid, he suggests that computing should be a utility, with organisations purchasing their data processing services off the IT equivalent of an ‘electricity grid’ instead of having their own individual datacentres.
In his book, The Big Switch (1) Nicholas Carr also expands on the concept of utility computing. He reports that businesses are switching from in-house IT departments to network services and that vendors are already moving to supply these services. For example:
Amazon is offering its Elastic Compute Cloud (EC2) and S3 (Simple Storage Service) as part of its web services (2).
Google is reportedly building a series of massive datacentres (which will incorporate container computer systems). It is expected to use these to offer ‘in the cloud’ computing services to businesses.
Sun offers a similar service through its Sun Grid Compute Utility.
Continuing the trend, Google has released another new product: the Google App Engine uses Google’s systems to allow organisations to make their applications available worldwide. They can produce their own web applications code, upload it to site and run it on the Google infrastructure, independent of their own systems.
Utility computing has been talked about for years; Oracle, IBM, HP and Sun have been promoting it since early this century. What is different now is the new emphasis on web-based access. It appears that the EC2 and S3 offerings have been somewhat successful, but few clients have taken up Sun’s to date.
Some organisations are making the transition to web-based utilities, but most are still buying their own computing equipment, their own software and hiring legions of IT personnel. Carr argues that all this will change once everyone moves to the computing grid. Computing, he claims, is now a commodity, like electricity was at the beginning of the last century. It is no longer cost effective for companies to try and differentiate themselves by doing all their IT services in-house when most of the services they need are available on the internet.
Implications of utility computing
The concept of utility computing is not new. Many organisations have moved their applications from the client-server model to a web-based model. The move to a utility model is logical. However, if it happens at the predicted scale and pace the impact will be significant. Some IT companies will prosper and others will suffer or become irrelevant.

Companies like Microsoft and Intel could be losers since their hardware and software markets will be drastically reduced when the client server model becomes obsolete. The new utilities will prosper as they will offer almost all the software applications that individual organisations previously needed to host internally. On the other hand, legacy applications are likely to continue if the cost of replacing them is too high and there is a high associated risk.
As organisations move to the utility model their IT departments will be drastically downsized and their IT expenditure will be significantly different. For example, if the organisation can pay a service provider for the IT resources as it uses them then it may be able to free up significant amounts of cash buried in datacentre resources.
The big issues that need addressing before utility computing can be used to support any organisations’ key business activities are around continuity of service, security (3) and service levels. The Red Shift
At the end of 2006 Sun Microsystems' CTO, Greg Papadopoulos advanced his Red Shift Theory for IT. It claims that an elite group of companies are consuming inordinate amounts of IT infrastructure, well beyond most other businesses, and that their demand is growing exponentially. Papadopoulos claimed that this has implications not just for IT's most insatiable consumers, but for the structure of the computing industry itself. He argues that Red-Shift companies will enjoy exponential business growth in the coming years. On the other hand, he claims that Blue-Shift companies — those whose processing needs aren't exploding — will grow at about the same rate as GDP.
Red Shift implications
If the Red Shift is valid and sustained, and the world collectively moves to SaaS on ‘big grunters’ at a Google or Amazon or other vendor’s server farms there will be less and less customers for products from traditional hardware and software suppliers such as IBM, HP, Sun, SAP, Oracle and so on. The hardware trend could be exacerbated if many power users copy Amazon and Google and move to unbranded commodity systems (‘white ware’). If this migration has the effect of reducing the product offerings of the traditional hardware suppliers, the move to SaaS by smaller organisations will be reinforced. Red shifts in the software arena will see a decrease in demand for systems-oriented specialists and a switch to an emphasis on business analysis.
Next steps
If this IT storm is as powerful as some predict and happens soon, it will catch many IT departments by surprise and they will be unprepared for the ensuing dramatic changes. To be prepared:
assess your application inventory and identify those that could be handled by a computing utility.
take one that is not core (so that key operations won’t be seriously affected if it malfunctions) and migrate it to a utility environment.
before starting the migration set your success criteria.
after migrating review against the success criteria.
aking advantage of the lessons learned, decide whether to repeat the process, but continue building skills and preparing for the ‘Big One’.
(1) Nicholas Carr The Big Switch: Rewiring the World, from Edison to Google, W. W. Norton (January 7, 2008)
(2) This site lists nearly 40 applications running on EC2. This one lists nearly 100 publicly available images running on EC2
(3) For example: If an organisation’s data is stored on the same servers as its rivals, and there’s some sort of computer glitch – what controls are in place to prevent unauthorised access? A few super-utilities servicing hundreds/thousands of organisations becomes attractive targets for hackers.

Performance Engineering & Enhancement Offering from Infosys

Its a giant leap forward to ensure that Projects are delivered on time, on budget and without performance and scalability issues. By the way, why do we need Performance Engineering any way? Very interestingly, with some 50-60 years of history, software development has still not reached a level of maturity where performance should be considered a bottomline before delivering software. We are still too obsessed with the functionalities and the look-and-feel and do not worry about the performance issues - even though we all know that at the end of the day, if the application or the product is not performing, it will be of very little use. Business will lose millions of dollars (there are umpteen cases to justify that), client relationships will get affected, credibility of the development team will be at stake - in spite of all these consequences, how many development teams focus on Performance and Scalability?
The Infosys offering is very unique that ways and it has a goal, when achieved, will change the way people think and develop software.
The Performance Engineering and Enhancement Solution Offering with its Proactive (because you start focussing on Performance right from the Requirements phase) & Holistic (because someone has to take a bottomline responsibility and the client will not have to run to multiple vendors) Performance Driven Development approach is a pathbreaking way of thinking and approaching in what was till date a disjointed solution space.
http://www.infosys.com/IT-services/architecture-services/service-offerings/performance-engineering.asp
More on this later......

Google and the Wisdom of Clouds

A lofty new strategy aims to put incredible computing power in the hands of many
by Stephen Baker (from Businessweek)

One simple question. That's all it took for Christophe Bisciglia to bewilder confident job applicants at Google (GOOG). Bisciglia, an angular 27-year-old senior software engineer with long wavy hair, wanted to see if these undergrads were ready to think like Googlers. "Tell me," he'd say, "what would you do if you had 1,000 times more data?"
What a strange idea. If they returned to their school projects and were foolish enough to cram formulas with a thousand times more details about shopping or maps or—heaven forbid—with video files, they'd slow their college servers to a crawl.
At that point in the interview, Bisciglia would explain his question. To thrive at Google, he told them, they would have to learn to work—and to dream—on a vastly larger scale. He described Google's globe-spanning network of computers. Yes, they answered search queries instantly. But together they also blitzed through mountains of data, looking for answers or intelligence faster than any machine on earth. Most of this hardware wasn't on the Google campus. It was just out there, somewhere on earth, whirring away in big refrigerated data centers. Folks at Google called it "the cloud." And one challenge of programming at Google was to leverage that cloud—to push it to do things that would overwhelm lesser machines. New hires at Google, Bisciglia says, usually take a few months to get used to this scale. "Then one day, you see someone suggest a wild job that needs a few thousand machines, and you say: Hey, he gets it.'"
What recruits needed, Bisciglia eventually decided, was advance training. So one autumn day a year ago, when he ran into Google CEO Eric E. Schmidt between meetings, he floated an idea. He would use his 20% time, the allotment Googlers have for independent projects, to launch a course. It would introduce students at his alma mater, the University of Washington, to programming at the scale of a cloud. Call it Google 101. Schmidt liked the plan. Over the following months, Bisciglia's Google 101 would evolve and grow. It would eventually lead to an ambitious partnership with IBM (IBM), announced in October, to plug universities around the world into Google-like computing clouds.
As this concept spreads, it promises to expand Google's footprint in industry far beyond search, media, and advertising, leading the giant into scientific research and perhaps into new businesses. In the process Google could become, in a sense, the world's primary computer.
"I had originally thought [Bisciglia] was going to work on education, which was fine," Schmidt says late one recent afternoon at Google headquarters. "Nine months later, he comes out with this new [cloud] strategy, which was completely unexpected." The idea, as it developed, was to deliver to students, researchers, and entrepreneurs the immense power of Google-style computing, either via Google's machines or others offering the same service.
What is Google's cloud? It's a network made of hundreds of thousands, or by some estimates 1 million, cheap servers, each not much more powerful than the PCs we have in our homes. It stores staggering amounts of data, including numerous copies of the World Wide Web. This makes search faster, helping ferret out answers to billions of queries in a fraction of a second. Unlike many traditional supercomputers, Google's system never ages. When its individual pieces die, usually after about three years, engineers pluck them out and replace them with new, faster boxes. This means the cloud regenerates as it grows, almost like a living thing.
A move towards clouds signals a fundamental shift in how we handle information. At the most basic level, it's the computing equivalent of the evolution in electricity a century ago when farms and businesses shut down their own generators and bought power instead from efficient industrial utilities. Google executives had long envisioned and prepared for this change. Cloud computing, with Google's machinery at the very center, fit neatly into the company's grand vision, established a decade ago by founders Sergey Brin and Larry Page: "to organize the world's information and make it universally accessible." Bisciglia's idea opened a pathway toward this future. "Maybe he had it in his brain and didn't tell me," Schmidt says. "I didn't realize he was going to try to change the way computer scientists thought about computing. That's a much more ambitious goal."
ONE-WAY STREET
For small companies and entrepreneurs, clouds mean opportunity—a leveling of the playing field in the most data-intensive forms of computing. To date, only a select group of cloud-wielding Internet giants has had the resources to scoop up huge masses of information and build businesses upon it. Our words, pictures, clicks, and searches are the raw material for this industry. But it has been largely a one-way street. Humanity emits the data, and a handful of companies—the likes of Google, Yahoo! (YHOO), or Amazon.com (AMZN)—transform the info into insights, services, and, ultimately, revenue.
This status quo is already starting to change. In the past year, Amazon has opened up its own networks of computers to paying customers, initiating new players, large and small, to cloud computing. Some users simply park their massive databases with Amazon. Others use Amazon's computers to mine data or create Web services. In November, Yahoo opened up a cluster of computers—a small cloud—for researchers at Carnegie Mellon University. And Microsoft (MSFT) has deepened its ties to communities of scientific researchers by providing them access to its own server farms. As these clouds grow, says Frank Gens, senior analyst at market research firm IDC, "A whole new community of Web startups will have access to these machines. It's like they're planting Google seeds." Many such startups will emerge in science and medicine, as data-crunching laboratories searching for new materials and drugs set up shop in the clouds.
For clouds to reach their potential, they should be nearly as easy to program and navigate as the Web. This, say analysts, should open up growing markets for cloud search and software tools—a natural business for Google and its competitors. Schmidt won't say how much of its own capacity Google will offer to outsiders, or under what conditions or at what prices. "Typically, we like to start with free," he says, adding that power users "should probably bear some of the costs." And how big will these clouds grow? "There's no limit," Schmidt says. As this strategy unfolds, more people are starting to see that Google is poised to become a dominant force in the next stage of computing. "Google aspires to be a large portion of the cloud, or a cloud that you would interact with every day," the CEO says. The business plan? For now, Google remains rooted in its core business, which gushes with advertising revenue. The cloud initiative is barely a blip in terms of investment. It hovers in the distance, large and hazy and still hard to piece together, but bristling with possibilities.
Changing the nature of computing and scientific research wasn't at the top of Bisciglia's agenda the day he collared Schmidt. What he really wanted, he says, was to go back to school. Unlike many of his colleagues at Google, a place teeming with PhDs, Bisciglia was snatched up by the company as soon as he graduated from the University of Washington, or U-Dub, as nearly everyone calls it. He'd never been a grad student. He ached for a break from his daily routines at Google—the 10-hour workdays building search algorithms in his cube in Building 44, the long commutes on Google buses from the apartment he shared with three roomies in San Francisco's Duboce Triangle. He wanted to return to Seattle, if only for one day a week, and work with his professor and mentor, Ed Lazowska. "I had an itch to teach," he says.
He didn't think twice before vaulting over the org chart and batting around his idea directly with the CEO. Bisciglia and Schmidt had known each other for years. Shortly after landing at Google five years ago as a 22-year-old programmer, Bisciglia worked in a cube across from the CEO's office. He'd wander in, he says, drawn in part by the model airplanes that reminded him of his mother's work as a United Airlines (UAUA) hostess. Naturally he talked with the soft-spoken, professorial CEO about computing. It was almost like college. And even after Bisciglia moved to other buildings, the two stayed in touch. ("He's never too hard to track down, and he's incredible about returning e-mails," Bisciglia says.)
On the day they first discussed Google 101, Schmidt offered one nugget of advice: Narrow down the project to something Bisciglia could have up and running in two months. "I actually didn't care what he did," Schmidt recalls. But he wanted the young engineer to get feedback in a hurry. Even if Bisciglia failed, he says, "he's smart, and he'd learn from it."
To launch Google 101, Bisciglia had to replicate the dynamics and a bit of the magic of Google's cloud—but without tapping into the cloud itself or revealing its deepest secrets. These secrets fuel endless speculation among computer scientists. But Google keeps much under cover. This immense computer, after all, runs the company. It automatically handles search, places ads, churns through e-mails. The computer does the work, and thousands of Google engineers, including Bisciglia, merely service the machine. They teach the system new tricks or find new markets for it to invade. And they add on new clusters—four new data centers this year alone, at an average cost of $600 million apiece.
In building this machine, Google, so famous for search, is poised to take on a new role in the computer industry. Not so many years ago scientists and researchers looked to national laboratories for the cutting-edge research on computing. Now, says Daniel Frye, vice-president of open systems development at IBM, "Google is doing the work that 10 years ago would have gone on in a national lab."
How was Bisciglia going to give students access to this machine? The easiest option would have been to plug his class directly into the Google computer. But the company wasn't about to let students loose in a machine loaded with proprietary software, brimming with personal data, and running a $10.6 billion business. So Bisciglia shopped for an affordable cluster of 40 computers. He placed the order, then set about figuring out how to pay for the servers. While the vendor was wiring the computers together, Bisciglia alerted a couple of Google managers that a bill was coming. Then he "kind of sent the expense report up the chain, and no one said no." He adds one of his favorite sayings: "It's far easier to beg for forgiveness than to ask for permission." ("If you're interested in someone who strictly follows the rules, Christophe's not your guy," says Lazowska, who refers to the cluster as "a gift from heaven.")
A FRENETIC LEARNER
On Nov. 10, 2006, the rack of computers appeared at U-Dub's Computer Science building. Bisciglia and a couple of tech administrators had to figure out how to hoist the 1-ton rack up four stories into the server room. They eventually made it, and then prepared for the start of classes, in January.
Bisciglia's mother, Brenda, says her son seemed marked for an unusual path from the start. He didn't speak until age 2, and then started with sentences. One of his first came as they were driving near their home in Gig Harbor, Wash. A bug flew in the open window, and a voice came from the car seat in back: "Mommy, there's something artificial in my mouth."
At school, the boy's endless questions and frenetic learning pace exasperated teachers. His parents, seeing him sad and frustrated, pulled him out and home-schooled him for three years. Bisciglia says he missed the company of kids during that time but developed as an entrepreneur. He had a passion for Icelandic horses and as an adolescent went into business raising them. Once, says his father, Jim, they drove far north into Manitoba and bought horses, without much idea about how to transport the animals back home. "The whole trip was like a scene from one of Chevy Chase's movies," he says. Christophe learned about computers developing Web pages for his horse sales and his father's luxury-cruise business. And after concluding that computers promised a brighter future than animal husbandry, he went off to U-Dub and signed up for as many math, physics, and computer courses as he could.
In late 2006, as he shuttled between the Googleplex and Seattle preparing for Google 101, Bisciglia used his entrepreneurial skills to piece together a sprawling team of volunteers. He worked with college interns to develop the curriculum, and he dragooned a couple of Google colleagues from the nearby Kirkland (Wash.) facility to use some of their 20% time to help him teach it. Following Schmidt's advice, Bisciglia worked to focus Google 101 on something students could learn quickly. "I was like, what's the one thing I could teach them in two months that would be useful and really important?" he recalls. His answer was "MapReduce."
Bisciglia adores MapReduce, the software at the heart of Google computing. While the company's famous search algorithms provide the intelligence for each search, MapReduce delivers the speed and industrial heft. It divides each task into hundreds, or even thousands, of tasks, and distributes them to legions of computers. In a fraction of a second, as each one comes back with its nugget of information, MapReduce quickly assembles the responses into an answer. Other programs do the same job. But MapReduce is faster and appears able to handle near limitless work. When the subject comes up, Bisciglia rhapsodizes. "I remember graduating, coming to Google, learning about MapReduce, and really just changing the way I thought about computer science and everything," he says. He calls it "a very simple, elegant model." It was developed by another Washington alumnus, Jeffrey Dean. By returning to U-Dub and teaching MapReduce, Bisciglia would be returning this software "and this way of thinking" back to its roots.
There was only one obstacle. MapReduce was anchored securely inside Google's machine—and it was not for outside consumption, even if the subject was Google 101. The company did share some information about it, though, to feed an open-source version of MapReduce called Hadoop. The idea was that, without divulging its crown jewel, Google could push for its standard to become the architecture of cloud computing.
The team that developed Hadoop belonged to a company, Nutch, that got acquired. Oddly, they were now working within the walls of Yahoo, which was counting on the MapReduce offspring to give its own computers a touch of Google magic. Hadoop remained open source, though, which meant the Google team could adapt it and install it for free on the U-Dub cluster.
Students rushed to sign up for Google 101 as soon as it appeared in the winter-semester syllabus. In the beginning, Bisciglia and his Google colleagues tried teaching. But in time they handed over the job to professional educators at U-Dub. "Their delivery is a lot clearer," Bisciglia says. Within weeks the students were learning how to configure their work for Google machines and designing ambitious Web-scale projects, from cataloguing the edits on Wikipedia to crawling the Internet to identify spam. Through the spring of 2007, as word about the course spread to other universities, departments elsewhere started asking for Google 101.
Many were dying for cloud knowhow and computing power—especially for scientific research. In practically every field, scientists were grappling with vast piles of new data issuing from a host of sensors, analytic equipment, and ever-finer measuring tools. Patterns in these troves could point to new medicines and therapies, new forms of clean energy. They could help predict earthquakes. But most scientists lacked the machinery to store and sift through these digital El Dorados. "We're drowning in data," said Jeannette Wing, assistant director of the National Science Foundation.
BIG BLUE LARGESSE
The hunger for Google computing put Bisciglia in a predicament. He had been fortunate to push through the order for the first cluster of computers. Could he do that again and again, eventually installing mini-Google clusters in each computer science department? Surely not. To extend Google 101 to universities around the world, the participants needed to plug into a shared resource. Bisciglia needed a bigger cloud.
That's when luck descended on the Googleplex in the person of IBM Chairman Samuel J. Palmisano. This was "Sam's day at Google," says an IBM researcher. The winter day was a bit chilly for beach volleyball in the center of campus, but Palmisano lunched on some of the fabled free cuisine in a cafeteria. Then he and his team sat down with Schmidt and a handful of Googlers, including Bisciglia. They drew on whiteboards and discussed cloud computing. It was no secret that IBM wanted to deploy clouds to provide data and services to business customers. At the same time, under Palmisano, IBM had been a leading promoter of open-source software, including Linux. This was a key in Big Blue's software battles, especially against Microsoft. If Google and IBM teamed up on a cloud venture, they could construct the future of this type of computing on Google-based standards, including Hadoop.
Google, of course, had a running start on such a project: Bisciglia's Google 101. In the course of that one day, Bisciglia's small venture morphed into a major initiative backed at the CEO level by two tech titans. By the time Palmisano departed that afternoon, it was established that Bisciglia and his IBM counterpart, Dennis Quan, would build a prototype of a joint Google-IBM university cloud.
Over the next three months they worked together at Google headquarters. (It was around this time, Bisciglia says, that the cloud project evolved from 20% into his full-time job.) The work involved integrating IBM's business applications and Google servers, and equipping them with a host of open-source programs, including Hadoop. In February they unveiled the prototype for top brass in Mountain View, Calif., and for others on video from IBM headquarters in Armonk, N.Y. Quan wowed them by downloading data from the cloud to his cell phone. (It wasn't relevant to the core project, Bisciglia says, but a nice piece of theater.)
The Google 101 cloud got the green light. The plan was to spread cloud computing first to a handful of U.S. universities within a year and later to deploy it globally. The universities would develop the clouds, creating tools and applications while producing legions of computer scientists to continue building and managing them.
Those developers should be able to find jobs at a host of Web companies, including Google. Schmidt likes to compare the data centers to the prohibitively expensive particle accelerators known as cyclotrons. "There are only a few cyclotrons in physics," he says. "And every one if them is important, because if you're a top-flight physicist you need to be at the lab where that cyclotron is being run. That's where history's going to be made; that's where the inventions are going to come. So my idea is that if you think of these as supercomputers that happen to be assembled from smaller computers, we have the most attractive supercomputers, from a science perspective, for people to come work on."
As the sea of business and scientific data rises, computing power turns into a strategic resource, a form of capital. "In a sense," says Yahoo Research Chief Prabhakar Raghavan, "there are only five computers on earth." He lists Google, Yahoo, Microsoft, IBM, and Amazon. Few others, he says, can turn electricity into computing power with comparable efficiency.
All sorts of business models are sure to evolve. Google and its rivals could team up with customers, perhaps exchanging computing power for access to their data. They could recruit partners into their clouds for pet projects, such as the company's clean energy initiative, announced in November. With the electric bills at jumbo data centers running upwards of $20 million a year, according to industry analysts, it's only natural for Google to commit both brains and server capacity to the search for game-changing energy breakthroughs.
What will research clouds look like? Tony Hey, vice-president for external research at Microsoft, says they'll function as huge virtual laboratories, with a new generation of librarians—some of them human—"curating" troves of data, opening them to researchers with the right credentials. Authorized users, he says, will build new tools, haul in data, and share it with far-flung colleagues. In these new labs, he predicts, "you may win the Nobel prize by analyzing data assembled by someone else." Mark Dean, head of IBM's research operation in Almaden, Calif., says that the mixture of business and science will lead, in a few short years, to networks of clouds that will tax our imagination. "Compared to this," he says, "the Web is tiny. We'll be laughing at how small the Web is." And yet, if this "tiny" Web was big enough to spawn Google and its empire, there's no telling what opportunities could open up in the giant clouds.
It's a mid-November day at the Googleplex. A jetlagged Christophe Bisciglia is just back from China, where he has been talking to universities about Google 101. He's had a busy time, not only setting up the cloud with IBM but also working out deals with six universities—U-Dub, Berkeley, Stanford, MIT, Carnegie Mellon, and the University of Maryland—to launch it. Now he's got a camera crew in a conference room, with wires and lights spilling over a table. This is for a promotional video about cloud education that they'll release, at some point, on YouTube (GOOG).
Eric Schmidt comes in. At 52, he is nearly twice Bisciglia's age, and his body looks a bit padded next to his protégé's willowy frame. Bisciglia guides him to a chair across from the camera and explains the plan. They'll tape the audio from the interview and then set up Schmidt for some stand-alone face shots. "B-footage," Bisciglia calls it. Schmidt nods and sits down. Then he thinks better of it. He tells the cameramen to film the whole thing and skip stand-alone shots. He and Bisciglia are far too busy to stand around for B footage.

The World's Eight Most Excellent Software Adventures

Joel Pobar's Blog

CIOs Share Worries Over Emerging Technologies

Web 2.0 and emerging mobile technologies have some CIOs taking stock of how to keep control and still give their users new freedoms.
In the midst of confronting emerging technologies, there can be feelings of loss of control, three CIOs at a panel at the Cisco Systems Inc. C-Scape conference in San Jose, California, admitted.
With Web 2.0, including social networking trends, trying to maintain control over technology in a large corporation can feel like piloting a sailing ship, said Randy Spratt, CIO at McKesson Corp., a medical services company.
"You never really know where the wind's coming from and how hard, but you have to ride with it and keep sailing," Spratt said.
Spratt said his biggest concerns with users who want to try new technologies is potential legal exposure for his company. "There could be loss of intellectual property, and inappropriate comments that could be discovered later," he said. "Whether it's a fax or a blog or an IM, [it] all relies on good-intentioned behaviors by employees."
Ultimately, relying on employees to behave responsibly is what matters most, he added. "They grow up with new technologies, many from outside work," he said. "The real question is how to turn [new technology] to your advantage. When is it appropriate to put a wiki up‏ When is it appropriate to let SharePoint grow virally‏"
Rebecca Jacoby, CIO at Cisco, acknowledged that she has "abandoned any idea that you could possibly control" the emerging trends, so she instead seeks to communicate with employees about appropriate use.
Cisco has 50,000 users of wikis, which pose challenges related to proper infrastructure support and to how the wikis correspond to Cisco's business processes, Jacoby added.
Integrating SharePoint at Chevron Oil is being done with care, said Louie Ehrlich, CIO for Chevron's global downstream operations, which primarily involve refinery functions. "We're playing with things and being cautious," he said. "We don't have a clear strategy." Ehrlich said the challenge becomes finding ways to integrate such new technologies into Chevron's overall budget imperatives.
McKesson has many new technologies at the corporate level, including four of Cisco's TelePresence systems for videoconferencing, which have been "phenomenally successful" in reducing travel costs and are "undeniably equivalent to face-to-face meetings," Spratt said.
However, he said, there are many small vendors that he called "annoying ankle biters" who approach workgroups throughout the organization to sell a few licenses for products in new technologies. "They create a lot of headaches for me [but] are often at the cutting edge of new technologies," Spratt said.
Regarding new mobile devices, Jacoby said she realizes that workers "like to make a personal choice about technologies closest to them, but the flip side is the [organization's] need to manage away from the device ... to manage that stuff in the network and then to have technologies to enable them."

Source: Computerworld

Virtualization in Two-thirds of Enterprises by '09

More than a third of enterprise IT shops have implemented x86 server virtualization, and nearly two-thirds expect to do so by 2009, Forrester Research finds in a new survey.
IT departments already using virtualization have virtualized 24% of servers, and that number is expected to grow to 45% by 2009.
Vendors need to get busy upgrading virtualization products, because many enterprises have been using the technology for two years or more and are ready to expand usage, Forrester reports.
"BMC Software, IBM Tivoli, HP Software, and Microsoft must repackage their offerings to create immediate tactical value by adding or buying tools for virtualization environment tasks, such as converting between physical and virtual servers and rapidly updating virtual server configurations," Forrester states.
The Forrester report -- "x86 virtualization adopters hit the tipping point" -- is based on a survey of 275 enterprise server decision-makers.
Previous Forrester research actually showed higher adoption of server virtualization, with 50% of IT shops using the technology in production and pilots in 2006.
Estimates tend to be "all over the map," and IT executives are sometimes too optimistic about predictions of future use, says report lead author Frank Gillett. But the survey results "show the power and popularity of the idea ... and demonstrates there is significant intent to increase usage."
The latest report finds that 37% of IT departments have virtualized servers already, and another 13% plan to do so by July 2008. An additional 15% think they will virtualize x86 servers by 2009.
As enterprises gain a couple years experience with virtualization, they will move from tactical, experimental approaches to strategic IT infrastructure initiatives that might involve upgrading servers, storage, networks and systems management.
But virtualization isn't close to being universally adopted throughout enterprises, Gillett says. IT executives typically aren't using the technology for critical applications, or platforms like grid computing and supercomputing, he says.
"Virtualization is working its way [up] from things where people are less uptight about performance," he says.
Virtualization is primarily about sharing machines and portability, but these may not be compelling reasons to virtualize critical workloads, according to Gillett. Machine sharing isn't that necessary if a machine is already busy, and portability might not be compelling when there are few other servers a workload can be moved to.

Source: CIO Magazine

Google:Rewriting the book on Datacentres (from ZDNet)

by Dan Farber, Larry Dignan, David Berlind

Om Malik had an interesting post about Google’s super-sized datacenters. He posited that Google’s massive infrastructure, customized for its processes, represents a the big barrier to entry for its rivals, and it will continue to give the company an edge as long as it keeps spending the billions on its custom infrastructure.

Indeed, Google is spending billions on equipment and buying up thousands of acres around the globe to better control its own infrastructure destiny and to deliver search results very fast and accurately. The company is spending over several billion dollars to build data centers collectively housing hundreds of thousands of servers running massively parallel computations in power-friendly locations in Iowa, Oklahoma, Oregon and North and South Carolina, and that’s just in the U.S.

Google data center under construction in The Dalles, Ore.

Lloyd Taylor, the former director of global operations at Google and now vice president of technology operation at LinkedIn, attributes part Google’s success to its data center design and science. “Everything goes back to physics,” the former member of the Johns Hopkins University Applied Physics Lab told me.

Google threw out the book on data centers and went back to basic heat transfer and electrical theory, he explained, eliminating everything not strictly necessary and coming up with a minimalist design.

It’s similar to how the integrated circuit industry applies a step and repeat concept for building circuit boards, he said. “Buy 1,000 acres in Oklahoma and build a data center and when you need more space, step and repeat, build another one next to it. As the business needs more space, a new data center is up and running in less than six months.”

The physics comes in understanding how to gain more efficiency in power and cooling, for example. “Most data centers have multiple transfers between the server, the air around the server, the cooling towers and so forth. Any time you do a transfer through any two medium heat transfer theory will tell you that you are losing efficiency. So, you eliminate the number of transfers and have a very simple connection from the server itself to the cooling towers,” Taylor said.

In addition, redundancy is aided by using the software and hardware layers to pay attention to things that aren’t working, and proactively redirecting loads. “If a meteor hits in Oregon, you may have to hit reload to get a search result, but that’s about it,” Taylor said.

Google has patented many of its hardware and software infrastructure innovations, and as Ethan Stock said in blog post, “Google is the server-side doppelganger to Apple, and their platforms like GFS, BigTable, MapReduce, and Sawzall are core to their competitive advantage.” The major difference is that Google’s servers are behind the curtain and Apple’s devices are center stage.

The hosting, backup generator and cooling tower businesses in general are booming. Dupont Fabros Technology, for example, is building a data center (below) in Northern Virginia that takes up 17.15 acres, with a 348,464 square foot facility and 171,200 square feet of raised floor, with output of 36.4 megawatts.

Over time, whatever Google has done to gain more efficiency and cost savings in its data center design will be in textbooks and less of a competitive advantage. The Red Shift, Sun CTO Greg Papadopoulos’ idea that we’ll end up with a handful of super-scaled data centers powering everything digital on the planet, is a long way off, and there will always be room for more niche operations.

Among the attendees at session at Gartner’s Data Center Conference in Las Vegas last month, only five percent said they would outsource their data center operations to a third-party provider.

Like ice gradually melting into the oceans in the Arctic regions, enterprises want to slowly migrate to the cloud, or utility, computing model, with services delivered by massive-scale data centers. But, the tech industry will have its own global warming, pushing companies to move faster to the cloud to stay competitive.

Virtualization Competition Heating Up

451 Group: Virtualization Competition Heating Up

NEW YORK, Dec. 6 -- The 451 Group believes that competition in virtualization technology is heating up and has moved to the management layer. Thanks to open source projects, the hypervisor -- the software layer that enables physical machines to run multiple virtual machines -- has become a commodity. As a result, hypervisor vendors have become virtualization management vendors and are now selling virtualization administration. These findings are contained in a report released today by New York-based The 451 Group, a technology-industry analyst company focused on the business of enterprise IT innovation.

"The number of companies jumping into the virtualization arena has exploded," said Rachel Chalmers, senior analyst for enterprise software at The 451 Group and author of the report. "Six companies represented VM management in our first major report on datacenter virtualization, published in December 2006. Less than 12 months later, in this report, we profile 50 privately held companies competing in exactly the same sector."

This report identifies 10 subsectors of virtualization management technology:

• Administration.

• Automation.

• Backup and high availability.

• Capacity planning.

• Infrastructure virtualization.

• Monitoring.

• Optimization.

• Security.

• Test lab automation.

• Workspace virtualization.

"Virtualization is the third-fastest-growing sector in IT buying, behind only storage and security," said Chalmers. "We believe Microsoft is the likeliest buyer, with potential interest in nine of the 10 subsectors -- if only because its as-yet-unreleased hypervisor is lagging an estimated two years behind that of market leader VMware. Judicious acquisition could help the Windows giant catch up."

Other public vendors highly likely to be acquisitive in order to flesh out patchy portfolios include OS vendors Hewlett-Packard, Novell, Red Hat and Sun Microsystems, and Windows management specialists Avocent, BMC Software, Citrix Systems, Quest Software and Symantec.

"The most likely target sectors, judging by the number of potential acquirers identified, are automation and monitoring," added Chalmers.

This 170-page report, "Virtualization 3: Managing the virtual revolution," was written by Chalmers. Two previous 451 Special Reports examined virtualization at the server level and at the desktop, respectively. In this Special Report, 451 analysts survey the public companies active in the space and identify gaps in their portfolios, and then investigate 50 startups, suggesting which ones might fill the acquirers' portfolio gaps. 451 analysts also take the opportunity to make predictions about the direction of the industry over the next 12-18 months.

Key Companies Covered

The report includes in-depth competitive assessments of the following companies (although this is not a complete list of companies covered in various sections of the report): 3Leaf Systems, Acronis, Akorri, Availigent, Avocent, BladeLogic, Blue Lane Technologies, BMC Software, CA, Cassatt, Catbird, CiRBA, Cisco Systems, Citrix Systems, CohesiveFT, CollabNet, Configuresoft, Desktone, DeviceVM, Egenera, eG Innovations, Embotics, Enigmatec, Enomaly, FastScale, Hewlett-Packard, Hyperic, IBM, illumita, InfoVista, InovaWave, Leostream, Marathon Technologies, Mendocino Software, Microsoft, Netuitive, Network Appliance, Nimsoft, Novell, Onaro, Pano Logic, PlateSpin, Platform Computing, Quest Software, Qumranet, Red Hat, Reflex Security, RingCube, Scalent Systems, ScienceLogic, SteelEye Technology, Stratavia, Surgient, SWsoft and Parallels, Sychron, Sun Microsystems, Symantec, ToutVirtual, Univa UD, Veeam Software, Virtual Iron, Virtugo Software, Vizioncore, VMLogix, VMware, XDS and Xsigo.

About The 451 Group

The 451 Group is an independent technology-industry analyst company focused on the business of enterprise IT innovation. The company's analysts provide critical and timely insight into the market and competitive dynamics of innovation in emerging technology segments. Clients of the company -- at vendor, investor, service-provider and end-user organizations -- rely on 451 insight to support both strategic and tactical decision-making for competitive advantage. For additional information on The 451 Group or to apply for trial access to its services, go to www.the451group.com.

-----

Source: The 451 Group and GRIDToday

CIOs see value in Web 2.0

A Forrester report suggests that corporate IT departments have seen demonstrable value from Web 2.0 technologies in the workplace and should continue to adopt more of those applications at their own pace.
But the report also reveals that the unsanctioned use of consumer, Web-based applications (a phenomenon known as Rogue IT or Shadow IT) (For more, see 'Shadow IT Culture' on the Rise for Businesses) remains high, behooving IT managers to get in the trenches to find out where sensitive corporate data could be exposed.
"Of the rogue usage going on, it's often difficult to see which poses privacy or security concerns," says Rob Koplowitz, a Forrester analyst and one of the authors of the study, "Web 2.0 Social Computing Dresses Up for Business."
Around 15 percent of the IT decision-makers surveyed at firms with 500 or more employees say their workers have used technologies like blogs, wikis and really simple syndication (RSS) for business purposes. On average, about 27 percent of those companies have already made formal enterprise investments in all three of those technologies and another 16 percent have at least considered it. At least 89 percent saw limited to substantial value from the use of blogs, RSS and wikis.
Meanwhile, Koplowitz says the numbers reported for rogue usage-which at Forrester's last count range from 3 percent to 8 percent-remain deceptively low.
"It could be a lot higher because unsanctioned use is, by definition, under the radar," he says. "The best an IT manager can do is have some anecdotal evidence and then work from there."
To avoid an ad-hoc approach to Web 2.0 adoption, Koplowitz says IT departments should start by getting a better handle on what applications users have flocked to and embrace them rather than shunning them. In doing so, IT eliminates an adversarial environment, allowing IT managers to form a long-term strategy with their users that encourages testing, setting usage policies and training.
"It's becoming increasingly difficult for IT to control what tools people use in their day-to-day activities," Koplowitz says. "It's in IT's best interest to find out what's going on and offer a sanctioned alternative."

Source: CIO(US)

Layer 7 Technologies Introduces First Virtual Appliance for Service Oriented Architectures

Tuesday 26 June 2007

New soft-appliance supports virtualized data center environments and reduces
deployment complexities.

Layer 7 Technologies, provider of XML security and networking for Service Oriented Architecture (SOA) and Web 2.0, today announced its SecureSpan™ XML Firewall is now available as a virtual soft-appliance. The first of its kind, this soft-appliance for SOA enables enterprises and services providers to rapidly deploy a SOA security appliance on commodity server hardware and VMWare virtualized computing platforms, reducing time to deployment and implementation costs while improving architectural flexibility.

The virtual soft-appliance features the same properties as Layer 7’s SecureSpan XML Firewall hardware appliance and provides advanced Web services identity and message level security to address the broadest range of SOA integration, portal and B2B security challenges. The solution enables end users to implement identical XML Firewall hardware appliance functionality on any server infrastructure running VMWare Player, VMWare Server or VMWare ESX through a simple download and installation.

"

"An increasing number of enterprises are exploring virtualization strategies for maximizing their hardware investment," said Jason Bloomberg, Senior Analyst at ZapThink. "Layer 7’s soft-appliance provides those organizations a production-ready SOA security and networking solution that can scale on demand to meet changing transaction loads."

"

Key benefits of the Layer 7 virtual appliance include:

• Real-time download and deployment

• Support for multi-tenant data center environments

• Full clustering for simplified load scaling

• Utility computing over shared server, CPU and memory resources

• Broad operating system (OS) support

• Interoperability with Layer 7’s SecureSpan hardware appliances

"

"By introducing the SecureSpan XML Firewall soft-appliance we are giving enterprises and service providers unmatched choice in where and how they implement their SOA architectures,” said Dimitri Sirota, VP of Marketing, Layer 7 Technologies. “Layer 7 is the first SOA security and networking vendor to offer advanced XML firewall functionality through a purpose built hardware-accelerated physical appliance or a software instantiated virtual appliance. This allows our customers to decide which platform best suits their needs.”

Teach Yourself Programming in Ten Years

Teach Yourself Programming in Ten Years
Peter Norvig


Why is everyone in such a rush?Walk into any bookstore, and you'll see how to Teach Yourself Java in 7 Days alongside endless variations offering to teach Visual Basic, Windows, the Internet, and so on in a few days or hours. I did the following power search at Amazon.com: pubdate: after 1992 and title: days and
(title: learn or title: teach yourself)and got back 248 hits. The first 78 were computer books (number 79 was Learn Bengali in 30 days). I replaced "days" with "hours" and got remarkably similar results: 253 more books, with 77 computer books followed by Teach Yourself Grammar and Style in 24 Hours at number 78. Out of the top 200 total, 96% were computer books.
The conclusion is that either people are in a big rush to learn about computers, or that computers are somehow fabulously easier to learn than anything else. There are no books on how to learn Beethoven, or Quantum Physics, or even Dog Grooming in a few days.
Let's analyze what a title like Learn Pascal in Three Days could mean:
Learn: In 3 days you won't have time to write several significant programs, and learn from your successes and failures with them. You won't have time to work with an experienced programmer and understand what it is like to live in that environment. In short, you won't have time to learn much. So they can only be talking about a superficial familiarity, not a deep understanding. As Alexander Pope said, a little learning is a dangerous thing.
Pascal: In 3 days you might be able to learn the syntax of Pascal (if you already knew a similar language), but you couldn't learn much about how to use the syntax. In short, if you were, say, a Basic programmer, you could learn to write programs in the style of Basic using Pascal syntax, but you couldn't learn what Pascal is actually good (and bad) for. So what's the point? Alan Perlis once said: "A language that doesn't affect the way you think about programming, is not worth knowing". One possible point is that you have to learn a tiny bit of Pascal (or more likely, something like Visual Basic or JavaScript) because you need to interface with an existing tool to accomplish a specific task. But then you're not learning how to program; you're learning to accomplish that task.
in Three Days: Unfortunately, this is not enough, as the next section shows.
Teach Yourself Programming in Ten YearsResearchers (Hayes, Bloom) have shown it takes about ten years to develop expertise in any of a wide variety of areas, including chess playing, music composition, painting, piano playing, swimming, tennis, and research in neuropsychology and topology. There appear to be no real shortcuts: even Mozart, who was a musical prodigy at age 4, took 13 more years before he began to produce world-class music. In another genre, the Beatles seemed to burst onto the scene with a string of #1 hits and an appearance on the Ed Sullivan show in 1964. But they had been playing small clubs in Liverpool and Hamburg since 1957, and while they had mass appeal early on, their first great critical success, Sgt. Peppers, was released in 1967. Samuel Johnson thought it took longer than ten years: "Excellence in any department can be attained only by the labor of a lifetime; it is not to be purchased at a lesser price." And Chaucer complained "the lyf so short, the craft so long to lerne."
Here's my recipe for programming success:
Get interested in programming, and do some because it is fun. Make sure that it keeps being enough fun so that you will be willing to put in ten years.
Talk to other programmers; read other programs. This is more important than any book or training course.
Program. The best kind of learning is learning by doing. To put it more technically, "the maximal level of performance for individuals in a given domain is not attained automatically as a function of extended experience, but the level of performance can be increased even by highly experienced individuals as a result of deliberate efforts to improve." (p. 366) and "the most effective learning requires a well-defined task with an appropriate difficulty level for the particular individual, informative feedback, and opportunities for repetition and corrections of errors." (p. 20-21) The book Cognition in Practice: Mind, Mathematics, and Culture in Everyday Life is an interesting reference for this viewpoint.
If you want, put in four years at a college (or more at a graduate school). This will give you access to some jobs that require credentials, and it will give you a deeper understanding of the field, but if you don't enjoy school, you can (with some dedication) get similar experience on the job. In any case, book learning alone won't be enough. "Computer science education cannot make anybody an expert programmer any more than studying brushes and pigment can make somebody an expert painter" says Eric Raymond, author of The New Hacker's Dictionary. One of the best programmers I ever hired had only a High School degree; he's produced a lot of great software, has his own news group, and made enough in stock options to buy his own nightclub.
Work on projects with other programmers. Be the best programmer on some projects; be the worst on some others. When you're the best, you get to test your abilities to lead a project, and to inspire others with your vision. When you're the worst, you learn what the masters do, and you learn what they don't like to do (because they make you do it for them).
Work on projects after other programmers. Be involved in understanding a program written by someone else. See what it takes to understand and fix it when the original programmers are not around. Think about how to design your programs to make it easier for those who will maintain it after you.
Learn at least a half dozen programming languages. Include one language that supports class abstractions (like Java or C++), one that supports functional abstraction (like Lisp or ML), one that supports syntactic abstraction (like Lisp), one that supports declarative specifications (like Prolog or C++ templates), one that supports coroutines (like Icon or Scheme), and one that supports parallelism (like Sisal).
Remember that there is a "computer" in "computer science". Know how long it takes your computer to execute an instruction, fetch a word from memory (with and without a cache miss), read consecutive words from disk, and seek to a new location on disk. (Answers here.)
Get involved in a language standardization effort. It could be the ANSI C++ committee, or it could be deciding if your local coding style will have 2 or 4 space indentation levels. Either way, you learn about what other people like in a language, how deeply they feel so, and perhaps even a little about why they feel so.
Have the good sense to get off the language standardization effort as quickly as possible. With all that in mind, its questionable how far you can get just by book learning. Before my first child was born, I read all the How To books, and still felt like a clueless novice. 30 Months later, when my second child was due, did I go back to the books for a refresher? No. Instead, I relied on my personal experience, which turned out to be far more useful and reassuring to me than the thousands of pages written by experts.
Fred Brooks, in his essay No Silver Bullets identified a three-part plan for finding great software designers:
Systematically identify top designers as early as possible.
Assign a career mentor to be responsible for the development of the prospect and carefully keep a career file.
Provide opportunities for growing designers to interact and stimulate each other.
This assumes that some people already have the qualities necessary for being a great designer; the job is to properly coax them along. Alan Perlis put it more succinctly: "Everyone can be taught to sculpt: Michelangelo would have had to be taught how not to. So it is with the great programmers".
So go ahead and buy that Java book; you'll probably get some use out of it. But you won't change your life, or your real overall expertise as a programmer in 24 hours, days, or even months.

(an Interview with Bjarne Stroustrup from MIT Technology Review)

The Problem with Programming

Bjarne Stroustrup, the inventor of the C++ programming language, defends his legacy and examines what's wrong with most software code.

In the 1980s and 90s, Bjarne Stroustrup designed and implemented the C++ programming language, which popularized object-oriented programming and influenced numerous other programming languages, including Java.

C++ remains the archetypal "high level" computer language (that is, one that preserves the features of natural, human language), and it is still used by millions of programmers. Many of the systems and applications of the PC and Internet eras were written in C++. For all that, the language remains controversial, largely because it is notoriously difficult to learn and use, and also because Stroustrup's design allows developers to make serious programming mistakes in the interest of preserving their freedom.

Stroustrup, for many years a researcher at AT&T Bell Labs, is now a professor of computer science in the Department of Engineering, at Texas A&M University, near Houston.

Technology Review: Why is most software so bad?

Bjarne Stroustrup: Some software is actually pretty good by any standards. Think of the Mars Rovers, Google, and the Human Genome Project. That's quality software! Fifteen years ago, most people, and especially most experts, would have said each of those examples was impossible. Our technological civilization depends on software, so if software had been as bad as its worst reputation, most of us would have been dead by now.

On the other hand, looking at "average" pieces of code can make me cry. The structure is appalling, and the programmers clearly didn't think deeply about correctness, algorithms, data structures, or maintainability. Most people don't actually read code; they just see Internet Explorer "freeze."

I think the real problem is that "we" (that is, we software developers) are in a permanent state of emergency, grasping at straws to get our work done. We perform many minor miracles through trial and error, excessive use of brute force, and lots and lots of testing, but--so often--it's not enough.

Software developers have become adept at the difficult art of building reasonably reliable systems out of unreliable parts. The snag is that often we do not know exactly how we did it: a system just "sort of evolved" into something minimally acceptable. Personally, I prefer to know when a system will work, and why it will.

TR: How can we fix the mess we are in?

BS: In theory, the answer is simple: educate our software developers better, use more-appropriate design methods, and design for flexibility and for the long haul. Reward correct, solid, and safe systems. Punish sloppiness.

In reality, that's impossible. People reward developers who deliver software that is cheap, buggy, and first. That's because people want fancy new gadgets now. They don't want inconvenience, don't want to learn new ways of interacting with their computers, don't want delays in delivery, and don't want to pay extra for quality (unless it's obvious up front--and often not even then). And without real changes in user behavior, software suppliers are unlikely to change.

We can't just stop the world for a decade while we reprogram everything from our coffee machines to our financial systems. On the other hand, just muddling along is expensive, dangerous, and depressing. Significant improvements are needed, and they can only come gradually. They must come on a broad front; no single change is sufficient.

One problem is that "academic smokestacks" get in the way: too many people push some area as a panacea. Better design methods can help, better specification techniques can help, better programming languages can help, better testing technologies can help, better operating systems can help, better middle-ware infrastructures can help, better understanding of application domains can help, better understanding of data structures and algorithms can help--and so on. For example, type theory, model-based development, and formal methods can undoubtedly provide significant help in some areas, but pushed as the solution to the exclusion of other approaches, each guarantees failure in large-scale projects. People push what they know and what they have seen work; how could they do otherwise? But few have the technical maturity to balance the demands and the resources.

TR: The idea behind C++ was that programmers would work harder in return for more-efficient code. Bell Labs wanted a language that a few really smart people would use to write code that would run on computers like Electronic Switching Systems (ESS) that weren't very fast. Today, there are a lot of software developers and computers are very fast. Does that vitiate the point of C++?

BS: C++ wasn't designed specifically for the large switching machines, but for a huge range of applications. Bell Labs was the home of an incredible range of interesting projects spanning every scale and using essentially every kind of computer and operating system. But yes, the average Bell Labs programmer was significantly more able than most people's notion of an "average programmer," and reliability and performance (in that order) were considered significantly more important than in most other places.

Performance is still an issue in many of the applications that I'm interested in: responsiveness of interfaces, start-up and close-down time of applications. Software developers have neutralized the astounding performance of modern computer hardware by adding layer upon layer of overelaborate [software] abstractions. We seem to have hit the limits of linear speedup for hardware, but in many cases, we could win a couple of orders of magnitude back from the software.

That said, C++ has indeed become too "expert friendly" at a time where the degree of effective formal education of the average software developer has declined. However, the solution is not to dumb down the programming languages but to use a variety of programming languages and educate more experts. There has to be languages for those experts to use--and C++ is one of those languages.

TR: In retrospect, in designing C++, wasn't your decision to trade off programmer efficiency, security, and software reliability for run time performance a fundamental mistake?

BS: Well, I don't think I made such a trade-off. I want elegant and efficient code. Sometimes I get it. These dichotomies (between efficiency versus correctness, efficiency versus programmer time, efficiency versus high-level, et cetera.) are bogus.

What I did do was to design C++ as first of all a systems programming language: I wanted to be able to write device drivers, embedded systems, and other code that needed to use hardware directly. Next, I wanted C++ to be a good language for designing tools. That required flexibility and performance, but also the ability to express elegant interfaces. My view was that to do higher-level stuff, to build complete applications, you first needed to buy, build, or borrow libraries providing appropriate abstractions. Often, when people have trouble with C++, the real problem is that they don't have appropriate libraries--or that they can't find the libraries that are available.

Other languages have tried to more directly support high-level applications.

That works, but often that support comes at the cost of specialization. Personally, I wouldn't design a tool that could do only what I wanted--I aim for generality.

TR: How do you account for the fact that C++ is both widely criticized and resented by many programmers but at the same time very broadly used? Why is it so successful?

BS: The glib answer is, There are just two kinds of languages: the ones everybody complains about and the ones nobody uses.

There are more useful systems developed in languages deemed awful than in languages praised for being beautiful--many more. The purpose of a programming language is to help build good systems, where "good" can be defined in many ways. My brief definition is, correct, maintainable, and adequately fast. Aesthetics matter, but first and foremost a language must be useful; it must allow real-world programmers to express real-world ideas succinctly and affordably.

The main reason for C++'s success is simply that it meets its limited design aims: it can express a huge range of ideas directly and efficiently. C++ was not designed to do just one thing really well or to prevent people doing things considered "bad." Instead, I concentrated on generality and performance.

I'm sure that for every programmer that dislikes C++, there is one who likes it. However, a friend of mine went to a conference where the keynote speaker asked the audience to indicate by show of hands, one, how many people disliked C++, and two, how many people had written a C++ program. There were twice as many people in the first group than the second. Expressing dislike of something you don't know is usually known as prejudice. Also, complainers are always louder and more certain than proponents--reasonable people acknowledge flaws. I think I know more about the problems with C++ than just about anyone, but I also know how to avoid them and how to use C++'s strengths.

And then, of course, you don't expect proponents of languages that lost out in competition with C++ to be polite about it. Software development doesn't have that degree of professionalism--though I hope it eventually will. Science is different in this respect: when a new tool, technique, or theory wins out, people see that as progress. In software, contributions by competitors and predecessors are not widely acknowledged, appreciated, or even understood.

TR: In The Design and Evolution of C++, you claim that Kierkegaard was an influence on your conception of the language. Is this a joke?

BS: A bit pretentious, maybe, but not a joke. A lot of thinking about software development is focused on the group, the team, the company. This is often done to the point where the individual is completely submerged in corporate "culture" with no outlet for unique talents and skills. Corporate practices can be directly hostile to individuals with exceptional skills and initiative in technical matters. I consider such management of technical people cruel and wasteful. Kierkegaard was a strong proponent for the individual against "the crowd" and has some serious discussion of the importance of aesthetics and ethical behavior. I couldn't point to a specific language feature and say, "See, there's the influence of the nineteenth-century philosopher," but he is one of the roots of my reluctance to eliminate "expert level" features, to abolish "misuses," and to limit features to support only uses that I know to be useful. I'm not particularly fond of Kierkegaard's religious philosophy, though.

TR: What do you regret the most?

BS: No regrets! Well, of course I dream of what I might have done differently and better, but seriously, who am I to second-guess, say, 1984 vintage Bjarne? He may have been less experienced than I, but he was no less smart, probably smarter, and he had a better understanding of the word of 1984 than I have. C++ has been used to build many systems that enhance our lives, and it has been a significant positive influence on later languages and systems. That's something to be proud of.

Ten Shining Examples of SOA in Action in 2006 (From the SOA In Action Blog)

Ten Shining Examples of SOA in Action in 2006

Here are ten good reasons to go to SOA, cited by ten companies that began their journey down the SOA path:

Integrate back-end legacy systems: International Truck had all types of back-end legacy systems that the company wanted to surface into a "Common Vehicle Tracking system" that could track truck production in near real time, and flag any defects or bottlenecks in production. If successfully deployed, such a system could save International Truck up to $3 million a year. The solution consisted of two Java EE-based interfaces to a homegrown order management system and Baan ERP system. Plans are to extend these interfaces to customers and trading partners. Eventually, the company expects to be able to swap out the order management system with little or no disruption to the tracking system.

Better connect with partners. MedicAlert embarked on the SOA path to achieve interoperability not only between its own internal applications, but also with partners -- hospitals, doctors’ offices, EMTs, and other medical professionals and establishments -- to provide up-to-date personal health records. The company's system, called E-HealthKEY, is intended to provide the foundation’s partners more seamless access to pertinent medical information. “The level of interoperability that is provided by implementing an SOA is really what we’re after,” said Jorge Mercado, lead architect of the software architecture group for the MedicAlert Foundation. “Keeping information in our repository and our repository alone is a good thing, but that’s not where we want to be in the future. We want to be able to share information with hospitals, doctor’s offices, labs, and pharmacies.”

Componentize product offerings: Experian leveraged SOA-based processes and technologies to develop a Customer Event Management system (CEMS) to support its base of leading financial institution customers. The system enables financial institutions to rapidly assess and process new accounts using Experian's online services. The credit giant's services are being delivered as standard components through Experian Web services, including Detect (application fraud prevention) and Delphi (credit scoring). The CEMS was built with the .NET Framework, and is currently being tested by a major financial customer. "We are taking our technology investment and re-engineering customer-related services as a set of Web services. We have smashed large monolithic applications into smaller components. Componentizing all our available product sets is a big and ongoing job, and has changed the way we work in terms of IT development and delivery," said John Finch, director of development and delivery for the information solutions division at Experian.

Abstract multiple ERP functions into a single service layer: Washington Group employed SOA-based middleware to move to an SOA that would abstract many of the functions used in various ERP systems across the company into a common service layer. "By integrating these systems, we can -- on the back end then on the front end -- build some business processes that aren't being executed within the legacy application execution systems, and have it done in a layer above that. We can then update the appropriate systems where needed," said Rich Colton, application integration manager with Washington Group International. "We have not found an ERP system that meets the engineering/construction industry's requirements. We ended up having to use the best of breed in different areas. Up until now, it's required a lot of duplication, triplication, and more, of information from one system to another."

Mirror the philosophy of the business: Ameriprise Financial's credo for its customers is "dream, plan, track" and this is reflected in its SOA effort as well. Ameriprise began developing an SOA several years ago, and lately has been moving its SOA effort from the realm of bits and bytes to a full-blown business proposition. "We realized is that in order for SOA to deliver the full business value, it has to become a business strategy," said Tracy Legrand, chief architect for Ameriprise. The SOA effort followed a roadmap which brought SOA functionality to new applications or services as they came online. Services covered within the company's SOA include customer management, asset management, and moving money between products, between funds, or accounts. Legrand estimates that Ameriprise has saved almost $10 million in cost-avoidance over a three-year period just from one of its leading services -- customer management.

Streamline requests to IT: The IT operations group at Siemens AG, which first built services around automating and streamlining the processes for fulfilling internal requests to IT for new equipment and passwords. Thomas Buse, section manager of concepts and processes for Siemens, said that "once users from various departments started using that system for new workers, they asked IT to similarly automate and improve the processes in their departments." He added that in the process, Siemens cut the time required to implement new processes by 83%. The company releases four to eight new business processes to run on its SOA every six to 12 weeks.

Maintain service quality: The Hartford has a very strong SOA governance effort, led by an SOA steering committee consisting of application architects. Committee members assess proposed new services based on such criteria as supportability, reusability and adherence to the company’s SOA reference architecture. Ben Moreland, architectural director at The Hartford, is also aware of the issues around services pushed into the registry without IT support to back the service up and maintain it, and therefore requires that IT sign off on all proposed services. “We don’t want a junk drawer of unsupported services in our SOA registry,” he said. This process keeps business and IT groups from proliferating "junk" services to the group’s master Universal Description, Discovery and Integration (UDDI) registry, the article noted. Moreland hopes to automate much of the approval process through workflow tools that integrate with its UDDI registry.

Keep vendors on their toes: The Federal Bureau of Investigation has been investigating SOA, and seen impressive results from a more outward-facing -- and smaller-scale -- SOA project. The agency launched a Regional Data Exchange, or R-DEx, a series of information sharing pilots with regional databases. Under R-DEx, the FBI has created plug-ins to Justice Department databases for four regional law enforcement data sharing associations, with more to come -- using an SOA registry built with off-the-shelf IT products. Because the project is built using SOA methodologies, the bureau has already been able to pull out and swap some of R-DEx's components, including an underused portal function and search engines. An interesting effect of this swappability is that it has kept "vendors on their toes," said project manager Margie Lonergan. The knowledge that they're easily replaced, "entices them to make sure they stay best of breed."

Expand a global reach: Monster -- the online jobsite -- recently expanded its reach to 24 countries across the globe, and needed a service oriented architecture that would stretch across separate regional units. Prior to the SOA implementation, new orders would be manually routed to a financial system for invoicing, explained Joan Lawson, director of global integration for Monster. Job postings were then distributed to appropriate regional units via a point to point extract, transform, and load process. "Due to a global marketplace and offshoring trends, our customers were demanding real-time integration of jobs and resumes across the various regional platforms," Lawson said. "An employer entering a job in North America may also have a job available in India." Monster developed an SOA-based middleware layer to take away these manual processes. When a customer posts a new opening, Lawson explained, the SOA middleware delivers the order to the applications within the various regional units. "If an employer sends a file feed of a job that's going to be available in North America as well as in Czech Republic, that will flow all the way through."

Get your motor running: Harley-Davidson employed SOA to to surpass its older, more rigid technology to a more flexible model that enabled it to better support its marketing efforts. Harley-Davidson's CIO, Jim Haney, explained that the company has marketing programs for turning showroom fantasies into realities. "We want to put together a good financial package to entice and incite people to get into the sport," he said. Previously, however, Harley’s credit and loan origination process wasn't up to the task. The company's financial services applications were tightly coupled with each other, and making a change in one program meant making changes to countless other applications as well. Harley's answer was to break it all up. "We actually busted apart all of those systems, and put the SOA with WebSphere in the middle of all that," Haney said. "We loosely coupled these things. Our goal is to be able to change any one of those systems. If we see key indicators in the industry, we want to very quickly put different marketing programs in place, and not have to go and touch and test every single system."

Code Kata from Dave Thomas Blog

Code Kata

Background

How do you get to be a great musician? It helps to know the theory, and to understand the mechanics of your instrument. It helps to have talent. But ultimately, greatness comes practicing; applying the theory over and over again, using feedback to get better every time.

How do you get to be an All-Star sports person? Obviously fitness and talent help. But the great athletes spend hours and hours every day, practicing.

But in the software industry we take developers trained in the theory and throw them straight in to the deep-end, working on a project. It’s like taking a group of fit kids and telling them that they have four quarters to beat the Redskins (hey, we manage by objectives, right?). In software we do our practicing on the job, and that’s why we make mistakes on the job. We need to find ways of splitting the practice from the profession. We need practice sessions.

CodeKata:	A description of how this all started
MoreKata:	Sometimes ‘kata’ isn’t quite the right word; karate uses other techniques to teach too.

The Kata

What makes a good practice session? You need time without interruptions, and a simple thing you want to try. You need to try it as many times as it takes, and be comfortable making mistakes. You need to look for feedback each time so you can work to improve. There needs to be no pressure: this is why it is hard to practice in a project environment. it helps to keep it fun: make small steps forward when you can. Finally, you’ll recognize a good practice session because you’ll came out of it knowing more than when you went in.

Code Kata is an attempt to bring this element of practice to software development. A kata is an exercise in karate where you repeat a form many, many times, making little improvements in each. The intent behind code kata is similar. Each is a short exercise (perhaps 30 minutes to an hour long). Some involve programming, and can be coded in many different ways. Some are open ended, and involve thinking about the issues behind programming. These are unlikely to have a single correct answer. I add a new kata every week or so. Invest some time in your craft and try them.

If you want to discuss kata, there’s a mailing list here, and a wiki here. However, remember that the point of the kata is not arriving at a correct answer. The point is the stuff you learn along the way.

KataOne:	Supermarket pricing. Pricing looks easy, but scratch the surface and there are some interesting issues to consider.
KataTwo:	Karate Chop. A binary chop algorithm is fairly boring. Until you have to implement it using five totally different techniques.
KataThree:	How Big, How Fast? Quick estimation is invaluable when it comes to making design and implementation decisions. Here are some questions to make you turn over the envelope.
KataFour:	Data Munging. Implement two simple data extraction routines, and see how much they have in common.
KataFive:	Bloom Filters. Implement a simple hash-based lookup mechanism and explore its characteristics.
KataSix:	Anagrams. Find all the anagram combinations in a dictionary.
KataSeven:	Reviewing. What does our code look like through critical eyes, and how can we make our eyes more critical?
KataEight:	Objectives. What effects do our objectives have on the way we write code?
KataNine:	Checkout. Back to the supermarket. This week, we’ll implement the code for a checkout system that handles pricing schemes such as "apples cost 50 cents, three apples cost $1.30."
KataTen:	Hash vs. Class. Is it always correct to use (for example) classes and objects to structure complex business objects, or couple simpler structures (hash as Hashes) do the job?
KataEleven:	Sorting it Out. Just because we need to sort something doesn’t necessarily mean we need to use a conventional sorting algorithm.
KataTwelve:	Best Sellers. Consider the implementation of a top-ten best sellers list for a high volume web store.
KataThirteen:	Counting Lines. Counting lines of code in Java source is not quite as simple as it seems.
KataFourteen:	Trigrams. Generating text using trigram analysis lets us experiment with different heuristics.
KataFifteen:	Playing with bits. A diversion to discover the pattern in some bit sequences.
KataSixteen:	Business Rules. How can you tame a wild (and changing) set of business rules?
KataSeventeen:	More Business Rules. The rules that specify the overall processing of an order can be complex too, particularly as they often involve waiting around for things to happen.
KataEighteen:	Dependencies. Let’s write some code that calculates how dependencies propagate between things such as classes in a program.
KataNineteen:	Word chains. Write a program that solves word chain puzzles (cat -> cot -> dot -> dog).
KataTwenty:	Klondike. Experiment with various heuristics for playing the game Klondike.
KataTwentyOne:	Simple Lists. Play with different implementations of a simple list.

Practicing Programming by Steve Yegge

Practicing Programming

Stevey's Drunken Blog Rants™

Back in October I wrote an essay in which I compared programming to other professions. In it, I made the unsubstantiated claim that programming is unusual, in that most programmers don't practice their craft -- at least, not in any disciplined or regular way. Those are, of course, fightin' words, so I figured I'd write a bit more about it. This essay is a sort of mini-manual about practicing to be a better programmer.

What exactly does it mean to practice becoming a better programmer?

Well, of course the boring dictionary definition is: "To do or perform something repeatedly in order to acquire or polish a skill." That definition feels a bit too narrow, though. I also want it to include the idea of studying, which the dictionary equally boringly defines as: "To apply one's mind purposefully to the acquisition of knowledge or understanding of a subject."

If you're still awake after that mind-numbing paragraph, then I think you're ready to give it a try! Studying has a weird way of putting me to sleep, unless I'm doing it because I want to. Homework just seems to kill the desire to learn. Don't read this blog if someone is insisting that you read it! Wait until you really think you want to get better at programming, then read it.

Incidentally, I toyed with the idea of coining a new word for "study and practice", like I did with servware. Unfortunately, "practudy" sounds like some sort of medical problem, and "studtice" is just unthinkably bad ("Honey, I'll be home late tonight -- I'm working on my studtice.") So for this essay I'll just use the word practice to mean study and practice.

Contrary to what you might believe, merely doing your job every day doesn't qualify as real practice. Going to meetings isn't practicing your people skills, and replying to mail isn't practicing your typing. You have to set aside some time once in a while and do focused practice in order to get better at something.

I know a lot of great engineers -- that's one of the best perks of working at Amazon -- and if you watch them closely, you'll see that they practice constantly. As good as they are, they still practice. They have all sorts of ways of doing it, and this essay will cover a few of them.

The great engineers I know are as good as they are because they practice all the time. People in great physical shape only get that way by working out regularly, and they need to keep it up, or they get out of shape. The same goes for programming and engineering.

It's a bit easier to tell if someone's in great shape physically than if they're in great shape mentally. You can't just stare at their brain and hope to find a six-pack in all those folds. It's easy to tell how physically fit someone is. You can make people run laps, lift things, take their physical measurements, etc.

But for determining someone's mental fitness, you pretty much have to interview them. It it's hard to do a good job of it, since it's like running backwards in front of the person, egging them to go faster. You have to be in pretty good shape yourself to be a good interviewer.

Smarts vs. Skills

If you need a cornerback for your football team, of course you want someone who's in outstanding shape. But they also have to know how to play football -- and in particular, how to play cornerback. Great football players can sometimes play multiple positions, but most positions aren't naturally fungible; you can't usually take an offensive lineman and make him a great cornerback or quarterback.

There are some parallels here with finding great programmers. You want people who are smart, and also who have common sense (both the regular kind, and the software kind.) Smarts and common sense are equivalent to being in good shape, and perhaps having good reflexes.

But football -- well, if you know anything about American football, you'll know that it's a pretty rich sport. It's more like playing chess than playing soccer. It has elaborate rules; simply knowing the edge-case rules in the NFL rule book is a significant feat. It also has elaborate plays with amazing complexity and diversity, even though it just looks like guys hitting each other for 10 seconds at a time. And football has traditions, uniforms, statistics, all kinds of different coaching roles, referee organizations, tournaments and championships, aftermarket products, games and video games, specialized recruiters, specialized announcers... Football is a way of life.

So for that cornerback we need, is it good enough for him to be in good shape? Hardly. Maybe it's sufficient for a position on a high-school junior-varsity team, but Amazon's supposed to be more like the NFL than high school J.V., isn't it? I think so.

We hire for smarts and for skills. Well, we try, anyway. Sometimes, in a pinch, we'll settle for one or the other. Our hiring philosophy is that if someone is smart and motivated, then they should be able to make up for any particular skills they might be lacking, provided they have "enough" of the basic skills according to some ill-defined but hopefully intuitive heuristic.

As it happens, we don't get much time on the job to improve our skills. For the most part, it's up to us, as engineers, to take our own training into our own hands. Yes, there's some training, and there's some mentoring, and there are book club discussions and so on. But in the NFL, people practice daily. That's what practicing is all about -- doing things repeatedly, daily, habitually, to get better as fast as possible.

I doubt we or any company is likely to set up organized daily practice for their engineers. In fact I personally don't think it should be necessary. The most important thing you learn in college is how to learn on your own. They teach you how to research, and how to apply the scientific method and question your own findings, and they give you the fundamentals of math/language/social sciences/etc., so that when you want to learn something, you know how to figure it out for yourself.

Unlike the NFL, programming is a profession for which practicing is usually most effective if you do it alone, or at most with one other person. It requires quiet concentration and deep thinking, and you absolutely must solve the problems yourself, rather than just seeing the solution done for you. So even if we wanted to set up organized daily practice, most of it would be spent reading or programming quietly.

In a nutshell: if you're a programmer, you need to take matters into your own hands, and train yourself.

Programming happens to be a discipline in which you can squeeze practice drills into nooks and crannies in your schedule; you don't need to get into uniform and allocate a 3-hour session for it. That's why I'm offering you these practice drills. It's so you can practice the things you need to be good at in order to play on an NFL-quality servware development team.

But I have a job already

Why bother practicing your skills if you've already made it past the interview, and you've landed yourself a job here?

Well, what do you suppose happens if you don't practice, and you gradually get all mentally fat and out of shape?

Easy enough to answer! Go to www.monster.com (or our resume pipeline) and look for SDE resumes from people with 20 to 30 years of experience. Well hey looky, they're mostly Fortran and Cobol experts, on IBM mainframes, and they've never done any web programming (or in fact anything you've even heard of before.) They don't even know Unix, which is only like 100 years old now, in dog years anyway.

Those folks, alas, are obsolete dinosaurs, and their programming knowledge is formally classified under "archaeology". They never pass our interviews, and they're becoming less and less able to find jobs. No wonder it's so hard to find senior SDEs -- most of them are just doing their jobs for 25 years, and now they don't know anything of value.

We do keep searching for senior SDEs, though, because every once in a while we find someone with great training and great experience and great practice habits. These folks are often phenomenal, and they make all the search efforts worthwhile.

Of all of your skills as a programmer, how many of them could be considered "timeless?" Face it: most of your technical knowledge has a shelf life, an expiration date. A good exercise (let's make it our first practice drill) is to enumerate all the skills you've acquired that are relevant to your job as a programmer. Sort them into two buckets, based on whether the skill will still be useful 100 years from now.

Practice Drill #1: Write your resume. List all your relevant skills, then note the ones that will still be needed in 100 years. Give yourself a 1-10 rating in each skill.

This drill will help you see where you need practice. It won't turn up your "blind spots" -- i.e., areas that you don't know anything about (hence aren't on your resume) but that you should know something about. But it'll at least help you see how current your working skillset is, and how long you expect it to stay current. And, hey, it's always nice to have an up-to-date resume.

What I think you'll likely find is that math, computer science, writing, and people skills are for the most part timeless, universal skills. Most specific technologies, languages and protocols eventually expire, to be replaced by better alternatives.

Technologies aren't replaced by switching them off overnight. Once a better alternative comes along, the original technology's usage follows an exponential decay curve, with a half-life: say, the time it takes for the technology to be used in only half as many places. And some technologies survive by evolving (XML is a good example); their half-life is the time it takes for your knowledge of that technology to become only half as valuable. With evolving technologies like Java or XML, your knowledge will decrease in value if you don't keep current.

So you need to practice -- not just to improve your skills, but also to keep yourself from becoming obsolete. Studying by itself doesn't cut it, either; you have to use a technology in order to gain any real familiarity with it. That's why this essay is really about study and practice. Great programmers do both.

The majority of programmers have literally no idea how to practice, since nobody teaches it. So they simply don't do it. The first step towards becoming good at practicing is knowing a thing or two about practice itself. Practicing for anything is generally best done via drills: short, high-intensity exercises designed to yield the highest return on the time you invest. In this essay I'll give you a dozen or so drills that you can practice, in any order, at any time, as often as you like. None of them should take more than an hour.

However, before I get into some standard drills that good programmers do regularly, I'm taking you on a quick detour, so we can look at how practicing works in professions that have been around for centuries. Why? Because those folks are pretty darn good at it by now, and they may have some lessons for us.

How most musicians practice

I've had a fair amount of classical music training, so I know how musicians practice -- and what differentiates the good ones from the bad ones. Because I used to be an extraordinarily bad one.

Picture the average amateur guitarist: A teenager. Messy hair. Cheap guitar. Plays alone, or for baked friends in bedroom in parents' house. Knows a few riffs, a few licks. Can play almost every track on the first two Nirvana CDs. Puts on a good show for an hour, if you're a forgiving listener.

OK, got it. The average guitarist sucks.

How does the average guitarist practice? In the years before I started getting serious about lessons, I played a lot -- 6 to 8 hours a day for about 5 years. I learned a lot of songs, all by memorization, and I had to play them constantly to keep them in memory, so at least 2 hours of every day was wasted just running through the pieces. Through brute-force effort I eventually started to sound like I knew what I was doing. Fooled myself and most of the people around me, anyway.

I was practicing WAY too much, and getting very little out of it.

I felt I could muddle through almost anything, but it was clear that I became progressively sloppier as the music became more technically challenging. And there were some things I just couldn't play. I could memorize them, sure, but I'd get halfway through them and my hands would just stop working, from sheer exhaustion. Then I'd watch professional guitarists play the piece (or solo, or whatever), and I'd be surprised at how effortless they made it seem. How could they be so relaxed?

The problem was that I had no idea how to practice correctly. The saying "practice makes perfect" is inaccurate, as any music teacher will happily tell you. Perfect practice makes perfect. I'd been practicing sloppily, and had become very good at being sloppy. For one thing, I was tensed up, trying to force my fingers to make the right moves. So I only knew how to play tensed up, which exhausts you quickly. I was actually doing all sorts of things wrong, more than I'd ever have guessed, but the details aren't important. What's important is that I was thinking about it all wrong.

I knew that everyone said you should take lessons, but I had convinced myself that I didn't need them. I was actually a bit afraid to take lessons, because instructors were telling me I'd have to "forget everything I knew and start from scratch." That was a stupid way to attract new students! Nobody's going to want to throw away years of work. It was also incorrect: lots of the stuff I knew carried forward. Learning the proper technique turned out to be more like learning a new song than learning a new instrument. But at the time, I thought: "Screw that. I know how to play guitar. I'm happy with my playing, and I'm not going to change the way I play."

I hope you don't think this discussion is too far afield, because my attitude towards guitar lessons was identical to the way most programmers feel about their technical skills. "I'm already great at Perl, so I don't want to go back to the beginning and learn C, or assembly-language. I like the way I program." Or: "I'm great at Java, and I don't see any reason I should have to learn how to write scripts. I can get by just fine without them."

The thing is: I wasn't a great guitarist, and Perl-only folks aren't great at Perl. But you can't see that until you've done the hard work of learning what your instructors are telling you to learn.

Practice Drill #2: Make a list of programmers who you admire. Try to include some you work with, since you'll be borrowing them for some drills. Make one or two notes about things they seem to do well — things you wish you were better at.

Simply thinking about good programmers you know, and what makes them good, is good practice in itself. But we'll also use the results of this drill in some later drills.

Anyway, let's compare the average guitarist's practice techniques (and philosophy) to the way real musicians do it.

How great musicians practice

Professional musicians have such a rigorous practice methodology that it takes a while to fully comprehend it, let alone apply it. To get the full picture, you need to take lessons from different people, watch other musicians (both good and bad ones) practice, and experiment a lot.

This isn't unique to music. People practice rigorously in other disciplines too. Take golf, for instance. Even Tiger Woods still takes regular lessons. He's working more on course management than on his swing, of course, but he still takes lessons. In hyper-competitive disciplines, even the practice techniques themselves are improving over time.

Real musicianship is the result of studying and applying the theory, history, and performance of music. Many musicians also advocate studying the physics of sound and music, the construction of musical instruments, the mechanics the human hand and ear, and the psychology of performers and audiences.

The average guitarist is no more aware of these sub-disciplines than your average laborador retriever. I sure wasn't. I just wanted to play guitar. Remember that awful Antonio Banderas movie, The Mask of Zorro? Yeah, I know. Painful. But at one point, Anthony Hopkins asks Banderas if he knows how to use a sword, and Banderas replies: "Yes. The pointy end goes in the other man." That remark actually hit home. It's right around the level of sophistication you find in the average guitarist, and (alas) in the average programmer.

Music theory is rich and complex. Most guitarists don't see any reason to bother with it, since you can "get by" without it. (I detest that phrase. Saying you can "get by" without learning anything new is a sure way to make my eyes glow red.)

Yes, you can get by without knowing any theory. But your playing is just parroting if you don't understand how the piece was constructed. Knowing music theory can improve your playing in a hundred subtle ways, with the net effect being a much more professional performance.

Real musicians read sheet music almost as easily as you and I read English. (Better, for some substitutions of the variable "you". Not you, of course. Other yous.) It's amazing to watch. But most guitarists can't read sheet music. Instead, they use a crude pictogram notation called "tabulature", which doesn't contain enough information to know how to play a piece without hearing it first. It's as lame as it sounds, trust me. Being able to read real sheet music means you don't have to worry as much about committing pieces to memory. It also gives you access to a lot of great music that's never been recorded.

Sheet music is also beautiful in its own right.

Musicians study the history of music for many reasons. One reason is simply to know why things are the way they are. Another is to learn about great past Masters and their extraordinary abilities, and perhaps set higher personal goals. Mostly, though, it's a matter of culture. All the best musicians are totally immersed in the musical world. The best musicians share ideas and push the envelope of their field; music history gives them a vocabulary and a set of examples to draw on when fleshing out new ideas. Knowing your history is important for innovation.

The history of programming, incidentally, is absolutely fascinating. They were solving problems 30 and 40 years ago that are completely relevant to our work today, and most of us (me included) still don't understand the solutions. The pioneers of Computer Science -- the Donald Knuths, the Von Neumanns, the Dijkstras and Hoares -- these folks are geniuses, like the Masters of music, and it pays to study their work and their lives. The history of our field makes for a a richly rewarding study.

Practice Drill #3: Go to Wikipedia's entry for computer science, scroll down to the "Prominent pioneers in computer science" section, pick a person from the list, and read about them. Follow any links from there that you think look interesting.

If you run out of people to read about, there's also a list of computer scientists. Spending a few hours a week in Wikipedia is a fun, easy way to get a feel for our field and related fields. Make sure you pay attention to the the history and the people. Makes it more fun.

Finally, there's music practice. There are sooo many types of practice. The common characteristic among them is that practice has to be habitual. Professional musicians develop daily and weekly practice habits that they keep up for their entire careers. Practice requires a recurring time commitment.

One type of practice is simply to go listen to other musicians play, as often as you can. You'll learn a lot just by watching and listening. The analogs in the programming world are watching other people program, and reading their code.

Practice Drill #4: Read through someone else's code for 20 minutes. For this drill, alternate between reading great code and reading bad code; they're both instructive. If you're not sure of the difference, ask a programmer you respect to show you examples of each. Show the code you read to someone else, and see what they think of it.

Another kind of musical practice, useful for avoiding memory lapses during recitals, is to close your eyes and envision yourself playing the piece, measure by measure. It helps to try doing the measures out of order -- starting at arbitrary points, working backwards, etc. The point is to build up multiple mental models of the piece, to gain a more complete and robust understanding of it. There are analogs for this in programming that I'll cover in some drills later on.

When can I play the dang thing?

A lot of practicing, of course, involves actually playing your instrument. I bet you thought I'd never get to that. But first (ha!) you need to learn a bit about the instrument's physical characteristics: how it responds to temperature and humidity changes, how the response decay varies with pitch and octave, how the strings sound when struck in different places, even how the instrument responds to the friction of your clothes. It'd be kind of embarrassing to drop the ol' guitar in the middle of a performance. But not as embarrassing as having your zipper scratch a $9000 guitar you haven't bought yet.

Heck, even tuning a guitar isn't a simple proposition. After you change a string's tension, it will stretch or contract slightly over the next few minutes in the direction opposite to the change you made. And you have the Pythagorean Comma to worry about; tuning an instrument "perfectly" is actually a physical impossibility, so you have to figure out how to strike the right balance for whatever key you're playing in. You could write a 30-page manual on the art and science of tuning a guitar.

Why am I belaboring the tuning thing? Because learning how to tune your guitar properly is basically a tools issue. Many guitarists are perfectly happy to get by with poor tuning, but then they sound bad even if they're playing well. Developers are often content to use whatever tools they've got, without digging in and figuring out how to "tune" the tools for maximum efficiency. Mastering the tools of the trade is an important part of every professional's ability to be effective.

Practice Drill #5: Make a list of your 10 favorite programming tools: the ones you feel you use the most, the ones you almost couldn't live without. Spend an hour reading the docs for one of the tools in your list, chosen at random. In that hour, try learn some new feature of the tool that you weren't aware of, or figure out some new way to use the tool.

By starting with the tools you already love using, it'll be that much easier to get into the habit of exploring their capabilities.

And finally, real musicians don't practice by playing the piece over and over from beginning to end. They dissect every piece of music into tiny components and work on each one individually -- every phrase, every note, every fingering, every transition, it's all worked through, mechanically and musically, for countless hours. They play it slow, fast, quietly, loudly, even in different time signatures and beats. And they do daily drills: right- and left-hand finger exercises for building stamina and dexterity.

Sound like a total pain? Actually, it's not too bad at all. Takes a bit of getting used to, but once you start doing it right, your overall technique improves rapidly. And you're no longer "capped" at a particular difficulty level, because you're not exhausting yourself, and you know how to tackle complex technical passages. Oh, and one hour of that kind of practice is as good as a week of playing songs over and over.

OK, enough about music already. The takeaway is that musicians have built up intricate and very effective practice techniques through centuries of experimentation. It takes formal instruction and a fair amount of personal discipline to master professional practice techniques. You wouldn't think of most of them on your own, at least until you get the feel for the overall goals of the practice.

We don't have centuries of accumulated experience about programming, so we're still sort of feeling our way along. But programming is a clearly a complex blend of art, science, logic, engineering, design, and craftsmanship, and we can borrow ideas from all those endeavors.

Hopefully I've convinced you that you can't become world-class (or even a competent professional) in any profession, including programming, without making a serious commitment to ongoing study, training, and practice.

You already knew that, though. The fact that you've read this far into my crazy blog means you're probably already the type who practices a lot. Or you like to read about it at least, like that Dilbert where he's reading a book about driving the cart around in a golf video game.

Practice Drill #6: Pick something you're good at that has nothing to do with programming. Think about how the professionals or great masters of that discipline do their practice. What can you learn from them that you can apply to programming?

Now, with your new-found insight, go chop a brick in half with your hand! Ouch. Better yet, go play some Doom 3. You can't study all the time.

Interviewing is great practice

One of the best ways to get better as a programmer is to participate in the recruiting process: resume screening, phone screening, interviewing, having people mock-interview you, coming up with new interview questions, solving other peoples' interview questions yourself, all that stuff.

Of course, if you just ask the same questions in interview after interview, you're not really practicing, and it won't help you much.

Here are some recruiting-related drills. Just fer fun 'n stuff. Is anyone really still reading this thing? You probably all think I've cracked. But read on! I believe I can make the unqualified promise, with no hedging whatsoever, and backed by my personal guarantee and my word as a sportsman and a gentleman, that this essay will eventually end.

Resume screening is a useful practice -- and not just for the comic relief, although that's certainly one of its draws. It's also useful so you can see what people in your industry are doing these days. Unemployed ones, anyway. And if you see a particular buzzword appearing more and more often, it's a sign that you might want to look into it, and see why it's popular.

Practice Drill #7: Get a pile of resumes and a group of reviewers together in a room for an hour. Make sure each resume is looked at by at least 3 reviewers, who write their initials and a score (1-3). Discuss any resumes that had a wide discrepancy in scoring.

Group resume screens are way more fun than doing it by yourself, and probably more accurate as well. You need at least four or five people, though. The discussion keeps you from falling asleep.

Phone screening is really hard to do well. It may not actually be that useful for improving any skills other than phone screening. However, recruiting is an important part of every programmer's job, so you should do phone-screens occasionally.

For the drill, we'll just have you listen in on someone else's phone screen. That way you might hear a question you don't know the answer to, which gives you something to go study.

Practice Drill #8: Listen in on a technical phone screen. Write up your feedback afterwards, cast your vote, and then talk about the screen with the screener to see if you both reached the same conclusions.

And, of course, go figure out how to solve anything the screener asked that you didn't know the answer to.

Incidentally, I sometimes listen in on phone screens, and I bring a pen and paper, so I can doodle during the boring lulls, where the screener is saying loudly into the speakerphone: "Did you say paren paren backslash slash? Or was it paren paren slash backslash slash?" I'll use the pen and paper to experiment with solutions to some of their questions, as well as take notes on the screen.

Interviewing is absolutely one of the best ways to improve your own technical skills.

For one thing, it requires you to invent and present challenging technical questions, then conduct deep-dive discussions of them with interview candidates. That's just plain good exercise.

For another, you can ask candidates to teach you new stuff. For instance, if they did a Masters thesis or Ph.D. on some exotic-sounding CS subject, you might have them try to explain it to you. If they do a great job, well, you'll have learned something new! And if they do a terrible job, you'll at least have learned something interesting about the candidate -- namely, that they can't explain something they know well, which implies that they may not understand it very well.

It doesn't necessarily have to be a thesis topic. You might run into a domain expert in a domain similar to one you're familiar with. For instance, if you're interviewing an SQL Server expert, you might ask how SQL Server handles a particularly thorny problem you've experienced with Oracle. Or if you're interviewing someone who's a fan of a technology you haven't heard of, you might ask them to give you a high-level overview of it.

Interviews can basically be free education, and you might as well take advantage of them.

Practice Drill #9: Conduct a technical interview with a candidate who's an expert in some field you don't know much about. Ask them to explain it to you from the ground up, assuming no prior knowledge of that field. Try hard to follow what they're saying, and ask questions as necessary.

Sometimes their explanation is just gobbledygook, and I don't get much out of it. That's usually because it requires more math than I have handy, so I'll spend a day hitting the math books after that happens. (It's never enough, but it's good practice anyway.)

Sometimes you'll be able to follow them, ramp up quickly on the idea, start probing in complex corners, and wind up with a really good discussion. Or sometimes you'll find the candidate can't answer even trivial questions about the subject, which can be useful data for the interview.

Pair-interviewing is also useful. All you need to do is listen in while someone else conducts the interview (although sometimes pair interviewers divide up the questions, or give each other a chance to ask stuff.)

Pair interviewing is great because it gives you new questions to ask, and it shows you new approaches to managing the flow of the interview. Everyone runs interviews a little differently, and being exposed to multiple styles will make you a better interviewer.

Practice Drill #10: Get yourself invited to someone else's technical interview. Listen and learn. Try to solve the interview questions in your head while the candidate works on them.

Nowadays I make it a habit to try to invite someone to every one of my interviews.

Some people do Mock Interviewing to hone their skills. You just pick a random person and have them ask you an interview question, and solve it right there on the board. If you fail the question, you have to go read up on that area. No big deal.

Ron Braunstein comes into my office every two or three weeks, writes a problem on the board, and asks me to solve it right then and there. (I do the same to him almost as often.) It's usually a problem that he just finished solving, and thinks it might be an interesting interview question or programming challenge. I work through it, then we talk about it and compare our approaches.

You should do this once in a while. It's good for you. Sometimes you'll do great, and sometimes you'll get stuck. Just like in real life.

Practice Drill #11: Find a buddy for trading practice questions. Ask each other programming questions, alternating weeks. Spend 10 or 15 minutes working on the problem, and 10 or 15 minutes discussing it (finished or not.)

If your buddy's questions are always too hard, my recommendation is to find yourself a new buddy. It can be a bit deflating when your buddy votes not to hire you 3 or 4 weeks in a row.

One last interviewing-related drill: make sure you can solve any coding question that another interviewer asks. I overhear interview questions occasionally that I'm not immediately sure how to solve. I file them away and go implement them later in the week.

Practice Drill #12: When you hear any interview coding question that you haven't solved yourself, go back to your desk and mail the question to yourself as a reminder. Solve it sometime that week, using your favorite programming language.

If you struggle with it, e.g. it takes more than an hour, you may have a gap in your knowledge, which is wonderful, because now you know and can fix it. However, it might also mean that the interview question is too hard. You might bring it to a Bar Raiser or another experienced interviewer and ask them if they think it's a reasonable SDE interview question.

Wrap-up

I was planning on trying to make up 20 practice drills, but the effort basically killed this blog entry. It sat unfinished for over 2 months. Instead, I'll stop at a dozen, and maybe mention some more drills in a future blog. There are plenty of others, but hopefully this gives you a few to work on.

If you have ideas for other practice drills, feel free to submit them in the comments section. I'm willing to try anything that's worked well for others.

(Published Jan 23, 2005)

---

Comments

---

Dave Thomas has a long blog thread on practicing programming. This gives 21 suggested exercises, or 'code kata'.

http://blogs.pragprog.com/cgi-bin/pragdave.cgi/Practices/Kata

On the music side of things, I'm learning to play piano, and teaching myself because two years of lessons did nothing but lighten my wallet. It seems that practicing techniques are still fairly hotly debated even now; some people maintain that non-musical drills such as those by Hanon and Czerny are worthless; it's better to use, say, Bach Two Part Inventions or Chopin Etudes, because technique should never be divorced from music.

Personally I found this free online book immensely helpful: http://members.aol.com/chang8828/contents.htm

(He has a good reviews second, where he recommends other materials; I went with Seymour Fink's book, and I'm about to buy the 'Freeing The Caged Bird' video.)

Also, I found this book reading this book right now, which combines elements of Tai Chi and Feldenkrais and an apparent unification of the arm-weight/finger-movement schools. The author is a bit full of himself, but the first couple of chapters (on the bus this morning) were good. I guess I should blog this...

http://www.amazon.com/exec/obidos/asin/0810845911

Different Definitions of Software Architecture

Community Architecture Definition at the Carnegie-Mellon University Web Site

What is a Software Architecture (from IBM Rational Edge)

Software Architecture

Computational Thinking

A nice article by Jeanette M. Wing (Head of the Computer Science Department, Carnegie Mellon University, Pittsburgh, PA)

Quality is #1

We just shipped v2 of our project - but few are cheering. To meet our dates we dropped quality on the floor (reliability, usability... you name it) and everyone knows it. There's already talk about what commitments we have for v3, but no one has articulated what we're going to do about raising the quality bar.

How do you (successfully) argue for time for higher quality? Has anyone worked on a project where quality was really job #1? How did it happen? Who defined (and defended) quality?

- Quality is job #1

Response by Neil Enns, Product Manager, Microsoft Visual Studio
---------------------------------------------------------------

In my product (Visual Studio) we took four months off feature work after shipping our last release, solely to focus on quality (called “Milestone Quality” or “MQ” for short). It wasn’t just product quality, but also engineering process quality. We automated thousands of test cases. We scrubbed our postponed bug database. We educated the development team on the latest in testing processes (such as TDD), and built tools to help developers implement those processes.

What was so cool about the four months is that it wasn’t just about fixing bugs in the product. It was much more about seriously looking at our engineering processes and being smart about what needed to change, and investing in the tools to support those changes.

This only works if everyone up the chain buys in, and it has to be driven from as senior a level as possible. In our case, Soma was adamant that we do this, and every single person in the division participated.

You can read a little more about this effort at Soma’s blog (http://blogs.msdn.com/somasegar/archive/2005/11/08/490694.aspx) and Eric’s blog (http://blogs.msdn.com/eric/archive/2005/11/04/489108.aspx).

Neil

The Original Google Paper

The Anatomy of a Large-Scale Hypertextual Web Search Engine

Structure & Interpretation of Computer Programs - The Complete Book Online

Structure & Interpretation of Computer Programs

IBM Rational Software Development Conference 2006

I'll be speaking at this year's conference on the Complexities of Modernizing a Legacy Software System.

IBM Rational Conference Website

Frameworks and The Hollywood Principle

THE HOLLYWOOD PRINCIPLE

The Template Method pattern leads to an inversion of control known as
the "Hollywood Principle," or, "Don't call us; we'll call you."
Subclasses can extend or reimplement the variable parts of the
algorithm, but they cannot alter the template method's flow of control
and other invariant parts. Therefore when you define a new subclass of
Node, you have to think not in terms of control flow but
*responsibility*---the operations you *must* override, those you *might*
override, and others you *mustn't* override. Structuring your
operations as template methods makes these responsibilities more
explicit.

The Hollywood Principle is a key to understanding frameworks. It lets
a framework capture architectural and implementation artifacts that
don't vary, deferring the variant parts to application-specific
subclasses.

The inversion of control is part of what makes framework programming
uncomfortable for some. When programming procedurally, one is very
much preoccupied with control flow. It's hard to imagine how you can
understand a procedural program without knowing the twists and turns
it takes, even with impeccable functional decomposition. But a good
framework will abstract away control flow details. You end up
focusing on objects, which can seem both more and less tangible than
control flow. You think in terms of object responsibilities and
collaborations. It's a higher-level, slightly more declarative view of
the world, with potentially greater leverage and flexibility. The
Template Method pattern realizes these benefits on a smaller scale
than a framework---at the operation level rather than the object
level.

Programmer's Bookshelf

Programmer's Bookshelf

Joel on Software

Joel On Software

How to be a Programmer

An interesting article by Robert L. Read

Software Architecture As A Discipline

What is a discipline? To be called a discipline, a field must satisfy three criteria:
a) it has a set of formalisms, methods, models, and processes that are used regularly to produce results,
b) it has a set of standard practices including generally accepted community standards for lcvels of competent performance and criteria for excellence, and
c) its practitioners can reliably fulfill standard promises to the satisfaction of clients.
The third criterion is the most important and least discussed. The question is not whether formalisms and practices are properly balanced, but what formalisms and practices are necessary so that practitioners can keep their promises to clients.

Architecture satisfies the criteria for a discipline: it has formalisms and methods for analyzing and constructing structures from
given materials; it has standard practices such as blueprints, styles of buildings, building codes, and means of interacting with building contractors; it has community standards for the levels of competence that architects can have; and its practitioners can make and fulfill promises to deliver plans and buildings on time and within budget. Schools of architecture teach aspiring architects with combinations of presentations, practice with formalisms, and projects of increasing sophistication
carried out under the guidance of an experienced architect.

One of the architect’s principal products is a map of the building-the blueprint. Working from sketches and interviews, the
architect must come to understand the interests of the client including form, function, social relations, styles of those who will use the building, efficiency, and affordability. This understanding is manifested in the blueprint. The same blueprint conveys sufficient information that the building contractor can calculate the time, materials, and costs to assemble the structure. It is used later to assess the building after people move in and to reconfigure portions of it if they demand changes.
A discipline of sofnvare architecture would be capable of training its practitioners to systematically fidfill promises to build and install software systems that are judged useful and dependable by their customers. To accomplish this we need ultimately to:
1. Understand the linguistic structure common to all domains of action, so that one can observe a particular domain for its basic distinctions, repetitive processes, standard practices, standards of assess ment, strategies, and driving concerns.
2. Know how to connect the linguistic structure of the domain to software structures that will assist in carrying out standard actions and know how to coordinate the implementation of those structures.
3. Know how to assess the satisfaction of people in the domain so that their effectiveness in the presence of the software can be measured.

The software architect must know the “‘cartography” of sofiware blueprints in general and must be able to develop specific blueprints for each new system. Wmograd and Flares discussed the framework in which design can take place . The common elements of every domain of action are:
 A set of “linguistic distinctions” (verbs, nouns, jargon, etc.) around which people in the domain organize their actions.
A set of “speech acts”-statements whose utterance initiates actions and signifies completions.
A set of “standard practices”-recurrent actions, organizational processes, roles, and standards of assessment-in which members of the domain engage.
A set of ‘tools and equipment” people use to perform actions. A tool is “ready to hand’ if the person using it does so with skill and without distraction.
Possible “breakdowns”-interruptions of progress caused by tools breaking, people failing to complete agreements, external circumstances, and the like.
A set of “ongoing concerns” of the people in the domain, for example, their common mission, interests, or fears.
The history and institutions of the domain.

They call the framework outlined above the “ontology” of the domain. Building on this terminology, we propose “ontological mapping’ as the name of the skill that a software architect must master [26]. The architect’s skill of creating a blueprint and using it to coordinate builders and later assessments is an example of this skill. The skill of ontological mapping can be learned and is the basis of a discipline of software architecture.


excerpt from 'A Discipline of Software Architecture' by Peter Denning and Pamela Dargan

The Ray Ozzie Memo

From: Ray Ozzie
Sent: Friday, October 28, 2005
To: Executive Staff and direct reports
Subject: The Internet Services Disruption

It is an exciting time, as we're at the beginning of the biggest product cycle in the company's history. In a week we ship new versions of Visual Studio, SQL Server and BizTalk Server. Later this month we ship Xbox 360. Next year we have a double barreled release of our two largest products with Windows Vista and Office "12". It's a great time for customers, our partners, and for those at Microsoft who have put so much of themselves into these products.

But we bring these innovations to market at a time of great turbulence and potential change in the industry. This isn't the first time of such great change: we've needed to reflect upon our core strategy and direction just about every five years. Such changes are inevitable because of the progressive and dramatic evolution of computing and communications technology, because of resultant changes in how our customers use and apply that technology, and because of the continuous emergence of competitors with new approaches and perspectives.

In 1990, there was actually a question about whether the graphical user interface had merit. Apple amongst others valiantly tried to convince the market of the GUI's broad benefits, but the non-GUI Lotus 1-2-3 and WordPerfect had significant momentum. But Microsoft recognized the GUI's transformative potential, and committed the organization to pursuit of the dream – through investment in applications, platform and tools – based on a belief that the GUI would dramatically expand and democratize computing.

When we reflected upon our dreams just five years later in 1995, the impetus for our new center of gravity came from the then-nascent web. With a clear view upon the challenges and opportunities it presented, the entire company pivoted to focus on the internet to pursue that 'fully connected' dream with support for internet standards throughout our product line: a web browser, server and development tools, and a service in MSN that was transformed into a web portal. Many things we developed in that era continue to fuel the growth of today's internet: the technologies of AJAX – DHTML and XMLHTTP – were created in 1998 and used in products such as OWA.

In 2000, in the waning days of the dot com bubble, we yet again reflected on our strategy and refined our direction. After taking a more deliberative look at the internet and its implications for software, we came to the conclusion that the internet would go beyond browsing and should support programmability on a global scale. We observed that certain aspects of our most fundamental platform – the tools and services that developers use when building their software – would not likely satisfy the emerging security and interoperability requirements of the internet. So we embarked upon .NET, a transformative new generation of the platform and tools built around managed code, the XML format and web services programming model. At the time, it was a risky bet to build natively around XML, but this bet paid off handsomely and .NET has become the most popular development environment in the world.

It is now 2005, and the environment has changed yet again – this time around services. Computing and communications technologies have dramatically and progressively improved to enable the viability of a services-based model. The ubiquity of broadband and wireless networking has changed the nature of how people interact, and they're increasingly drawn toward the simplicity of services and service-enabled software that 'just works'. Businesses are increasingly considering what services-based economics of scale might do to help them reduce infrastructure costs or deploy solutions as-needed and on subscription basis.

Most challenging and promising to our business, though, is that a new business model has emerged in the form of advertising-supported services and software. This model has the potential to fundamentally impact how we and other developers build, deliver, and monetize innovations. No one yet knows what kind of software and in which markets this model will be embraced, and there is tremendous revenue potential in those where it ultimately is.

Just as in the past, we must reflect upon what's going on around us, and reflect upon our strengths, weaknesses and industry leadership responsibilities, and respond. As much as ever, it's clear that if we fail to do so, our business as we know it is at risk. We must respond quickly and decisively.

The Landscape
Since 1995, inexpensive computing and communications technologies have advanced at a rapid rate that even exceeded our expectations. It's so very difficult now for us to imagine a world without the PC, the web and the cell phone. In the US, there are more than 100MM broadband users, 190MM mobile phone subscribers, and WiFi networks blanket the urban landscape. This pattern is mirrored in much of the developed world. Computing has become linked to the communications network; when a PC is purchased, it's assumed that the PC will have high-speed internet connectivity. At work, at home, in a hotel, at school or in a coffee shop, the networked laptop has become our 'virtual office' where we file our information and interact with others. The broad accessibility and rapid pace of innovation in hardware, networks, software and services has catalyzed a virtuous cycle whose pace isn't slowing. There has never been a more exciting time to be a developer or a user of technology.

Our products have embraced the internet in many amazing ways. We've transformed the desktop into a rich platform for interactive internet browsing, media and communications-centric applications. We've transformed Windows into best-of-breed infrastructure for internet applications and services. We've created, in .NET, the most popular development platform in the world. We've got amazing products in Office and our other IW offerings, having fully embraced standards such as XML, HTML, RSS and SIP. Our MSN team has demonstrated great innovation and has held its own in a highly competitive and rapidly changing environment – particularly with Spaces and in growing a base of 180M active Messenger users worldwide. The Xbox team has also built a huge user community and has demonstrated that internet-based "Live" interaction is a high-value, strong differentiator.

But for all our great progress, our efforts have not always led to the degree that perhaps they could have. We should've been leaders with all our web properties in harnessing the potential of AJAX, following our pioneering work in OWA. We knew search would be important, but through Google's focus they've gained a tremendously strong position. RSS is the internet's answer to the notification scenarios we've discussed and worked on for some time, and is filling a role as 'the UNIX pipe of the internet' as people use it to connect data and systems in unanticipated ways. For all its tremendous innovation and its embracing of HTML and XML, Office is not yet the source of key web data formats – surely not to the level of PDF. While we've led with great capabilities in Messenger & Communicator, it was Skype, not us, who made VoIP broadly popular and created a new category. We have long understood the importance of mobile messaging scenarios and have made significant investment in device software, yet only now are we surpassing the Blackberry.

And while we continue to make good progress on these many fronts, a set of very strong and determined competitors is laser-focused on internet services and service-enabled software. Google is obviously the most visible here, although given the hype level it is difficult to ascertain which of their myriad initiatives are simply adjuncts intended to drive scale for their advertising business, or which might ultimately grow to substantively challenge our offerings. Although Yahoo also has significant communications assets that combine software and services, they are more of a media company and – with the notable exception of their advertising platform – they seem to be utilizing their platform capabilities largely as an internal asset. The same is true of Apple, which has done an enviable job
integrating hardware, software and services into a seamless experience with dotMac, iPod and iTunes, but seems less focused on enabling developers to build substantial products and businesses.

Even beyond our large competitors, tremendous software-and-services activity is occurring within startups and at the grassroots level. Only a few years ago I'd have pointed to the Weblog and the Wiki as significant emerging trends; by now they're mainstream and have moved into the enterprise. Flickr and others have done innovative work around community sharing and tagging based on simple data formats and metadata. GoToMyPC and GoToMeeting are very popular low-end solutions to remote PC access and online meetings. A number of startups have built interesting solutions for cross-device file and remote media access. VoIP seems on the verge of exploding – not just in Skype, but also as indicated by things such as the Asterisk soft-PBX. Innovations abound from small developers – from RAD frameworks to lightweight project management services and solutions.

Many startups treat the 'raw' internet as their platform. At the grassroots level, such projects actively use standards such as vCards and iCal for sharing contacts and calendars. Most all use RSS in one way or another for data sharing. Remixing and mashing of multiple web applications using XML, REST and WS is common; interesting mash-ups range from combining maps with apartment listings, to others that place RSS feeds on top of systems and data not originally intended for remixing. Developers needing tools and libraries to do their work just search the internet, download, develop & integrate, deploy, refine. Speed, simplicity and loose coupling are paramount.

And the work of these startups could be improved with a 'services platform'. Ironically, the same things that enable and catalyze rapid innovation can also be constraints to their success. Many hard problems are often ignored – the most significant of which is achieving scale. Some scale issues are technological and result from the fact that they are generally built on application server platforms rather than high-scale service platforms. But new services also need to build user communities from scratch – generally by word of mouth. Many fund their sites using syndicated ads, but have a difficult time transforming their services into higher levels of commerce. Some seek to incorporate client software into their user experience, but then need to reinvent software deployment, update, communications and synchronization mechanisms. User identity and cross-service interoperability mechanisms are still needlessly fragmented. Intuitively there seems to be a platform opportunity in providing such capabilities to developers in a form that retains the speed, simplicity and loose coupling that is so very important for rapid innovation.

Key Tenets
Today there are three key tenets that are driving fundamental shifts in the landscape – all of which are related in some way to services. It's key to embrace these tenets within the context of our products and services.

1. The power of the advertising-supported economic model.

Online advertising has emerged as a significant new means by which to directly and indirectly fund the creation and delivery of software and services. In some cases, it may be possible for one to obtain more revenue through the advertising model than through a traditional licensing model. Only in its earliest stages, no one yet knows the limits of what categories of hardware, software and services, in what markets, will ultimately be funded through this model. And no one yet knows how much of the world's online advertising revenues should or will flow to large software and service providers, medium sized or tail providers, or even users themselves.

2. The effectiveness of a new delivery and adoption model.

A grassroots technology adoption pattern has emerged on the internet largely in parallel to the classic methods of selling software to the enterprise. Products are now discovered through a combination of blogs, search keyword-based advertising, online product marketing and word-of-mouth. It's now expected that anything discovered can be sampled and experienced through self-service exploration and download. This is true not just for consumer products: even enterprise products now more often than not enter an organization through the internet-based research and trial of a business unit that understands a product's value.

Limited trial use, ad-monetized or free reduced-function use, subscription-based use, on-line activation, digital license management, automatic update, and other such concepts are now entering the vocabulary of any developer building products that wish to successfully utilize the web as a channel. Products must now embrace a "discover, learn, try, buy, recommend" cycle – sometimes with one of those phases being free, another ad-supported, and yet another being subscription-based. Grassroots adoption requires an end-to-end perspective related to product design. Products must be easily understood by the user upon trial, and useful out-of-the-box with little or no configuration or administrative intervention.

But enabling grassroots adoption is not just a product design issue. Today's web is fundamentally a self-service environment, and it is critical to design websites and product 'landing pages' with sophisticated closed-loop measurement and feedback systems. Even startups use such techniques in conjunction with pay-per-click advertisements. This ensures that the most effective website designs will be selected to attract discovery of products and services, help in research and learning, facilitate download, trial and purchase, and to enable individuals' self-help and making recommendations to others. Such systems can recognize and take advantage of opportunities to up-sell and cross-sell products to individuals, workgroups and businesses, and also act as a lead generation front-end for our sales force and for our partners.

3. The demand for compelling, integrated user experiences that "just work".

The PC has morphed into new form factors and new roles, and we increasingly have more than one in our lives – at work, at home, laptops, tablets, even in the living room. Cell phones have become ubiquitous. There are a myriad of handheld devices. Set-top boxes, PVRs and game consoles are changing what and how we watch television. Photos, music and voice communications are all rapidly going digital and being
driven by software. Automobiles are on a path to become smart and connected. The emergence of the digital lifestyle that utilizes all these technologies is changing how we learn, play games, watch TV, communicate with friends and family, listen to music and share memories.

But the power of technology also brings with it a cost. For all the success of individual technologies, the array of technology in a person's life can be daunting. Increasingly, individuals choose products and services that are highly-personalized, focused on the end-to-end experience delivered by that technology. Products must deliver a seamless experience, one in which all the technology in your life 'just works' and can work together, on your behalf, under your control. This means designs centered on an intentional fusion of internet-based services with software, and sometimes even hardware, to deliver meaningful experiences and solutions with a level of seamless design and use that couldn't be achieved without such a holistic approach.

The Opportunities
These three tenets are causing a shift in the software landscape that started with consumers and is progressively working its way toward the enterprise – changing how software is monetized, how software is delivered, and what kind of software is ultimately embraced. With our presence in so many markets serving so many audiences, and with such a broad variety of products and solutions, we are well positioned to deliver seamless experiences to customers, enabled by services and service-enhanced software, including:

SEAMLESS OS – The operating system as it would be designed for today's multi-PC, multi-device, work anywhere, web-based world. Enabling you to login using any of your service-based or enterprise identities. Deploying software automatically and as appropriate to all your devices, and roaming application data and settings. Permitting seamless access to storage across all your PCs, devices, servers and the web.

SEAMLESS COMMUNICATIONS – Communications and notifications – from voice to typing to shared screen; from PC to service-based agent to phone. Maintaining continuous co-presence with intimate friends and family; improving the coordination amongst individuals who need to work together by reducing latency and adding clarity through shared context.

SEAMLESS PRODUCTIVITY – Enabling you to create, find and organize documents and data among all the desktops, devices, servers and services to which you have access, and with all the others with whom you need to work, through 'shared space' products that are internet service-based, enterprise server-based and directly peer-to-peer. Working within and across homes, small businesses, virtual workgroups and enterprises.

SEAMLESS ENTERTAINMENT – Enabling you to create, store, organize, present, consume and interact with media of all kinds; accessing, caching and viewing it anywhere you like regardless of where the media resides. Gaming experiences that bring two or two million people together across PCs, devices and the web.

SEAMLESS MARKETPLACE – Enabling you to research, find, buy and sell whatever you want through a seamlessly integrated purchase, billing & payment & points, advertising & lead generation & sales management system designed to satisfy the needs of both buyers and sellers.

SEAMLESS SOLUTIONS – Enabling workgroups and businesses to rapidly create and customize any of a broad class of template-driven, semi-structured data-based applications and solutions that "just work" and provide instant value – whether using them from the web, from enterprise servers, or from mobile client PCs.

SEAMLESS IT – Enabling enterprises to seamlessly and cost-effectively manage many of the things they've classically done within their data centers – e.g. PCs, messaging, content and applications. The management experience might be wholly within the cloud, or with the cloud seamlessly integrating enterprise server assist.

Moving Forward
In order to adapt to the requirements underlying these key tenets, groups must reflect upon their existing plans, and assess their designs in the context of the end-to-end experiences they need deliver in order to understand how services might make a substantive impact. Groups should consider how new delivery and adoption models might impact plans, and whether embracing new advertising-supported revenue models might be market-relevant.

In assessing where we are and where we need to be, some new efforts will surely require incubation. But in many areas we have 80% of the product and technical infrastructure already built – we just need to close the 20% gap. Following are but a few thoughts for each division intended to catalyze a "services-enhanced software" mindset.

Platform Products & Services Division
a. BASE vs. ADDITIVE EXPERIENCES – In MSN, and in Windows Update and software deployed by it, we have quite a bit of experience with methods and practices for getting innovations to market on a rapid cycle. In the form of a newly combined division, we should consider many options as to how we might bring user experience innovations and enhancements to users worldwide. Specifically, we should consider the achievability, desirability, and methods of increasing the tempo for both 'base' OS experiences as well as 'additive' experiences that might be delivered on a more rapid tempo. In doing so, we would better serve a broad range of highly-influential early adopters.

b. SERVICES PLATFORM – Through years of experience, the MSN team understands the methods and practices of building 'internet scale' services. The Platform team understands developers and has deep experience in communications and storage architectures. These teams must work together, benefiting from each others' strengths, to develop a next generation internet services platform – a platform for both internal and external innovation. A platform with capabilities and an operations infrastructure that takes those services to a scale never yet seen on the internet - to our benefit, and to the benefit of our partners and customers.

c. SERVICE/SERVER SYNERGY – A tension has emerged between our products designed for the enterprise and those for the internet. Exchange/Hotmail, AD/Passport, and Messenger/Communicator are but three examples. All our enterprise clients and servers must interoperate with and complement our internet services. Our functional aspirations are generally "server/service symmetry", but architectural considerations dictate that different implementations may be required to economically reach internet scale. We must quickly find the best path to achieve seamless user, developer, and administration experiences involving servers and services.

d. LIGHTWEIGHT DEVELOPMENT – The rapid growth of application assembly using things such as REST, JavaScript and PHP suggests that many developers gravitate toward very rapid, lightweight ways to create and compose solutions. We have always appreciated the need for lightweight development by power users in the form of products such as Access and SharePoint. We should revisit whether we're adequately serving the lightweight model of development and solution composition for all classes of development.

e. RESPONSIBLE COMPETITION – We will compete energetically but also responsibly and with recognition of our high legal responsibilities. We will design and license Windows and our internet-based services as separate products, so customers can choose Windows with or without Microsoft's services. We'll design and license Windows and our services on terms that provide third parties with the same
ability to benefit from the Windows platform that Microsoft's services enjoy. Our services innovations will include tight integration with the Windows client via documented interfaces, so that competing services can plug into Windows in the same manner as Microsoft's services. We will compete hard and responsibly in services on the basis of software innovation and price – and on that basis we will offer consumers and businesses the best value in the market.

Business Division
a. CONNECTED OFFICE - How would we extend or re-conceptualize Office modules to fit in this seamless model of connectedness to others, and to other applications? Should PowerPoint directly 'broadcast to the web', or let the audience take notes and respond? How should we increase the role of Office Online as the portal for productivity? What should we do to bring Office's classic COM-based publish-and-subscribe capabilities to a world where RSS and XML have become the de facto publish-and-subscribe mechanisms?

b. TELECOM TRANSFORMATION - How should our investments in RTC evolve to serve not just the enterprise, but also fully embrace the concept of grassroots adoption? How can RTC begin as an individual phenomenon, growing into a small business offering with a level of function that they'd never imagine possible, growing into the enterprise? How should we utilize service-based federation and hosting to ensure a 'just works' experience for all users, whether or not an administrator was ever involved?

c. RAPID SOLUTIONS - How can we utilize our extant products and our knowledge of the broad historical adoption of forms-based applications to jump-start an effort that could dramatically surpass offerings from Quickbase to Salesforce.com? How could we build it to scale to hundreds of millions of users at an unimaginably low cost that would change the game? How could we re-shape our client-side software offerings such as Access and Groove, and our server offerings such as SharePoint, to grow and thrive in the presence of such a service? Could these rapid solutions encourage a new ISV ecosystem and business model?

Entertainment & Devices Division
a. CONNECTED ENTERTAINMENT - How can XBox Live benefit from interconnection with other services assets, such as PC-based and mobile-based IM and VoIP? How might both the PC and XBox mutually benefit from a common marketplace? Might PC users act as spectators/participants in XBox games, and vice-versa?

b. GRASSROOTS MOBILE SERVICES – How might the Windows Mobile device experience be transformed by for consumers by connection to a services infrastructure – in particular one enabled by RTC-based unified communications? How might unmediated connection to a rich services infrastructure transform mobile phones into a mass market messaging, media and commerce phenomenon?

c. DEVICE/SERVICE FUSION – What new devices might emerge if we envision hardware/software/service fusion? What new kinds of devices might be enabled by the presence of a service?

What's Different?
One perspective on this memo might be to say "This is in many ways is pretty close to what we're already working on. What's the big deal?" Or "We tried something similar years ago; why will we succeed this time?" These are understandable reactions. Many visions of the future going all the way back to "Information at Your Fingertips" contain elements of what has been laid out here.

That said, I have a number of reasons for optimism that we can deliver well on this vision. First, I know that Bill, Steve and the senior leadership team understand that Microsoft's execution effectiveness will be improved by eliminating obstacles to developing and shipping products. The recent reorganization into three divisions is a significant step, and the division presidents are committed to changes to improve our agility.

Second, we are just now completing a wave of innovation that has never been seen in this company. 2006 is going to be an amazing year for shipping products, and many across the company will be ready to take on a new mission.

Third, regardless of past aspirations, this is the right time to be focusing on services for two specific reasons: the increasing ubiquity of broadband has made it viable, and the proven economics of the advertising model has made it profitable. It can be argued, for example, whether or not Hailstorm was the 'right' undertaking. But regardless, the effort would certainly have benefited from having a known-viable services business model for which to design.

Finally, I believe at this juncture it's generally very clear to each of us why we need to transform – the competitors, the challenges, and the opportunities. As an outsider, I was repeatedly impressed and awed over the years by how this company's talent has swarmed to effectively respond to huge business challenges and transitions.

That said, even when we've been solidly in pursuit of a common vision, our end-to-end execution of key scenarios has often been uneven – in large part because of the complexity of doing such substantial undertakings. In any large project, the sheer number of moving parts sometimes naturally causes compartmentalization of decisions and execution. Some groups might lose sight of how their piece fits in, or worse, might develop features without a clear understanding of how they'll be used. In some cases by the time the vision is delivered, the pieces might not quite fit into the originally-envisioned coherent whole. We cannot allow the seams in our organization, or our methods of making decisions, show through in our products, or result in the failure to deliver on key end-to-end experiences.

Complexity kills. It sucks the life out of developers, it makes products difficult to plan, build and test, it introduces security challenges, and it causes end-user and administrator frustration. Moving forward, within all parts of the organization, each of us should ask "What's different?", and explore and embrace techniques to reduce complexity.

Some problems are inherently complex; there is surely no silver bullet to reducing complexity in extant systems. But when tackling new problems, I've found it useful to dip into a toolbox of simplification approaches and methods. One such tool is the use of extensive end-to-end scenario-based design and implementation. Another is that of utilizing loosely-coupled design of systems by introducing constraints at key junctures – using standards as a tool to force quick agreement on interfaces. Many such tools are not rocket science: for example, by forcing a change in practices to increase the frequency of release cycles, scope and complexity of any given release by necessity is greatly reduced. Another simple tool I've used involves attracting developers to use common physical workspaces to naturally catalyze ad hoc face-time between those who need to coordinate, rather than relying solely upon meetings and streams of email and document reviews for such interaction. Embracing change at a local level through such tools can make a real difference – one project at a time.

Next Steps
We're off to a great start with many initiatives already under way – from efforts occurring now within MSN, to the IW services being launched imminently. We're in a tremendous position to succeed, but doing so will require your belief, creativity, support, leadership, follower-ship and action.

This memo was intended to get all of us roughly on the same page, and to get you thinking. The next steps are:

1) I am working with the division presidents to assign, by December 15th, "scenario owners" – a role intended to improve our execution of key services-based initiatives through leadership. These leaders will provide an outside-in perspective in mapping out and communicating specific market objectives, while at the same time working with developers and others at the detail level to ensure expedient decision making and continuity. These individuals will be responsible for driving critical decisions such as feature re-prioritization and cuts while appreciating the business tradeoffs and impact of such decisions. They'll listen. They'll rapidly effect changes in plans to ensure execution and improve agility, even for scenarios that span divisions. Initial scenarios to be assigned ownership will include the seven seamless experiences described earlier.

2) Beginning in January these individuals will work with me and with product groups to concretely map out scenarios and pragmatically assess changes needed in product and go-to-market plans related to services and service-based scenarios. For some groups this will impact short-term plans; for many others on path to shipping soon, it will factor significantly into planning for future releases.

3) All Business Groups have been asked to develop their plans to embrace this mission and create new service offerings that deliver value to customers and utilize the platform capabilities that we have today and are building for the future. We expect both technical and non-technical communities to be increasingly engaged on the topic of services and service-enhanced software. As we begin planning the next waves of innovation – such as those beyond Vista and Office "12" – we will mobilize execution around those plans.

4) I have created an internal blog that will be used to notify you of further plans as they emerge. There, I'll point you to libraries of documents that you will find interesting to read, and I'll be experimenting with ways that you can directly engage in the conversation.

These steps are important and necessary, but not sufficient, for us to deliver on our aspirations. The most important step is for each of us to internalize the transformative and disruptive potential of services. We must then focus on the need for agility in execution, and take actions as appropriate where each of us can.

The opportunities to deliver greater value to our customers, to our developer and partner communities, and to our shareholders are significant. I very much look forward to embarking on this journey with all of you.

-- Ray

A Controlled Software Development Process - whether Open Source or Enterprise Scale

Torvalds gets tough on kernel coders (from www.news.com)

Linus Torvalds, the creator of Linux and the maintainer of the development kernel, is cracking down on developers who add last-minute changes to the kernel.

The kernel development team recently set a policy that new features must be added to the next version of the kernel during the two weeks after the release of the previous version.

But James Bottomley, who currently maintains the code for SCSI support in the kernel, said Wednesday that he is finding it difficult to keep to the two-week merge window as contributors are leaving it to him to test whether their patches work with the rest of the system.

"That's a nice theory, except that it's my contributors who drop me in it by leaving their patch sets until you declare a kernel, dumping the integration testing on me in whatever time window is left," Bottomley wrote in a posting to the kernel mailing list.

Torvalds replied that Bottomley needs to get tough on his contributors.

"You tell them to stop it, and stop accepting their patches in that window, so that it's their code that gets delayed, not yours," he said in an e-mail.

Torvalds added that he plans to get tough on people who add things to the kernel too late.

"People always complain that I'm being too soft. Not so this time," Torvalds said.

"If people miss the merge window or start abusing it with hurried last-minute things that just cause problems for -rc1 (the first release candidate), I'll just refuse to merge, and laugh in their faces derisively when they whine plaintively at me, and tell them there's going to be a new opening soon enough," he said.

The latest release of the Linux kernel, version 2.6.14, was released almost a month late due to last-minute mistaken bug reports.