With great art, genius looks effortless. Think of Fred Astaire dancing or Jascha Heifitz playing violin. A whole symphonic work can appear graceful and natural, even easy, but it is merely a mask: you do not observe the blood, sweat, and tears. The audience only participates in the thrill of achievement at its final stage.
This is as it should be.
It can be the same with technology. The multitudes who used the Mises Institute website see only fabulous commentary, hear great lectures, read the classics of the Austrian School in many different formats, discover vast bibliographic resources, purchase books from the catalog, read about the lives of great thinkers, follow the blog for news and comments, all toward the goal of explaining and applying the economics of liberty. Hundreds of writers and volunteers work with a small technical staff to prove that Mises was right: "There cannot be too much of a correct theory."
Well, readers of the site cannot help but notice that some things have lately changed around Mises.org. The site is as vast and radical as always but now it is especially beautiful, fast, efficient, and reliable. For example, you can now listen to the latest lectures live, as people did last week during the seminar with Charles Adams, broadcast to Moscow to Beijing to LA, and will again this week.
What readers do not see is the astonishing upheaval in the site that began a mere two weeks ago. For all of us involved, it has been the professional experience of a lifetime, watching the combination of intelligence, dedication, and technology work together to achieve something that few would believed possible. I still do not quite believe the marvel that has unfolded before us. Yet here we sit today looking at a site that reaches more than ever as it hums along as if it is just an inherent feature of the structure of reality, as natural and normal as the sunrise.
Background to the Drama
It began with the most absurdly ambitious agenda for development—or what we thought was ambitious. Now, keep in mind that changing anything on a site this big is a huge undertaking, especially for a small institute with very few resources to spend on web development. When the Wall Street Journal goes through an upgrade, it has a vast team of top professionals who plow through millions of dollars and test the site for months on end before going live. Even then, teams of designers coordinate sleep schedules to stay up with the site 24/7, working out bugs minute by minute.
All of this is out of the question for us, of course. But not being the WSJ doesn’t mean we can't dream of ever better ways to make it possible for the message to get out. It has been through extreme economization of resources, hard work, and unrelenting passion that the site has become the most trafficked institutional site dealing with economics in the world today, bigger than the IMF, the World Bank, and the Fed itself.
Mises.org has been constantly improved from the first day it went online in 1995. Our dreams this round began with the hope of expanding the header and site width to permit more information to display. This is necessary because of today's larger screens and more efficient delivery systems.
But even this proposal was daunting. The site consists of 33,000 pages, uses 5 gigs of hard disk, delivers 15 gigs of logfiles, and produces 1.8 million page views for 750,000 visitors per month generating a monthly hit total of 16 million. Any change on such a vast contraption requires careful thought and no small amount of chutzpah.
The site was packed with antique code and built-in tables and font tags dating from all ages. The content bar on the side seemed to work on some pages but not others; it has to be periodically removed even though it is part of the footer. The header alone is something of a technological marvel. When it was first revealed, it was way ahead of its time. It is built of tables within tables, and graphics within graphics, and with a rotating picture of Mises, and hover effects of all sorts. It cannot be big and lumbering since it sites on every page of the site. It must be and remain light and agile.
An Old-Growth Tree
Let me pause and explain that Mises.org, in its technical features, is not like any site on the web. From the day we moved from .html to .asp (active server pages) we've had a cardinal rule against any broken links (yes, I know they appear from time to time). This has meant building many redirects, patches, and troubleshooting devices along the way. The site looked slick but beneath the surface you could see the reality of a site built not last year or the year before but rather over 8 years.
Mises.org like an old growth tree. If you could have sliced through the code from top to bottom, each ring would reveal a stage of technological development during eight of the fastest moving years in all of human history. The oldest ring in the slice was technologically obsolete while the newest ring was impressive enough to delight a college-age whiz kid. But somehow it all worked together, even if no one could account for the why of it all.
To employ another metaphor, Mises.org is not at all like Lubbock, Texas, planned and mapped so that every street begins with a number or a letter, and so that the city looks like only squares within squares from an aerial photograph. No, it is more like Boston, Massachusetts, a city built over long years that began with paths trod by cows that became dirt roads for wagons and eventually paved streets but which still maintain their original look and feel. An aerial photograph of Boston looks like chaos itself but somehow the place still works—and no one would ever suggest that Boston be restructured to look like Lubbock.
Any change made to Mises.org, then, must take all of this in account. We can dream all we want, but from a pure technical point of view, there has to be a strong assumption at the outset to think very carefully about the limits in addition to the possibilities.
And yet the hullabaloo about the site-width change ended up being much ado about nothing: with the assistance of David Veksler, a student interning with us this summer from Texas A&M, the conversion of the site from 700 to 800 pixels went off without a hitch. One day the site looked one way, and the next day, it was over: a new Mises.org. The header width had changed and the new reality was dazzling to one and all.
This event, accomplished in less than 24 work hours, opened up a new world. David's surprising expertise and efficiency caused us to think more broadly. What about cleaning out unused pages? Eliminating duplicate files? Upgrading the store? Speeding matters up? Perfecting and unifying fonts and display? All of this began to happen to, as the newest tools combined with expertise began to race through the site burning out dated code and replacing it with better code. Hour by hour the house of Mises began to look cleaner and work better. The sense of exhilaration was overwhelming.
Then came the matter of cost. We had been uploading ever more media files, which use vast bandwidth and sometimes lead to overage charges—the web equivalent of a margin call that eats up the budget. Just the plain monthly fees of running a site on this scale began to be exorbitant. Because we lease space instead of attempting to run our servers from in-house, we began to shop around for web space, and, much to our astonishment, we found that prices had been plummeting—a glorious and much well "deflation."
Move? Surely Not
Now, competition for server space is extremely rivalrous for new sites. Everyone starting up wants to pay the least amount possible. But what about existing sites? They are too often stuck. The transition costs of moving are huge. There are many vendors to deal with, user names and passwords galore, DNS switches, and secure cert changes, and redirects, and more three-letter things than any one person could possibly keep up with. Sure, you can threaten to leave your current host of server space. And yes, you are the customer and so you should be able to dictate terms. But on a practical level, it is so very difficult to move, most people are stuck where they are.
To put it another way, Auburn University could move from Auburn, Alabama--in theory. Yes, the city desires to serve the university as it does other businesses that could pick up and move. It could happen. But everyone knows it won't. They are bound together because the costs moving are just too high to make it within the realm of possibility. So life goes on, and everyone tries to get along as best they can. A site and a host are like a horse and carriage, right? Like salt and pepper, they are married. Surely.
We asked for, but did not receive, a cut in the rates. That's when we began to think the unthinkable. Perhaps a move would be in order. Maybe it can be done. Perhaps so. Surely not! Well, who is to say? What are the advantages and what are the disadvantages? These were troubling, even unthinkable, questions. We were thinking of leaping across the abyss, with not a clue about what would emerge on the other side.
We all took deep breaths, cross our fingers, knocked on wood, threw salt over our shoulders, and did every other superstitious thing we could think of, and decided: yes, we would go ahead. David, the least superstitious person among us, had every reason to believe it could be done, though it turned out that even he had actually underestimated the Herculian task.
To return to the Boston metaphor, moving Mises.org was like putting the city on a forklift, and letting it come to rest gently in Iowa, with the goal of not alerting any of the residents or otherwise disturbing life in the city.
Still, we began to accumulate the information necessary for the move, and work through a series of steps to discover precisely what was proprietary and what was not. There were codes, passwords, leases, applications, and a thousand details spread far and wide. Juggling all this in the course of one day, while still making no public announcement concerning our intentions, was daunting indeed, but we had only begun.
The actual move came much faster than we expected, because the new server was assigned to us much more quickly than we imagined. We contracted for it on Friday, and moved the entire site on the following Tuesday, complete with an upgraded operating system. The experience was very much what people say about high-risk sports such as bungy-jumping or sky-diving: you have to experience it to believe it. At 2:00 pm that day, hearts racing and feeling on top of the world, we notified our existing company that we had left.
Effortless? Not Yet
Then we went live. It takes time for the new server to populate through the internet and during this time users would see parts of the old site. Sometimes within the same region or same building, two computers would generate different versions. It was extremely disorienting and scary for about 12 hours until the DNS had fully populated and the pieces of the mosaic grew closer and closer to form a unified picture.
Meanwhile, issues were cropping up by the minute. This doesn't work, this looks funny, here's a broken link, oh we forgot about this software, who has the key to this application, how many domain names do we own anyway and what that's password again? David swatted down the bugs as they flew by, like an action-packed video game in action. So far, it was all in a day's work.
We left from work that day with a sense of spectacular accomplishment but also a sense of foreboding, for some unexplained things had happened later in the day. The server did not go down. It had just began to…hang. We had to restart the machinery that drives the site from database to display. Now, this problem had been around awhile. A long while in fact. It wasn't taking down the site altogether. It was just hanging it. But it didn't matter either way for the user. For the rest of the world the site might as well be down.
Death's Head at the Feast
Here was the shock that none of us had quite expected: the hanging was worse, far worse, than it had been on our old server with our old host. The mystery of the hanging server led to our first real crisis of faith, to four of the most harrowing days of our lives.
The hang was easy to fix, temporarily. It was just a matter of a couple of clicks inside the server. The trouble was that it would not stay fixed. But one had no idea when it would go down. It would stay up for 4 or even 10 hours. Then it would go down in repeated moments of 6 minutes. There was no way to know.
It was like an elaborate prison torture in which a heavy drop of water would randomly fall on the prisoner's head in random intervals over a very long period of time. It seemed calculated to drive us all mad, and as it became difficult to work on other aspects of the site, it became clear that this had to be solved.
The mystery of the hanging server seemed to have everyone stumped. Now, one theory was that this was caused by the installation of a language called Perl which runs the blog. The blog software we chose was better suited for a Linux than windows system. This was the preferred theory of some because the hanging server problem began about the same time as the blog was installed about one year ago.
It was David's view that this theory was incorrect. Coincidence in time doesn't necessarily imply causation (Austrians are always alert to reject the fallacy of post hoc ergo propter hoc). But to eliminate this possibility, he shielded the blog from the rested of the site by running it off a different an open-source software package. Would it work? We waited. And waited.
Part of the frustration was that the hang was random and the only way to tested whether a particular fix worked was to wait, hour after hour after hour. We didn't even have a way to cause the hang to happen. Finally, the dreaded event happened again. The server hung. Our hearts sunk.
In the meantime a better theory was in the works. The hangs hand something to do with the speed at which the site was drawing from the database. It seemed as if heavy loads gummed up the works but we couldn't be sure, since the server logs were ambiguous. There was one cause, or perhaps many, but isolating it or was the difficult task.
Meanwhile, a 24-hour-a-day site watch began. It felt like the middle ages. One of us had to be available at all hours to monitor the site and check it to see that it stayed up, even as David continued to think and work toward a fix. We traded off sleeping times. We woke at unusual hours. We had IM notifications, cell phones, emails, and every other form of communication device to alert us. Finally, it seemed obvious that all this frenzy and hysteria was getting in the way of the David's ability to think through the problem. The answer was there. It was a matter of finding the time and space to discover it.
The Bunker
It was Sunday morning. We were exhausted, sleepy, weary, frustrated. We needed a new strategy, one that could only emerge if David could somehow gain some distance. Making sure that he had enough provisions to last him, we arranged for the site to be otherwise monitored while he took the time to think. We joked later that his dorm room had become his bunker. The hangs continued the entire day but we did our best to somehow protect David from following it blow-by-blow but he couldn't avoid it: he had set up a system that automatically pinged him very aggressively when the server hung.
So it went for the whole harrowing day. Detailed technical queries were sent out to the web's leading on-line help sources—people actually paid lots of money to figure these sorts of things out, but no plausible theories emerged. As midnight approached, the problem still not solved, I gave out and prepared for another week of what David called "our own private Hell." He stayed up--all night it turns out.
The next morning was like every morning of the last five days, filled with dread. And yet, there was a very obscure message waiting for me. "See Mises.org. You are in for a surprise." Nothing more. The rest of us had to wait until mid morning. Whatever had been done, it seemed to be working. We were practically holding our breath as our hero slept for the first time in what seemed like days.
Later, we got the whole story. It turned out that the database was indeed the issue. The coincidence in time with the implementation of the blog was easy to explain: the blog increased traffic, which increased database load. Why was it hanging more than ever? With new software and new speed, the pages were delivering ever faster, and thereby increasing the load. Still it wasn't enough to cause the hang, but perhaps several factors working together, among them ancient code, were the cause.
Essentially we had become victim of our own success: more use than the underlying architecture could support. (Read this more detailed technical explanation by David.)
Overnight, David had begun a mass conversion from our existing language of .asp to the most modern language of .net with the extension of .aspx. This is something sites pay tens of thousands of dollars to do. They plot for months. Our switch took place on a dime in the wee hours, in desperation.
In addition, he put a cache on the front page and the most popular pages of the site. He downloaded newer version of the mechanism that linked the database to the website. He came up with many other tricks to reduce the load. All together, the fix worked. We were all in a state of ebullient shock, afraid to celebrate just yet but very optimistic.
As the day went on, it became ever clearer that total victory had been achieved. The conversion of the site into the new language proceeded apace, vender software for all the site functionality was being upgraded by the hour. Our up time for that bright and brilliant Monday: 100%.
Even at the worst point, we were very pleased that thanks to constant efforts and dedication, we never went below 85% up time, and most of down time was in the middle of the night and lasted only a little while. We hear of bugs constantly but we it is all to the good: every bug report becomes the basis of an ever better site.
As Elegant as a Dance
I write on Friday, the end of the first week of the new Mises.org. We have been streaming audio for two days to all parts of the world. The Mises University starts, on the one week anniversary of that dramatic Sunday when the world went from dark to light. We will be streaming audio and video.
I talk with students who are here with us studying for the summer, writing dissertations and papers and doing deep research. We do our best to spare them the gory details explained above. But I mention in passing that we have been though through some exciting times on the site. They say, hey, it looks good. Otherwise they noticed nothing.
Perfect! Exactly as it should be, seemingly effortless. What kind of world have we entered? One where the greatest ideas ever imagined are available to all at virtually no cost, a world in which the state's conspiracy of silence against the old liberal tradition is ended and the monopoly on information demolished.
Praise be to the free market and the entrepreneurial drive that make it all possible. Praise be to the hundreds of writers and volunteers who make the ideas come to life, and, mostly, to the financial supporters of this work, without whom none of this would be possible. It is they who make it possible to unite message and method to become a powerhouse of freedom in our time.
How the world has change. How we are changing the world. And now, forget about the technical side, and spend time on Mises.org, and see how economics can be as elegant as dance, as beautiful as a symphony, and as inevitable as the sunrise.
The Technical Details of the Mises.org Transformation
by David Veksler
I faced five major challenges during my stay at the Mises Institute: cleaning up hundreds (thousands?) of dead pages, implementing a new header on over 15,000 pages, moving the server to a new web host running a totally different operating system, web server, and database server, converting the daily articles to ASP.Net, and porting the online store to a major new version. Other than the last, the first four have been largely accomplished. Here is how it was done:
I used five tools to manage Mises.org: Macromedia Dreamweaver MX, Visual Studio.Net 2003, SmartFTP, WinRAR, and the "HandyFile Find And Replace" tool.
I started by zipping the entire Mises.org website and installing a fully functional mirror of the website on my new Dell Inspiron 9100 laptop – possibly the biggest, fastest, and heaviest laptop on the market today. I initially exported the database to Access for testing, but found that I could connect the server database for my testing after optimizing the SQL. Installing the website on my laptop had the additional (dubious) benefit of allowing me work on the site everywhere I lugged my 10lbs behemoth. I used the 404 redirect trick and my search/replace utility to move hundreds of pages around without breaking thousands of links. Doing this for a week gave me a feel for the website and prepared me for the move to a new host.
Once the decision to move was made, we spent a bit of time researching a new web host to host the website. We decided to go with a dedicated Windows 2003 server provided by iPowerWeb. This gave us the additional benefit and challenge of the newest OS. The webhost provided us with Windows 2003 Server, but no database application, so I had to think quick about installing a DBMS. Initially I decided to go with MySQL, but found that this would require a week of conversion, and some commercial components only worked with SQL Server, so I installed MSDE, a free, low-end version of SQL Server, and planned to move to MySQL later. I then proceeded to zip all 5.5GB of Mises.org and transfer it directly to the new server. After installing MySQL and MSDE, I used the SQL Server data export function to install copies of the Mises.org database in MSDE and MySQL. A day and a half was lost attempting to export the ancient flat-file database used for the blog into MySQL, which was an adventure unto itself.
Having moved the files and databases over, I spent some time with my handy search/replace utilities fixing all the connection strings and replacing all the hardcoded file paths. I then spent another day re-installing or recording in pure ASP all the commercial components used on the old server. More time was spent replacing relative file paths and unused functions caused by the Windows 2000-2003 switch. As I went along, I removed all the <font> tags I came across, replacing them by CSS (Cascading Style Sheet) styles and header tags. Dreamweaver was invaluable in the process, proving advanced search functions, just the right level of synchronicity between the live and development servers, and beautifully supporting whatever programming language (ASP, VB.Net, Perl, SQL, C#, JavaScript) I came across.
Having moved the server, we installed the SSL cert for the secure pages, installed the new store, the stats packages, changed the DNS pointers, and broke out the champagne. As soon as the DNS changes propagated, the server hangings started. This was a nightmare episode I would rather forget. I spent hours surfing through Google results, MS TechNet articles, online forums, and making random changes in search of the magic fix. Nothing I did fixed it, so I installed four different server monitoring services (the hang was notoriously difficult monitor to detect because of server and client side caching.) Finally, in total exhaustion and desperation, I noticed that the hangings roughly corresponded to high server loads. Unable to sleep, I sat down for two hours sometime around 2 AM, and rewrote the article pages of Mises.org in VB.Net. As I suspected, the caching implemented by ASP.Net and the redesigned SQL code solved the hanging problem. The database load was reduced sufficiently that I decided to keep the website running on MSDE, which is designed to max out at 25 users. In the next few days, I ported my hack .Net code to a two-tier architecture, and prepared the website for a full-scale conversion. I also reduced the database load by optimizing all SQL queries to only select the data actually needed my a particular page.
Some details on the new configuration:
- Hardware: 1.8 Ghz Celeron, 512MB RAM, 40GB HD.
- Operating System: Windows 2003 Standard Server
- Web Servers: IIS 6.0, Apache 2.0.50
- Scripting languages: VB.Net, C#, ASP, Perl
- Database Servers: SQL Server 2000 Desktop Edition, MySQL 4.1
- Size: 15,000+ ASP pages, 33,000+ total files; ~5.5 GB
