In-Depth
Top Tips for Migrating Your Applications to the Azure Cloud
From architecture to security to database compatibility, here's what you need to know to make your cloud migration a success.
One of our favorite aspects of technology is that it is constantly
evolving and continually changing—there's always more to learn! As
students and followers of cloud computing, we're tremendously excited
about the Windows Azure. As technical evangelists for Microsoft, we have
the great fortune to work with customers in the adoption of new
technology. As a result, we've seen a host of different ways in which to
apply Windows Azure.
Early on, George had a personal reason for wanting to use Windows
Azure. George is involved in many community activities, and the ability to
quickly spin up temporary applications and spin them down when no longer
needed proved tremendously useful. For developers with experience writing
Microsoft .NET Framework code, there's hardly any learning curve—build
your application, deploy it and run it.
Because of the interest many of our corporate customers expressed
in Windows Azure, we decided to hold a set of Windows Azure Migration Labs
at the Microsoft Technology Centers. The intent was for customers to bring
their applications into the lab and actually migrate them to Windows
Azure. Through this process, every single customer was able to
successfully migrate its Web applications and SQL databases to the Windows
Azure.
We weren't surprised—we already had plenty of experience with Windows
Azure and were confident our customers would meet with success. But in the
course of helping the lab attendees migrate their various applications, we
learned quite a few tricks that help migrations go smoothly. In this
article, we'll share some of those tips and tricks we discovered working
with customers on real-world migrations.
Migration Basics
When deciding to migrate an application from on-premises to the cloud
(or to create a new application on a cloud service), there are several
aspects of the application architecture that need to be considered:
- Application management
- Application security
- Application compatibility
- Database compatibility
The questions and concerns we heard most frequently during the
migration labs tended to revolve around these four areas. As a result,
we'll focus our discussion around these topics.
One misconception we often encountered was the idea that, by using
Windows Azure, developers don't have to worry about common architectural
patterns regarding issues such as availability, scalability, reliability
and security when moving to or creating applications in the cloud. The
truth is that architectural patterns in the context of distributed
computing are equally valid for applications architected for on-premises
deployment or Windows Azure deployment.
Application Management
No matter whether your application is running on-premises or in the
cloud, the operations management team needs data that will enable them to
make effective decisions. The issues you'll need to consider include
service-level agreements, capacity planning, customer billing, auditing,
application monitoring, traffic analysis and managing costs (knowing when
to scale up or down). These need to be resolved before the application is
deployed to production—and for best results, often before the
application is created.
These were just some of the issues that were considered during the
Windows Azure Migration Labs. By utilizing the Windows Azure Diagnostics
API provided in the Windows Azure SDK
(Microsoft.WindowsAzure.Diagnostics), customers were able to expose
application crash dumps, failed request tracing, Windows event logs, IIS
logs, Windows Azure logs and performance counters.
This is much more straightforward than you might expect. You tell the
diagnostic monitor which types of diagnostic information to collect (see
Listing 1 for an example) and set the data transfer
schedule for the information to be transferred to a central Windows Azure
storage location.
For more information about Windows Azure diagnostics, see the article
"Cloud
Diagnostics: Take Control of Logging and Tracing in Windows Azure" by
Mike Kelley, in the June 2010 issue of MSDN Magazine.
Application Security
A top concern of any organization moving to the cloud is security. Most
companies have invested a substantial amount of time, money and
engineering into designing and developing a security model and it's
important that they're able to leverage existing investments such as
identity stores, single sign-on solutions and firewalls.
While there are many ways for a company to go about securing
cloud-based applications, an increasingly popular pattern is a
claims-based approach.
This process is shown in Figure 1. In order for an
application to be able to process security tokens from a Security Token
Service (STS), a trust relationship must be established between the STS
and the application.
[Click on image for larger view.] |
| Figure 1. Claims-Based Identity in an Application Context |
The access control rules (step 1) are defined to meet business
requirements (who can log into the application). These rules are stored
with the STS. When a user tries to access the application, she's
redirected to the STS so she can receive a valid token (step 2). The user
provides a set of input claims (for example, a Live ID or a domain
account) to the STS for authentication purposes. The STS will map these
claims to a set of output claims once the user is authenticated (step 3).
In step 4, the output claims are packaged into a Security Assertions
Markup Language (SAML) token, signed by the STS, and returned to the user
for forwarding to the application (the relying partner in step 5). The
application confirms that the SAML token is valid and from the trusted STS
(step 6). Once the token is validated, the application checks the claims
in the token and sends back the appropriate response (step 7). Pretty
simple! The beauty of this approach is that it fits in extremely well
within the ASP.NET provider model. The process of making your ASP.NET
application claims-aware is really quite simple.
In order to make life easier for the developer, Microsoft introduced
the Windows Identity Foundation (WIF) SDK. This does all of the grunt work
of parsing SAML 2.0 tokens, letting the developer concentrate on his
application without having to worry about the underlying security
technology.
The first thing to do is download
WIF and the WIF
SDK. Once these are installed, you'll have what you need to make your
application claims-aware.
In the Visual Studio solution with your ASP.NET Web application,
right-click and select Add | Add New Web Site. Select the ASP.NET Security
Token Service Web Site template. This will allow you to set up an STS for
your development environment.
Once you have your STS created, you can add a reference to the STS by
right-clicking on your application and clicking "Add STS reference." This
starts a wizard that walks you through the process of establishing a
relationship between your application and the STS. Point to the
application's web.config file for your site and specify the Application
URI (see Figure 2).
[Click on image for larger view.] |
| Figure 2. Starting the Federation Utility Wizard |
In the next step, choose "Use an existing STS" and then specify the
location of the FederationMetadata.xml file in the STS project (see
Figure 3). Choose the defaults for the remainder of the
process.
[Click on image for larger view.] |
| Figure 3. Configuring the STS |
Take a look at your web.config file. You'll see that the
FedUtil.exe wizard changed a substantial amount of code. The most
important changes were made to the microsoft.identityModel node of the
web.config file. Here you'll see references to your STS project, along
with the claim types expected by the application. To ensure that your
application is appropriately receiving the claims back from your STS, put
the following code in your default.aspx page (note that you'll have to add
a reference to the Microsoft.IdentityModel from the WIF SDK):
IClaimsIdentity ici =
(IClaimsIdentity)Thread.CurrentPrincipal.Identity;
foreach (Claim c in ici.Claims) {
Response.Write(c.ClaimType + " - " + c.Value + "<br/>");
}
When you next run your application, you will be automatically
redirected to your STS. By default, the STS allows you to authenticate as
"Adam Carter." Just click the Login button (you don't need a
password).
After the STS authenticates the login, you will be redirected back to
your Web application, along with a SAML token needed for authentication.
Your application will accept the token and allow the default.aspx page to
run. Because the WIF modules intercept your security credentials, you're
able to cast the identity principal as an IClaimsIdentity, and,
consequently, you can extract the claims type and value out of the
identity object (see Figure 4).
[Click on image for larger view.] |
| Figure 4. The Claims Type and Value of an Identity
Object |
Now that the Web application is claims-aware, it's easy to adapt it to
your existing identity model. Simply update your configuration file so
that it points to your production STS and ensure that you've configured
your application as a relying party. Additionally, you can use this
information to create a custom role provider so that you can translate
claims types into roles.
This is an extremely powerful technique, and will let you move your
applications to almost any environment—on-premises, the cloud or even a
partner datacenter—and still validate against your identity stores through
a publicly exposed STS.
Application Compatibility
Windows Azure is an application platform, so it's important to
understand the types of applications suited to the Windows Azure. While
you have the ability to run native code and you can run applications with
full trust, you must package your application before deploying it to the
cloud, which means it's important to evaluate your application to see if
it's a good fit.
Here's an example. One of our customers at the Windows Azure Migration
Labs had an existing application consisting of a SQL Server 2005 back end,
a LINQ to SQL data-access layer and a front end using both MVC Framework
1.0 and ASP.NET 3.5 SP1 running on IIS.
The application sat in a Web farm with a load balancer routing the
traffic. The application itself was stateless so it didn't matter to which
server the user was ultimately directed.
An interesting detail about this application was that the MVC
application manages more than 220 separate Web sites. The company used a
combination of MVC routing and information stored in the SQL Server
database to determine which content should be loaded for each Web site.
There were five Web servers behind the load balancer serving more than 4
million page visits per month for the collection of Web sites.
The primary challenge the company faced was the length of time it took
to provision a new Web server for its environment: months! When the
company considered migrating the application to Windows Azure, its primary
motivation was saving a tremendous amount of time. Scaling out would
become a configuration detail rather than a quarter-long nightmare.
The migration process to Windows Azure is actually quite
straightforward. Here's the generic process we used:
- Verify that the application is running correctly in the development
environment.
- Migrate the SQL Server back end to SQL Azure using the SQL Azure
Migration Wizard (we'll discuss the details later in this article).
- Update the local application to work with the SQL Azure database.
This was as simple as changing the connection string.
- Convert the application into a Web Role project.
- Validate that the application runs on the local development
fabric.
- Package up the Web Role and deploy it to Windows Azure.
- Validate that the application runs from Windows Azure.
In order to reduce the size of the Web role package, we ended up
pulling all the images and CSS files out of their content folders and
placing them in Windows Azure blob storage.
Because all the content was in Windows Azure blob storage, GGP was able
to leverage the Windows Azure content-delivery network (CDN). This allowed
caches of data to sit closer to the end users.
For an overview of development, testing and deployment for Windows
Azure, see the article "Windows
Azure: Developing and Deploying Windows Azure Apps in Visual Studio
2010" in the April 2010 issue of MSDN Magazine. For a deeper look at
storage issues, see "Cloud
Storage: Fueling Your Application's Engine with Windows Azure Storage"
in the January 2010 issue.
Database Compatibility
When SQL Azure first came out, we migrated a couple of our SQL Server
databases to it. Along with our experience holding the Windows Azure
Migration Labs, we learned a few important things you should consider
before embarking on the migration process.
First, it's important to check the size of your database and how it
fits within the database allowances used by SQL Azure. Currently, SQL
Azure offers Web Editions in 1GB and 5GB sizes and Business Editions in
10, 20, 30, 40 and 50GB sizes. You need to check your database and make
sure it isn't larger than 50GB. If your database is larger than 50GB, then
you'll need to examine your database and see if it can be broken down into
smaller databases (in other words, sharding your database) or moving large
data to blobs.
SQL Azure supports only SQL Authentication, so you'll need to consider
whether changes are needed to the authentication scheme used by your
application. On top of that, SQL Azure has a resource throttle that limits
connection time. We'll discuss both of these issues a little later in the
article.
The version of your SQL Server database is another item you need to
take into consideration before migrating your database to SQL Azure. SQL
Azure is built on top of SQL Server 2008. This means that if you want to
migrate your SQL Server 2000 or SQL Server 2005 databases to SQL Azure,
you need to make sure your databases are compatible with SQL Server 2008.
For example, earlier versions of SQL Server support the old style TSQL
joins such as *= and =* operators in the WHERE clause. SQL Server 2008
only supports ANSI style joins. For example:
SELECT ProcessClassTypeName
, bpa.PropertyMetadata AS PropertyMetadataOverride
, act.PropertyMetadata AS PropertyMetadataDefault
FROM dbo.BusinessProcessActivities bpa
LEFT JOIN dbo.Activities act ON act.Activity_ID = bpa.Activity_ID
When the compatibility level of the database is set to SQL Server 2005
or SQL Server 2008, the old style TSQL joins (*= and =*) are not
supported. This is only one example of compatibility issues you'll find
when migrating to SQL Server 2008.
It's beyond the scope of this article to discuss in detail the
migration process to SQL Server 2008. If you're interested in database
migration best practices, please check out the "Ultimate
guide for upgrading to SQL Server 2008". There's also a wealth of
resources available from the MSDN SQL Server developer
center.
You'll find the best path is to migrate from a SQL Server
2008-compatible database to SQL Azure. This means that if you want to
migrate your SQL Server 2000 or 2005 database to SQL Azure, you can go
through an on-premises upgrade to SQL Server 2008 before you migrate to
SQL Azure.
Microsoft offers a great tool called SQL
Server Upgrade Advisor that analyzes instances of SQL Server 2000 and
SQL Server 2005 to identify features and configuration changes that might
affect your upgrade. It provides links to documentation that describes
each identified issue and how to resolve it. Once you've verified that
your database is compatible with SQL Server 2008, you can fast-forward to
migrating the database to SQL Azure.
That said, you also need to be aware that SQL Azure doesn't support 100
percent of new SQL Server 2008 functionality. For example, filestream is
not currently supported in SQL Azure. There are a few ways to check for
compatibility issues when going to SQL Azure.
The blunt force approach is to simply ride the wild side—run your TSQL
scripts against SQL Azure and look for errors. Correct any errors that
occur and run again. Repeat until successful. Perhaps not the best use of
your time, but that's for you to decide.
You can use the SQL Server Management Studio script generator wizard to
generate your TSQL script. Note that when you walk through the wizard, be
sure to select the advanced scripting option and select SQL Azure Database
for the "Script for the database engine type" property. If you miss this
step, SQL Server will generate TSQL that's not compatible with SQL
Azure.
Another option is to download SQL Azure Migration Wizard (SQLAzureMW)
from sqlazuremw.codeplex.com.
SQLAzureMW will do its best to identify compatibility issues, fix them
where possible and notify you of all issues it knows about.
To better understand general guidelines and limitations to SQL Azure,
see msdn.microsoft.com/library/ee336245.
Once you have your database schema (tables, views, stored procedures
and so on) in SQL Azure, you'll need to upload your data. Here are the
most common ways:
- SQL Server Integration Services
- Bulk Copy Program (BCP)
- SqlBulkCopy for data migration
- SQL Azure Migration Wizard (which uses BCP in the
background)
Using SQLAzureMW
George created SQLAzureMW to help our customers with the SQL database
migration process. Figure 5 shows SQLAzureMW in
action.
[Click on image for larger view.] |
| Figure 5. Using SQLAzureMW |
SQLAzureMW analyzes SQL Server databases for compatibility issues with
SQL Azure. It also allows you to migrate database objects and data from
the source database to SQL Azure.
By using SQLAzureMW, database developers can get an idea of how much
work will be required in migrating their databases to SQL Azure. If
SQLAzureMW flags a lot of compatibility issues with a SQL Server 2000 or
2005 database, we recommend upgrading your database to SQL Server 2008
first, then migrating to SQL Azure. The process of migrating to SQL Server
2008 is well-documented and there's a lot of guidance and expertise you
can leverage. For more information on migrating to SQL Server 2008, see
the SQL Server 2008 Upgrade Technical Reference Guide (microsoft.com/downloads/details.aspx?FamilyID=66d3e6f5-6902-4fdd-af75-9975aea5bea7).
There's also a wealth of resources available from the MSDN SQL Server
developer center (msdn.microsoft.com/sqlserver).
Note that if you don't have SQL Server 2008 R2, this doesn't stop you
from going through the upgrade process. Just download SQL Server 2008 R2
Express Edition and go through the side-by-side upgrade process.
Other good sources for database developers to understand the
differences between SQL Server and SQL Azure—what's compatible and what's
not, and general guidance and limitations—include the Transact-SQL
Reference (SQL Azure Database) at msdn.microsoft.com/library/ee336281
and the General Guidelines and Limitations (SQL Azure Database) at msdn.microsoft.com/library/ee336245.
No matter whether you decide to upgrade to SQL Server 2008 first or
just migrate directly from SQL Server 2000 or 2005, you'll still need a
way to analyze your database for compatibility issues and generate SQL
Azure-compatible SQL. This is where SQLAzureMW really helps out. Not
only can SQLAzureMW analyze the database, it will also analyze SQL
Profiler trace files when you want to check dynamic SQL for compatibility
issues.
During the migration labs, we were able to migrate all of the SQL
databases (SQL Server 2000 and SQL Server 2005) to SQL Azure with little
to no modification. The two remaining issues that needed to be addressed
were authentication and SQL Azure resource throttling.
The authentication issue was due to the fact that SQL Azure supports
only SQL Authentication, not Windows Authentication. In the case of one
customer, they had to modify their connection string to reflect username
and password instead of a trusted connection. For example, we started out
with something like this:
<add key="ConStr"
value="server=DbSvr;database=CRMDB;Trusted_Connection=yes" />
We simply changed the connection string to something like this:
<add key="ConStr"
value="Server=avl6qnn22s.database.windows.net;Database=CRMDB;User
ID=WebSvrAdmin@avl6qnn22s;Password=password;Trusted_Connection=False;" />
To get more information on connecting to SQL Azure using ADO.NET, see msdn.microsoft.com/library/ee336243.
Resource Throttling
Addressing SQL Azure resource throttling took a little more work for
some applications. For the applications that followed best practices in
getting a connection to a SQL database only when needed and at the last
possible second, doing all of the transactions in a quick and efficient
manner and letting go of the connection as soon as possible, SQL Azure
throttling wasn't an issue. On the other hand, the applications that
grabbed a connection to a SQL database at startup and held onto the
connection for the life of the program, or for long periods of time, had
to be modified to implement retry logic or refactored to follow best
practices and not hold resources.
One misconception we ran into a lot was that SQL Azure would only
disconnect your connection if the connection sat idle for five minutes.
SQL Azure takes into account several factors in determining when to
disconnect an application, including excessive resource usage,
long-running queries, long-running single transactions and idle
connections.
The SQL Azure team will continue to tweak the resource throttling
parameters, but the net effect is that the application has to have retry
logic built into it because SQL Azure will force a disconnection on any
application that exceeds its resource utilization parameters.
In general, SQL Azure will provide specific error messages if it ever
throttles the connection. For a full list of errors, see msdn.microsoft.com/library/ff394106.
If you have a high volume of small transactions, you should use the
following pattern:
- Use the connection pool. The connection pool manager will keep the
connections open for you and there will be little to no impact on
performance for the application to open and close connections.
- Keep the connection for as small a duration as possible. Open the
connection, execute your transaction and close the connection.
- Use the try-catch pattern around your database activity.
- Catch the exceptions and retry the transaction, if appropriate.
- Log your failures and exceptions to help facilitate problem
resolution. Make sure to get a UTC timestamp (or provide time and time
zone), the connection context ID and the exception number.
SQLAzureMW is a good example of an application that had to deal with
resource throttling in SQL Azure. As we mentioned earlier, SQLAzureMW can
migrate an on-premises SQL database to SQL Azure. If you're migrating a
database that has more than 1,000 tables, 1,500 stored procedures and
millions of rows of data, it could easily take longer than five hours
depending on the actual amount of data that needs to be uploaded.
In this scenario, SQLAzureMW would have well over 2,500 TSQL statements
to execute against SQL Azure, as well as the data to migrate via BCP.
Executing more than 2,500 TSQL statements in a single statement (or
transaction) would more than likely exceed SQL Azure resource throttling
parameters, thus resulting in a terminated connection.
As a solution, SQLAzureMW breaks down the transactions into smaller
bunches and runs until SQL Azure terminates the connection. When
SQLAzureMW encounters the connection error, it reestablishes a new
connection with SQL Azure and picks up processing after the last
successful command. In the same manner, when using BCP to upload the data
to SQL Azure, SQLAzureMW chunks the data into smaller sections and uses
retry logic to figure out the last successful record uploaded before the
connection was closed. Then it has BCP restart the data upload with the
next set of records.
The SQL Azure product team has made vast improvements to SQL Azure
since SQL Azure was first released. For example, a lot of the throttling
issues we ran into during the migration labs have gone away—though we
still recommend that your applications use retry logic to handle
terminated connections gracefully.
Next Steps
As you can see, although there are a number of issues you need to
consider in planning for a smooth Windows Azure migration, in practice
we've found that the amount of work required to migrate an application
from on-premises to Windows Azure is often minimal. Of course, this is
going to be different for each application.
You'll need to do your own analysis to determine whether it makes sense
to migrate to Windows Azure and what issues you'll need to address. During
the Windows Azure Migration Labs, our customers found they were able to
migrate their applications with little to no modification and that they
were able to utilize the Windows Azure with very little learning curve and
investment. The information here, along with tools like SQLAzureMW, should
help you achieve a similarly successful result.