Code Library: Azure

Showing posts with label Azure. Show all posts

Adding tenant VMs to backup As the new VMs are deployed on the CPS stamp, customers can run a runbook (called Protect-TenantVMs) to protect new tenant VMs that were just created. All VMs are configured to protect once daily with a retention period of one week. Test VMs that do not need DPM protection can be excluded by specifying an exclusion VM list using a runbook (called Add-DPMExclusionItems).

You must run the Protect-TenantVMs runbook to manage tenant VM protection. This runbook adds up to 75 newly created VMs to a protection group in DPM. You should run this runbook manually or through a scheduled task once each day. After a tenant VM is added to a protection group, by default, the tenant VM is configured for daily backup, with a retention period of seven days. This runbook is designed to protect 75 new VMs per run per day to ensure enough time to complete tenant VM backups in the backup window and enough time for the deduplication process to complete. If more than 75 new VMs were created (on one rack) and you need to add them to a protection group on the same day, you can run this runbook more than once to protect the additional VMs.

The data deduplication process reduces backup storage usage. There is a default schedule for data deduplication and for tenant backups.

You should plan to run the Protect-TenantVMs runbook so that it does not interfere with the backup window. Therefore, run it any time between 6:00 AM and 6:00 PM local time (at least three to four hours before the backup window starts).

If you need to prevent protection of some VMs, you can run the Add-DPMExclusionItems runbook and specify VM names (wildcard characters are supported) that should be excluded during VM protection.

Source of Information : Microsoft System Center

What is distributed caching? A cache provides high throughput, low-latency access to commonly accessed application data by storing the data in memory. For a cloud app, the most useful type of cache is a distributed cache, which means that the data is not stored in the individual web server's memory but on other cloud resources, and the cached data is made available to all of an application's web servers (or other cloud VMs that are used by the application).

When the application scales by adding or removing servers, or when servers are replaced because of upgrades or faults, the cached data remains accessible to every server that runs the application.

By avoiding the high-latency data access of a persistent data store, caching can dramatically improve application responsiveness. For example, retrieving data from cache is much faster than retrieving it from a relational database.

A side benefit of caching is reduced traffic to the persistent data store, which may result in lower costs when there are data egress charges for the persistent data store.

When to use distributed caching
Caching works best for application workloads that do more reading than writing of data and when the data model supports the key/value organization that you use to store and retrieve data in cache. Caching is also more useful when application users share a lot of common data; for example, cache would not provide as many benefits if each user typically retrieves data unique to that user. An example where caching could be very beneficial is a product catalog, because the data does not change frequently and all customers are looking at the same data.

The benefit of caching becomes increasingly measurable the more an application scales, because the throughput limits and latency delays of the persistent data store become more of a limit on overall application performance. However, you might implement caching for reasons other than performance as well. For data that doesn't have to be perfectly up to date when shown to a user, cache access can serve as a circuit breaker for when the persistent data store is unresponsive or unavailable.

Popular cache population strategies
To be able to retrieve data from cache, you have to store it there first. There are several strategies for getting the data you need in a cache into the cache:

• On demand/cache aside The application tries to retrieve data from cache, and when the cache doesn't have the data (a “miss”), the application stores the data in the cache so that it will be available the next time. The next time the application tries to get the same data, it finds what it's looking for in the cache (a “hit”). To prevent fetching cached data that has changed in the database, you invalidate the cache when making changes to the data store.

• Background data push Background services push data into the cache on a regular schedule, and the app always pulls from the cache. This approach works great with high-latency data sources that don't require that you always return the latest data.

• Circuit breaker The application normally communicates directly with the persistent data store, but when the persistent data store has availability problems, the application retrieves data from cache. Data may have been put in cache using either the cache aside or background data push strategy. This is a fault-handling strategy rather than a performance-enhancing strategy.

To keep data in the cache current, you can delete related cache entries when your application creates, updates, or deletes data. If it's all right for your application to sometimes get data that is slightly out of date, you can rely on a configurable expiration time to set a limit on how old cache data can be.

You can configure absolute expiration (the amount of time since the cache item was created) or sliding expiration (the amount of time since a cache item was last accessed). Absolute expiration is used when you depend on the cache expiration mechanism to prevent data from becoming too stale. Regardless of the expiration policy you choose, the cache will automatically evict the oldest (least recently used, or LRU) items when the cache's memory limit is reached.

Source of Information : Building Cloud Apps With Microsoft Azure

Circuit breakers There are several reasons why you don’t want to retry too many times over too long a period:

• Too many users persistently retrying failed requests might degrade other users’ experience. If millions of people are all making repeated retry requests, you could tie up IIS dispatch queues and prevent your app from servicing requests that it otherwise could handle successfully.

• If everyone is retrying an operation because of a service failure, so many requests could be queued up that the service gets flooded when it starts to recover.

• If the error is the result of throttling and there’s a window of time the service uses for throttling, continued retries could move that window out and cause the throttling to continue.

• You might have a user waiting for a webpage to render. Making people wait too long might be more annoying that relatively quickly advising them to try again later.

Exponential back-off addresses some of these issue by limiting the frequency of retries that a service can get from your application. But you also need to have circuit breakers: this means that at a certain retry threshold your app stops retrying and takes some other action, such as one of the following:

• Custom fallback. If you can’t get a stock price from Reuters, maybe you can get it from Bloomberg; or if you can’t get data from the database, maybe you can get it from cache.

• Fail silently. If what you need from a service isn’t all-or-nothing for your app, just return null when you can’t get the data. For example, if you're displaying a Fix It task and the Blob service isn't responding, you could display the task details without the image.

• Fail fast. Error out the user to avoid flooding the service with retry requests that could cause service disruption for other users or extend a throttling window. You can display a friendly “try again later” message.

There is no one-size-fits-all retry policy. You can retry more times and wait longer in an asynchronous background worker process than you would in a synchronous web app where a user is waiting for a response. You can wait longer between retries for a relational database service than you would for a cache service. Here are some sample recommended retry policies to give you an idea of how the numbers might vary. ("Fast First" means no delay before the first retry.)

Source of Information : Building Cloud Apps With Microsoft Azure

Built-in logging support in Azure Azure supports the following kinds of logging in the Websites service:

• System.Diagnostics tracing (you can turn on and off and set levels on the fly without restarting the site)

• Windows events

• IIS logs (HTTP/FREB)

Azure supports the following kinds of logging in Cloud Services:
• System.Diagnostics tracing

• Performance counters

• Windows events

• IIS logs (HTTP/FREB)

• Custom directory monitoring

The Fix It app uses System.Diagnostics tracing. All you need to do to enable System.Diagnostics logging in an Azure website is flip a switch in the portal or call the REST API. In the portal, click the Configuration tab for your site and scroll down to see the Application Diagnostics section. You can turn logging on or off and select the logging level you want. You can have Azure write the logs to the file system or to a storage account.

Source of Information : Building Cloud Apps With Microsoft Azure

Log for insight A telemetry package is a good first step, but you still have to instrument your own code. The telemetry service tells you when there’s a problem and tells you what customers are experiencing, but it may not give you a lot of insight into what’s going on in your code.

You don’t want to have to remote into a production server to see what your app is doing. That might be practical when you have one server, but what about when you’ve scaled to hundreds of servers and you don’t know which ones you need to remote into? Your logging should provide enough information that you never have to remote into production servers to analyze and debug problems. You should be logging enough information so that you can isolate issues solely through the logs.

Log in production
A lot of people turn on tracing in production only when there’s a problem and they want to debug. This approach can introduce a substantial delay between the time you become aware of a problem and the time you obtain useful troubleshooting information about it. And the information you get might not be helpful for intermittent errors.

What we recommend for the cloud environment, where storage is cheap, is that you always leave logging on in production. That way, when errors happen, you already have them logged and have historical data that can help you analyze issues that develop over time or happen regularly at different times. You could automate a purge process to delete old logs, but you might find that it's more expensive to set up such a process than it is to keep the logs.

The added expense of logging is trivial compared with the amount of troubleshooting time and money you can save by having all the information you need already available when something goes wrong. Then, when someone tells you they had a random error sometime around 8:00 last night, but they don’t remember the error, you can readily find out what the problem was.

For less than $4 a month, you can keep 50 gigabytes of logs on hand, and the performance impact of logging is trivial so long as you keep one thing in mind— be sure your logging library is asynchronous

Differentiate logs that inform from logs that require action
Logs are meant to INFORM (I want you to know something) or ACT (I want you to do something). Be careful to write ACT logs only for issues that genuinely require a person or an automated process to take action. Too many ACT logs will create noise, requiring too much work to sift through all the log records to find genuine issues. And if your ACT logs trigger some action, such as sending email to support staff, avoid having a single issue trigger thousands of such actions.

In .NET System.Diagnostics tracing, logs can be assigned to the Error, Warning, Info, or Debug/Verbose level. You can differentiate ACT from INFORM logs by reserving the Error level for ACT logs and using the lower levels for INFORM logs.

Configure logging levels at run time
While it’s worthwhile to always have logging on in production, another best practice is to implement a logging framework that enables you to adjust at run time the level of detail that you’re logging, without redeploying or restarting your application. For example, when you use the tracing facility in System.Diagnostics, you can create Error, Warning, Info, and Debug/Verbose logs. We recommend that you always log Error, Warning, and Info logs in production and be able to dynamically add Debug/Verbose logging for troubleshooting on a case-by-case basis.

The Azure Websites service has built-in support for writing System.Diagnostics logs to the file system, Table storage, or Blob storage. You can select different logging levels for each storage destination, and you can change the logging level on the fly without restarting your application. Logging support in Blob storage makes it easier to run HDInsight analysis jobs on your application logs because HDInsight knows how to work with Blob storage directly.

Log exceptions
Don’t just put exception.ToString() in your logging code. That leaves out inner exceptions and contextual information. In the case of SQL errors, it leaves out the SQL error number. For all exceptions, include context information, the exception itself, and inner exceptions to be sure that you provide everything that’s needed for troubleshooting. For example, context information might include the server name, a transaction identifier, and a user name (but not the password or any secrets!).

Not every developer will do the right thing with exception logging if you rely on them to do so individually. To ensure that logging is done the right way every time, build exception handling into your logger interface: pass the exception object itself to the logger class and log the exception data properly in the logger class.

Log calls to services
We highly recommend that you write a log every time your app calls out to a service, whether to a database, a REST API, or any external service. Include in your logs not only an indication of success or failure but how long each request took. In the cloud environment you’ll often see problems related to slowdowns rather than complete outages. Something that normally takes 10 milliseconds might suddenly start taking a second. When someone tells you your app is slow, you want to be able to look at New Relic or whichever telemetry service you have and validate the user’s experience, and then you want to be able to look at your own logs to dive into the details of why your app is slow.

Use an ILogger interface
What Microsoft recommends doing when you create a production application is to create a simple ILogger interface and stick some methods in it. This makes changing the logging implementation later much easier, and you don’t have to go through all your code to do it. We could use the System.Diagnostics.Trace class throughout the Fix It app, but instead we’re using it under the covers in a logging class that implements ILogger, and we make ILogger method calls throughout the app.

With an approach such as this, if you ever want to make your logging richer, you can replace System.Diagnostics.Trace with whatever logging mechanism you want. For example, as your app grows, you might decide that you want to use a more comprehensive logging package, such as NLog or Enterprise Library Logging Application Block. (Log4Net is another popular logging framework, but it doesn't perform asynchronous logging.)

One reason for using a framework such as NLog is to divide logging output into separate high-volume and high-value data stores. Doing that helps you efficiently store large volumes of INFORM data that you don’t need to execute fast queries against, while maintaining quick access to ACT data.

Semantic logging
For a relatively new way to do logging that can produce more useful diagnostic information, see Enterprise Library Semantic Logging Application Block (SLAB). SLAB uses Event Tracing for Windows (ETW) and EventSource support in .NET 4.5 to enable you to create more structured and queryable logs. You define a different method for each type of event that you log, which enables you to customize the information you write. For example, to log a SQL Database error you might call a LogSQLDatabaseError method. For that kind of exception, you know that a key piece of information is the error number, so you could include an error number parameter in the method’s signature and record the error number as a separate field in the log record you write. Because the number is in a separate field, you can more easily and reliably get reports based on SQL error numbers than you could if you were just concatenating the error number into a message string.

Source of Information : Building Cloud Apps With Microsoft Azure

Buy or rent a telemetry solution One of the things that’s great about the cloud environment is that it’s really easy to buy or rent your way to victory. Telemetry is an example. Without a lot of effort, you can get a really good telemetry system up and running, very cost-effectively. There are a bunch of great Microsoft partners that integrate with Azure, and some of them have free tiers—so you can get basic telemetry for nothing. Here are just a few of the ones currently available on Azure:

• New Relic
• AppDynamics
• MetricsHub
• Dynatrace

As June 2014, Microsoft Application Insights for Visual Studio Online is not released but is available in preview. Microsoft System Center also includes monitoring features.

Source of Information : Building Cloud Apps With Microsoft Azure

SLAs People often hear about service-level agreements (SLAs) in the cloud environment. Basically, these are promises that companies make about how reliable their service is. A 99.9 percent SLA means you should expect the service to be working correctly 99.9 percent of the time. That's a fairly typical value for an SLA, and it sounds like a very high number, but you might not realize how much down time .1 percent actually amounts to. Here’s a table that shows how much downtime various SLA percentages amount to over a year, a month, and a week.

So a 99.9 percent SLA means your service could be down 8.76 hours a year or 43.2 minutes a month. That’s more downtime than most people realize. As a developer, you want to be aware that a certain amount of downtime is possible and handle it in a graceful way. At some point someone is going to be using your app, and a service is going to be down, and you want to minimize the negative impact of that on the customer.

One thing you should know about an SLA is what time frame it refers to: is the clock reset every week, every month, or every year? In Azure, the clock is reset every month, which is better for you than a yearly SLA, since a yearly SLA could hide bad months by offsetting them with a series of good months.

Of course, Microsoft aspires to do better than the SLA; usually, your app will be down much less time than what’s shown in the previous table.. The promise is that if Azure’s services are ever down for longer than the maximum downtime, you can ask for money back. The amount of money you get back probably won’t fully compensate you for the business impact of the excess downtime, but that aspect of the SLA acts as an enforcement policy and lets you know that Microsoft does take its SLA levels very seriously.

Composite SLAs
An important thing to think about when you’re looking at SLAs is the impact of using multiple services in an app, with each service having a separate SLA. For example, the Fix It app uses the Website, Storage, and SQL Database services. Here are their SLA numbers as of June 2014 (note that a 99.99% SLA is available for Storage at extra cost):

What is the maximum downtime you would expect for the app on the basis of these service SLAs? You might think that your downtime would be equal to the worst SLA percentage, or 99.9 percent in this case. That would be true if all three services always failed at the same time, but that isn’t necessarily what actually happens. Each service may fail independently at different times, so you have to calculate the composite SLA by multiplying the individual SLA numbers.

This calculation means that your app could be down not just 43.2 minutes a month but three times that amount—108 minutes a month—and still be within the Azure SLA limits.

This issue is not unique to Azure. Microsoft actually offers the best cloud SLAs of any cloud service available, and you’ll have similar issues to deal with if you use any vendor’s cloud services. What this highlights is the importance of thinking about how you can design your app to handle the inevitable service failures gracefully, because they might happen often enough to impact your customers or users.

Cloud SLAs compared with enterprise downtime experience
People sometimes say, “In my enterprise app I never have these problems.” If you ask how much downtime they actually have per month, they usually say, “Well, it happens occasionally.” And if you ask how often, they admit that, “Sometimes we do need to back up or install a new server or update software.” Of course, that counts as downtime. Most enterprise apps, unless they are especially mission-critical, are actually down for more than the amount of time allowed by Microsoft’s service SLAs. But when it’s your server and your infrastructure and you’re responsible for it and in control of it, you tend to feel less angst about down times. In a cloud environment, you’re dependent on someone else, and you don’t know what’s going on, so you might tend to be more worried about it.

When an enterprise achieves a greater uptime percentage than comes with a cloud SLA, it does so by spending a lot more money on hardware. A cloud service could do that but would have to charge much more for its services. Instead, you take advantage of a cost-effective service and design your software so that the inevitable failures cause minimum disruption to your customers. Your job as a cloud app designer is not so much to avoid failure as to avoid catastrophe, and you do that by focusing on software, not on hardware. Whereas enterprise apps strive to maximize mean time between failures, cloud apps strive to minimize mean time to recover.

Not all cloud services have SLAs
Be aware also that not every cloud service even has an SLA. If your app is dependent on a service with no uptime guarantee, your app could be down far longer than you might imagine. For example, if you enable login to your site using a social provider such as Facebook or Twitter, check with the service provider to find out whether there is an SLA, and you might find there isn’t one. But if the authentication service goes down or is unable to support the volume of requests you throw at it, your customers are locked out of your app. You could be down for days or longer. The creators of one new app expected hundreds of millions of downloads and took a dependency on Facebook authentication—but they didn’t talk to Facebook before going live and discovered too late that there was no SLA for that service.

Not all downtime counts toward SLAs
Some cloud services may deliberately deny service if your app over uses them. This is called throttling. If a service has an SLA, it should state the conditions under which your app might be throttled, and your app design should avoid those conditions and react appropriately to the throttling if it happens. For example, if requests to a service start to fail when you exceed a certain number of requests per second, you want to be sure that automatic retries don't happen so fast that they cause the throttling to continue.

Source of Information : Building Cloud Apps With Microsoft Azure

Design to survive failures - Failure scope You also have to think about failure scope—whether a single machine is affected, a whole service such as SQL Database or Storage, or an entire region.

Machine failures
In Azure, a failed server is automatically replaced by a new one, and a well-designed cloud app recovers from this kind of failure automatically and quickly. Earlier, we stressed the scalability benefits of a stateless web tier, and ease of recovery from a failed server is another benefit of statelessness. Ease of recovery is also one of the benefits of platform-as-a-service (PaaS) features such as SQL Database and Websites. Hardware failures are rare, but when they occur, these services handle them automatically; you don’t even have to write code to handle machine failures when you’re using one of these services.

Service failures
Cloud apps typically use multiple services. For example, the Fix It app uses the SQL Database service and the Storage service, and it’s deployed to the Websites service. What will your app do if one of the services you depend on fails? For some service failures a friendly “Sorry, try again later” message might be the best you can do. But in many scenarios you can do better. For example, when your back-end data store is down, you can accept user input, display “Your request has been received,” and store the input someplace else temporarily. Then, when the service you need is operational again, you can retrieve the input and process it.

The Fix It app stores tasks in SQL Database, but it doesn’t have to quit working when SQL Database is down. In that chapter you'll see how to store user input for a task in a queue and use a worker process to read the queue and update the task. If SQL Database is down, the ability to create Fix It tasks is unaffected; the worker process can wait and process new tasks when SQL Database is available.

Region failures
Entire regions may fail. A natural disaster might destroy a data center—it might be flattened by a meteor, the trunk line into the datacenter could be cut by a farmer burying a cow with a backhoe, etc. If your app is hosted in the stricken data center, what do you do? It’s possible to set up your app in Azure to run in multiple regions simultaneously so that if a disaster occurs in one, your app continues running in another region. Such failures are extremely rare occurrences, and most apps don’t jump through the hoops necessary to ensure uninterrupted service through failures of this sort. See the Resources section at the end of the chapter for information about how to keep your app available even through a region failure.

Source of Information : Building Cloud Apps With Microsoft Azure

What is Blob storage? The Azure Blob Storage service provides a way to store files in the cloud. The Blob service has a number of advantages over storing files in a local network file system:

• It's highly scalable. A single storage account can store 100 terabytes, and you can have multiple storage accounts. Some of the biggest Azure customers store hundreds of petabytes. Microsoft OneDrive uses Blob storage.

• It's durable. Every file you store in the Blob service is automatically backed up.

• It provides high availability. The SLA for Storage promises 99.9 percent or 99.99 percent uptime, depending on which geo-redundancy option you choose.

• It's a platform-as-a-service (PaaS) feature of Azure, which means you just store and retrieve files, paying only for the actual amount of storage you use, and Azure automatically takes care of setting up and managing all of the VMs and disk drives required for the service.

• You can access the Blob service by using a REST API or by using a programming language API. SDKs are available for .NET, Java, Ruby, and other languages.

• When you store a file in the Blob service, you can easily make it publicly available over the Internet.

• You can secure files in the Blob service so that they can accessed only by authorized users, or you can provide temporary access tokens that makes the files available to someone only for a limited period of time.

Anytime you're building an app for Azure and you want to store a lot of data that in an on-premises environment would go in files—such as images, videos, PDFs, spreadsheets, and so on—consider the Blob service.

Source of Information : Building Cloud Apps With Microsoft Azure

Vertical partitioning Vertical portioning is like splitting up a table by columns: one set of columns goes into one data store, and another set of columns goes into a different data store.

When you represent this data as a table and look at the different varieties of data, you can see that the three columns on the left have string data that can be efficiently stored by a relational database, whereas the two columns on the right are essentially byte arrays that come from image files. It's possible to storage image-file data in a relational database, and a lot of people do that because they don’t want to save the data to the file system. They might not have a file system capable of storing the required volumes of data, or they might not want to manage a separate backup and restore system. This approach works well for on-premises databases and for small amounts of data in cloud databases. In the on-premises environment, it might be easier to just let the database administrator (DBA) take care of everything.

But in a cloud database, storage is relatively expensive, and a high volume of images could make the size of the database grow beyond the limits at which it can operate efficiently. You can address these problems by partitioning the data vertically, which means you choose the most appropriate data store for each column in your table of data. What might work best for this example is to put the string data in a relational database and the images in Blob storage.

Storing images in Blob storage instead of in a database is more practical in the cloud than in an on- premises environment because you don’t have to worry about setting up file servers or managing backup and restore of data stored outside the relational database: all that is handled for you by the Blob storage service.

Without this partitioning scheme, and assuming an average image size of 3 megabytes (MB), the Fix It app would be able to store only about 40,000 tasks before it hit the maximum database size of 150 gigabytes. After removing the images, the database can store 10 times as many tasks; the application can handle a much larger number of people before you have to think about implementing a horizontal partitioning scheme. And as the app scales, your expenses grow more slowly because the bulk of your storage needs are going into very inexpensive Blob storage.

Source of Information : Building Cloud Apps With Microsoft Azure

Choosing a data storage option No one approach is right for all scenarios. If anyone says that a particular technology is the answer, the first thing to ask is "What is the question?" because different solutions are optimized for different things. The relational model has definite advantages; that’s why it’s been around for so long. But there are also downsides to SQL that can be addressed with a NoSQL solution.

Often, what we see work best is a composite approach in which SQL and NoSQL are used in a single solution. Even when people say they’re embracing NoSQL, a closer looks reveals that they’re using several different NoSQL frameworks—they’re using CouchDB, Redis, and Riak for different things. Even Facebook, which uses NoSQL solutions extensively, uses different NoSQL frameworks for different parts of the service. The flexibility to mix and match data storage approaches is one of the qualities that’s nice about the cloud; it’s easy to use multiple data solutions and integrate them in a single app.

Here are some questions to think about when you’re choosing an approach:

Data semantic
What is the core data storage and data access semantic (are you storing relational or unstructured data)?
Unstructured data such as media files fits best in Blob storage; a collection of related data such as products, inventories, suppliers, customer orders, etc., fits best in a relational database.

Query support
How easy is it to query the data?
What types of questions can be efficiently asked?

Key/value data stores are very good at getting a single row when given a key value, but they are not so good for complex queries. For a user-profile data store in which you are always getting the data for one particular user, a key/value data store could work well. For a product catalog from which you want to get different groupings based on various product attributes, a relational database might work better.

NoSQL databases can store large volumes of data efficiently, but you have to structure the database around how the app queries the data, and this makes ad hoc queries harder to do. With a relational database, you can build almost any kind of query.

Functional projection
Can questions, aggregations, and so on be executed on the server?

If you run SELECT COUNT(*) from a table in SQL, the DBMS will very efficiently do all the work on the server and return the number you’re looking for. If you want the same calculation from a NoSQL data store that doesn't support aggregation, this operation is an inefficient “unbounded query” and will probably time out. Even if the query succeeds, you have to retrieve all the data from the server and bring it to the client and count the rows on the client.

What languages or types of expressions can be used?
With a relational database, you can use SQL. With some NoSQL databases, such as Azure Table storage, you’ll be using OData, and all you can do is filter on the primary key and get projections (select a subset of the available fields).

Ease of scalability
How often and how much will the data need to scale?
Does the platform natively implement scale-out?
How easy is it to add or remove capacity (size and throughput)?

Relational databases and tables aren’t automatically partitioned to make them scalable, so they are difficult to scale beyond certain limitations. NoSQL data stores such as Azure Table storage inherently partition everything, and there is almost no limit to adding partitions. You can readily scale Table storage up to 200 terabytes, but the maximum database size for Azure SQL Database is 500 gigabytes. You can scale relational data by partitioning it into multiple databases, but setting up an application to support that model involves a lot of programming work.

Instrumentation and Manageability
How easy is the platform to instrument, monitor, and manage?

You need to remain informed about the health and performance of your data store, so you need to know up front what metrics a platform gives you for free and what you have to develop yourself.

Operations
How easy is the platform to deploy and run on Azure? PaaS? IaaS? Linux?

Azure Table storage and Azure SQL Database are easy to set up on Azure. Platforms that aren’t built-in Azure PaaS solutions require more effort.

API Support
Is an API available that makes it easy to work with the platform?

The Azure Table Service has an SDK with a .NET API that supports the .NET 4.5 asynchronous programming model. If you're writing a .NET app, the work to write and test the code will be much easier for the Azure Table Service than for a key/value column data store platform that has no API or a less comprehensive one.

Transactional integrity and data consistency
Is it critical that the platform support transactions to guarantee data consistency?

For keeping track of bulk emails sent, performance and low data-storage cost might be more important than automatic support for transactions or referential integrity in the data platform, making the Azure Table Service a good choice. For tracking bank account balances or purchase orders, a relational database platform that provides strong transactional guarantees would be a better choice.

Business continuity
How easy are backup, restore, and disaster recovery?

Sooner or later production data will become corrupted and you’ll need an undo function. Relational databases often have more fine-grained restore capabilities, such as the ability to restore to a point in time. Understanding what restore features are available in each platform you’re considering is an important factor to consider.

Cost
If more than one platform can support your data workload, how do they compare in cost?

For example, if you use ASP.NET Identity, you can store user profile data in Azure Table Service or Azure SQL Database. If you don't need the rich querying facilities of SQL Database, you might choose Azure Table storage in part because it costs much less for a given amount of storage.

Microsoft generally recommends that you should know the answer to the questions in each of these categories before you choose your data storage solutions.

In addition, your workload might have specific requirements that some platforms can support better than others. For example:
• Does your application require audit capabilities?
• What are your data longevity requirements—do you require automated archival or purging capabilities?
• Do you have specialized security needs? For example, your data might include personally identifiable information (PII), but you have to be sure that PII is excluded from query results.
• If you have some data that can't be stored in the cloud for regulatory or technological reasons, you might need a cloud data storage platform that facilitates integration with your on-premises storage.

Source of Information : Building Cloud Apps With Microsoft Azure

Platform as a Service (PaaS) versus Infrastructure as a Service (IaaS) The data storage options listed earlier include both Platform-as-a-Service (PaaS) and Infrastructure-as-a-Service (IaaS) solutions.

In a PaaS solution, Microsoft manages the hardware and software infrastructure and you just use the service. SQL Database is a PaaS feature of Azure. You ask for databases, and behind the scenes Azure sets up and configures the virtual machines (VMs) and sets up the databases on them. You don’t have direct access to the VMs and don’t have to manage them.

In an IaaS solution, you set up, configure, and manage VMs that run in Microsoft’s data center infrastructure, and you put whatever you want on them. Microsoft provides a gallery of preconfigured VM images for common VM configurations. For example, you can install preconfigured VM images for Windows Server 2008, Windows Server 2012, BizTalk Server, Oracle WebLogic Server, Oracle Database, and others.

PaaS data solutions that Azure offers include:
• Azure SQL Database (formerly known as SQL Azure) A cloud relational database based on SQL Server.
• Azure Table storage A column-oriented NoSQL database.
• Azure Blob storage File storage in the cloud.

For IaaS, you can run any software that you can load onto a VM, for example:
• Relational databases such as SQL Server, Oracle, MySQL, SQL Compact, SQLite, or Postgres.
• Key/value data stores such as Memcached, Redis, Cassandra, and Riak.
• Column data stores such as HBase.
• Document databases such as MongoDB, RavenDB, and CouchDB.
• Graph databases such as Neo4j.

The IaaS option gives you almost unlimited data storage options, and many of them are especially easy to use because you can create VMs using preconfigured images. For example, in the management portal, go to Virtual Machines, click the Images tab, and then click Browse VM Depot.

You then see a list of hundreds of preconfigured VM images, and you can create a VM from an image that has a database management system such as MongoDB, Neo4J, Redis, Cassandra, or CouchDB preinstalled:

Azure makes IaaS data storage options as easy to use as possible, but the PaaS offerings have many advantages that make them more cost-effective and practical for many scenarios:

• You don’t have to create VMs; you just use the portal or a script to set up a data store. If you want a 200-terabyte data store, you just click a button or run a command, and in seconds it’s ready for you to use.

• You don’t have to manage or patch the VMs used by the service; Microsoft does that for you automatically.

• You don’t have to worry about setting up infrastructure for scaling or high availability; Microsoft handles all that for you.

• You don’t have to buy licenses; license fees are included in the service fees.

• You pay only for what you use.

PaaS data storage options in Azure include offerings by third-party providers. For example, you can choose the MongoLab Add-On from the Azure Store to provision a MongoDB database as a service.

Source of Information : Building Cloud Apps With Microsoft Azure

Hadoop and MapReduce The high volumes of data that you can store in NoSQL databases may be difficult to analyze efficiently in a timely manner. To perform this type of analysis, you can use a framework such as Hadoop, which implements MapReduce functionality. Essentially, what a MapReduce process does is the following:

• Limits the size of the data that needs to be processed by selecting out of the data store only the data you actually need to analyze. For example, if you want to know the makeup of your user base by birth year, the process selects only birth years out of your user profile data store.

• Breaks down the data into parts and sends them to different computers for processing. Computer A calculates the number of people with dates between 1950 and 1959, computer B works on dates between 1960 and 1969, and so on. This group of computers is called a Hadoop cluster.

• Puts the results of each part back together after the processing on the parts is complete. You now have a relatively short list of how many people have each birth year, and the task of calculating percentages in this overall list is manageable.

On Azure, HDInsight enables you to process, analyze, and gain new insights from big data by using the power of Hadoop. For example, you could use HDInsight to analyze web server logs in the following manner:

• Enable web server logging to your storage account. This sets up Azure to write logs to the Blob service for every HTTP request to your application. The Blob service is basically cloud file storage and integrates nicely with HDInsight.

• As the app gets traffic, web server IIS logs are written to Blob storage.

• In the Azure management portal, click New, Data Services, HDInsight, Quick Create, and then specify an HDInsight cluster name, cluster size (number of HDInsight cluster data nodes), and a user name and password for the HDInsight cluster.

You can now set up MapReduce jobs to analyze your logs and get answers to questions such as:

• What times of day does my app get the most or least traffic?

• What countries is my traffic coming from?

• What is the average neighborhood income of the areas my traffic comes from? (There's a public dataset that provides neighborhood income by IP address, and you can match that data against the IP addresses in the web server logs.)

• How does neighborhood income correlate to specific pages or products in the site?

You could then use the answers to questions such as these to target ads based on the likelihood that a customer would be interested in or would be likely to buy a particular product.

Most functions that you can perform in the management portal can be automated, and that includes setting up and executing HDInsight analysis jobs. A typical HDInsight script might contain the following steps:

• Provision an HDInsight cluster and link it to your storage account for Blob storage input.

• Upload the MapReduce job executables (.jar or .exe files) to the HDInsight cluster.

• Submit a MapReduce job that stores the output data to Blob storage.

• Wait for the job to complete.

• Delete the HDInsight cluster.

• Access the output from Blob storage.

By running a script that performs these steps, you minimize the amount of time that the HDInsight cluster is provisioned, which minimizes your costs.

Source of Information : Building Cloud Apps With Microsoft Azure

Data storage options on Azure The cloud makes it relatively easy to use a variety of relational and NoSQL data stores. Here are some of the data storage platforms that you can use in Azure.

The illustration shows four types of NoSQL databases:

• Key/value databases store a single serialized object for each key value. They’re good for storing large volumes of data in situations where you want to get one item for a given key value and you don’t have to query based on other properties of the item.

• Azure Blob storage is a key/value database that functions like file storage in the cloud, with key values that correspond to folder and file names. You retrieve a file by its folder and file name, not by searching for values in the file contents.

• Azure Table storage is also a key/value database. Each value is called an entity (similar to a row, identified by a partition key and row key) and contains multiple properties (similar to columns, but not all entities in a table have to share the same columns). Querying on columns other than the key is extremely inefficient and should be avoided. For example, you can store user profile data, with one partition storing information about a single user. You could store data such as user name, password hash, birth date, and so forth, in separate properties of one entity or in separate entities in the same partition. But you wouldn't want to query for all users with a given range of birth dates, and you can't execute a join query between your profile table and another table. Table storage is more scalable and less expensive than a relational database, but it doesn't enable complex queries or joins.

• Document databases are key/value databases in which the values are documents. "Document" here isn't used in the sense of a Word or an Excel document but means a collection of named fields and values, any of which could be a child document. For example, in an order history table, an order document might have order number, order date, and customer fields, and the customer field might have name and address fields. The database encodes field data in a format such as XML, YAML, JSON, or BSON, or it can use plain text. One feature that sets document databases apart from other key/value databases is the capability they provide to query on nonkey fields and define secondary indexes, which makes querying more efficient. This capability makes a document database more suitable for applications that need to retrieve data on the basis of criteria more complex than the value of the document key. For example, in a sales order history document database, you could query on various fields, such as product ID, customer ID, customer name, and so forth. MongoDB is a popular document database.

• Column-family databases are key/value data stores that enable you to structure data storage into collections of related columns called column families. For example, a census database might have one group of columns for a person's name (first, middle, last), one group for the person's address, and one group for the person's profile information (date of birth, gender, and so on). The database can then store each column family in a separate partition while keeping all of the data for one person related to the same key. You can then read all profile information without having to read through all of the name and address information as well. Cassandra is a popular column-family database.

• Graph databases store information as a collection of objects and relationships. The purpose of a graph database is to enable an application to efficiently perform queries that traverse the network of objects and the relationships between them. For example, the objects might be employees in a human resources database, and you might want to facilitate queries such as "find all employees who directly or indirectly work for Scott." Neo4j is a popular graph database.

Compared with relational databases, the NoSQL options offer far greater scalability and are more cost effective for storage and analysis of unstructured data. The tradeoff is that they don't provide the rich querying and robust data integrity capabilities of relational databases. NoSQL options would work well for IIS log data, which involves high volume with no need for join queries. NoSQL options would not work so well for banking transactions, which require absolute data integrity and involve many relationships to other account-related data.

A newer category of database platforms, called NewSQL, combines the scalability of a NoSQL database with the querying capability and transactional integrity of a relational database.

NewSQL databases are designed for distributed storage and query processing, which are often hard to implement in "OldSQL" databases. NuoDB is an example of a NewSQL database that can be used on Azure.

Source of Information : Building Cloud Apps With Microsoft Azure

Async support in ASP.NET 4.5 In ASP.NET 4.5, support for asynchronous programming has been added not just to the language but also to the MVC, Web Forms, and Web API frameworks. For example, an ASP.NET MVC controller action method receives data from a web request and passes the data to a view, which then creates the HTML to be sent to the browser. Frequently, the action method needs to get data from a database or web service to display it in a webpage or to save data entered in a webpage. In those scenarios it's easy to make the action method asynchronous: instead of returning an ActionResult object, you return Task and mark the method with the async keyword. Inside the method, when a line of code kicks off an operation that involves wait time, you mark it with the await keyword.

Under the covers the compiler generates the appropriate asynchronous code. When the application makes the call to FindTaskByIdAsync, ASP.NET makes the FindTask request and then unwinds the worker thread and makes it available to process another request. When the FindTask request is done, a thread is restarted to continue processing the code that comes after that call. During the interim, between when the FindTask request is initiated and when the data is returned, you have a thread available to do useful work which otherwise would be tied up waiting for the response.

There is some overhead for asynchronous code, but under low load conditions, that overhead is negligible, while under high load conditions you’re able to process requests that otherwise would be held up waiting for available threads.

It has been possible to do this kind of asynchronous programming since ASP.NET 1.1, but it was difficult to write, prone to error, and difficult to debug. Now that the coding for it is simplified in ASP.NET 4.5, there's no reason anymore not to do it.

Source of Information : Building Cloud Apps With Microsoft Azure

Use .NET 4.5’s async support to avoid blocking calls .NET 4.5 enhanced the C# and Visual Basic programming languages to make it much simpler to handle tasks asynchronously. The benefit of asynchronous programming applies not just to parallel processing situations, such as when you want to kick off multiple web service calls simultaneously. It also enables your web server to perform more efficiently and reliably under high load conditions. A web server has only a limited number of threads available, and under high load conditions, when all of the threads are in use, incoming requests have to wait until threads are freed up. If your application code doesn't handle tasks like database queries and web service calls asynchronously, many threads are unnecessarily tied up while the server is waiting for an I/O response. This limits the amount of traffic the server can handle under high load conditions. With asynchronous programming, threads that are waiting for a web service or database to return data are freed up to service new requests until the data is received. In a busy web server, hundreds or thousands of requests that would otherwise be waiting for threads to be freed up can then be processed promptly.

As you saw earlier, it's as easy to decrease the number of web servers handling your website as it is to increase them. So, if a server can achieve greater throughput, you don't need as many of them, and you can decrease your costs because you need fewer servers for a given traffic volume than you otherwise would.

Support for the .NET 4.5 asynchronous programming model is included in ASP.NET 4.5 for Web Forms, MVC, and Web API; in Entity Framework 6; and in the Azure Storage API.

Source of Information : Building Cloud Apps With Microsoft Azure

Stateless web tier behind a smart load balancer Stateless web tier means you don't store any application data in the web server memory or file system. Keeping your web tier stateless enables you to both provide a better customer experience and save money:

• If the web tier is stateless and it sits behind a load balancer, you can quickly respond to changes in application traffic by dynamically adding or removing servers. In the cloud environment, where you pay for server resources only for as long as you actually use them, that ability to respond to changes in demand can translate into huge savings.

• A stateless web tier is architecturally much simpler for scaling out the application. That enables you to respond to scaling needs more quickly, and spend less money on development and testing in the process.

• Cloud servers, like on-premises servers, need to be patched and rebooted occasionally. If the web tier is stateless, rerouting traffic when a server goes down temporarily won't cause errors or unexpected behavior.

Most real-world applications do need to store state for a web session; the main point here is not to store it on the web server. You can store state in other ways, such as on the client in cookies or out of process server-side in ASP.NET session state by using the Redis cache provider. You can store files in
Azure Blob storage instead of the local file system.

Source of Information : Building Cloud Apps With Microsoft Azure

Azure Mobile Apps In today’s world, mobile devices—from tablets to phones to watches to fitness bands—are everywhere you look. Having a mobile application can be a big plus for a company, whether it’s used externally, internally, or both.

Azure Mobile Apps, included as part of Azure App Service, is a backend as a service that provides multiple features to make it easier and quicker to create a mobile application. Mobile Apps is both flexible and scalable, so when your application becomes widely used, you can scale appropriately to handle your customers’ needs.

Another advantage of Azure Mobile Apps is that you only have to write one version of your backend. The backend can be used by devices running iOS, Android, and Windows, allowing you to reach every user on every platform without extra work.

The following are some of the features provided by Azure Mobile Apps. You can certainly program a service to implement these features from the ground up, but using Azure Mobile Apps saves you the time and money it would take to do that.

 Data storage You can choose for your data storage to be powered by SQL Database, which has an interface simple enough to use without being a DBA. You can also integrate with SQL Server, Azure Table Storage, MongoDB, DocumentDB, or via an API to software as a service (SaaS) providers such as Salesforce.com and Office 365.

You can write your application to work offline and synchronize the data when the application can go online again. This is helpful when the customer loses Internet connectivity—the customer can continue to work, knowing the work will be stored on the backend when connectivity is regained.

 User authentication and data authorization are greatly simplified You can easily implement single sign-on (SSO) with Azure AD, a Microsoft account, Facebook, Twitter, and Google.

 Push notifications You can send information for customer and enterprise applications to any customer’s mobile device by using Microsoft Azure Notification Hubs. This can come from any backend, whether it runs in Azure or is on-premises. Notification Hubs automatically handles the server-side code to push messages to the push notification services for iOS, Android, and Windows devices.

Notification Hubs has a tagging feature that can be used to target audiences based on activity, interest, location, or preference. In addition, the templates feature of Notification Hubs enables you to send localized push notifications in the customer’s own language.

 Because Mobile Apps runs in Azure, you can easily scale in and out to meet customer demand You can even set up autoscaling that will automatically scale out as demand increases, handling millions of devices.

 You can use Microsoft Azure WebJobs to perform backend processing on the server at a scheduled time For example, you might want to create a scheduled job that requests an update from your on-premises database and stores the new information in a table, waiting to be retrieved by your mobile application.

 You can create a hybrid connection This connection can be used to connect the mobile application to on-premises systems, Office 365, and SharePoint.

Source of Information : Microsoft Azure Essentials Fundamentals of Azure Second Edition

Application and infrastructure modernization and migration There comes a time in every application’s life when it is time to upgrade. It could be a user interface redesign or a hardware refresh. The Azure platform cannot help create an appealing, modern user interface, but it can modernize the supporting infrastructure.

Many organizations will go through a periodic hardware refresh cycle; typically, this happens about every three years. When it is time for a hardware refresh, organizations today have a new question to ask: Should we buy new on-premises hardware, or should we leverage our infrastructure and services to the cloud?

Besides a required hardware refresh, an organization might choose to migrate to the cloud because it has reached physical capacity limits in its existing on-premises datacenter or because it is going to in the very near future. Perhaps the current datacenter does not have enough physical space for more servers or cannot supply the necessary power or cooling. Maybe there is a desire to eliminate or reduce the management of hardware infrastructure going forward. Moving to the cloud might enable the organization to get out of the datacenter business completely, or at least partially. In this case, Microsoft is responsible for the hardware and related infrastructure components of the datacenter, and the organization can focus on providing great business solutions.

Some organizations will choose to migrate to the cloud to get capacity in new geographies they can’t currently support because they have no presence in that area or because it would be cost-prohibitive. There are Azure datacenters in over 22 regions around the world from Melbourne to Amsterdam and from Sao Paulo to Singapore. Additionally, Microsoft has an arrangement with 21Vianet, making Azure available in two regions in China. Microsoft has also announced the deployment of Azure to another eight regions. Instead of building and maintaining a global datacenter presence, an organization can elect to take advantage of Microsoft’s existing investments and deploy to multiple regions with ease.

Should the choice be to modernize or migrate to the cloud, there is certainly a wealth of Azure resources available. In choosing to adopt these resources, an organization could have many questions to answer, including these:

 Do we leverage platform as a service (PaaS), infrastructure as a service (IaaS), or both?

 Instead of maintaining a custom solution, should we leverage platform-provided services such as Azure Search or Azure Media Services?

 Should we move everything, or just some components? What hybrid model works best for our requirements?

 Which Azure region(s) should we use?

 How does using Azure affect our business and operations model?

 What is our service level agreement (SLA)? What is our disaster recovery story?

Source of Information : Microsoft Azure Essentials Fundamentals of Azure Second Edition

Hybrid scenarios The number of companies running solutions in the cloud is increasing at an incredible rate. Their success encourages other organizations to take the same step. Some organizations will not be able to move all of their workloads into the cloud, either because of regulatory issues or because some workloads cannot run in a virtualized environment. In these cases, hybrid computing, in which a company runs part of its infrastructure in the cloud and part on-premises, will be an important strategy.

The Microsoft Azure platform provides a great hybrid computing story. There are multiple ways to connect an on-premises datacenter to one or more Azure regions. Azure provides both site-to-site and point-to-site virtual network connectivity. Either option provides a secure VPN connection between on-premises assets and resources hosted in Azure. An additional hybrid connectivity option is Azure ExpressRoute, which enables a private connection between Azure and your on-premises infrastructure or colocation facility, all without going over the public Internet.

Network connectivity
Regardless of the chosen option—site-to-site, point-to-site, or ExpressRoute—hybrid connectivity is a key scenario for the Azure platform. Creating a hybrid connection opens a wide range of possibilities to extend an on-premises infrastructure to the cloud. Two common scenarios for network-enabled hybrid connectivity are the following:

 Hosting a website in Azure but keeping the database on premises In an organization’s journey to the cloud, migrating the on-premises data to Azure can be one of the more difficult tasks. The difficulty usually comes in one of two forms: a technical issue or a compliance requirement. On the technical front, as an example, the application in question is designed to use a database that is not supported in Azure. On the compliance front, perhaps there is a regulatory requirement that cannot be met with Azure SQL Database or by running a database (SQL Server, MongoDB, and so on) on Azure Virtual Machines. In these cases, an organization might choose to host the website in Azure using Azure Web Apps or Azure Virtual Machines, with the database remaining on premises. Connectivity between the website and the database could then be established using one of the aforementioned technologies: a site-to-site connection, a point-to-site connection, or ExpressRoute.

 Accessing an on-premises service Sometimes, a website has a dependency on a particular service that cannot be moved to the cloud. Perhaps the website depends on an API that performs a crucial business calculation, and that API cannot be moved due to security because other on-premises services also depend on the service or because it is legacy technology that is not supported in Azure. In such a scenario, a hybrid connection is established between Azure and the on-premises infrastructure to allow the Azure-hosted website to freely communicate with the necessary API that continues to reside on-premises.

Besides using a network connection in this scenario, an Azure Service Bus Relay could be used to access an on-premises service. For information on how to use the Azure Service Bus Relay service, please refer to http://azure.microsoft.com/documentation/articles/service-bus-dotnet-how-to-use-relay/.

Internet connectivity
There are many scenarios in which all that is needed is an Internet connection rather than a special hybrid connectivity solution. After all, the ability to connect to Internet-accessible services is one of the attractive features of cloud computing. A few common scenarios include these:

 Storage of archival data Large amounts of data, especially archival data that is rarely accessed, can be very expensive to store on-premises. The cost in terms of infrastructure, people, software licenses, and physical space can quickly put a tremendous financial burden on an organization. Azure provides virtually limitless storage capacity at an incredibly low price. An organization might wish to use the scalable storage provided by Azure Blob storage as a data archival store. When the data is needed, the on-premises service(s) download the data from Azure Blob storage and perform the necessary processing. A basic Internet connection will often suffice, but an ExpressRoute connection could also be used for improved speed and security.
Another option for storage of archival data is Microsoft Azure StorSimple. StorSimple includes a hardware appliance that is installed on-premises. The appliance keeps frequently accessed data local (on the device). As data ages (is accessed less frequently), it is automatically moved to Azure Blob storage. For more information on StorSimple, please refer to http://azure.microsoft.com/documentation/services/storsimple/.

 Organizations can choose to synchronize their Azure AD users and groups with user and group information from their on-premises Active Directory. In doing so, they can use Azure Active Directory Connect to synchronize the user data and a password hash, making Azure AD the authority for user authentication. Alternatively, an organization might wish to synchronize the user data but require users to authenticate via an Active Directory Federation Services (AD FS) endpoint residing on-premises, effectively redirecting the user to an on-premises AD FS site for authentication before redirecting to the desired location.

 Burst to the cloud Sometimes, an organization’s on-premises infrastructure is not able to handle the required load. Maybe there is a holiday season rush or a government-mandated period to sign up for an important service. Instead of building the on-premises infrastructure to handle the temporary surge in demand, an organization might choose to leverage the elastic nature of the cloud to burst to the cloud when needed and scale back to only on-premises services when the load returns to normal. In this scenario, an organization could use Azure Web Apps or Azure Virtual Machines to host the service and could implement autoscale rules to ensure capacity keeps up with user demand.

Source of Information : Microsoft Azure Essentials Fundamentals of Azure Second Edition

Code Library

Categories