Code Library: Cloud

Showing posts with label Cloud. Show all posts

Why back up Azure VMs? The reasons for backing up VMs on-premises and in Azure are similar. Different customer segments derive different value from the solution, but here is a list of commonly encountered reasons for backup of VMs:

 Reduced restore time during disasters The VM contains all the information needed to get the application up and running. This includes the operating system, the application software, the configuration settings, and the data. It would typically take more time to piece together all of these at the time of restore, so restoring the complete VM is much faster.

 Greater suitability for long-term retention The VM includes the user data and the associated application software to work with it. Thus, the restores done from very old backup points are more likely to achieve any data retrieval goals.

 Easier management The VM acts as an encapsulating entity for the data. Rather than manage tens to hundreds of smaller data entities, the VM makes it simpler for the backup administrator. This makes patch rollback scenarios easy to address.

Source of Information : Microsoft System Center

Backup scenarios This section examines some of the different backup scenarios that can be implemented using Azure Backup.

Tape replacement
Many organizations store their backup data on-premises on disk media and store their long-term retention data on tapes. They invest in significant tape infrastructure to meet their compliance requirement. Besides the cost of tape infrastructure, tapes require manual intervention to replace the older ones. Tapes must be labelled correctly and risk potential errors and data loss if they are mishandled. To store tape data offsite, organizations must arrange tape pick-up on a daily or weekly basis. In addition, to recover offsite data, organizations must request all the relevant tapes and restore the data.

Cloud-based backup inherently addresses all of the preceding issues. As long as organizations have network connectivity to the cloud provider, backup to the cloud saves money. Organizations can leverage the pay-as-you-go model with the cloud rather than investing in upfront costs for tape storage.

Beyond the cost savings, there are other inherent advantages to storing backup data in the cloud:
 Data can be retrieved even if there is a disaster on-premises.
 Restore times are cut down and there is no need to wait for the tape delivery from offsite.
 There is no need to restore all data to retrieve a single item.

Azure Backup is clearly a good solution to address the tape replacement scenario because it provides a competitive tape replacement strategy for businesses.

Branch office backup
Branch offices typically have fewer machines and smaller infrastructure than a large datacenter. However, the data generated in branch offices is often critical for the business. Some organizations back up this data locally in the branch office, which means they need to purchase additional storage for each branch in addition to managing the complexity of the storage and backup infrastructure in each branch.

More than a handful of branch offices increases the management complexity multi-fold. In this case, some organizations back up their branch office data to the main office. But this again means that the main office must purchase the storage necessary to support all the workloads that are backed up from each of the branches.

With cloud-based backup, however, organizations can eliminate their local storage and back up data directly to the cloud. Azure Backup enables businesses to back up their Windows-based servers directly to the cloud, thereby eliminating local storage at each of the branch offices.

Windows client backup
With Azure Backup, organizations can back up files and folders on their Windows-based desktops and laptop computers directly to the cloud with an entirely self-service model where the IT administrator needs to take minimal or no action on behalf of the user. Since the data is encrypted when it leaves the computer, data is always secure. Individuals in a small organization can either share the same vault or each user can have a dedicated vault or subscription, depending on the sharing needs among the individuals in the organization.

Protection of Microsoft Azure assets
With an ever-increasing number of enterprises and small businesses moving their workloads to the cloud, organizations need a simple mechanism to ensure that data created in the cloud is also backed up, just as is done on-premises. Microsoft Azure inherently provides high availability and redundancy of storage with the guarantee that if there is a storage or computer outage, the application can continue to run using redundant storage or computers. With the support for backup of Azure IaaS virtual machines (VMs), however, (which is currently in preview at the time of this writing) organizations can get the benefit of additional protection since they can protect their workload data from software corruption or data loss scenarios as well. In addition, organizations can always test their backups by performing a restore of data periodically.

Source of Information : Microsoft System Center

Advantages of Azure Backup Azure Backup makes a great case for moving on-premises tape and disk infrastructure to the cloud. As with all cloud solutions, it is cost effective, with a pay-as-you-go model and no upfront costs. But unlike other cloud-connect strategies, Azure Backup is built as a cloud-first software as a service.

This model has several advantages. The service comes with 99.9 percent availability time. As users create a backup vault to store data, the data is stored in geo-replicated storage, protecting it from disasters. Even if there is an outage of one of the Azure datacenters, the data is accessible.

But it is not sufficient for the data to be geo-redundant; the service that enables access to data should also be geo-redundant. Azure Backup is available in two or more regions per geography and has a built-in business continuity plan so that even when the primary Azure datacenter experiences an outage, the service fails over to a new datacenter. Therefore, regardless of whether the organization loses on-premises data or whether the Azure datacenter has an outage, both the data and the backup service are available for customers to retrieve their data. If Azure fails over to a secondary data center, customers are able to browse all the recovery points associated with backup, pick any recovery point, and perform a restore, as well as continue backing up data to the service post failover.

With Azure Backup, backed up data is always encrypted on both the wire and at rest on Azure such that it is always secure before it leaves the on-premises datacenter. The Azure Backup service also maintains backup metadata that enables customers to restore data anywhere from Azure to an alternate Windows-based or DPM server.

Source of Information : Microsoft System Center

Recovering tenant VMs All tenant VMs are deployed with a single parent VHD. DPM’s original location recovery workflow will not work for tenant VMs. Complete the following steps to recover tenant VMs:

1. In the VMM console, determine the name of the host on which the VM that you want to recover is located by doing the following:

a. In the VMs And Services workspace, expand All Hosts, and then click Compute Clusters.

b. In the VMs pane, type the name of the VM.

c. Note the value in the Host column that is associated with the VM.

d. Note which compute cluster the host is a member of. (Under Compute Clusters, click each cluster to view the members.)

e. Right-click the VM, and then click Properties. Click the Hardware Configuration tab. Under Bus Configuration, the VHDs that are attached to the VM are listed. Click the operating system VHD (typically the first one under IDE Devices) to see if there is a VHD chain. Note the value in the Fully Qualified Path To Parent Virtual Hard Disk box (for example, copy and save it to Notepad). If the VM properties are corrupted and you cannot access them, you can skip this step.

2. In the VMM console, find a tenant share that has enough available capacity to store the recovered VM by doing the following:

a. In the Fabric workspace, expand Storage, and then click File Servers.

b. In the File Servers, File Shares pane, expand the file server that is in the same rack as the compute cluster where the Hyper-V host that you identified in step 1c resides.

c. Use the Available Capacity column to find a TenantShare with enough free space. (This procedure uses the example share \\-FS-02.contoso.com\TenantShare14.)

3. On the Console VM, open Failover Cluster Manager, and connect to the compute cluster on which the host that you identified in step 1c is a member of.

4. Under the cluster name, click Roles.

5. In the Roles pane, find the cluster resource name of the VM that you want to recover. The name will be in the format SCVMM VMName Resources.

6. On the Console VM, open Windows PowerShell, and run the following commands to delete the VM. Press Enter after each command. Note that the Hyper-V host is the host on which the VM that you want to recover is located.
Stop-VM -ComputerName HyperVHostName -Name VMName Remove-VM -ComputerName HyperVHostName -Name VMName

7. Create a symbolic link to the tenant share that you identified in step 2 by first running the following command:
Enter-PSSession -ComputerName HyperVHostName
In the remote session, run the following commands:
cd c:\ cmd /c "mklink /d DirectoryName \\SharePath" exit

8. On the Console VM, find the DPM server that backs up the VM that you want to recover. To do this, complete the following steps:

a. Open the Operations console.

b. In the Monitoring workspace, expand System Center 2012 R2 Data Protection Manager, select State Views, and then click Protected Servers.

c. In the Look For box, enter the cluster resource name of the VM.

d. In the DPM server column, note the name of the DPM server that backs up the VM.
You can also do this by running the following Windows PowerShell command from the Operations Manager Shell:
Get-SCOMClassInstance | where {$_.DisplayName -like '*clusterresourcename*'} | foreach { $_.'[Microsoft.SystemCenter.DataProtectionManager.

9. Open the DPM administrator console, and connect to the DPM server that you identified in step 8. Find and note the name of the protection group that the VM that you want to recover was added to.

10. On the Console VM, recover the VM by running the following Windows PowerShell commands as an elevated user. Press Enter after each command. Note that DPM-TenantVM-0# is the name of the DPM server that you identified in step 8, ProtectionGroupName is the protection group that the VM is a member of, VMName is the NetBIOS name of the VM that you want to recover, and SymbolicLinkOnHyperVHost is the symbolic link that you created earlier, for example c:\test1.
$pg = Get-DPMPRotectionGroup -DPMServerName DPM-TenantVM-0# | where {$_.Name -eq "ProtectionGroupName"} $ds = Get-DPMDatasource -ProtectionGroup $pg | where {$_.Computer -eq "VMName"} Get-DPMRecoveryPoint -Datasource $ds | select Name, BackupTime ## this is used for display only $rps = Get-DPMRecoveryPoint -Datasource $ds $rpo = New-DPMRecoveryOption -HyperVDatasource -TargetServer HyperVHostName -RecoveryLocation AlternateHyperVServer -RecoveryType Recover -TargetLocation $rp = $rps[$rps.Length - 1] ## Value of - 1 indicates the latest recover point. A value of - 2 would be the recovery point before that. $ri = Get-DPMRecoverableItem $rp -BrowseType Child Recover-RecoverableItem -RecoverableItem $rp -RecoveryOption $rpo

11. On the Hyper-V host on which the VM is located, open Windows PowerShell as an elevated user.

12. Perform a storage migration by running the following command where SOFSShare is the share that you identified in step 2.
Move-VMStorage -ComputerName HyperVHostName -VMName VMName -DestinationStoragePath SOFSShare

13. Re-parent the VM to its original parent that you identified in step 1e by running the following command. (You can skip this step and continue to step 15 if the original VM configuration was corrupted and you could not get this property value in step 1e.)
Get-VMHardDiskDrive VMName | Get-VHD | where {$_.parentPath -ne $null} | Set-VHD -ParentPath "\\SharePathofParentVHD"

14. Delete the "local" parent VHD (that was just recovered).

15. Delete the symbolic link. To do this, open a Windows PowerShell session as an elevated user, and then run the following commands. (Press Enter after each command.)
Enter-PSSession -ComputerName HyperVHostName
del DirectoryName
exit

16. From a Console VM, run the following Windows PowerShell commands to configure the VM as highly available. Press Enter after each command. (You must connect the VM to its original cluster resource role.) Note that in the following commands, VMClusterResourceName is the cluster resource name for the VM (for example "SCVMM VMName Resources"), ComputeClusterName is the compute cluster name on which the Hyper-V host resides, and VMConfigLocation is the location that is identified in the Get-VM command that you run in this procedure.
Get-VM –Name VMName | Select VMId, ConfigurationLocation $res = Get-ClusterResource -Name "VMClusterResourceName" -Cluster ComputeClusterName Set-ClusterParameter -InputObject $res -Name VMId -Value -Cluster ComputeClusterName Set-ClusterParameter -InputObject $res -Name VmStoreRootPath -Value "VMConfigLocation" -Cluster ComputeClusterName

Source of Information : Microsoft System Center

Using DPM servers for tenant backup By default, the CPS installation process provisions eight tenant DPM servers per rack that can be used for tenant backup. These servers are deployed to the compute clusters and use the naming convention DPM-TenantVM-0# (-01 through -08 on the first rack, -09 through 16 on the second rack, and so on). All of these DPM servers are pre-configured and ready to protect.

To provide spindle isolation and to keep backups on a separate pool, one storage pool on each rack is assigned for backup. This backup pool is configured as dual parity to maintain N+2 redundancy and has total usable capacity of 115.2 TB. Each tenant backup DPM server is provisioned with 20 TB of allocated disk space (20 VHDs of 1 TB each) that is provisioned on 15.4 TB of physical space. This difference between allocated and physical disk space is addressed by data deduplication that runs on the backup pool.

Source of Information : Microsoft System Center

Recovering infrastructure VMs There are three infrastructure VMs (for Active Directory, DNS, and DHCP) in the management cluster. All of them are backed up locally by using Windows Server Backup.

If one or more of the Active Directory/DNS/DHCP instances fails because of corruption or deletion of critical directories, you can use bare metal recovery to recover the instance. The procedure for using bare metal recovery to recover a single instance is described in this section. If there are multiple instance failures, you must repeat this procedure sequentially for all failed instances.

1. From the VMM console, connect to the failed domain controller VM. Boot the Active Directory/DNS/DHCP server into Windows Recovery Environment (WinRE). The server automatically boots into WinRE if it fails to boot into normal mode twice. If the server boots normally, run the following commands at a command prompt to restart in WinRE mode:
reagent /boottore shutdown /r /t 0

2. In WinRE mode, click Troubleshoot.

3. On the Advanced Options screen, click System Image Recovery.

4. Select the Administrator account on the System Image Recovery screen.

5. Type the password on the next screen.

6. In the Re-image Your Computer Wizard, you can see the latest available system image for recovery. If you want to recover to an older point in time, click Select A System Image, and choose the desired point in time. Click Next.

7. Click Next on the Choose Additional Restore Options page.

8. Click Finish to complete the Re-image Your Computer Wizard. The following screens display the progress of the recovery as all volumes are restored.

9. In the dialog box that is displayed, click Restart to restart the computer.

10. After recovery completes, schedule full server backups by using the Windows Server Backup
tool, as described in Configure Automatic Backups to a Volume at http://technet.microsoft.com//library/dd851674.aspx. You can do this as follows:
a. On the Select Backup Configuration page, click Full Server (recommended).
b. On the Specify Backup Time page, click Once A Day, and then select 12:00 AM as the backup time.
c. On the Specify Destination Type page, click Back Up To A Volume.
d. On the Select Destination Volume page, select Local Disk (F:) as the destination volume.

Source of Information : Microsoft System Center

Recovering Virtual Machine Manager VMM plays a key role in managing the hosts and VMs in the CPS environment. If you have exhausted all options to try to recover from application failure, you can use DPM to recover the VMM database to an older point in time.

To recover the VMM database, complete the following steps:
1. From the console VM, open Failover Cluster Manager.

2. Connect to the management cluster.

3. Shut down the two VMM VMs (-VMM-01, -VMM-02) that are located on the management cluster.

4. Use the steps in the section "Recovering a database to its original location" to recover the VMM database (called VirtualManagerDB in SCSHAREDDB SQL Server instance). To minimize data loss, be sure to select the latest recovery point.

5. Open Failover Cluster Manager, and connect to the management cluster.

6. Start the VMM VMs.

7. In the VMM console, verify that the content in the Fabric workspace is updated.

8. Detect and repair any data consistency issues by following the required steps in the “How to use data consistency runbooks” section in the CPS Admin Guide.

To recover the VMM VMs, complete the following steps:
1. From the console VM, open Failover Cluster Manager.

2. Connect to the management cluster.

3. Shut down the two VMM VMs (-VMM-01, -VMM-02) that are located on the management cluster.

4. Use the steps in the "Recovering VMs to their original location" section to recover the VMM VMs. To minimize data loss, be sure to select the latest recovery point.

5. In Failover Cluster Manager, connect to the management cluster, and then click Roles. In the Roles pane, right-click each VMM VM, and then click Start.

6. In Failover Cluster Manager, connect to the VMM guest cluster -HA-VMM, and then click Roles. If the -HA-VMM clustered role is not running, right-click the role, and then click Start Role.

7. Detect and repair any data consistency issues by following the required steps in the “How to use data consistency runbooks” section in the CPS Admin Guide.

Source of Information : Microsoft System Center

The three Vs of data storage To determine whether you need a partitioning strategy and what it should be, consider three questions about your data:

• Volume How much data will you ultimately store? A couple gigabytes? A couple hundred gigabytes? Terabytes? Petabytes?

• Velocity What is the rate at which your data will grow? Is it an internal app that isn’t generating a lot of data? An external app to which customers will be uploading images and videos?

• Variety What type of data will you store? Relational, images, key-value pairs, social graphs?

If you think you’re going to have a lot of volume, velocity, or variety, you have to carefully consider what kind of partitioning scheme will best enable your app to scale efficiently and effectively as it grows, and to ensure that you don’t run into any bottlenecks.

There are basically three approaches to partitioning:
• Vertical partitioning
• Horizontal partitioning
• Hybrid partitioning

Source of Information : Building Cloud Apps With Microsoft Azure

Managing access control for containers and blobs The Windows Azure Storage Service authenticates all requests against the Table service and Queue service. However, the storage service allows the possibility of unauthenticated access against the Blob service. The reason is that blobs provide an ideal location for storing large static content for a website. For example, the images in a photo-sharing site could be stored as blobs and downloaded directly from the Blob service without being transferred through a web role.

Public access control for the Blob service is managed at the container level. The Blob service supports the following three types of access control:

» No public read access in which all access must be authenticated
» Public read access which allows blobs in a container to be readable without authentication
» Full public read access in which authentication is not required to read the container data and the blobs contained in it

No public read access is the same access control as for the Queue service and Table service. The other two access control types both allow anonymous access to a blob, so that, for example, the blob can be downloaded into a browser by providing its full URL.

In the Windows Azure Storage Client library, the BlobContainerPublicAccessType enumeration specifies the three types of public access control for a container. The BlobContainerPermissions class exposes two properties: PublicAccess specifying a member of the BlobContainerPublicAccessType enumeration and SharedAccessPolicies specifying a set of shared access policies. The SetPermissions() method of the CloudBlobContainer class is used to associate a BlobContainerPermissions instance with the container. The GetPermissions() method retrieves the access permissions for a container.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Connecting to the storage emulator The Windows Azure SDK provides a compute emulator and a storage emulator that work in a development environment to provide a local emulation of Windows Azure hosted services and storage services. There are some differences in functionality between storage services and the storage emulator. Prior to Windows Azure SDK v1.3, the storage emulator was named development storage.

An immediate difference is that the storage emulator supports only one account name and access key. The account name is hard-coded to be devstoreaccount1. The access key is hard-coded to be:

Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==

Another difference is that the storage endpoints are constructed differently for the storage emulator. The storage service uses URL subdomains to distinguish the endpoints for the various types of storage. For example, the endpoint for the Blob service for a storage account named myaccount is:

myaccount.blob.core.windows.net

The endpoints for the other storage types are constructed similarly by replacing the word blob with either table or queue.

This differentiation by subdomain name is not used in the storage emulator which is hosted on the local host at 127.0.0.1. Instead, the storage emulator distinguishes the endpoints for various types of storage through use of different ports. Furthermore, the account name, rather than being part of the subdomain, is provided as part of the URL. Consequently, the endpoints used by the storage emulator are as follows:

127.0.0.1:10000/devstoreaccount1 Blob
127.0.0.1:10001/devstoreaccount1 Queue
127.0.0.1:10002/devstoreaccount1 Table

The Windows Azure Storage Client library hides much of this complexity but an understanding of it remains important in case something goes wrong. The account name and access key are hard-coded into the Storage Client library, which also provides simple access to an appropriately constructed CloudStorageAccount object.

The Storage Client library also supports a special value for the DataConnectionString in the service configuration file. Instead of specifying the account name and access key, it is sufficient to specify the following:

UseDevelopmentStorage=true

For example, this is specified as follows in the service configuration file:

This value can also be used for the Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString data connection string required for Windows Azure Diagnostics.

The CloudStorageAccount.Parse() and CloudStorageAccount.FromConnectionString() methods handle this value in a special way to create a CloudStorageAccount object that can be used to authenticate against the storage emulator.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Using SetConfigurationSettingPublisher() The CloudStorageAccount class in the Windows Azure Storage Client library encapsulates a StorageCredential instance that can be used to authenticate against the Windows Azure Storage Service. It also exposes a FromConfigurationSetting() factory method that creates a CloudStorageAccount instance from a configuration setting.

This method has caused much confusion since, without additional configuration, it throws an InvalidOperationException with a message of "SetConfigurationSettingPublisher needs to be called before FromConfigurationSetting() can be used." Consequently, before using FromConfigurationSetting(), it is necessary to invoke SetConfigurationSettingPublisher() once. The intent of this method is that it can be used to specify alternate ways of retrieving the data connection string that FromConfigurationSetting() uses to initialize the CloudStorageAccount instance. This setting is process-wide, so is typically done in the OnStart() method of the RoleEntryPoint class for the role.

CloudStorageAccount.SetConfigurationSettingPublisher(
(configName, configSetter) =>
{
configSetter(RoleEnvironment.GetConfigurationSettingValue(
configName));
});

There are several levels of indirection here, but the central feature is the use of a method that takes a String parameter specifying the name of the configuration setting and returns the value of that setting. In the example here, the method used is RoleEnvironment.GetConfigurationSettingValue(). The configuration-setting publisher can be set to retrieve configuration settings from any location including app.config or web.config.

The use of SetConfigurationSettingPublisher() is no longer encouraged. Instead, it is better to use CloudStorageAccount.Parse(), which takes a data connection string in canonical form and creates a CloudStorageAccount instance from it. We see how to do this in the Connecting to the Windows Azure Storage Service recipe.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Connecting to the Windows Azure Storage Service In a Windows Azure hosted service, the storage account name and access key are stored in the service configuration file. By convention, the account name and access key for data access are provided in a setting named DataConnectionString. The account name and access key needed for Windows Azure diagnostics must be provided in a setting named Microsoft.WindowsAzure.Plugins.Diagnostics.ConnectionString.

The DataConnectionString setting must be declared in the ConfigurationSettings section of the service definition file. However, unlike other settings, the connection string setting for Windows Azure diagnostics is implicitly defined when the diagnostics module is specified in the Imports section of the service definition file. Consequently, it must not be specified in the ConfigurationSettings section.

A best practice is to use different storage accounts for application data and diagnostic data. This reduces the possibility of application data access being throttled by competition for concurrent writes from the diagnostics monitor. It also provides a security boundary between application data and diagnostics data, as diagnostics data may be accessed by individuals who should have no access to application data.

In the Windows Azure Storage Client library, access to the storage service is through one of the client classes. There is one client class for each of Blob service, Queue service, and Table service—CloudBlobClient, CloudQueueClient, and CloudTableClient respectively. Instances of these classes store the pertinent endpoint, as well as the account name and access key.

The CloudBlobClient class provides methods to access containers list their contents and get references to containers and blobs. The CloudQueueClient class provides methods to list queues and to get a reference to the CloudQueue instance used as an entry point to the Queue service functionality. The CloudTableClient class provides methods to manage tables and to get the TableServiceContext instance used to access the WCF Data Services functionality used in accessing the Table service. Note that CloudBlobClient, CloudQueueClient, and CloudTableClient instances are not thread safe so distinct instances should be used when accessing these services concurrently.

The client classes must be initialized with the account name and access key, as well as the appropriate storage service endpoint. The Microsoft.WindowsAzure namespace has several helper classes. The StorageCredentialsAccountAndKey class initializes a StorageCredential instance from an account name and access key while the StorageCredentialsSharedAccessSignature class initializes a StorageCredential instance from a shared access signature. The CloudStorageAccount class provides methods to initialize an encapsulated StorageCredential instance directly from the service configuration file.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Managing Windows Azure Storage Service access keys The data stored by the Windows Azure Storage Service must be secured against unauthorized access. To ensure that security, all storage operations against the table service and the queue service must be authenticated. Similarly, other than inquiry requests against public containers and blobs, all operations against the blob service must also be authenticated. The blob service supports public containers so that, for example, blobs containing images can be downloaded directly into a web page.

Each storage account has a primary access key and a secondary access key that can be used to authenticate operations against the storage service. When creating a request against the storage service, one of the keys is used along with various request headers to generate a 256-bit, hash-based message authentication code (HMAC). This HMAC is added as an Authorization request header to the request. On receiving the request, the storage service recalculates the HMAC and rejects the request if the received and calculated HMAC values differ. The Windows Azure Storage Client library provides methods that manage the creation of the HMAC and attaching it to the storage operation request.

There is no distinction between the primary and secondary access keys. The purpose of the secondary access key is to enable continued use of the storage service while the other access key is being regenerated. While the primary access key is used for authentication against the storage service, the secondary access key can be regenerated without affecting the service—and vice versa. This can be extremely useful in situations where storage access credentials must be rotated regularly.

As possession of the storage account name and access key is sufficient to provide full control over the data managed by the storage account, it is essential that the access keys be kept secure. In particular, access keys should never be downloaded to a client, such as a Smartphone, as that exposes them to potential abuse.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Controlling Access in the Windows Azure Platform The various components of the Windows Azure Platform are exposed using Internet protocols. Consequently, they need to support authentication so that access to them can be controlled. The Windows Azure Storage Service manages the storage of blobs, queues, and tables. It is essential that this data be kept secure, so that there is no unauthorized access to it. Each storage account has an account name and an access key which are used to authenticate access to the storage service. The management of these access keys is important. The storage service provides two access keys for each storage account, so that the access key not being used can be regenerated. We see how to do this in the Managing Windows Azure Storage Service access keys recipe.

The storage service supports hash-based message authentication (HMAC), in which a storage operation request is hashed with the access key. On receiving the request, the storage service validates it and either accepts or denies it. The Windows Azure Storage Client library provides several classes that support various ways of creating an HMAC, and which hide the complexity of creating and using one. We see how to use them in the Connecting to the Windows Azure Storage Service recipe. The SetConfigurationSettingPublisher() method has caused some programmer grief, so we look at it in the Using SetConfigurationSettingPublisher() recipe.

The Windows Azure SDK provides a compute emulator and a storage emulator. The latter uses a hard-coded account name and access key. We see the support provided for this in the Connecting to the storage emulator recipe.

Blobs are ideal for storing static content for web roles, so the storage service provides several authentication methods for access to containers and blobs. Indeed, a container can be configured to allow anonymous access to the blobs in it. Blobs in such a container can be downloaded without any authentication. We see how to configure this in the Managing access control for containers and blobs recipe.

There is a need to provide an intermediate level of authentication for containers and blobs, a level that lies between full authentication and anonymous access. The storage service supports the concept of a shared access signature, which is a pre-calculated authentication token and can be shared in a controlled manner allowing the bearer to access a specific container or blob for up to one hour. We see how to do this in the Creating a shared access signature for a container or blob recipe.

A shared access policy combines access rights with a time for which they are valid. A container-level access policy is a shared access policy that is associated by name with a container. A best practice is to derive a shared access signature from a container-level access policy. Doing this provides greater control over the shared access signature as it becomes possible to revoke it. We see how do this in the Using a container-level access policy recipe.

There is more to the Windows Azure Platform than storage. The Windows Azure Service Management REST API is a RESTful API that provides programmatic access to most of the functionality available on the Windows Azure Portal. This API uses X.509 certificates for authentication. Prior to use, the certificate must be uploaded, as a management certificate, to the Windows Azure Portal. The certificate must then be added as a certificate to each request made against the Service Management API. We see how to do this in the Authenticating against the Windows Azure Service Management REST API recipe.

The Windows Azure AppFabric services use a different authentication scheme, based on a service namespace and authentication token. In practice, these are similar to the account name and access key used to authenticate against the storage service, although the implementation is different. The Windows Azure AppFabric services use the Windows Azure Access Control Service (ACS) to perform authentication. However, this is abstracted away in the various SDKs provided for the services. We see how to authenticate to one of these services in the Authenticating with the Windows Azure AppFabric Caching Service recipe.

Source of Information : MICROSOFT WINDOWS AZURE DEVELOPMENT COOKBOOK

Case Studies of some Platform as Service (PaaS) offerings. Aneka. Aneka is a .NET-based service-oriented resource management and development platform. Each server in an Aneka deployment (dubbed Aneka cloud node) hosts the Aneka container, which provides the base infrastructure that consists of services for persistence, security (authorization, authentication and auditing), and communication (message handling and dispatching). Cloud nodes can be either physical server, virtual machines (XenServer and VMware are supported), and instances rented from Amazon EC2.

The Aneka container can also host any number of optional services that can be added by developers to augment the capabilities of an Aneka Cloud node, thus providing a single, extensible framework for orchestrating various application models.

Several programming models are supported by such task models to enable execution of legacy HPC applications and MapReduce, which enables a variety of data-mining and search applications.

Users request resources via a client to a reservation services manager of the Aneka master node, which manages all cloud nodes and contains scheduling service to distribute request to cloud nodes.

App Engine. Google App Engine lets you run your Python and Java Web applications on elastic infrastructure supplied by Google. App Engine allows your applications to scale dynamically as your traffic and data storage requirements increase or decrease. It gives developers a choice between a Python stack and Java. The App Engine serving architecture is notable in that it allows real-time auto-scaling without virtualization for many common types of Web applications. However, such auto-scaling is dependent on the application developer using a limited subset of the native APIs on each platform, and in some instances you need to use specific Google APIs such as URLFetch, Datastore, and memcache in place of certain native API calls. For example, a deployed App Engine application cannot write to the file system directly (you must use the Google Datastore) or open a socket or access another host directly (you must use Google URL fetch service). A Java application cannot create a new Thread either.

Microsoft Azure. Microsoft Azure Cloud Services offers developers a hosted . NET Stack (C#, VB.Net, ASP.NET). In addition, a Java & Ruby SDK for .NET Services is also available. The Azure system consists of a number of elements. The Windows Azure Fabric Controller provides auto-scaling and reliability, and it manages memory resources and load balancing. The .NET Service Bus registers and connects applications together. The .NET Access Control identity providers include enterprise directories and Windows LiveID. Finally, the .NET Workflow allows construction and execution of workflow instances.

Force.com. In conjunction with the Salesforce.com service, the Force.com PaaS allows developers to create add-on functionality that integrates into main Salesforce CRM SaaS application. Force.com offers developers two approaches to create applications that can be deployed on its SaaS plaform: a hosted Apex or Visualforce application. Apex is a proprietary Java-like language that can be used to create Salesforce applications. Visualforce is an XML-like syntax for building UIs in HTML, AJAX, or Flex to overlay over the Salesforce hosted CRM system. An application store called AppExchange is also provided, which offers a paid & free application directory.

Heroku. Heroku is a platform for instant deployment of Ruby on Rails Web applications. In the Heroku system, servers are invisibly managed by the platform and are never exposed to users. Applications are automatically dispersed across different CPU cores and servers, maximizing performance and minimizing contention. Heroku has an advanced logic layer than can automatically route around failures, ensuring seamless and uninterrupted service at all times.

Source of Information : Wiley - Cloud Computing Principles and Paradigms 2011

Source of information :

PLATFORM AS A SERVICE PROVIDERS Public Platform as a Service providers commonly offer a development and deployment environment that allow users to create and run their applications with little or no concern to low-level details of the platform. In addition, specific programming languages and frameworks are made available in the platform, as well as other services such as persistent data storage and inmemory caches.

Programming Models, Languages, and Frameworks. Programming models made available by IaaS providers define how users can express their applications using higher levels of abstraction and efficiently run them on the cloud platform. Each model aims at efficiently solving a particular problem. In the cloud computing domain, the most common activities that require specialized models are: processing of large dataset in clusters of computers (MapReduce model), development of request-based Web services and applications; definition and orchestration of business processes in the form of workflows (Workflow model); and high-performance distributed execution of various computational tasks. For user convenience, PaaS providers usually support multiple programming languages. Most commonly used languages in platforms include Python and Java (e.g., Google AppEngine), .NET languages (e.g., Microsoft Azure), and Ruby (e.g., Heroku). Force.com has devised its own programming language (Apex) and an Excel-like query language, which provide higher levels of abstraction to key platform functionalities. A variety of software frameworks are usually made available to PaaS developers, depending on application focus. Providers that focus on Web and enterprise application hosting offer popular frameworks such as Ruby on Rails, Spring, Java EE, and .NET.

Persistence Options. A persistence layer is essential to allow applications to record their state and recover it in case of crashes, as well as to store user data. Traditionally, Web and enterprise application developers have chosen relational databases as the preferred persistence method. These databases offer fast and reliable structured data storage and transaction processing, but may lack scalability to handle several petabytes of data stored in commodity computers.

In the cloud computing domain, distributed storage technologies have emerged, which seek to be robust and highly scalable, at the expense of relational structure and convenient query languages. For example, Amazon SimpleDB and Google AppEngine datastore offer schema-less, automatically indexed database services. Data queries can be performed only on individual tables; that is, join operations are unsupported for the sake of scalability.

Source of Information : Wiley - Cloud Computing Principles and Paradigms 2011

Case Studies of the most popular public IaaS clouds Amazon Web Services. Amazon WS4 (AWS) is one of the major players in the cloud computing market. It pioneered the introduction of IaaS clouds in 2006. It offers a variety cloud services, most notably: S3 (storage), EC2 (virtual servers), Cloudfront (content delivery), Cloudfront Streaming (video streaming), SimpleDB (structured datastore), RDS (Relational Database), SQS (reliable messaging), and Elastic MapReduce (data processing).
The ElasticCompute Cloud (EC2) offers Xen-based virtual servers (instances) that can be instantiated from Amazon Machine Images (AMIs). Instances are available in a variety of sizes, operating systems, architectures, and price. CPU capacity of instances is measured in Amazon Compute Units and, although fixed for each instance, vary among instance types from 1 (small instance) to 20 (high CPU instance). Each instance provides a certain amount of nonpersistent disk space; a persistence disk service (Elastic Block Storage) allows attaching virtual disks to instances with space up to 1TB.

Elasticity can be achieved by combining the CloudWatch, Auto Scaling, and Elastic Load Balancing features, which allow the number of instances to scale up and down automatically based on a set of customizable rules, and traffic to be distributed across available instances. Fixed IP address (Elastic IPs) are not available by default, but can be obtained at an additional cost.

In summary, Amazon EC2 provides the following features: multiple data centers available in the United States (East and West) and Europe; CLI, Web services (SOAP and Query), Web-based console user interfaces; access to instance mainly via SSH (Linux) and Remote Desktop (Windows); advanced reservation of capacity (aka reserved instances) that guarantees availability for periods of 1 and 3 years; 99.5% availability SLA; per hour pricing; Linux and Windows operating systems; automatic scaling; load balancing.

Flexiscale. Flexiscale is a UK-based provider offering services similar in nature to Amazon Web Services. However, its virtual servers offer some distinct features, most notably: persistent storage by default, fixed IP addresses, dedicated VLAN, a wider range of server sizes, and runtime adjustment of CPU capacity (aka CPU bursting/vertical scaling). Similar to the clouds, this service is also priced by the hour.

In summary, the Flexiscale cloud provides the following features: available in UK; Web services (SOAP), Web-based user interfaces; access to virtual server mainly via SSH (Linux) and Remote Desktop (Windows); 100% availability SLA with automatic recovery of VMs in case of hardware failure; per hour pricing; Linux and Windows operating systems; automatic scaling (horizontal/vertical).

Joyent. Joyent’s Public Cloud offers servers based on Solaris containers virtualization technology. These servers, dubbed accelerators, allow deploying various specialized software-stack based on a customized version of Open- Solaris operating system, which include by default a Web-based configuration tool and several pre-installed software, such as Apache, MySQL, PHP, Ruby on Rails, and Java. Software load balancing is available as an accelerator in addition to hardware load balancers.

A notable feature of Joyent’s virtual servers is automatic vertical scaling of CPU cores, which means a virtual server can make use of additional CPUs automatically up to the maximum number of cores available in the physical host.

In summary, the Joyent public cloud offers the following features: multiple geographic locations in the United States; Web-based user interface; access to virtual server via SSH and Web-based administration tool; 100% availability SLA; per month pricing; OS-level virtualization Solaris containers; Open-Solaris operating systems; automatic scaling (vertical).

GoGrid. GoGrid, like many other IaaS providers, allows its customers to utilize a range of pre-made Windows and Linux images, in a range of fixed instance sizes. GoGrid also offers “value-added” stacks on top for applications such as high-volume Web serving, e-Commerce, and database stores. It offers some notable features, such as a “hybrid hosting” facility, which combines traditional dedicated hosts with auto-scaling cloud server infrastructure. In this approach, users can take advantage of dedicated hosting (which may be required due to specific performance, security or legal compliance reasons) and combine it with on-demand cloud infrastructure as appropriate, taking the benefits of each style of computing.

As part of its core IaaS offerings, GoGrid also provides free hardware load balancing, auto-scaling capabilities, and persistent storage, features that typically add an additional cost for most other IaaS providers.

Rackspace Cloud Servers. Rackspace Cloud Servers is an IaaS solution that provides fixed size instances in the cloud. Cloud Servers offers a range of Linux-based pre-made images. A user can request different-sized images, where the size is measured by requested RAM, not CPU.

Like GoGrid, Cloud Servers also offers hybrid approach where dedicated and cloud server infrastructures can be combined to take the best aspects of both styles of hosting as required. Cloud Servers, as part of its default offering, enables fixed (static) IP addresses, persistent storage, and load balancing (via A-DNS) at no additional cost.

Source of Information : Wiley - Cloud Computing Principles and Paradigms 2011

Diagnostics in the cloud At some point you might need to debug your code, or you’ll want to judge how healthy your application is while it’s running in the cloud. We don’t know about you, but the more experienced we get with writing code, the more we know that our code is less than perfect. We’ve drastically reduced the amount of debugging we need to do by using test-driven development (TDD), but we still need to fire up the debugger once in a while.

Debugging locally with the SDK is easy, but once you move to the cloud you can’t debug at all; instead, you need to log the behavior of the system. For logging, you can use either the infrastructure that Azure provides, or you can use your own logging framework. Logging, like in traditional environments, is going to be your primary mechanism for collecting information about what’s happening with your application.

Using Azure Diagnostics to find what’s wrong
Logs are handy. They help you find where the problem is, and can act as the flight data recorder for your system. They come in handy when your system has completely burned down, fallen over, and sunk into the swamp. They also come in handy when the worst hasn’t happened, and you just want to know a little bit more about the behavior of the system as it’s running. You can use logs to analyze how your system is performing, and to understand better how it’s behaving. This information can be critical when you’re trying to determine when to scale the system, or how to improve the efficiency of your code.

The drawback with logging is that hindsight is 20/20. It’s obvious, after the crash, that you should’ve enabled logging or that you should’ve logged a particular segment of code. As you write your application, it’s important to consider instrumentation as an aspect of your design.

Logging is much more than just remote debugging, 1980s-style. It’s about gathering a broad set of data at runtime that you can use for a variety of purposes; debugging is one of those purposes.

Challenges with troubleshooting in the cloud
When you’re trying to diagnose a traditional on-premises system, you have easy access to the machine and the log sources on it. You can usually connect to the machine with a remote desktop and get your hands on it. You can parse through log files, both those created by Windows and those created by your application. You can monitor the health of the system by using Performance Monitor, and tap into any source of information on the server. During troubleshooting, it’s common to leverage several tools on the server itself to slice and dice the mountain of data to figure out what’s gone wrong.

You simply can’t do this in the cloud. You can’t log in to the server directly, and you have no way of running remote analysis tools. But the bigger challenge in the cloud is the dynamic nature of your infrastructure. On-premises, you have access to a static pool of servers. You know which server was doing what at all times. In the cloud, you don’t have this ability. Workloads can be moved around; servers can be created and destroyed at will. And you aren’t trying to diagnose the application on one server, but across a multitude of servers, collating and connecting information from all the different sources. The number of servers used in cloud applications can swamp most diagnostic analysis tools. The shear amount of data available can cause bottlenecks in your system.

For example, a typical web user, as they browse your website and decide to check out, can be bounced from instance to instance because of the load balancer. How do you truly find out the load on your system or the cause for the slow response while they were checking out of your site? You need access to all the data that’s available on terrestrial servers and you need the data collated for you.

You also need close control over the diagnostic data producers. You need an easy way to dial the level of information from debug to critical. While you’re testing your systems, you need all the data, and you need to know that the additional load it places on the system is acceptable. During production, you want to know only about the most critical issues, and you want to minimize the impact of these issues on system performance.

For all these reasons, the Windows Azure Diagnostics platform sits on top of what is already available in Windows. The diagnostics team at Microsoft has extended and plugged in to the existing platform, making it easy for you to learn, and easy to find the information you need.

Source of Information : Manning Azure in Action 2010

Code Library

Categories