Category Archives: Azure Storage

Azure IaaS and Azure Stack: announcements and updates (May 2021 – Weeks: 17 and 18)

This series of blog posts includes the most important announcements and major updates regarding Azure infrastructure as a service (IaaS) and Azure Stack, officialized by Microsoft in the last two weeks.

Azure

Compute

Azure Hybrid Benefit for Linux with RI and VMSS Support

Azure Hybrid Benefit is available for Linux, extending the ability to easily migrate RHEL and SLES servers to Azure beyond existing pay-as-you-go instances to include support for Azure Reserved Instance (RI) and virtual machine scale set (VMSS).

While previous Bring-Your-Own-Subscription cloud migration options available to Red Hat and SUSE customers allowed them to use their pre-existing RHEL and SLES subscriptions in the cloud, Azure Hybrid Benefit for Linux improves upon this with several capabilities unique to Azure making enterprise Linux cloud migration even easier than before:

  • Applies to all Red Hat Enterprise Linux and SUSE Linux Enterprise Server pay-as-you-go images available in the Azure Marketplace or Azure Portal. No need to provide your own image.
  • Save time with seamless post-deployment conversions—production redeployment is unnecessary. Simply convert the pay-as-you-go images used during your proof-of-concept testing to bring-your-own-subscription billing.
  • Lower ongoing operational costs with automatic image maintenance, updates, and patches: Microsoft maintains the converted RHEL and SLES images for you.
  • Enjoy the convenience of unified user interface integration with the Azure CLI, providing the same UI as other Azure virtual machines, as well as scalable batch conversions.
  • Get co-located technical support from Azure, Red Hat, and SUSE with just one ticket.
  • Combine with recently announced Red Hat and SUSE support for Azure shared disks to lift-and-shift failover clusters and parallel file systems, like Global File System.
  • Fully compatible with Azure Arc, providing end-to-end hybrid cloud operations management for Windows, RHEL, and SLES servers in one solution.

New Azure VMs for general purpose and memory intensive workloads (preview)

The new Dv5, Dsv5, Ddv5, Ddsv5, and Ev5, Edv5 series Azure Virtual Machines, now in preview, are based on the 3rd Generation Intel® Xeon® Platinum 8370C (Ice Lake) processor in a hyper-threaded configuration. This custom processor can reach an all-core Turbo clock speed of up to 3.5GHz and features Intel® Turbo Boost Technology 2.0, Intel® Advanced Vector Extensions 512 (Intel® AVX-512) and Intel® Deep Learning Boost. These new offerings deliver a better value proposition for general-purpose, and memory intensive workloads compared to the prior generation (e.g., increased scalability and an upgraded CPU class) including better price to performance.

The Dv5, Dsv5, Ddv5, Ddsv5 VM sizes offer a combination of vCPUs and memory able to meet the requirements associated with most general-purpose workloads and can scale up to 96 vCPUs. The Ddv5 and Ddsv5 VM sizes feature high performance, large local SSD storage (up to 2,400 GiB). The Dv5 and Dsv5 VM series offer a lower price of entry since they do not feature any local temporary storage. If you require temporary storage select the latest Ddv5 or Ddsv5 Azure virtual machines, which are also in Preview.

The Ev5 and Edv5 VM sizes feature up to 672 GiB of RAM and are ideal for memory-intensive enterprise applications. You can attach Standard SSDs and Standard HDDs disk storage to these VMs. If you prefer to use Premium SSD or Ultra Disk storage, please select the Esv5 and Edsv5 VM series, which will be in preview in the near future. The Ev5 and Esv5 VMs offer a lower price of entry since they do not feature any local temporary storage. If you require temporary storage select the latest Edv5 VM series which are also in preview, or the Edsv5 VM series, which will be in preview in the near future.

New NPv1 virtual machines

NPv1 series virtual machines are a new addition to the Azure product offering. These instances are powered by Xilinx Alveo U250 FPGAS. These highly-programmable accelerators benefit a variety of computationally intensive workloads such as genomics, image-processing, security, data analysis and more. The NP series offering is based upon the commercially available U250 from Xilinx and uses a standard shell easing the difficulties of migrating existing FPGA workloads & solutions to the cloud. New Xilinx Alveo U250 FPGA NPv1 VMs are now generally available in West US 2, East US, West Europe, and Southeast Asia.

Microsoft acquires Kinvolk to accelerate container-optimized innovation

Microsoft is excited to bring the expertise of the Kinvolk team to Azure and having them become key contributors to the engineering development of Azure Kubernetes Service (AKS), Azure Arc, and future projects that will expand Azure’s hybrid container platform capabilities and increase Microsoft’s upstream open source contributions in the Kubernetes and container space. Microsoft is also committed to maintaining and building upon Kinvolk’s open source culture. The Kinvolk team will continue to remain active in their existing open source projects and will be essential to driving further collaboration between Azure engineering teams and the larger open source container community.

Storage

Azure Blob storage: NFS 3.0 protocol support public preview now expands to all regions

Azure Blob storage is the only public cloud storage platform that supports NFS 3.0 protocol over object storage natively (no gateway or data copying required), with object storage economics. This new level of support is optimized for high-throughput, read-heavy workloads where data will be ingested once and minimally modified further, such as large-scale analytic data, backup and archive, media processing, genomic sequencing, and line-of-business applications. Azure Blob Storage NFS 3.0 preview supports general purpose v2 (GPV2) storage accounts with standard tier performance in all publicly available regions. Further, Microsoft is enabling a set of Azure blob storage features in premium blockblob accounts with NFS 3.0 feature enabled such as blob service REST API and lifecycle management.

Attribute-based Access Control (ABAC) in preview

Attribute-based access control (ABAC) is an authorization strategy that defines access levels based on attributes associated with security principals, resources, requests, and the environment. Azure ABAC builds on role-based access control (RBAC) by adding conditions to Azure role assignments in the existing identity and access management (IAM) system. This preview includes support for role assignment conditions on Blobs and ADLS Gen2, and enables you to author conditions based on resource and request attributes.

Prevent Shared Key authorization for an Azure Storage account

Every secure request to an Azure Storage account must be authorized. By default, requests can be authorized with either Azure Active Directory (Azure AD) credentials, or by using the account access key for Shared Key authorization. Of these two types of authorization, Azure AD provides superior security and ease of use over Shared Key and is recommended by Microsoft. To require clients to use Azure AD to authorize requests, you can disallow requests to the storage account that are authorized with Shared Key. Microsoft is announcing the general availability of the ability to disable Shared Key authorization for Azure Storage.

Append blob support in Azure Data Lake Storage

Append blobs provide a simple and effective way of adding new content to the end of a file or blob when the existing content does not need to be modified. This makes append blobs great for applications such as logging that need to add information to existing files efficiently and continuously. Until now, only block blobs were supported in Azure Data Lake Storage accounts. Applications can now also create append blobs in these accounts and write to them using Append Block operations. These append blobs can be read using existing Blob APIs and Azure Data Lake Storage APIs.

Networking

Multiple features for Azure VPN Gateway

The following features for Azure VPN Gateway are general available:

  • Multiple authentication types for point-to-site VPN – You can now enable multiple authentication types on a single gateway for OpenVPN tunnel type. Azure AD, certificate-based and RADIUS can all be enabled on a single gateway.
  • BGP diagnostics – You can now see the Border Gateway Protocol session status, route advertised and routes learnt by the VPN Gateway.
  • VPN packet capture in Azure portal – Support for packet capture on the VPN Gateway is now availbe in the Azure portal.
  • VPN connection management – With new enhancements in VPN connection management capabilities, you can now reset an individual connection instead of resseting the whole gateway. You can also set the Internet Key Exchange (IKE) mode of the gateway to responder-only, initiator-only or both and view the Security Association (SA) of a connection.

Azure IaaS and Azure Stack: announcements and updates (April 2021 – Weeks: 15 and 16)

This series of blog posts includes the most important announcements and major updates regarding Azure infrastructure as a service (IaaS) and Azure Stack, officialized by Microsoft in the last two weeks.

Azure

Compute

New M-series Msv2/Mdsv2 Medium Memory VMs for memory-optimized workloads

Azure Msv2/Mdsv2 Medium Memory Series offering up to 192vCPU and 4TB memory configurations and running on Cascade Lake processor are now generally available. Msv2/Mdsv2 medium memory VM sizes providing a 20% increase in CPU performance, increased flexibility with local disks, and a new intermediate scale up-option. These virtual machines provide unparalleled computational performance to support large in-memory databases and workloads such as SAP HANA and SQL Hekaton.

Azure Virtual Machines DCsv2-series in Azure Government (public preview)

Azure Government customers can build secure, enclave-based applications to protect code and data while it’s in use, in a dedicated cloud that meets stringent government security and compliance requirements. Confidential computing DCsv2-series virtual machines are now in preview for Azure Government customers (federal, state, local governments, and their partners) in US Government Virginia and Arizona regions. These VMs are backed by Intel XEON E-2288G processors with Intel Software Guard Extensions (SGX) technology.

Microsoft announces plans to establish first datacenter region in Malaysia

The new datacenter region is part of the “Bersama Malaysia” initiative to support inclusive economic growth in Malaysia.

Storage

Azure Blob storage supports objects up to 200 TB in size

Workloads that utilize larger file sizes such as backups, media, and seismic analysis can now utilize Azure Blob storage and ADLS Gen2 without breaking these large files into separate blobs. Each blob is made up of up to 50,000 blocks. Each block can now be 4GB in size for a total of 200 TB per blob or ADLS Gen2 file.

Lustre HSM tools to import from or export to Azure Storage

Lustre HSM (Hierarchical Storage Management) provides the capability to associate a Lustre file system with an external storage system and migrate file data between them.

Now available are the File System Hydrator and Copy Tool, which enables integrating a Lustre file system with an Azure storage account:

  • The File System Hydrator is used to import a file system namespace from an Azure storage account into a Lustre file system with the imported files left in the ‘released’/’exist’ state.
  • The Copy Tool is used to hydrate the content of the files in the storage account into the Lustre file system on-demand. The copy tool can also be used to archive content of files back into the storage account, including changed or added files.

Networking

Application Gateway URL Rewrite

Azure Application Gateway now supports the ability to rewrite host name, path and query string of the request URL. In addition to header rewrites, you can now also rewrite URL of all or some of the client requests based on matching one or more conditions as required. You can choose to route the request based on the original URL or the rewritten URL. This feature enables several important scenarios such as allowing path based routing for query string values and support for hosting friendly URLs.

Azure IaaS and Azure Stack: announcements and updates (April 2021 – Weeks: 13 and 14)

This series of blog posts includes the most important announcements and major updates regarding Azure infrastructure as a service (IaaS) and Azure Stack, officialized by Microsoft in the last two weeks.

Azure

Compute

Virtual machine (VM) level disk bursting available on all Dsv3 and Esv3 families

Virtual machine level disk bursting allows your virtual machine to burst its disk IO and MiB/s throughput performance for a short time daily. This feature is now enabled on all our Dsv3-series and Esv3-series virtual machines, with more virtual machine types and families support soon to come. There is no additional cost associated with this new capability or adjustments on the VM pricing and it comes enabled by default.

Cloud Services (extended support) is generally available

Cloud Services (extended support), which is a new Azure Resource Manager (ARM)-based deployment model for Azure Cloud Services, is generally available. Cloud Services (extended support) has the primary benefit of providing regional resiliency along with feature parity with Azure Cloud Services deployed using Azure Service Manager (ASM). It also offers some ARM capabilities such as role-based access and control (RBAC), tags, policy, private link support, and use of deployment templates. The ASM-based deployment model for Cloud Services has been renamed Cloud Services (classic). Customers retain the ability to build and rapidly deploy web and cloud applications and services. Customers will be able to scale cloud services infrastructure based on current demand and ensure that the performance of applications can keep up while simultaneously reducing costs. The platform-supported tool for migrating existing cloud services to Cloud Services (extended support) also goes into preview. Migrating to ARM will allow customers to set up a robust infrastructure platform for their applications. 

Storage

Azure File Sync agent v12 

Improvements and issues that are fixed in the v12 release:

  • New portal experience to configure network access policy and private endpoint connections
    • You can now use the portal to disable access to the Storage Sync Service public endpoint and to approve, reject and remove private endpoint connections. To configure the network access policy and private endpoint connections, open the Storage Sync Service portal, go to the Settings section and click Network.
  • Cloud Tiering support for volume cluster sizes larger than 64KiB
  • Measure bandwidth and latency to Azure File Sync service and storage account
    • The Test-StorageSyncNetworkConnectivity cmdlet can now be used to measure latency and bandwidth to the Azure File Sync service and storage account. Latency to the Azure File Sync service and storage account is measured by default when running the cmdlet. Upload and download bandwidth to the storage account is measured when using the “-MeasureBandwidth” parameter. To learn more, see the release notes.
  • Improved error messages in the portal when server endpoint creation fails
    • We heard your feedback and have improved the error messages and guidance when server endpoint creation fails.
  • Miscellaneous performance and reliability improvements
    • Improved change detection performance to detect files that have changed in the Azure file share.
    • Performance improvements for reconciliation sync sessions.
    • Sync improvements to reduce ECS_E_SYNC_METADATA_KNOWLEDGE_SOFT_LIMIT_REACHED and ECS_E_SYNC_METADATA_KNOWLEDGE_LIMIT_REACHED errors.
    • Files may fail to tier on Server 2019 if Data Deduplication is enabled on the volume.
    • AFSDiag fails to compress files if a file is larger than 2GiB.

To obtain and install this update, configure your Azure File Sync agent to automatically update when a new version becomes available or manually download the update from the Microsoft Update Catalog.

More information about this release:

  • This release is available for Windows Server 2012 R2, Windows Server 2016 and Windows Server 2019 installations.
  • A restart is required for servers that have an existing Azure File Sync agent installation.
  • The agent version for this release is 12.0.0.0.
  • Installation instructions are documented in KB4568585.

Encryption scopes in Azure Storage

Encryption scopes introduce the option to provision multiple encryption keys in a storage account for blobs. Previously, customers using a single storage account for multi-tenancy scenarios were limited to using a single account-scoped encryption key for all the data in the account. With encryption scopes, you now can provision multiple encryption keys and choose to apply the encryption scope either at the container level (as the default scope for blobs in that container) or at the blob level. 

Azure Data Explorer external tables

An external table is a schema entity that references data stored outside the Azure Data Explorer database. Azure Data Explorer Web UI can create external tables by taking sample files from a storage container and creating schema based on these samples. You can then analyze and query data in external tables without ingestion into Azure Data Explorer.

Azure Security Center: Azure Storage protection

Azure Security Center, the cloud solution that allows you to prevent, detect and respond to security threats affecting hybrid architectures, it also provides enhanced protection for storage resources in Azure. The solution detects unusual and potentially harmful attempts to access or use Azure Storage. This article describes how to effectively protect storage in Azure with this solution, looking at the news recently announced in this area.

Azure Security Center (ASC) is possible to activate it in two different tiers:

  • Free tier. In this tier ASC is totally free and performs a continuous assessment, providing recommendations relating to the security of the Azure environment.
  • Standard tier. Compared to tier free adds enhanced threat detection, using behavioral analysis and machine learning to identify zero-day attacks and exploits. Through machine learning techniques and through the creation of whitelist is possible to control the execution of applications to reduce exposure to network attacks and malware. Furthermore, the standard level adds the ability to perform in an integrated manner a Vulnerability Assessment for virtual machines in Azure. Azure Security Center Standard supports several resources including: VMs, Virtual machine scale sets, App Service, SQL servers, and Storage accounts.

Advanced Threat Protection (ATP) for Azure Storage, it is one of several features in Azure Security Center Standard.

Figure 1 – Comparison of the features of the different tiers of ASC

Enabling the Security Center Standard tier is strongly recommended to improve security postures in your Azure environment.

The Advanced Threat Protection feature (ATP) for Azure Storage was announced last year, allowing you to detect common threats such as malware, access from suspicious sources (including TOR nodes), data exfiltration activities and more, but all limited to blob containers. Support for Azure Files and Azure Data Lake Storage Gen2 has also been included recently. This also helps customers protect data stored in file shares and data stores designed for the analysis of corporate big data.

Enabling this feature from the Azure portal is very simple and can be done at the Security Center-protected subscription level or selectively on individual storage accounts.

To enable this protection on all storage accounts in your subscription, you must go to the "Pricing & Settings” of Security Center and activate the protection of Storage Accounts.

Figure 2 – ATP activation for Azure Storage at the subscription level

If you prefer to enable it only on certain storage accounts, you need to activate it in the respective settings of Advanced security.

Figure 3 – ATP activation on the single storage account

When anomaly occurs on a storage account, security alerts are sent by email to Azure subscription administrators, with details of detected suspicious activity and related recommendations on how to investigate and resolve threats.

Details included in the event notification include::

  • The nature of the anomaly
  • The name of the storage account
  • The time of the event
  • The type of storage
  • Potential causes
  • The recommended steps to investigate what has been found
  • The actions to be taken to remedy what happened

Figure 4 – Example of a security alert sent in the face of a detection of a threat

In this example, the EICAR test file was used to validate that the solution was working correctly.. This is a file developed by the’European Institute for Computer Anti-Virus Research (EICAR) which is used to securely validate security solutions.

Security alerts can be viewed and managed directly from Azure Security Center, where details and actions to investigate current threats and address future threats are displayed..

Figure 5 – Example of a security alert in the ASC Security alerts tile

To get the full list of possible alerts generated by unusual and potentially malicious attempts to log in or use storage accounts, you can access the Threat protection for data services in Azure Security Center.

This protection is very useful even if you have architecture that uses the service Azure File Sync (AFS), which allows you to centralize the network folders of your infrastructure in Azure Files.

Conclusions

Business companies are increasingly moving their data to the cloud, looking for distributed architecture, high performance and cost optimization. All features offered by the public cloud require you to strengthen cybersecurity, particularly given the increasing complexity and sophistication of cyberattacks. By adopting Advanced Threat Protection (ATP) for Azure Storage, you can increase the level of storage security used in your Azure environment easily and effectively.

Protect Azure File Sync through Azure Backup

Azure File Sync service allows you to centralize your infrastructure's network folders in Azure Files, allowing you to maintain the typical characteristics of a file server on-premises, in terms of performance, compatibility and flexibility and at the same time to benefit from the potential offered by cloud. Azure File Sync integrates with Azure Backup making it possible to centrally manage protection policies in the cloud. This article describes how these two solutions are integrated and what you need to consider to enable effective protection.

The main features of Azure File Sync are the following:

  • Cloud tiering: are maintained locally only recently accessed data.
  • Multi-site sync: you have the option to sync between different sites, allowing write access to the same data between different Windows Servers and Azure Files.
  • Integration with Azure backup: ability to enable content protection using Azure Backup.
  • Disaster recovery: you have the option to immediately restore metadata files and retrieve only the data you need, for faster service reactivation in Disaster Recovery scenarios.
  • Direct access to the cloud: you can directly access content on the File share from other Azure resources (IaaS and PaaS).

Azure File Sync can turn Windows Server into a "cache" to quickly access content on a given Azure File share. Local access to data can occur with any protocol available in Windows Server. You have the possibility to have multiple "cache" servers in different geographic locations.

The ability to enable the Cloud Tiering makes Azure File Sync an increasingly popular solution, but this aspect in particular requires you to make the necessary considerations in the strategy to be adopted for data protection. As well as antivirus solutions, backup solutions may cause files stored in the cloud to be recalled through the Cloud Tiering feature. Microsoft recommends a cloud backup solution to back up Azure File share instead of an on-premises backup solution. If you are using a local backup solution, backups must be performed on a server belonging to a sync group where cloud tiering is disabled.

How the backup job works

Azure File share security is done under the following architecture:

Figure 1 – Architecture for securing Azure File share

The Azure File Share security process involves the following steps::

  1. The presence of a Recovery Service Vault is required in order to configure backups. Therefore, you should proceed with the creation of it if it is not available.
  2. Azure Backup will perform a discovery required to complete the process of registering the storage account that hosts the Azure File shares to be protected.
  3. Completed the registration process, Azure Backup will store the list of File shares present on the storage account in its catalog.
  4. You can select the Azure File share to protect and associate them with its backup policies, specific scheduling and data retention policies.
  5. Based on the policies configured Azure Backup performs backups. A key aspect to consider is that the backup is currently being backed up by generating a snapshot of the Azure File share. Data in Azure File share are never transferred to the Recovery Service Vault, but Azure Backup simply creates and manages snapshots that are part of the storage account.
  1. In the event of a restore, snapshots will be used, the relative URL of the backups, is taken from the metadata store in the Recovery Service Vault.
  2. The backup and restore job monitor is sent to the Azure Backup Monitoring service. This allows you to get an overall view of all backups, including Azure File Share. Furthermore, you can also configure alerts or e-mail notifications if you have problems performing backups.

 

Benefits of adopting this security strategy

  • Zero infrastructure: no infrastructure is required to enable environmental protection.
  • Customizing retention policies: backups can be configured with data retention policies daily, weekly, monthly and yearly, based on your needs. Annual backups can now be kept up to 10 years.
  • Built-in management capabilities: you can schedule your backups and specify the retention period you want in a way that is fully integrated into the platform.
  • Instant Restore: Azure File Share backup uses snapshots, this allows you to select only the files you want to restore instantly.
  • Alerts and reports: you can configure alerts for backup and restore operations that present errors. You can also use the reporting solution provided by Azure Backup to get detailed information about backup jobs.

Protect against accidental deletion of Azure File shares

To provide greater protection against cyberattacks and accidental deletion, Azure Backup recently added an extra layer of security to the Azure File shares snapshot management solution. If you delete the File shares, content and its recovery points (Snapshots) are retained for a configurable period of time, enabling full recovery without data loss. When you configure protection for a File share, Azure Backup enables soft-delete functionality at the account storage level with a retention period of 14 days, which is configurable according to your needs. This setting determines the time window in which File Share content and snapshots can be restored after any accidental deletion operations. Once the File share is restored, backups resume working without the need for additional configurations.

Conclusions

This solution allows in very simple, reliable and secure way to configure protection for Azure File shares and easily recover data when needed. The integration between Azure File Sync and Azure Backup will surely see the release of several new features in the coming months, including, very much heard, the ability to configure data transfer to the Recovery Service Vault instead of keeping snapshots in the same storage account where the data resides. To understand all the support scopes and limits in using the Azure Backup service to protect Azure File shares, you can see this Microsoft article.