Guest Author: Randy Weis Date: June 14, 2022 There are different levels of Disaster Recovery in an IT organization depending on the acceptable data loss (RPO or Recovery Point Objective) and acceptable time to recovery (RTO or Recovery Time Objective). In addition, there are other variables involved when planning DR. Such as: Disaster Recovery Types Business Continuity – covers more than IT, such as facilities and communications Disaster Recovery – recovery of business applications Disaster Resiliency – avoiding disasters with resilient infrastructure such as clustering at platform, datacenter or network level (SDWAN) High Availability – application-level redundancy and failover Scope of the Disaster Partial Site Disaster – Some portion of the IT infrastructure is taken offline by human error/malice, burst pipe, hardware failure Total Site Disaster – An entire site (datacenter, server room, server closet) goes down due to natural disaster, fire, flood, local power outage/generator failure, and will be down for more than a few hours (days, weeks) Disaster Recovery Location Veeam works at the Disaster Recovery Level and complements the other levels of workplace disaster recovery. Veeam provides orchestrated Disaster Recovery from one location to another. In addition, Veeam offers Direct Restore to AWS. GCP or Azure for VMs and physical machines. Regardless of workload, Veeam meets the level of DR necessary in one license and UI. Level 0: Recovery of Backup Server Scope: Partial or Total Site Disaster The Veeam Backup & Replication Server can be recovered quickly from a Configuration Backup. A new VM can be created (from template, preferably) at the same or alternate site depending on the scope of the disaster. Once the configuration database is recovered, or the local repository of backup copies is rescanned, restores can begin (see Level 1). This is an internal recovery option for IT, prior to site or application recovery. One of the main differentiators of Veeam though compared to the market is you can still easily restore your data if there is no configuration backup. All backup files are portable and have the metadata needed for recovery. Simply install a new Veeam server or turn on the idle one in your DR site and rescan the primary or DR repository whether it's on-prem or in the cloud. Level 1: Instant Recovery Scope: Partial Site Disaster RPO = hours RTO = minutes This is an “on-site” disaster recovery. Veeam offers Instant Recovery to VMware/Hyper-V/AHV for backups of: Virtual Machines/disks in VMware/Hyper-V/Nutanix AHV/RHV VMs in Storage Snapshots Physical Windows and Linux machines (Veeam does the P2V conversion) NAS File Shares: R/W for SMB, Read-only for NFS Databases for Oracle or SQL Backups of VMs in Azure/AWS EC2/GCP to VMware Level 2: Recovery from Backups at another location Scope: Partial or Total Site Disaster RPO = hours RTO = hours/minutes Veeam copies backup files from one repository to another in a Backup Copy Job. A Veeam Backup & Replication Server (VBR) at that site can restore the backups from the target repository for Backup Copy Jobs. If log shipping for SQL or Oracle is enabled, those logs will also be available at the recovery location. Backup Copy Jobs can copy & Restore any backup that is stored in a Veeam repository: VMs from VMware, Hyper-V, Nutanix AHV, RHV Agent backups for Windows, Linux, Solaris, AIX RMAN, SAP HANA, SQL VMS from AWS/Azure/GCP stored in an External Repository (Object Storage container) Level 3: Replication & Orchestrated Recovery of VM from VMware Snapshots (and Hyper-V) Scope: Total Site Disaster RPO = 30 minutes to 4 hours for replicas, 24 hours for recovery from Backup RTO = minutes to hours, depending on scope and hardware resources Veeam replicates Virtual Machine files from one datastore to another using VMware Snapshots with built-in orchestration that can re-map to DR virtual networks, re-IP addressing, boot order and boot delay. Replication can use Backup Copy Jobs as a source, so that Recovery from Backups RTO can be reduced to minutes. Veeam can build failover plans that can be triggered on demand or planned; offers commit failover or failback (testing), failback and commit failback (after original site is available). Supported workloads are: VMware Hyper-V Level 4: Continuous Data Replication & Orchestrated Recovery of VMware VMs Scope: Total Site Disaster RPO = 2 seconds to 5 minutes RTO = minutes Veeam replicates VMware Virtual Machine files from one datastore to another using VMWare API for IO Filtering (no snapshots), with built-in orchestration that can re-map to DR virtual networks, re-IP addressing, boot order and boot delay. Veeam can build failover plans that can be triggered on demand or planned; offers commit failover or failback (testing), failback and commit failback (after original site is available). Level 5: Disaster Recovery Orchestrator Scope: Total Site Disaster RPO: 2 seconds to 24 hours RTO: minutes Veeam provides a higher level of orchestration and automation with Veeam’s Orchestrator software. This software can recover in an automated manner from backups, from snapshot replicas and from CDP replicas. DRO offers a more complete and customized automation of application recovery with Runbooks that are updated as the plan is updated Scheduled testing, bubble or actual Audit Logs Readiness Reports Test Result Reports Delegation of recovery by role and by site Veeam’s Disaster Recovery Orchestrator can be implemented as an enterprise strategy and solution. Scenarios for orchestrated recovery are: Replication Jobs (snap or CDP) Backup Jobs Storage Replication (NetApp and HPE 3PAR) Disaster Recovery to the Cloud Scope: Partial or Total Site Disaster RPO = 24 hours RTO = hours Veeam copies backup files from a local repository to a Capacity Tier repository (S3 Bucket) in AWS/Azure/GCP. A Veeam Backup & Replication Server (VBR) in the cloud can restore the backups from the S3 repository directly into a running VM into one of the major public cloud providers.

This one will be short and sweet... Validating database backups are crucial for ensuring restores are actually useable in the event of a disaster or attack. With Veeam's Oracle RMAN plug-in DBAs can continue to leverage the same validate commands they are accustom to, while also providing visibility and governance to the data protection team. validate backupset There are a number of validate commands a DBAs can run. Anything from validating a database to a control file to a specific table. As you can see above, DBAs can run the same RMAN commands they are familiar with, but if the Veeam plug-in is installed the verification will come from the backup file on the Veeam repository. This enables data protection teams to still have governance over all the backups the business is responsible for, while also enabling DBAs to keep responsibility over database protection and recovery activities. Best of both worlds. As shown above, Veeam administrators have visibility into any validate, restore or backup commands Oracle DBAs run.

An under the radar enhancement in Veeam v11a is that protecting AIX servers can now be centrally monitored and managed on the Veeam Backup & Replication (VBR) server. In addition, you can now perform file-level recovery from Veeam Backup Enterprise Manager. This post will be a quick and dirty guide to the configuration of just that. So let's get started. Create a Protection Group On the VBR server create a Protection Group and select "Computers with pre-installed agents." This will download the Veeam AIX agent packages needed for installation to a location of your choosing. As you can see below there are packages for AIX as well Solaris. Then use whichever file transfer tool you prefer to move the files from VBR to the AIX server. I am using WinSCP below. Install Veeam Agent for AIX Untar the Veeam agent file using the below command. tar -xvzf VeeamAgentSolaris-3.0.0.561-i386.tar.gz Install the agent. pkgadd -G -d ./VeeamAgent-3.0.0.561-i386.pkg Connect the AIX server to VBR by executing the below XML file. This can take several minutes. veeamconfig mode setvbrsettings --cfg AIXProd.xml The server should now display in your protection group. Create a Backup Job On the Veeam Backup server we can now create a backup job. Rather than waiting the default 6 hours for the AIX server to sync with VBR after creating the backup job, you can run the below command. This can take several minutes to run. veeamconfig mode syncnow You can see the configuration applied on the AIX server as shown below. The backup job I created will run daily at 10pm and protect everything from the root directory. veeamconfig job list (to find the name of the job) veeamconfig job info --name Once the backup job runs you can view the details of the job both in VBR and on the AIX server. If you don't want to wait until 10pm you can run an ad-hoc backup. veeamconfig job start --name Like I said, it would be a quick and dirty setup guide. That's all I have for you today. Hope this helps!

Protect, Detect and Recover! The three key pillars a backup vendor is expected to offer when it comes to the NIST Cybersecurity Framework. The top priority on that list is protecting your data. A secure backup both on-premise and in a private or public cloud offers immense protection. This can protect from both a hacker accessing the backup server as well as the backup repository. Secondly, the ability to recovery quickly is crucial. If the business cannot recover quickly all the ransomware defense planning you did was for nothing. It is key to view ransomware in the same limelight as disaster recovery in order to meet these demands. Lastly, detecting malware and ransomware is critical for a backup vendor to provide, but it should be viewed as a last resort measure for companies who have other dedicated tools detecting ransomware. Malware and Ransomware Detection with Veeam With that said, scanning backup files to assess their health and recoverability is crucial and something Gartner recommends as part of their Isolated Recovery Environment for backup vendors. Verifying backups to ensure no known vulnerabilities get re-injected into the production environment during restores can be a massive timesaver. Veeam can automagically do this with SureBackup. As the name implies, this enables users to verify backups are usable by both scanning the backup contents for malware/ransomware and checking the integrity of the backup via a CRC test. As a side note, you can use any scanning tool that has a CLI. For example, Trend Micro, Bitdefender, Windows Defender, etc. Simply edit the XML file here. I am using ESET. Linking SureBackup jobs with a daily backup policy means you can come into work not only with finished backups, but also backups that have been inspected. As you can see below, there are many other possibilities with SureBackup though. This post is specifically leveraging the tool to scan backup contents, but you could also create a small DR test by ensuring VMs are connected to the network in a restore and inserting any custom scripts. Results of the SureBackup job are in the task log of the UI and/or the emailed report. Below is a glance of the UI. We can see the AV scan took a little over 12 minutes and the CRC test was less than a minute. Not only is this helpful for compliance purposes to prove backups are tested regularly, but also for peace of mind. Lastly, below is what an emailed report will look like if you instructed SureBackup to send a notification. You can send to a group or to multiple addresses. This is great, but how would a larger organization use this? A great way to set this up in the real-world if you're a larger business is to run a weekly SureBackup on a handful of your important VMs in each backup policy. For example, create a SureBackup for each backup job and stagger them throughout the week. This will keep the load balanced on the server you're mounting the file to. In addition, you can increase the performance if you have the compute resources to mount backup files to multiple servers. This will allow you to scan multiple backups in parallel. Conclusion Nearly all meetings lately are about a ransomware defense strategy. During these conversations though, nobody acknowledges the gigantic elephant in the room shouting, "If your backup vendor is the one detecting ransomware you're probably opening a Coinbase account." Don't get me wrong, a backup solution should absolutely provide a way to help detect ransomware, but companies need a holistic approach for this fight. There is an entire industry dedicated to detecting ransomware. Having a secure copy you can quickly recovery from should be a company's top priority in a backup solution. After all, it doesn't matter what you detect if you can't recover a safe copy quickly. #################################################################### I Hate UIs. I Want to Script. If you prefer scripting, you can run the below script and link it your backup policy under the post-script option. Connect to your Veeam backup server. Add-PSSnapin VeeamPSSnapin -ErrorAction SilentlyContinue Connect-VBRServer -Server "servername" Create variables for the VMs you want to scan. You will want to change the name of the VM and target server that will be used to inspecting and cleansing. $restorepoint = Get-VBRRestorePoint -Name "VMname" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 $targetServerName = "servername" $targetAdminCredentials = Get-VBRCredentials -name "credentials" | where {$_.description -eq "description"} $restorepoint = Get-VBRRestorePoint -Name "ATLNIMBLE_WIN" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Now that your variables are set, you're ready to mount the VM to the server. $session = Publish-VBRBackupContent -RestorePoint $restorepoint -TargetServerName $targetServerName -TargetServerCredentials $targetAdminCredentials Below is an example using ESET to scan the contents of the mounted disks and dump the script output to "ecls.txt." Fun fact, starting your PS command with "&" allows you to run CLI commands in PS. The mount will always be under C:\VeeamFLR\. & "C:\Program Files\ESET\ESET Security\ecls.exe" /base-dir="C:\Program Files\ESET\ESET Security\Modules" /subdir "C:\VeeamFLR\" /log-file=c:\ecls.txt /aind /unsafe /unwanted /suspicious /clean-mode=standard Unmount the server and mark script as complete. Unpublish-VBRBackupContent -Session $session If you want to run it all at once. $restorepoint = Get-VBRRestorePoint -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 $targetServerName = "servername" $targetAdminCredentials = Get-VBRCredentials -name "creds" | where {$_.description -eq "description"} $session = Publish-VBRBackupContent -RestorePoint $restorepoint -TargetServerName $targetServerName -TargetServerCredentials $targetAdminCredentials & "C:\Program Files\ESET\ESET Security\ecls.exe" /base-dir="C:\Program Files\ESET\ESET Security\Modules" /subdir "C:\VeeamFLR\" /log-file=c:\ecls.txt /aind /unsafe /unwanted /suspicious /clean-mode=standard Unpublish-VBRBackupContent -Session $session

ServiceNow Integration with Veeam comes up frequently in conversations to which SEs famously respond, "You can easily script integration." The goal of this post is to help users do exactly that. Easily script integration between Veeam ONE and ServiceNow. Luckily, there are already great resources from Veeam SEs that document how-to do this via Slack or Teams and also a great script on Github that I used for this blog post from Carlos Talbot. Veeam ONE has over 200+ built-in alarms for VMware, Hyper-v and Veeam Backup and Replication. Veeam ONE goes well beyond the basic alarming and reporting for backup and dives deep into the hypervisor to extract details about disk/CPU usage, snapshot size/age, VM power status, orphaned snapshots and much more. In addition, Veeam ONE can monitor specific processes or services on a per-VM level, which we will expand on more later. Prerequisites ServiceNow URL Username and password for ServiceNow Admin access to the Veeam ONE Client server Ideally a little familiarity with PowerShell but this post should be straightforward enough if you have zilch Setup To get started, download both createticket.ps1 and resolveticket.ps1 scripts to the same directory on Veeam ONE. Edit both files under "Configure the variables" section and ONLY change the $SNOWURL and $ScriptDir variables as shown below. Do the same for the resolveticket.ps1 file as well. If troubleshooting is needed setting $Debug to true can be leveraged. ############################################### ###Configure the variables below, you will be prompted for SNOW login ###during the first run which is then saved securely $SNOWURL = "https://dev71578.service-now.com/" $ScriptDir = "C:\createticket.ps1" $Debug = $false #enables writing to SNOWDebug.log file Run the below command in PowerShell which will prompt you for ServiceNow credentials. You will only need to enter the creds once, and they will be stored where your script directory is under the name SNOWCredentials.xml. c:\createticket.ps1 "VM power status" "EXCH2K16" "virtual machine is not Running" "12/13/2021 9:56:38 PM" "Error" "Reset/Resolved" "21117" This also acts as a test to verify if we can properly connect between Veeam ONE and ServiceNow. Login to ServiceNow and confirm there is an incident created with the details of the command we just ran. Should look similar to the screenshot below. Configuration Congratulations! You now have the power to create incidents in ServiceNow with over 200+ alarms in Veeam ONE. Go head and log in to Veeam ONE and click-on Alarm Management in the bottom left. Feel free to edit any alarm you like to test this functionality. For those following along, let's search for "VM with no Backup." This is a great alarm to validate our work and for practical use. Make sure you enable the alarm and set the rule to whatever RPO you like. The alarm can be set on the entire virtual infrastructure or just a specific cluster/host. In the Notification setting of the alarm paste the below two commands and set the action to Run script as shown below. powershell.exe C:\createticket.ps1 '%1' '%2' '%3' '%4' '%5' '%6' '%7' powershell.exe C:\resolveticket.ps1 '%1' '%2' '%3' '%4' '%5' '%6' '%7' Assuming there are VMs in your environment that have not been protected log in to ServiceNow and confirm a ticket was created. Should look similar to the below screenshot. As I mentioned earlier, there are hundreds of alarms you can enable. Feel free to set any of them by following the same workflow we just went through. To help give you some guidance though, another popular alarm is Snapshot Age. It never ceases to amaze me how some customers have snapshots in VMware that are months old taking up space on production datastores. To alleviate this you can create a ServiceNow ticket for any snap older than x amount of days as shown below. The Notification tab should be identical to the previous alarm. Some additional popular alarms I see customers use are Guest OS disk space, datastore free space, backup job status, and ransomware activity just to name a few. But one last capability that is worth illustrating is the ability to set an alarm on a specific service. For example, the below screenshot shows how you can create an alarm if the VSS SQLWriter is Stopped. Something that might be useful to know if you're protecting SQL through Veeam. You can enable an alarm for any service or process on a VM though. Once again the Notification tab should be the same as the previous alarms. I hope you find this post helpful as it's meant to be more of a guided how-to. Big shootouts to Jorge De La Cruz and Carlos Talbot who provided the majority of the heavy lifting. Let us know if you have any questions! #ServiceNow #Veeam #VeeamONE #Alarms #Monitoring

Ransomware attacks needs to be viewed under the same category as power outages and natural disasters. The requirement to recover quickly is a necessity. Recently, I'm seeing many vendors in the data protection industry advertise immutability and ransomware detection features. Both of which should absolutely be part of a company's ransomware strategy, but an immutable copy coming from spinning disk or tape can result in too much downtime for the business. It's an easy decision for a CEO or CFO if an attacker's ransom is $100,000, and the cost of downtime for a day is $500,000. The only question at that point becomes how do we make a Coinbase account to transfer Bitcoin? In addition, companies most likely already have ransomware detection and prevention tools. Realistically, if your backup software is what's detecting ransomware from a backup taken days or even weeks ago the cost of data loss might be too great to restore anyway. It's not that immutability and detection capabilities aren't great features to have. Veeam takes those features seriously, but the top priority to the business should be the ability to recover quickly in the event of a ransomware disaster. Without further ado, below are Veeam recovery capabilities that can provide fast RTOs to give companies a realistic chance at avoiding paying ransoms. Replica from Backup - Replicated VMs from backups which keeps load off production Recovery from Storage Snapshot - Quick file or VM restores off storage snapshots Recovery from fast performing repository - Backup to fast performing media Failover/failback capabilities - Traditional DR capabilities in the same UI and license Replica from Backup Replica from backup is one of the most underrated features Veeam offers. The beauty of replica from backup is it creates a VM in the DR site off the backup repository that is ready to be failed over to in the event of a disaster. Meaning, it creates a replica without putting any load on the production VM. RPOs will most likely be ~24 hours coming from a backup source, but it significantly increases your RTOs, since the VMs in DR really just need to be turned on for the most part! As a side note, you can also do this off the backup copy job! To dig into the how-to of this a little more just be sure to select "source" when choosing your virtual machines for the replication job. As you can see below, you have the option to replicate from backup instead of the production storage. Lastly, you can see how this achieves insanely fast RTOs as the VM is already built and ready to go. All you have to do is failover in the event of a disaster and the VM will power on. Recovery from Storage Array Snapshot Another undervalued feature Veeam offers is recovery from storage array (Pure, NetApp, EMC, HPE and many more) snapshots whether Veeam orchestrated them or not. Something that continually wows customers when I show them this feature is how Veeam can act as a catalogue for snapshots it didn't even take and provide a tree view into the Array > LUN > Snapshot > VMs as shown below. In the case of an instant VM Recovery you can see how it clones the snapshot and mounts it to the ESXi host, so it's ready for immediate use. Lastly, just in case there are any doubters that think these are mockups below is the restored VM in vSphere ready to be logged into. Recover from a Fast Performing Backup Repository One of the greatest advantages of Veeam is that it is software only. There is no vendor lock-in or mandatory hardware platform that needs to be used. Now, I wouldn't expect a company to backup all of their workloads to a flash based repository, but it is not uncommon to protect Level 0 or Level 1 workloads (5-10%) of the environment to a repository that can achieve fast RTOs via an Instant VM Recovery. In my experience, I see Veeam users that apply this strategy and backup to EMC Unity, Pure Flash Array C, HPE Nimble and many other similar offerings in the marketplace. As a side note, you can also instantly restore physical machines to VMware providing a great alternative to bare metal recovery. Failover and Failback with Snapshot Based Replication or CDP The more commonly known capability Veeam is famous for is traditional snapshot based replication or Continuous Data Protection (CDP) for SLAs that demand second type RPOs. There is already so much great content out there on this topic, so there is no need for me to recreate the wheel. The goal here is to highlight that for those workloads that don't just require fast RTOs but also low RPOs, CDP and snapshot based replication are offered in the same UI and license. There is no additional cost or management overhead. In summary, the elephant in the room when strategizing with companies on an effective ransomware recovery plan is that it's not cheap. It needs to be viewed in the same light as a datacenter power outage or natural disaster. Restoring from tape or spinning disk might not be worth the downtime compared to just paying the ransom in the eyes of the people making the big bucks up top. If an effective ransomware strategy means a little more compute to replicate to or some more fast performing storage to backup to that might not be so bad compared to paying a ransom or dealing with branding/reputation issues post-attack.

Curiosity got the best of me when I saw AWS Backup announced support for VMware. Considering AWS is well on its way to takeover the world in IT, groceries, movies, shopping etc, I needed to ensure for my livelihood that backup wouldn't be next. AWS does a fairly good job documenting how-to protect VMware, so I will just add a few tid-bits given my experience. AWS Gateway Deploying the AWS Backup Gateway server is a fairly straight forward process via an OVF template in VMware. The major hiccup for me was the networking piece. You need to know the IP of the Gateway in order to register it in the AWS console. After some research, I found that the default username and password is admin/password. The IP will be located at the top after you login as shown below. Backup Backing up the virtual machines are relatively straight forward too, but I hope for your sake you only have a small handful. You can only specify by VM attribute, so you have to individually select the VMs you want to protect. There is no option to protect by cluster, datastore or VMware tag. In addition, I find all the options between a Backup Plan, Job and Backup Policy a tad confusing, but like I said this is a biased review. It is important to note, there are no application consistency options for Linux and high-level backup details are limited. The main item to highlight though is to specify "only include specific resources," or you will receive a hefty bill when AWS charges you all the storage and network costs associated with protecting all the VMs in vCenter. Restore Restoring virtual machines is my main gripe. First of all, you cannot restore a VMware backup to AWS as an EC2. This is something easily done in Veeam for several years. Secondly, you can only restore full VMs. There is no option to restore a disk, file or application item. Thirdly, you have to manually enter the destination for the ESXi host, datastore and file path! I have never seen this in my 7 years in the backup industry. Any other vendor populates these fields, so you don't have to login into vSphere on a treasure hunt expedition. Lastly, the error messaging and logging are barebones. It's nearly impossible to troubleshoot based on the information provided. It took me over 10 attempts for a restore to work. Error message when troubleshooting restore. Pricing In attempt to not come off as totally biased (if it's not too late), AWS Backup for VMware is very affordable. It's so affordable that it is free. There is no per-instance or Front End Per TB license consumption model. AWS Backup charges for the backup storage, retrieval of data, network costs associated with backup and restore, and cross-region data transfer for copy jobs. AWS does charge $.05 per GB of consumed storage though, whereas Veeam can backup to more cost-effective storage. In addition, it is worth noting that Veeam can also protect EC2/EBS to S3 which is $.022 per GB, whereas AWS charges $.05 per GB to store that data on snapshots. High-Level Comparison In summary, AWS has created a v1.0 product in a space that is surrounded by products that are well into v10 and v20s. The one advantage AWS provides is minimal upfront cost, but as time goes on and data grows the high storage consumption costs will create a total cost of ownership that is the same or potentially even higher than other players in the industry. AWS Backup for VMware is meant for users with minimal backup and recovery requirements.

One of the more frequent questions I'm asked from Veeam users is, "Should we do forever forward incremental or forward incremental with periodic full?" This question quickly leads down a path to, "Should we do an active full or synthetic full?" Both of these are great questions but require context and background information to answer. The goal of this post is to arm you with the proper tools to know what's best for your environment. First, it's important to understand the three backup methods within Veeam; forward incremental, forever forward incremental, and reverse incremental. Forever forward incremental is 3 I/Os per backup on the repository because on every backup a merge process takes place to combine the full backup file (.VBK) with the oldest incremental file (.VIB). The same is true for reverse incremental. The only difference is that the VBK in reverse incremental is always your latest recovery point, allowing for faster restores from the most recent point in time. Forward incremental on the other hand is 1 write I/O which allows for much faster backups. With the power of forward incremental comes great responsibility though. Weekly synthetic or active fulls must be ran. The benefit of having a weekly full to reference increases restore performance, whereas forever forward incremental can have poor restore performance if it's rehydrating data from a full taken weeks ago. Below is a table that is a good quick reference to understand the impact each method has on the backup repository. The key takeaway here is that without fast performing destination storage and with a retention north of 14 days, reverse/forever forward incremental are not sustainable backup methods. The reason being is every backup requires 3 I/Os and restore performance can be impacted if the full being rehydrated is from weeks ago. Here is a nice cheat sheet though if you're like me and words are too hard to process. Hopefully at this point, I've given you the proper ammunition to decide what backup method makes sense for your business. If you came to the conclusion that forward incremental is the backup method of choice the next questions is, "Should we do active fulls or synthetic fulls." The key benefit synthetic fulls has over active full is that it takes all the burden off your important compute resources. Synthetic full simply takes all the incrementals in your backup chain and merges them into a full backup file. When you add in fast cloning technology with Windows ReFS or Linux XFS this becomes a great option because rather than creating one giant new VBK, fast cloning technology creates a VBK that is a pointer file to the previous incrementals. This can be a huge space saver on your backup repository. With the power of synthetic fulls and fast cloning technology comes great responsibility though. As time goes on fast cloning finds less space savings because if the last active full was taken months or even years ago the newly created synthetic full is referencing blocks that are no longer applicable. That's why the answer to the question is, "Both if you're leveraging ReFS or XFS." Luckily, Veeam makes it very easy to achieve this in the job settings. You can specify weekly synthetics and bi-monthly active fulls. Lastly, I'll leave you with a nice illustration on what each backup chain looks like depending on the backup method with a 7 day retention. In this scenario periodic fulls might not be needed with only a 7 day retention. The furthest VBK Veeam will need to reference in forever forward incremental is 7 days ago, whereas forward incremental with periodic fulls is going to take up twice as much space on your repository, since two full 7 day backups chains are required before Veeam can delete backup files. In the case of a 30 day retention though, forever forward might create poor restore performance with the latest full always being 30 days ago. This is when forward incremental with periodic fulls is recommended with a combination of both synthetic and active fulls with Windows ReFS or Linux XFS. #Veeam #syntheticfull #backup #dataprotection

Veeam Backup and Replication is much more than just backup. I can prove it because it is in the name of the product. By providing both backup AND replication in a single UI and license users can easily meet SLAs required for different workloads. When we refer to replication we are referring to hypervisor based replication. Replicating data from one virtual host to another virtual host, whether it is in the same data center for a quick migration or part of a DR strategy that requires minimal RTO and RPO. Veeam also has Backup Copy jobs which is not to be confused with replication. Veeam Replication is meant for workloads that have stringent SLAs that require minimal data loss and quick uptime, whereas Backup Copy jobs is meant for everything else. It takes an exact copy of the backup files on your primary repository and creates a secondary copy to a DR site. This is a great option for workloads that don't require stringent SLAs. It takes all the load off the hypervisor compute resources and puts them on the repositories. In addition, the destination of the Backup Copy job doesn't have to match the source storage repository. This gives users flexibility to leverage existing hardware in their data center and keep total cost of ownership at a minimum. In a DR situation, Veeam can see all the restore points in the secondary site, and run instant VM, VMDK, file, and application recoveries the same way it would if they were in the primary site. In the case of Data Domains, ExaGrids, StoreOnce, and other deduplication devices, it is common to leverage the hardware replication those products come with instead of Veeam Backup Copy Jobs. Veeam can still leverage these replicas as recovery points by simply rescanning the destination repository. You can right click on the repository and run a rescan and the recovery points will show in the Disk (imported) section giving you all the same recovery options as if Veeam orchestrated it as a Backup Copy Job. Now, if you are a masochist then manually running that rescan everyday might be fun for you, but there is a simple PowerShell script you can run/schedule, so those recovery points are always waiting for you in a DR scenario. Get-VBRBackupRepository -Name "Local Repository Test" | Sync-VBRBackupRepository If your target repository is a Veeam Scale-Out Backup Repository then you can run the below PS command. Sync-VBRSOBREntityState -Repository There's more though! Much much more! Not only can you do these automagical rescans on backup repositories, but you can also do them on storage arrays. If you're leveraging storage array based replication for an integrated Veeam storage system (NetApp, EMC, and HPE) or one that falls within the Veeam Universal Storage Plug-in then you can rescan the destination storage array and use those snapshots as Veeam recovery points. Much like Veeam replication this is a great use-case for VMs with stringent SLAs. You can perform the same recoveries off the secondary snapshot as you can on the primary storage array. That includes instant VM, file, and application recovery. And don't worry! If you're not a masochist there is a PS command to run/schedule these rescans for the storage arrays too! $volume = Get-StoragePluginVolume -Name "VOLUME-01" Sync-StoragePluginVolume -Volume $volume If you have a Veeam integrated storage systems like Nimble, HPE 3Par/Primera EMC Unity/VNX, and NetApp than below are the PS commands. Nimble: $volume = Get-NimbleVolume -Name "VOL01" Sync-NimbleVolume -Volume $volume HPE: $storage = Get-HP3Storage -Name "HPE 3PAR StoreServe Storage" $volume = Get-HP3Volume -Storage $storage Sync-HP3Volume -Volume $volume EMC Unity/VNX: $storage = Get-VNXHost -Name "VNX Storage" $volume = Get-VNXVolume -Name "VOLUME1" -Host $storage ync-VNXVolume -Volume $volume NetApp: $volume = Get-NetAppVolume -Name "VOL01" Sync-NetAppVolume -Volume $volume I've said it before, and I'll say it again. Veeam is much more than just backup. Veeam provides users the capabilities to meet every level of SLA within a single product, license and UI. Not only giving users ease of management, but also enabling users to keep total cost of ownership to a minimum. Below is a helpful diagram illustrating how Veeam can be the data fabric that helps users meet different SLA requirements. #Veeam #DataProtection #Backup #Linchtips #DR

The power struggle between SQL DBAs and the Data Protection team is a tale as old as...well me probably. Let's not get too dramatic. The SQL team wants to protect their own databases to ensure simple and quick recovery, while the backup team wants to ensure those databases are truly being protected. In order for backup admins to take ownership of SQL backups though, two fundamental requirements must be met: DBAs must be able to easily recover their databases DBAs must be able to quickly recover their databases Veeam v10 offers a capability that has been hovering under the radar that we are going to shine a bright light on. This capability provides SQL DBAs the ability to quickly recover databases off of storage snapshots, while the Veeam admin is the one protecting both the image and the databases. Before we go into detail on the magic of v10, it is important to understand what Veeam already offers for SQL protection and recovery. Restoring SQL databases from storage array snapshots has been in the product for several years. It is as simple as right clicking on the VM and selecting SQL application recovery, but the restores are triggered by the Veeam admin. This functionality serves a purpose for many customers with HPE, Nimble, NetApp, Pure, EMC Unity/VNX, Infinidat and several other supported storage arrays, but for larger customers who require SQL DBAs to control the recoveries this isn't always a perfect match. Despite not being able to control recoveries from snapshots, it is important to note that SQL DBAs certainly can control recoveries from backups on the disk repository. This gives DBAs control of the recoveries, while the Veeam admin handles SQL protection (including transaction logs at an interval of your choosing). Once the backups are setup, DBAs login to the Veeam web portal, select the server they want to restore from, choose the database and point the restore to the original or new location. The backup team defines the scope and type of restores available to SQL DBAs by creating a role. The scope can be by vSphere tag, host, cluster or datastore for virtual machines. For physical machines the Veeam admin can define the scope by Protection Group. In the example below, when DBAs login to the web portal they'll be able to restore SQL databases from VMs that are tagged, "MS SQL." Although this is a great option for many customers, it doesn't meet the restore performance some SQL DBAs are accustomed to. If you ask a DBA how long they can afford to be down they will say, "None," and look at you like you're from Mars for asking the question. The web portal certainly provides an easy way for DBAs to restore their databases, but it is restricted by the performance of the backup repositories which is typically spinning disk. DBAs are used to doing their dumps to all-flash arrays which offer rapid recoveries when you pull a .BAK file from there. All of this leads to the exciting new capability in Veeam v10! SQL DBAs can now leverage storage array snapshots as rapid recovery points, when they kickoff database restores. This is made possible because not only can the Veeam admin backup the SQL image once a day, but also they can now create additional recovery points on the storage array along with the transaction logs at an interval of your choosing. This functionality works with HPE, Nimble, NetApp, Pure, EMC Unity/VNX, Infinidat and several other storage arrays. Although recovering databases from storage snapshots through the web portal did not make v10, we can provide the next best thing! A PowerShell script! The below script will publish a database to the SQL Server from the most recent storage array snapshot. This will give you a side-by-side comparison of your production database and the most recent point-in-time like in the screenshot below. # Install remote Veeam console on machine this script will be run # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Publish Database $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "database-name" Publish-VESQLDatabase -Database $database -ServerName "SQL-Server-name" -DatabaseName "new-name-of-published-db" # Login to SQL Studio to see the published database # Unpublish Database When Done $session = Get-VESQLRestoreSession $database = Get-VESQLPublishedDatabase -Session $session[0] -Name "name-of-published-database" Unpublish-VESQLDatabase -Database $database # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session In the Veeam console the admin can see an application item restore for SQL taking place. Also, we can confirm this is coming from a NetApp snapshot clone by the log messages, and that the database published successfully! In addition, the below script will restore the database from the most recent storage snapshot rather than publish it. If the database already exists you will be given the option to overwrite it. # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Restore Database $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "database-name" Restore-VESQLDatabase -Database $database -ServerName "SQL-Server-name" # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session The restore was successful and we can confirm it came from a NetApp snapshot clone. Lastly, exporting a .BAK file is a great option for SQL DBAs to have whether it is from the snapshot or backup file. Below is a script to do just that. # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Export .BAK File $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "db-name" Export-VESQLDatabase -Database $database -Path "C:\export\Export.bak" -Server "SQL Server" -ToBackupFile # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session The struggle between SQL DBAs and the data protection team may continue for years, but hopefully this post provides guidance on how to have the best of both worlds. This enables the data protection team to ensure databases are protected, while enabling DBAs to perform quick and simple recoveries. #SQL #DBA #backup #dataprotection #linchtips #powershell

A simple google search will yield tons of statistics on the cost of ransomware, and how many companies have fallen victim to its attacks. The numbers vary so greatly that I'm not going to cite any particular source, but it is clear that ransomware is real, not going anywhere and can cost your business a significant amount of time and money. What is Ransomware? First, let's start with a high-level overview on what ransomware is and how it works. Ransomware most commonly sneaks in via email attachments or links, but it can also come from clicking or downloading a link from a malicious website. Once the infection has been downloaded to its new home it begins to spread. It starts by installing a program on the local machine it gained access to and ensures the program starts on reboot. From there ransomware will look to spread across the network to gain access to as many endpoints as possible. Once it has achieved that it will begin to search for files with important extensions such as .doc, .xlsx, ppt, etc and encrypt them, so they are unusable. More sophisticated attackers will wait until you're infected for 30 days or more before demanding payment because at that point you may have fallen out of your retention policy. Your backups would now be useless, since you've been protecting encrypted files. Spoiler alert: later we will cover how Veeam can detect this behavior! How to defend against Ransomware? As part of your ransomware strategy your first line of defense should be yourself! Similar to a cold or flu, ransomware is an infection. If you never wash your hands, constantly eat finger-foods, and always touch your face, then your chances of infection are significantly greater! The same is true for ransomware. If you don't take proper precautions by using strong passwords, applying software patches, and limiting/restricting access to endpoints on the network, then you're putting yourself at greater risk. Your second line of defense are tools and vendors that offer solutions specifically to fight ransomware! Think of these as your yearly flu shots. They aren't guaranteed to keep illness away, but they are usually effective. They will scan the network for suspicious activity, analyze emails for phishing content, prevent suspicious downloads and much more. Your final line of defense should be a backup solution that can save the day if needed and has ransomware built-in to the heart of the product. Just like the flu there is no cure you can take, but with proper fluids and medicine you'll be back to normal in no time! With that, let's cover the 5 ways Veeam can help. 5 ways Veeam can help: 1. Alarm notification if backup file size growth is abnormal! Despite the spoiler above this is a great defensive strategy to have in a backup solution. The best way not to pay attackers their ransom is to simply restore from a backup, but if it's a smart attacker who waited days or even weeks to demand payment then you might not have a valid backup. Veeam provides an alarm that can notify you if backup file size growth is abnormal, moreover, Veeam can take automatic remediation via a provided script to turn off the VM or remove it from the network. The reason this alarm can be helpful is because if an attacker is encrypting all your files than the backup sizes should be significantly larger since there are more changed files and less files that can be compressed and deduplicated. This is a great indicator to check if these VMs have encrypted files on them. In the alarm, you can define rules for what abnormal file size growth looks like. For example, below we've specified Veeam to look for any backup files that exceed 150% growth, and we've instructed Veeam to compare the file to the previous 3 backup files in the "analysis depth" field. You can setup Veeam to simply send you an alarm notification when it gets to the warning (150% growth) or error (200% growth) stage, or you can have Veeam take automatic remediation actions. In the actions tab, you can insert a script that will run if the above rule is meant. A common script might be to shut off the VM or remove it from the network. As you can see in the "execution type" field you can choose to receive an email first to approve the execution of the script. 2. Scan for possible malware activity. A second capability Veeam provides as a layer of protection against ransomware is an alarm that looks for high cpu usage or a high write rate on the underlying datastore. Similar to the previous alarm you can set rules to look for high cpu usage, high write rates, latency on the datastore, network transmission rate, and many other variables. High CPU usage is a great indicator that files on the VM are being encrypted as that is a CPU intensive task. Now, it might be common in your environment to have VMs with high CPU usage. In that case, leaving it as an alarm notification with no automatic remediation might be best. In addition, you can suppress the alarm during snapshot deletion or creation activities. 3. Immutable backups offsite. Offsite backups in and of itself is a great strategy against ransomware. This gets a copy offsite and to a separate network. In some situations this isn't enough though. Veeam offers a capability with AWS to make secondary copies of data immutable. This functionality will extend to other public cloud and on-premises vendors in the future, but there was no reason to let this capability with AWS sit idle. When adding an AWS S3 bucket to the Veeam console, you can choose to select the number of days you want backups to be immutable. For this period of time nobody can modify or delete the backup. Not a Veeam admin user. Not a root AWS user. It locks the objects in what AWS calls compliance mode. Not even a support call can lead to deletion of this data. As a side note, I really like that there is an option to limit storage consumption if you know the business shouldn't exceed a certain amount of TBs or PBs. As expected, if we try and delete objects in the AWS bucket as a root user it gives us an access denied because the object is locked for 7 days. And if we try to delete the backup files in Veeam as an admin account it also fails because the objects are locked. 4. Automatically test backups in an isolated environment for ransomware or malware. Veeam SureBackup and SureReplica have been in the product for years. This is a powerful capability that enables users to put their backups to work. Whether it is to automate DR tests, spin up RA environments for test/dev, security, DBAs, analytics, etc. The use-cases are countless. One of those use-cases though is to test your backups for malware or ransomware. First, you create what I refer to as the plumbing. Veeam calls this a Virtual Lab though to be more official. This is where you define the host, datastore, isolated network, and masquerade IP you want these VMs to spin-up in. Once the plumbing is setup you can point VMs to the virtual lab and have automatic tests run once they are turned on. For example, you can check if a DNS port is open, a SQL database is online, guest tools is installed, or you can insert your own scripts to execute. In addition, you can have Veeam scan the files for any suspicious malware or ransomware. Not only can you do this in an isolated lab, but also when you do any type of restore back to production. Before restoring the data Veeam will mount the backup and scan it for viruses. If anything is found it will abort the recovery. As you can see, this quickly becomes an option to take annual DR testing to the next level. I've seen customers who run this on a daily basis after their backups. I understand that might not work for customers with thousands of VMs, but why not make what used to be a yearly DR operational restore or backup validity test and turn it into a quarterly test? 5. API to mount any virtual disk to any machine for ransomware or malware testing. Veeam offers a Data Integration API that enables users to easily re-use and access their backup data for a number of use-cases. Scanning for viruses being one of them. Here is how it works! $backup = Get-VBRBackup -Name "Name_of_Backup_Job" $targetServerName = “192.18.138.125” $targetAdminCredentials = Get-VBRCredentials -name "DEMO\Administrator" $restorepoint = Get-VBRRestorePoint -Backup $backup | Sort-Object –Property CreationTime | Select -Last 1 $session = Publish-VBRBackupContent -RestorePoint $restorepoint -TargetServerName $targetServerName -TargetServerCredentials $targetAdminCredentials Ransomware is prevalent and any company that doesn't have or is not forming a ransomware strategy to prevent, detect, and recover from a ransomware attack is at a significantly higher risk of infection. As a first line of defense, do what you can to harden your infrastructure. That includes Veeam as well. If I can only recommend one thing it is to enable 2FA on all Veeam login server sessions. See this post from Rhys Hammond for a how-to. Second, find tools or vendors that can help with prevention and detection. Lastly, choose a backup software that has ransomware protection built-in to the product. God Speed! #dataprotection #Veeam #ransomware #backup #LinchTips

Much like protection for SQL Servers, that we covered in the last post, there is typically a line in the sand on who is responsible for protecting Oracle databases within larger organizations. It is either the Oracle database administrators (DBAs) or the Data Protection team. Unlike SQL though, it is much less of a power struggle to determine who is responsible. This is because Oracle makes it simple for DBAs to take ownership of database protection through the RMAN (Oracle Recovery Manager) utility. What is Oracle RMAN? As the name implies RMAN is a tool built into Oracle that enable DBAs to easily manage database backups and recoveries. It has been part of Oracle for over 20 years, and it comes at no additional cost. Through the command-line of RMAN DBAs can easily trigger full database and archivelog backups. Protecting archivelogs allows databases to be recovered from any particular point in time, and it allows the databases to be protected in hot backup mode, meaning the database does not need to be shutoff during backup. If you do not protect the archivelogs (noarchivelogs) then the database has to be shutoff during backup. In addition, from the RMAN command-line it is simple to recover whole databases or just a tablespace partition. RMAN is the recommended backup method for Oracle databases, so understandably Oracle DBAs would be a little confused if you came knocking on their door asking to start protecting their databases. Why am I reading a blog from a backup vendor then? Despite Oracle offering their own backup and recovery tool for DBAs, it is still crucial that any enterprise level backup solution plug-in to RMAN. It is the data protection team's responsibility to ensure all workloads are meeting required RPOs, and that data is recoverable in an outage, disaster, ransomware attack, etc. Veeam offers an Oracle RMAN plug-in that provides the best of both worlds. It enables DBAs to create backups and restores as they've always done through RMAN, while giving the data protection team insight into those backups. All while never speaking a word to each other! How great is that? Veeam Plug-in for Oracle RMAN in Depth Oracle DBAs simply need to install the Veeam RMAN plug-in on their servers and go through a basic setup wizard, where they'll enter the Veeam server, credentials and select Veeam repositories to be the destination of backups. As you can see below, the Veeam plug-in will then update the backend RMAN configuration settings, enabling DBAs to use the same RMAN backup and restore commands they're accustomed to. OracleRMANConfigTool --wizard Once the Veeam Plug-in is installed, the data protection team will see all RMAN backups in the Veeam console. As you can see below, when an Oracle DBA triggers a RMAN backup command the Veeam plug-in is utilized, and the Veeam admin will see the backups taking place on the backup server. RMAN> backup database plus archivelog; RMAN backup statistics seen in the Veeam Console. Oracle DBAs have many tools available to them thanks to RMAN. For example, prior to restoring a database, they can check to see if the current database is valid by checking for any missing files. In addition, before performing a restore DBAs can validate the backup itself is not missing any database files. This will look at the backup file on the Veeam repository and verify it is recoverable and has the proper files in place. RMAN> restore database validate; Now, you could run a restore with peace of mind, knowing that the point-in-time isn't corrupted. As you can see below, when an Oracle DBA triggers a restore from the RMAN command-line, RMAN knows to pull data files from the backup on the Veeam repository. RMAN> restore database; RMAN> recover database; The Veeam plug-in for RMAN offers more than just visibility into backups the Oracle database team is taking though. Veeam admins also have the capability to easily restore databases from the Veeam console. In a situation where the DBA doesn't show up for work or is locked out of their machines in a ransomware attack, the Veeam admin can trigger a restore from the backup server. They simply select the point-in-time and choose to restore the database to the original or alternate location. It is important to note, that Oracle DBAs have the ability to restore more granularly than the entire database with RMAN. They can restore a specific tablespace or tablespace partition. The same way you can validate a database before restoring it, you can do the same for a tablespace. As you can see below, restoring the tablespace from RMAN pulled the data from a Veeam repository. RMAN> validate tablespace users; RMAN> restore tablespace users; It is much more common for Oracle DBAs to own backup and recovery of their databases because RMAN is such a proven and effective utility. That does not mean the data protection team shouldn't be part of ensuring these workloads are protected though. It's important an enterprise backup solution provides levels of protection across all the critical workloads in an organization. An ideal data protection solution should allow Oracle DBAs to backup and recover their databases as if a third party tool isn't involved, while also providing visibility into the protection of Oracle databases to the backup team. #RMAN #Oracle #Veeam #linchtips #dataprotection #backups