Search Results
Search this site
45 results found with an empty search
- Veeam ONE's 10 Best Alarms for Ransomware Protection
If you use Veeam and aren't taking advantage of Veeam ONE's breadth of capabilities, you are missing out. Veeam ONE is part of Veeam Availability Suite (VAS), which is a combination of Veeam Backup & Replication (VBR) and Veeam ONE (VONE). I discover in most my end-user conversations that either owners of Veeam ONE vastly underutilize it, or they didn't spring for VAS and are in need of many VONE capabilities. Today, I want to focus on VONE's 10 best alarms to enable as part of a ransomware defense strategy. This will just be a fraction of what VONE can do as there are over 350 reports and 150 alarms. The possibilities are endless as VONE ingests data from vSphere, vCD, VBR, AWS, Azure and GCP and has API integration for those who want to showoff. 1. Possible Ransomware Activity Possible Ransomware Activity analyzes on a per-VM basis anomaly like behavior. For example, spikes in CPU or datastore usage rates, both of which can be signs of encryption. The key to this alarm is the historical view though. Understandably, the first thing a business wants to do in a ransomware attack is recover data, but from what day do you recover from? How can you be sure you haven't been backing up encrypted data for days or even weeks? The historical view enables users to pinpoint the day ransomware potentially took foothold in the environment. The above shows spike in CPU and the below shows spikes in datastore usage rates. 2. Attempted Backup Deletion There are two scenarios an attacker will attempt to delete backup data. The first is the hacker will gain access to the backup console and start deleting backups. The second is the hacker will try to access the backup target itself and remove files from there. We will cover both , but let's focus on the first. This alarm will notify users if a bad actor has gained access to the console and is trying to delete backups and which account they've compromised. Deleting backups is a custom alarm that can be enabled by creating the below rule as shown. 3. Suspicious Backup File Size VONE analyzes previous backup sizes and alerts the user if backup sizes look out of the ordinary. Larger backup files can mean there is a lot of natural changed data, but it can also mean encrypted files are being backed up. 4. Unusual Job Duration Along the same lines a job that runs an unusual amount of time can be a sign of protecting encrypted files as there would be much more changed blocks to backup. 5. Job Modification Audit Sticking with scenario one where a bad actor gains access to the backup server, they might start editing or deleting jobs. This report will inform you who's account is making changes and when they made them. Knowing exactly when the ransomware attack started can be powerful knowledge in a recovery scenario so cycles aren't wasted recovering invalid data. 6. Immutability Retention Modified Immutable backups are for the second scenario where the hacker attempts to gain access to the backup target itself. By leveraging Veeam's immutable repository this becomes nearly impossible as Veeam doesn't store the credentials to that server. Furthermore, SSH can be disabled on the box, so now the hacker would need both physical access and credentials from somewhere outside Veeam. 7. State of Immutability The best bet by a hacker at that point is to gain access to the backup server and reduce the amount of immutable copies going forward or turnoff immutability altogether. In both scenarios users would be alerted. 8. Disabled Jobs It might sound basic but setting alerts on disabled jobs can be a great way to notify the business something is afoot. It is not uncommon in a ransomware attack for a hacker who gained access to the backup server to disable backup jobs. The rule for the alarm can be set down to 1 minute, so you are notified nearly instantly. 9. Failed vCenter Logon Attempt Who isn't guilty of a fat finger? I sure am. Certainly this will create some false positives, but you can set the alarm to only notify if x amount of failed logon attempts have taken place. Albeit a hail marry, this could still potentially alert you of a bad actor in the network. 10. vSphere Modifications As was first mentioned, VONE is a powerful tool capable of many things. Ingesting vSphere data is one of those. Users can receive a report on modifications made within vCenter. This can be great for real-time monitoring, but also for trying to pinpoint which account was compromised and when. With Veeam ONE the possibilities are endless. This post was a peak behind the curtain into what VONE can do. There are hundreds of additional reports and alarms and many additional use-cases users have accomplished through API and/or PowerShell integration. For those who want to keep digging into the power Veeam ONE offers below are my 5 favorite blog posts from the Veeam community. The Power of Veeam ONE Pro Tips with VONE VONE integration with Slack VONE integration with ServiceNOW My favorite VONE alarms
- Integrating Veeam NAS Backup with Dell EMC Isilon
If you still think Veeam only protects VMs you are stuck in 2015. Fortunately for you, I would be lucky to get you to the next paragraph if I dove into everything Veeam can protect, so I will narrow it down to just NAS. More specifically, Veeam's integration with Dell EMC Isilon. Veeam leverages Isilon's ChangeList API for their SnapshotIQ technology during the backup process to grab only the deltas (changed files and folders) between the current and previous snapshot. Veeam only keeps one snapshot per share on the Isilon. Once the next snapshot is taken the previous is deleted. Minimizing the amount of snaps needed on Isilon can be a massive space saver if the business is retaining 30-plus days worth of snapshots. The integration between Veeam and Isilon allow for both fast backups and potentially reduced consumed space on the Isilon itself. Increased performance is great and all but saving the business money is even better. In addition to faster backup speeds and less snapshots retained, Veeam provides the capability to instantly recover a share to the same or alternate server whether it is an Isilon or not. Rather than spend money on multiple Isilons for replicas, you can backup your Isilon to something more cost effective and instantly recover file shares in a disaster scenario. Now, while the main purpose of this post is to inform you of the business value Veeam brings to the table by integrating with Isilon, I also enjoy getting my hands dirty. The rest of this post is a runbook on how to setup/configure Veeam and Isilon together. NOTE: There are a couple references to Veeam's upcoming v12 release and the enhancements around NAS. Brownie points if you can call out what they are in the comment section. Isilon Setup In order for Veeam to leverage Isilon's Snapshot IQ technology, Veeam needs a role with the proper permissions as described here. Instead of making you read that document and manually enter in each privilege though, you can simply take note of the screenshot and Isilon CLI commands below if you prefer. isi auth user create veeamadmin --zone System --enabled True --set-password isi auth roles create --name VeeamStorageIntegration --zone System isi auth roles modify --zone System VeeamStorageIntegration --description "veeam backup role" isi auth roles modify --zone System VeeamStorageIntegration --add-priv ISI_PRIV_LOGIN_PAPI --add-priv ISI_PRIV_AUTH --add-priv ISI_PRIV_DEVICES --add-priv ISI_PRIV_NETWORK --add-priv ISI_PRIV_NFS --add-priv ISI_PRIV_SMB --add-priv ISI_PRIV_SNAPSHOT isi auth roles modify --zone System VeeamStorageIntegration --add-user veeamadmin isi auth roles view Practice --zone System This creates a custom "VeeamStorageIntegration" role with the user "veeamadmin" associated to it. This role/user needs to be created in the System zone. The BackupAdmin role that comes out of the box with Isilon does not have sufficient permissions. Lastly, apply the service account you use to any share that needs protecting. We will use this same service account when we add Isilon in the Veeam console. Veeam Setup From the Veeam server add the Isilon storage system as described here using the "veeamadmin" user we created above. You can use a root user but the user/role we created is more granular if you prefer. The documentation does a great job guiding you through this step-by-step, so no need for me to plagiarize. Next, add the Isilon under NAS Filer in the Inventory section using your backup service account that was added to the SMB shares. Check the "Use native changed file tracking" box to ensure you leverage the Isilon SnapshotIQ integration. Now, it's time to create a backup as described well here. During the backup Veeam creates a snapshot on Isilon as shown below. In my lab, the original backup was 14 folders and 312 files as I am just protecting a small share here for instructional purposes. The next incremental run compared the previous and current snapshot and only found 2 new folders and 7 new files making the backup very performant. BONUS: See the immutability line item? To prove I'm not full of it, you can also see the snap in Isilon under Data Protection > SnapshotIQ. With Veeam v12 coming out later in 2022, NAS backups can target an immutable repository. My lab is using an Object-Lock enabled AWS S3 bucket as the target. There is no way to delete those backups from the Veeam console until the natural expiration date is applied as shown below. In addition, if you login to AWS as a root user and call support they will tell you, "Too bad. We want your storage money." The Money Saver Like I mentioned in the beginning, faster backups are great but saving money is better. Many companies store weeks if not months worth of snapshots as well as replicate shares to another Isilon in a DR site. This can be costly if you rarely use your secondary Isilon, or it's main purpose is only for DR. Leveraging Veeam can both reduce the consumed space from snaps as well as replace the replica Isilon with a more cost effective solution. Instant File Share Recoveries enable companies to store backups in something cheaper and deeper, while also providing quick recovery to the same or alternate server. For example, you could instantly restore to a windows share and have immediate user read/write access. Starting in v12, instant recoveries of SMB shares in a read/write state will be available as well as NFS shares in a read-only state. In addition, companies will have the ability to migrate the share back to the original source or to a new server. Whether it's the fact that site A is never coming back or simply the business is shifting away from Isilon, you will have the ability to recover NAS data quickly and affordably. In summary, Veeam's integration with Isilon enables companies to protect their data fast, reduce the amount of snaps taking up consumed space and quickly recover data in a cost effective way.
- Levels of Disaster Recovery with Veeam
Guest Author: Randy Weis Date: June 14, 2022 There are different levels of Disaster Recovery in an IT organization depending on the acceptable data loss (RPO or Recovery Point Objective) and acceptable time to recovery (RTO or Recovery Time Objective). In addition, there are other variables involved when planning DR. Such as: Disaster Recovery Types Business Continuity – covers more than IT, such as facilities and communications Disaster Recovery – recovery of business applications Disaster Resiliency – avoiding disasters with resilient infrastructure such as clustering at platform, datacenter or network level (SDWAN) High Availability – application-level redundancy and failover Scope of the Disaster Partial Site Disaster – Some portion of the IT infrastructure is taken offline by human error/malice, burst pipe, hardware failure Total Site Disaster – An entire site (datacenter, server room, server closet) goes down due to natural disaster, fire, flood, local power outage/generator failure, and will be down for more than a few hours (days, weeks) Disaster Recovery Location Veeam works at the Disaster Recovery Level and complements the other levels of workplace disaster recovery. Veeam provides orchestrated Disaster Recovery from one location to another. In addition, Veeam offers Direct Restore to AWS. GCP or Azure for VMs and physical machines. Regardless of workload, Veeam meets the level of DR necessary in one license and UI. Level 0: Recovery of Backup Server Scope: Partial or Total Site Disaster The Veeam Backup & Replication Server can be recovered quickly from a Configuration Backup. A new VM can be created (from template, preferably) at the same or alternate site depending on the scope of the disaster. Once the configuration database is recovered, or the local repository of backup copies is rescanned, restores can begin (see Level 1). This is an internal recovery option for IT, prior to site or application recovery. One of the main differentiators of Veeam though compared to the market is you can still easily restore your data if there is no configuration backup. All backup files are portable and have the metadata needed for recovery. Simply install a new Veeam server or turn on the idle one in your DR site and rescan the primary or DR repository whether it's on-prem or in the cloud. Level 1: Instant Recovery Scope: Partial Site Disaster RPO = hours RTO = minutes This is an “on-site” disaster recovery. Veeam offers Instant Recovery to VMware/Hyper-V/AHV for backups of: Virtual Machines/disks in VMware/Hyper-V/Nutanix AHV/RHV VMs in Storage Snapshots Physical Windows and Linux machines (Veeam does the P2V conversion) NAS File Shares: R/W for SMB, Read-only for NFS Databases for Oracle or SQL Backups of VMs in Azure/AWS EC2/GCP to VMware Level 2: Recovery from Backups at another location Scope: Partial or Total Site Disaster RPO = hours RTO = hours/minutes Veeam copies backup files from one repository to another in a Backup Copy Job. A Veeam Backup & Replication Server (VBR) at that site can restore the backups from the target repository for Backup Copy Jobs. If log shipping for SQL or Oracle is enabled, those logs will also be available at the recovery location. Backup Copy Jobs can copy & Restore any backup that is stored in a Veeam repository: VMs from VMware, Hyper-V, Nutanix AHV, RHV Agent backups for Windows, Linux, Solaris, AIX RMAN, SAP HANA, SQL VMS from AWS/Azure/GCP stored in an External Repository (Object Storage container) Level 3: Replication & Orchestrated Recovery of VM from VMware Snapshots (and Hyper-V) Scope: Total Site Disaster RPO = 30 minutes to 4 hours for replicas, 24 hours for recovery from Backup RTO = minutes to hours, depending on scope and hardware resources Veeam replicates Virtual Machine files from one datastore to another using VMware Snapshots with built-in orchestration that can re-map to DR virtual networks, re-IP addressing, boot order and boot delay. Replication can use Backup Copy Jobs as a source, so that Recovery from Backups RTO can be reduced to minutes. Veeam can build failover plans that can be triggered on demand or planned; offers commit failover or failback (testing), failback and commit failback (after original site is available). Supported workloads are: VMware Hyper-V Level 4: Continuous Data Replication & Orchestrated Recovery of VMware VMs Scope: Total Site Disaster RPO = 2 seconds to 5 minutes RTO = minutes Veeam replicates VMware Virtual Machine files from one datastore to another using VMWare API for IO Filtering (no snapshots), with built-in orchestration that can re-map to DR virtual networks, re-IP addressing, boot order and boot delay. Veeam can build failover plans that can be triggered on demand or planned; offers commit failover or failback (testing), failback and commit failback (after original site is available). Level 5: Disaster Recovery Orchestrator Scope: Total Site Disaster RPO: 2 seconds to 24 hours RTO: minutes Veeam provides a higher level of orchestration and automation with Veeam’s Orchestrator software. This software can recover in an automated manner from backups, from snapshot replicas and from CDP replicas. DRO offers a more complete and customized automation of application recovery with Runbooks that are updated as the plan is updated Scheduled testing, bubble or actual Audit Logs Readiness Reports Test Result Reports Delegation of recovery by role and by site Veeam’s Disaster Recovery Orchestrator can be implemented as an enterprise strategy and solution. Scenarios for orchestrated recovery are: Replication Jobs (snap or CDP) Backup Jobs Storage Replication (NetApp and HPE 3PAR) Disaster Recovery to the Cloud Scope: Partial or Total Site Disaster RPO = 24 hours RTO = hours Veeam copies backup files from a local repository to a Capacity Tier repository (S3 Bucket) in AWS/Azure/GCP. A Veeam Backup & Replication Server (VBR) in the cloud can restore the backups from the S3 repository directly into a running VM into one of the major public cloud providers.
- Validate Oracle Backups with Veeam
This one will be short and sweet... Validating database backups are crucial for ensuring restores are actually useable in the event of a disaster or attack. With Veeam's Oracle RMAN plug-in DBAs can continue to leverage the same validate commands they are accustom to, while also providing visibility and governance to the data protection team. validate backupset There are a number of validate commands a DBAs can run. Anything from validating a database to a control file to a specific table. As you can see above, DBAs can run the same RMAN commands they are familiar with, but if the Veeam plug-in is installed the verification will come from the backup file on the Veeam repository. This enables data protection teams to still have governance over all the backups the business is responsible for, while also enabling DBAs to keep responsibility over database protection and recovery activities. Best of both worlds. As shown above, Veeam administrators have visibility into any validate, restore or backup commands Oracle DBAs run.
- A Quick Setup Guide to Veeam Agent for AIX
An under the radar enhancement in Veeam v11a is that protecting AIX servers can now be centrally monitored and managed on the Veeam Backup & Replication (VBR) server. In addition, you can now perform file-level recovery from Veeam Backup Enterprise Manager. This post will be a quick and dirty guide to the configuration of just that. So let's get started. Create a Protection Group On the VBR server create a Protection Group and select "Computers with pre-installed agents." This will download the Veeam AIX agent packages needed for installation to a location of your choosing. As you can see below there are packages for AIX as well Solaris. Then use whichever file transfer tool you prefer to move the files from VBR to the AIX server. I am using WinSCP below. Install Veeam Agent for AIX Untar the Veeam agent file using the below command. tar -xvzf VeeamAgentSolaris-3.0.0.561-i386.tar.gz Install the agent. pkgadd -G -d ./VeeamAgent-3.0.0.561-i386.pkg Connect the AIX server to VBR by executing the below XML file. This can take several minutes. veeamconfig mode setvbrsettings --cfg AIXProd.xml The server should now display in your protection group. Create a Backup Job On the Veeam Backup server we can now create a backup job. Rather than waiting the default 6 hours for the AIX server to sync with VBR after creating the backup job, you can run the below command. This can take several minutes to run. veeamconfig mode syncnow You can see the configuration applied on the AIX server as shown below. The backup job I created will run daily at 10pm and protect everything from the root directory. veeamconfig job list (to find the name of the job) veeamconfig job info --name Once the backup job runs you can view the details of the job both in VBR and on the AIX server. If you don't want to wait until 10pm you can run an ad-hoc backup. veeamconfig job start --name Like I said, it would be a quick and dirty setup guide. That's all I have for you today. Hope this helps!
- Automagically Scan Backups for Ransomware with Veeam
Protect, Detect and Recover! The three key pillars a backup vendor is expected to offer when it comes to the NIST Cybersecurity Framework. The top priority on that list is protecting your data. A secure backup both on-premise and in a private or public cloud offers immense protection. This can protect from both a hacker accessing the backup server as well as the backup repository. Secondly, the ability to recovery quickly is crucial. If the business cannot recover quickly all the ransomware defense planning you did was for nothing. It is key to view ransomware in the same limelight as disaster recovery in order to meet these demands. Lastly, detecting malware and ransomware is critical for a backup vendor to provide, but it should be viewed as a last resort measure for companies who have other dedicated tools detecting ransomware. Malware and Ransomware Detection with Veeam With that said, scanning backup files to assess their health and recoverability is crucial and something Gartner recommends as part of their Isolated Recovery Environment for backup vendors. Verifying backups to ensure no known vulnerabilities get re-injected into the production environment during restores can be a massive timesaver. Veeam can automagically do this with SureBackup. As the name implies, this enables users to verify backups are usable by both scanning the backup contents for malware/ransomware and checking the integrity of the backup via a CRC test. As a side note, you can use any scanning tool that has a CLI. For example, Trend Micro, Bitdefender, Windows Defender, etc. Simply edit the XML file here. I am using ESET. Linking SureBackup jobs with a daily backup policy means you can come into work not only with finished backups, but also backups that have been inspected. As you can see below, there are many other possibilities with SureBackup though. This post is specifically leveraging the tool to scan backup contents, but you could also create a small DR test by ensuring VMs are connected to the network in a restore and inserting any custom scripts. Results of the SureBackup job are in the task log of the UI and/or the emailed report. Below is a glance of the UI. We can see the AV scan took a little over 12 minutes and the CRC test was less than a minute. Not only is this helpful for compliance purposes to prove backups are tested regularly, but also for peace of mind. Lastly, below is what an emailed report will look like if you instructed SureBackup to send a notification. You can send to a group or to multiple addresses. This is great, but how would a larger organization use this? A great way to set this up in the real-world if you're a larger business is to run a weekly SureBackup on a handful of your important VMs in each backup policy. For example, create a SureBackup for each backup job and stagger them throughout the week. This will keep the load balanced on the server you're mounting the file to. In addition, you can increase the performance if you have the compute resources to mount backup files to multiple servers. This will allow you to scan multiple backups in parallel. Conclusion Nearly all meetings lately are about a ransomware defense strategy. During these conversations though, nobody acknowledges the gigantic elephant in the room shouting, "If your backup vendor is the one detecting ransomware you're probably opening a Coinbase account." Don't get me wrong, a backup solution should absolutely provide a way to help detect ransomware, but companies need a holistic approach for this fight. There is an entire industry dedicated to detecting ransomware. Having a secure copy you can quickly recovery from should be a company's top priority in a backup solution. After all, it doesn't matter what you detect if you can't recover a safe copy quickly. #################################################################### I Hate UIs. I Want to Script. If you prefer scripting, you can run the below script and link it your backup policy under the post-script option. Connect to your Veeam backup server. Add-PSSnapin VeeamPSSnapin -ErrorAction SilentlyContinue Connect-VBRServer -Server "servername" Create variables for the VMs you want to scan. You will want to change the name of the VM and target server that will be used to inspecting and cleansing. $restorepoint = Get-VBRRestorePoint -Name "VMname" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 $targetServerName = "servername" $targetAdminCredentials = Get-VBRCredentials -name "credentials" | where {$_.description -eq "description"} $restorepoint = Get-VBRRestorePoint -Name "ATLNIMBLE_WIN" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Now that your variables are set, you're ready to mount the VM to the server. $session = Publish-VBRBackupContent -RestorePoint $restorepoint -TargetServerName $targetServerName -TargetServerCredentials $targetAdminCredentials Below is an example using ESET to scan the contents of the mounted disks and dump the script output to "ecls.txt." Fun fact, starting your PS command with "&" allows you to run CLI commands in PS. The mount will always be under C:\VeeamFLR\. & "C:\Program Files\ESET\ESET Security\ecls.exe" /base-dir="C:\Program Files\ESET\ESET Security\Modules" /subdir "C:\VeeamFLR\" /log-file=c:\ecls.txt /aind /unsafe /unwanted /suspicious /clean-mode=standard Unmount the server and mark script as complete. Unpublish-VBRBackupContent -Session $session If you want to run it all at once. $restorepoint = Get-VBRRestorePoint -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 $targetServerName = "servername" $targetAdminCredentials = Get-VBRCredentials -name "creds" | where {$_.description -eq "description"} $session = Publish-VBRBackupContent -RestorePoint $restorepoint -TargetServerName $targetServerName -TargetServerCredentials $targetAdminCredentials & "C:\Program Files\ESET\ESET Security\ecls.exe" /base-dir="C:\Program Files\ESET\ESET Security\Modules" /subdir "C:\VeeamFLR\" /log-file=c:\ecls.txt /aind /unsafe /unwanted /suspicious /clean-mode=standard Unpublish-VBRBackupContent -Session $session
- Veeam Integration with ServiceNow
ServiceNow Integration with Veeam comes up frequently in conversations to which SEs famously respond, "You can easily script integration." The goal of this post is to help users do exactly that. Easily script integration between Veeam ONE and ServiceNow. Luckily, there are already great resources from Veeam SEs that document how-to do this via Slack or Teams and also a great script on Github that I used for this blog post from Carlos Talbot. Veeam ONE has over 200+ built-in alarms for VMware, Hyper-v and Veeam Backup and Replication. Veeam ONE goes well beyond the basic alarming and reporting for backup and dives deep into the hypervisor to extract details about disk/CPU usage, snapshot size/age, VM power status, orphaned snapshots and much more. In addition, Veeam ONE can monitor specific processes or services on a per-VM level, which we will expand on more later. Prerequisites ServiceNow URL Username and password for ServiceNow Admin access to the Veeam ONE Client server Ideally a little familiarity with PowerShell but this post should be straightforward enough if you have zilch Setup To get started, download both createticket.ps1 and resolveticket.ps1 scripts to the same directory on Veeam ONE. Edit both files under "Configure the variables" section and ONLY change the $SNOWURL and $ScriptDir variables as shown below. Do the same for the resolveticket.ps1 file as well. If troubleshooting is needed setting $Debug to true can be leveraged. ############################################### ###Configure the variables below, you will be prompted for SNOW login ###during the first run which is then saved securely $SNOWURL = "https://dev71578.service-now.com/" $ScriptDir = "C:\createticket.ps1" $Debug = $false #enables writing to SNOWDebug.log file Run the below command in PowerShell which will prompt you for ServiceNow credentials. You will only need to enter the creds once, and they will be stored where your script directory is under the name SNOWCredentials.xml. c:\createticket.ps1 "VM power status" "EXCH2K16" "virtual machine is not Running" "12/13/2021 9:56:38 PM" "Error" "Reset/Resolved" "21117" This also acts as a test to verify if we can properly connect between Veeam ONE and ServiceNow. Login to ServiceNow and confirm there is an incident created with the details of the command we just ran. Should look similar to the screenshot below. Configuration Congratulations! You now have the power to create incidents in ServiceNow with over 200+ alarms in Veeam ONE. Go head and log in to Veeam ONE and click-on Alarm Management in the bottom left. Feel free to edit any alarm you like to test this functionality. For those following along, let's search for "VM with no Backup." This is a great alarm to validate our work and for practical use. Make sure you enable the alarm and set the rule to whatever RPO you like. The alarm can be set on the entire virtual infrastructure or just a specific cluster/host. In the Notification setting of the alarm paste the below two commands and set the action to Run script as shown below. powershell.exe C:\createticket.ps1 '%1' '%2' '%3' '%4' '%5' '%6' '%7' powershell.exe C:\resolveticket.ps1 '%1' '%2' '%3' '%4' '%5' '%6' '%7' Assuming there are VMs in your environment that have not been protected log in to ServiceNow and confirm a ticket was created. Should look similar to the below screenshot. As I mentioned earlier, there are hundreds of alarms you can enable. Feel free to set any of them by following the same workflow we just went through. To help give you some guidance though, another popular alarm is Snapshot Age. It never ceases to amaze me how some customers have snapshots in VMware that are months old taking up space on production datastores. To alleviate this you can create a ServiceNow ticket for any snap older than x amount of days as shown below. The Notification tab should be identical to the previous alarm. Some additional popular alarms I see customers use are Guest OS disk space, datastore free space, backup job status, and ransomware activity just to name a few. But one last capability that is worth illustrating is the ability to set an alarm on a specific service. For example, the below screenshot shows how you can create an alarm if the VSS SQLWriter is Stopped. Something that might be useful to know if you're protecting SQL through Veeam. You can enable an alarm for any service or process on a VM though. Once again the Notification tab should be the same as the previous alarms. I hope you find this post helpful as it's meant to be more of a guided how-to. Big shootouts to Jorge De La Cruz and Carlos Talbot who provided the majority of the heavy lifting. Let us know if you have any questions! #ServiceNow #Veeam #VeeamONE #Alarms #Monitoring
- Recover Quickly in a Ransomware Attack
Ransomware attacks needs to be viewed under the same category as power outages and natural disasters. The requirement to recover quickly is a necessity. Recently, I'm seeing many vendors in the data protection industry advertise immutability and ransomware detection features. Both of which should absolutely be part of a company's ransomware strategy, but an immutable copy coming from spinning disk or tape can result in too much downtime for the business. It's an easy decision for a CEO or CFO if an attacker's ransom is $100,000, and the cost of downtime for a day is $500,000. The only question at that point becomes how do we make a Coinbase account to transfer Bitcoin? In addition, companies most likely already have ransomware detection and prevention tools. Realistically, if your backup software is what's detecting ransomware from a backup taken days or even weeks ago the cost of data loss might be too great to restore anyway. It's not that immutability and detection capabilities aren't great features to have. Veeam takes those features seriously, but the top priority to the business should be the ability to recover quickly in the event of a ransomware disaster. Without further ado, below are Veeam recovery capabilities that can provide fast RTOs to give companies a realistic chance at avoiding paying ransoms. Replica from Backup - Replicated VMs from backups which keeps load off production Recovery from Storage Snapshot - Quick file or VM restores off storage snapshots Recovery from fast performing repository - Backup to fast performing media Failover/failback capabilities - Traditional DR capabilities in the same UI and license Replica from Backup Replica from backup is one of the most underrated features Veeam offers. The beauty of replica from backup is it creates a VM in the DR site off the backup repository that is ready to be failed over to in the event of a disaster. Meaning, it creates a replica without putting any load on the production VM. RPOs will most likely be ~24 hours coming from a backup source, but it significantly increases your RTOs, since the VMs in DR really just need to be turned on for the most part! As a side note, you can also do this off the backup copy job! To dig into the how-to of this a little more just be sure to select "source" when choosing your virtual machines for the replication job. As you can see below, you have the option to replicate from backup instead of the production storage. Lastly, you can see how this achieves insanely fast RTOs as the VM is already built and ready to go. All you have to do is failover in the event of a disaster and the VM will power on. Recovery from Storage Array Snapshot Another undervalued feature Veeam offers is recovery from storage array (Pure, NetApp, EMC, HPE and many more) snapshots whether Veeam orchestrated them or not. Something that continually wows customers when I show them this feature is how Veeam can act as a catalogue for snapshots it didn't even take and provide a tree view into the Array > LUN > Snapshot > VMs as shown below. In the case of an instant VM Recovery you can see how it clones the snapshot and mounts it to the ESXi host, so it's ready for immediate use. Lastly, just in case there are any doubters that think these are mockups below is the restored VM in vSphere ready to be logged into. Recover from a Fast Performing Backup Repository One of the greatest advantages of Veeam is that it is software only. There is no vendor lock-in or mandatory hardware platform that needs to be used. Now, I wouldn't expect a company to backup all of their workloads to a flash based repository, but it is not uncommon to protect Level 0 or Level 1 workloads (5-10%) of the environment to a repository that can achieve fast RTOs via an Instant VM Recovery. In my experience, I see Veeam users that apply this strategy and backup to EMC Unity, Pure Flash Array C, HPE Nimble and many other similar offerings in the marketplace. As a side note, you can also instantly restore physical machines to VMware providing a great alternative to bare metal recovery. Failover and Failback with Snapshot Based Replication or CDP The more commonly known capability Veeam is famous for is traditional snapshot based replication or Continuous Data Protection (CDP) for SLAs that demand second type RPOs. There is already so much great content out there on this topic, so there is no need for me to recreate the wheel. The goal here is to highlight that for those workloads that don't just require fast RTOs but also low RPOs, CDP and snapshot based replication are offered in the same UI and license. There is no additional cost or management overhead. In summary, the elephant in the room when strategizing with companies on an effective ransomware recovery plan is that it's not cheap. It needs to be viewed in the same light as a datacenter power outage or natural disaster. Restoring from tape or spinning disk might not be worth the downtime compared to just paying the ransom in the eyes of the people making the big bucks up top. If an effective ransomware strategy means a little more compute to replicate to or some more fast performing storage to backup to that might not be so bad compared to paying a ransom or dealing with branding/reputation issues post-attack.
- A Biased Opinion on AWS Backup for VMware
Curiosity got the best of me when I saw AWS Backup announced support for VMware. Considering AWS is well on its way to takeover the world in IT, groceries, movies, shopping etc, I needed to ensure for my livelihood that backup wouldn't be next. AWS does a fairly good job documenting how-to protect VMware, so I will just add a few tid-bits given my experience. AWS Gateway Deploying the AWS Backup Gateway server is a fairly straight forward process via an OVF template in VMware. The major hiccup for me was the networking piece. You need to know the IP of the Gateway in order to register it in the AWS console. After some research, I found that the default username and password is admin/password. The IP will be located at the top after you login as shown below. Backup Backing up the virtual machines are relatively straight forward too, but I hope for your sake you only have a small handful. You can only specify by VM attribute, so you have to individually select the VMs you want to protect. There is no option to protect by cluster, datastore or VMware tag. In addition, I find all the options between a Backup Plan, Job and Backup Policy a tad confusing, but like I said this is a biased review. It is important to note, there are no application consistency options for Linux and high-level backup details are limited. The main item to highlight though is to specify "only include specific resources," or you will receive a hefty bill when AWS charges you all the storage and network costs associated with protecting all the VMs in vCenter. Restore Restoring virtual machines is my main gripe. First of all, you cannot restore a VMware backup to AWS as an EC2. This is something easily done in Veeam for several years. Secondly, you can only restore full VMs. There is no option to restore a disk, file or application item. Thirdly, you have to manually enter the destination for the ESXi host, datastore and file path! I have never seen this in my 7 years in the backup industry. Any other vendor populates these fields, so you don't have to login into vSphere on a treasure hunt expedition. Lastly, the error messaging and logging are barebones. It's nearly impossible to troubleshoot based on the information provided. It took me over 10 attempts for a restore to work. Error message when troubleshooting restore. Pricing In attempt to not come off as totally biased (if it's not too late), AWS Backup for VMware is very affordable. It's so affordable that it is free. There is no per-instance or Front End Per TB license consumption model. AWS Backup charges for the backup storage, retrieval of data, network costs associated with backup and restore, and cross-region data transfer for copy jobs. AWS does charge $.05 per GB of consumed storage though, whereas Veeam can backup to more cost-effective storage. In addition, it is worth noting that Veeam can also protect EC2/EBS to S3 which is $.022 per GB, whereas AWS charges $.05 per GB to store that data on snapshots. High-Level Comparison In summary, AWS has created a v1.0 product in a space that is surrounded by products that are well into v10 and v20s. The one advantage AWS provides is minimal upfront cost, but as time goes on and data grows the high storage consumption costs will create a total cost of ownership that is the same or potentially even higher than other players in the industry. AWS Backup for VMware is meant for users with minimal backup and recovery requirements.
- Veeam Backup Methods Explained
One of the more frequent questions I'm asked from Veeam users is, "Should we do forever forward incremental or forward incremental with periodic full?" This question quickly leads down a path to, "Should we do an active full or synthetic full?" Both of these are great questions but require context and background information to answer. The goal of this post is to arm you with the proper tools to know what's best for your environment. First, it's important to understand the three backup methods within Veeam; forward incremental, forever forward incremental, and reverse incremental. Forever forward incremental is 3 I/Os per backup on the repository because on every backup a merge process takes place to combine the full backup file (.VBK) with the oldest incremental file (.VIB). The same is true for reverse incremental. The only difference is that the VBK in reverse incremental is always your latest recovery point, allowing for faster restores from the most recent point in time. Forward incremental on the other hand is 1 write I/O which allows for much faster backups. With the power of forward incremental comes great responsibility though. Weekly synthetic or active fulls must be ran. The benefit of having a weekly full to reference increases restore performance, whereas forever forward incremental can have poor restore performance if it's rehydrating data from a full taken weeks ago. Below is a table that is a good quick reference to understand the impact each method has on the backup repository. The key takeaway here is that without fast performing destination storage and with a retention north of 14 days, reverse/forever forward incremental are not sustainable backup methods. The reason being is every backup requires 3 I/Os and restore performance can be impacted if the full being rehydrated is from weeks ago. Here is a nice cheat sheet though if you're like me and words are too hard to process. Hopefully at this point, I've given you the proper ammunition to decide what backup method makes sense for your business. If you came to the conclusion that forward incremental is the backup method of choice the next questions is, "Should we do active fulls or synthetic fulls." The key benefit synthetic fulls has over active full is that it takes all the burden off your important compute resources. Synthetic full simply takes all the incrementals in your backup chain and merges them into a full backup file. When you add in fast cloning technology with Windows ReFS or Linux XFS this becomes a great option because rather than creating one giant new VBK, fast cloning technology creates a VBK that is a pointer file to the previous incrementals. This can be a huge space saver on your backup repository. With the power of synthetic fulls and fast cloning technology comes great responsibility though. As time goes on fast cloning finds less space savings because if the last active full was taken months or even years ago the newly created synthetic full is referencing blocks that are no longer applicable. That's why the answer to the question is, "Both if you're leveraging ReFS or XFS." Luckily, Veeam makes it very easy to achieve this in the job settings. You can specify weekly synthetics and bi-monthly active fulls. Lastly, I'll leave you with a nice illustration on what each backup chain looks like depending on the backup method with a 7 day retention. In this scenario periodic fulls might not be needed with only a 7 day retention. The furthest VBK Veeam will need to reference in forever forward incremental is 7 days ago, whereas forward incremental with periodic fulls is going to take up twice as much space on your repository, since two full 7 day backups chains are required before Veeam can delete backup files. In the case of a 30 day retention though, forever forward might create poor restore performance with the latest full always being 30 days ago. This is when forward incremental with periodic fulls is recommended with a combination of both synthetic and active fulls with Windows ReFS or Linux XFS. #Veeam #syntheticfull #backup #dataprotection
- Automagically Rescan Storage System and Repository in DR Site for easy Recoverability
Veeam Backup and Replication is much more than just backup. I can prove it because it is in the name of the product. By providing both backup AND replication in a single UI and license users can easily meet SLAs required for different workloads. When we refer to replication we are referring to hypervisor based replication. Replicating data from one virtual host to another virtual host, whether it is in the same data center for a quick migration or part of a DR strategy that requires minimal RTO and RPO. Veeam also has Backup Copy jobs which is not to be confused with replication. Veeam Replication is meant for workloads that have stringent SLAs that require minimal data loss and quick uptime, whereas Backup Copy jobs is meant for everything else. It takes an exact copy of the backup files on your primary repository and creates a secondary copy to a DR site. This is a great option for workloads that don't require stringent SLAs. It takes all the load off the hypervisor compute resources and puts them on the repositories. In addition, the destination of the Backup Copy job doesn't have to match the source storage repository. This gives users flexibility to leverage existing hardware in their data center and keep total cost of ownership at a minimum. In a DR situation, Veeam can see all the restore points in the secondary site, and run instant VM, VMDK, file, and application recoveries the same way it would if they were in the primary site. In the case of Data Domains, ExaGrids, StoreOnce, and other deduplication devices, it is common to leverage the hardware replication those products come with instead of Veeam Backup Copy Jobs. Veeam can still leverage these replicas as recovery points by simply rescanning the destination repository. You can right click on the repository and run a rescan and the recovery points will show in the Disk (imported) section giving you all the same recovery options as if Veeam orchestrated it as a Backup Copy Job. Now, if you are a masochist then manually running that rescan everyday might be fun for you, but there is a simple PowerShell script you can run/schedule, so those recovery points are always waiting for you in a DR scenario. Get-VBRBackupRepository -Name "Local Repository Test" | Sync-VBRBackupRepository If your target repository is a Veeam Scale-Out Backup Repository then you can run the below PS command. Sync-VBRSOBREntityState -Repository There's more though! Much much more! Not only can you do these automagical rescans on backup repositories, but you can also do them on storage arrays. If you're leveraging storage array based replication for an integrated Veeam storage system (NetApp, EMC, and HPE) or one that falls within the Veeam Universal Storage Plug-in then you can rescan the destination storage array and use those snapshots as Veeam recovery points. Much like Veeam replication this is a great use-case for VMs with stringent SLAs. You can perform the same recoveries off the secondary snapshot as you can on the primary storage array. That includes instant VM, file, and application recovery. And don't worry! If you're not a masochist there is a PS command to run/schedule these rescans for the storage arrays too! $volume = Get-StoragePluginVolume -Name "VOLUME-01" Sync-StoragePluginVolume -Volume $volume If you have a Veeam integrated storage systems like Nimble, HPE 3Par/Primera EMC Unity/VNX, and NetApp than below are the PS commands. Nimble: $volume = Get-NimbleVolume -Name "VOL01" Sync-NimbleVolume -Volume $volume HPE: $storage = Get-HP3Storage -Name "HPE 3PAR StoreServe Storage" $volume = Get-HP3Volume -Storage $storage Sync-HP3Volume -Volume $volume EMC Unity/VNX: $storage = Get-VNXHost -Name "VNX Storage" $volume = Get-VNXVolume -Name "VOLUME1" -Host $storage ync-VNXVolume -Volume $volume NetApp: $volume = Get-NetAppVolume -Name "VOL01" Sync-NetAppVolume -Volume $volume I've said it before, and I'll say it again. Veeam is much more than just backup. Veeam provides users the capabilities to meet every level of SLA within a single product, license and UI. Not only giving users ease of management, but also enabling users to keep total cost of ownership to a minimum. Below is a helpful diagram illustrating how Veeam can be the data fabric that helps users meet different SLA requirements. #Veeam #DataProtection #Backup #Linchtips #DR
- SQL Protection and Rapid Recovery from a DBA Lens
The power struggle between SQL DBAs and the Data Protection team is a tale as old as...well me probably. Let's not get too dramatic. The SQL team wants to protect their own databases to ensure simple and quick recovery, while the backup team wants to ensure those databases are truly being protected. In order for backup admins to take ownership of SQL backups though, two fundamental requirements must be met: DBAs must be able to easily recover their databases DBAs must be able to quickly recover their databases Veeam v10 offers a capability that has been hovering under the radar that we are going to shine a bright light on. This capability provides SQL DBAs the ability to quickly recover databases off of storage snapshots, while the Veeam admin is the one protecting both the image and the databases. Before we go into detail on the magic of v10, it is important to understand what Veeam already offers for SQL protection and recovery. Restoring SQL databases from storage array snapshots has been in the product for several years. It is as simple as right clicking on the VM and selecting SQL application recovery, but the restores are triggered by the Veeam admin. This functionality serves a purpose for many customers with HPE, Nimble, NetApp, Pure, EMC Unity/VNX, Infinidat and several other supported storage arrays, but for larger customers who require SQL DBAs to control the recoveries this isn't always a perfect match. Despite not being able to control recoveries from snapshots, it is important to note that SQL DBAs certainly can control recoveries from backups on the disk repository. This gives DBAs control of the recoveries, while the Veeam admin handles SQL protection (including transaction logs at an interval of your choosing). Once the backups are setup, DBAs login to the Veeam web portal, select the server they want to restore from, choose the database and point the restore to the original or new location. The backup team defines the scope and type of restores available to SQL DBAs by creating a role. The scope can be by vSphere tag, host, cluster or datastore for virtual machines. For physical machines the Veeam admin can define the scope by Protection Group. In the example below, when DBAs login to the web portal they'll be able to restore SQL databases from VMs that are tagged, "MS SQL." Although this is a great option for many customers, it doesn't meet the restore performance some SQL DBAs are accustomed to. If you ask a DBA how long they can afford to be down they will say, "None," and look at you like you're from Mars for asking the question. The web portal certainly provides an easy way for DBAs to restore their databases, but it is restricted by the performance of the backup repositories which is typically spinning disk. DBAs are used to doing their dumps to all-flash arrays which offer rapid recoveries when you pull a .BAK file from there. All of this leads to the exciting new capability in Veeam v10! SQL DBAs can now leverage storage array snapshots as rapid recovery points, when they kickoff database restores. This is made possible because not only can the Veeam admin backup the SQL image once a day, but also they can now create additional recovery points on the storage array along with the transaction logs at an interval of your choosing. This functionality works with HPE, Nimble, NetApp, Pure, EMC Unity/VNX, Infinidat and several other storage arrays. Although recovering databases from storage snapshots through the web portal did not make v10, we can provide the next best thing! A PowerShell script! The below script will publish a database to the SQL Server from the most recent storage array snapshot. This will give you a side-by-side comparison of your production database and the most recent point-in-time like in the screenshot below. # Install remote Veeam console on machine this script will be run # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Publish Database $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "database-name" Publish-VESQLDatabase -Database $database -ServerName "SQL-Server-name" -DatabaseName "new-name-of-published-db" # Login to SQL Studio to see the published database # Unpublish Database When Done $session = Get-VESQLRestoreSession $database = Get-VESQLPublishedDatabase -Session $session[0] -Name "name-of-published-database" Unpublish-VESQLDatabase -Database $database # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session In the Veeam console the admin can see an application item restore for SQL taking place. Also, we can confirm this is coming from a NetApp snapshot clone by the log messages, and that the database published successfully! In addition, the below script will restore the database from the most recent storage snapshot rather than publish it. If the database already exists you will be given the option to overwrite it. # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Restore Database $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "database-name" Restore-VESQLDatabase -Database $database -ServerName "SQL-Server-name" # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session The restore was successful and we can confirm it came from a NetApp snapshot clone. Lastly, exporting a .BAK file is a great option for SQL DBAs to have whether it is from the snapshot or backup file. Below is a script to do just that. # Add Veeam Snap-in Add-PSSnapin VeeamPSSnapin # Connect to Veeam Server Connect-VBRServer -User "username" -Password "password" -Server "VBR-server-name" # start the SQL Explorer $restorepoint = Get-VBRApplicationRestorePoint -SQL -Name "VM-name" | Sort-Object -Property CreationTime -Descending | Select-Object -First 1 Start-VESQLRestoreSession -RestorePoint $restorepoint # Export .BAK File $session = Get-VESQLRestoreSession $database = Get-VESQLDatabase -Session $session[0] -Name "db-name" Export-VESQLDatabase -Database $database -Path "C:\export\Export.bak" -Server "SQL Server" -ToBackupFile # Shutdown SQL Explorer $session = Get-VESQLRestoreSession Stop-VESQLRestoreSession -Session $session The struggle between SQL DBAs and the data protection team may continue for years, but hopefully this post provides guidance on how to have the best of both worlds. This enables the data protection team to ensure databases are protected, while enabling DBAs to perform quick and simple recoveries. #SQL #DBA #backup #dataprotection #linchtips #powershell












