Thursday, September 26, 2013

Computer Browser service on Windows 2008

The Computer Browser service on Windows 2008 either won't start, or starts and then stops. This can be caused by Windows Firewall. Enable all the File and Printer sharing inbound rules and then try starting the service.

Tuesday, September 10, 2013

Editing Cisco ASA configuration by line

The Cisco ASA parses its configuration line by line, so the order the configuration is in can be critical. When we need to add a line to the config it's always in the middle somewhere so we have to insert it into the access-list by line number. To do so we go to an enabled sessions and use "sh access-list <ACL name>" to show the access list with the lines numbers. So if the access list in question is called acl-dmz then the command would be:

sh access-list acl-dmz

Looking at the lines you figure out where you want the new line inserted, then you go into configuration, and preface the line with "access-list <ACL name> line <number> <command>". So if I want to allow traffic on a new external IP address in the acl-out access list at line 10 the command would be:

access-list acl-out line 10 extended permit  tcp any host 192.168.0.1 eq www 

Tuesday, September 3, 2013

Prevent Veeam from locking tape drive

With Veeam v.7 tape drive support has been introduced. However, it can be a problem if you're running Backup Exec on the same server as Veeam. We run Backup Exec to back up the few physical servers we still have, and also to move Veeam backups to tape for off site storage.

Veeam regularly rescans for new devices, and can temporarily lock the tape during the scan, which Backup Exec sees as the tape being offline.

You can set Veeam to only scan at start up by adding DWORD ChangerElementFillCompletionTimeoutSec to HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication and setting it at the maximum value.

Friday, June 14, 2013

Upgrading firmware on Sonicwall email security appliance

This is the procedure we follow to upgrade the firmware on our Sonicwall 3300 email security appliance:

- Download new firmware and check MD5 checksum.
- Backup appliance settings.
- SSH to appliance using Putty and log in as "snwlcli".
- When prompted enter the appliance's admin's username and password.
- Run "stop appservices". It'll take a minute.
- Run "start tomcat".
- Wait for GUI to restart and then log in again.
- Update the firmware through "Advanced" under "System". Updating the firmware can take 10 minutes or more depending how how many users and spam you have on the appliance.

Not following this can lead to issues. Unfortunately, Sonicwall doesn't make it easy to find the correct procedure on their site.

Thursday, June 13, 2013

Using dssec.dat to change properties in ADUC

So, we wanted to delegate control of specific OUs in Active Directory to users, but only allow them to change phone numbers and titles. I was able to restrict nearly everything by allowing or denying the user specific security permissions for user objects in the OU. However, I couldn't seem to find how to restrict access to the "Last Name", "Initials", and "E-mail" fields in ADUC. 

The trick was editing the dssec.dat file in the system32 folder per the below MS Knowledge Base article. This change needs to be made on every DC you want to edit those permissions on.

The one thing which threw me is even after restarting ADUC it wouldn't show the new properties. It turns out there is a delay between making the changes in dssec.dat and them taking effect. I'd say make the change in the file and then way at least 30 minutes before open ADUC.

Friday, May 3, 2013

Errors 9877 and 9646 in Exchange 2010 application log

Seeing error 9877 in application log of your Exchange 2010 mailbox server:


Log Name:      Application
Source:        MSExchangeIS Mailbox Store
Date:          5/2/2013 1:18:21 PM
Event ID:      9877
Task Category: Content Indexing
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <Server Name>
Description:
Content Indexing function 'CISearch::EcGetRowsetAndAccessor' received an unusual and unexpected error code from MSSearch. 

along with error 9646:

Log Name:      Application
Source:        MSExchangeIS
Date:          5/3/2013 7:57:28 AM
Event ID:      9646
Task Category: General
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <Server Name>
Description:
Mapi session "<MAPI session ID> <Object>" exceeded the maximum of 250 objects of type "objtMessage".

These are normal and shouldn't affect anything and can be safely ignored. However, if you want to resolve them you can use MS KB2616127 article for the 9877 and MS KB830829 for the 9646: http://support.microsoft.com/kb/2616127
http://support.microsoft.com/kb/830829

Friday, April 26, 2013

Migrating datastores with eagerzeroedthick disks

When you migrate a VM with one or more disks that are eagerzeroedthick from one datastore to another the disks will no longer be eagerzeroedthick. This is especially relevant if you have an Exchange server which is under heavy load because a lazy zeroed disk will have an I/O penalty.

Exchange 2010 management shell not connecting to local Exchange server

We've had a couple times with the local Exchange Management Shell would not connecting to the local Exchange server and would give us an error related to the Windows Management Interface (WMI). We found the solution is simply restarted the WWW Publishing service.

Tuesday, April 23, 2013

Warnings after promoting 2008 R2 to domain controller

When promoting the first Windows 2008 or Windows 2008 R2 domain controller in our 2003 domain we saw 1181, 1182, and 1185 information events, and 1153 warning events. These all appear to be normal and part of the promotion process and initial synchronization process for the new DC.

Exchange 2010 EMC errors out after demoting domain controller

We demoted one of our domain controllers and the Exchange 2010 EMC started erroring out on start up because it couldn't find the domain controller. We tried setting it to another DC but it wouldn't allow it since it couldn't call up the domain list from the now decommissioned DC. We ended up deleting the configuration file for EMC which allowed it to rediscover the active domain controllers. The configuration file, along with other MMC configuration files, can be found in C:\Users\<username>\AppData\Roamin\Microsoft\MMC.

Wednesday, April 17, 2013

1083 and 1955 NTDS replication warning in Directory Services log

On two of our 2003 domain controllers we were seeing an intermittently 1083 warning followed immediately by a 1955 information event in the Directory Service long at a rate of once or twice on month. Neither DCdiag nor repadmin return any errors. After working with Microsoft support they confirmed this is an expected transient error which can be safely ignored.


Event Type: Warning
Event Source: NTDS Replication
Event Category: Replication 
Event ID: 1083
Date: 4/15/2013
Time: 8:44:57 AM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: <DC>
Description:
Active Directory could not update the following object with changes received from the domain controller at the following network address because Active Directory was busy processing information. 

Object:
<Object>
Network address:
<Address> 



Event Type: Information
Event Source: NTDS Replication
Event Category: Replication 
Event ID: 1955
Date: 4/15/2013
Time: 8:44:57 AM
User: NT AUTHORITY\ANONYMOUS LOGON
Computer: <DC>
Description:
Active Directory encountered a write conflict when applying replicated changes to the following object. 

Object: 
<Object>
Time in seconds: 
0  

Event log entries preceding this entry will indicate whether or not the update was accepted. 

A write conflict can be caused by simultaneous changes to the same object or simultaneous changes to other objects that have attributes referencing this object. This commonly occurs when the object represents a large group with many members, and the functional level of the forest is set to Windows 2000. This conflict triggered additional retries of the update. If the system appears slow, it could be because replication of these changes is occurring. 

User Action 
Use smaller groups for this operation or raise the functional level to Windows Server 2003.


Wednesday, April 3, 2013

Veeam SureBackup scripts for Exchange 2010

Veeam's SureBackup is an excellent tool to verify the viability of your backups, and is able to run test scripts against your backed up VMs. This is a test script which runs against an Exchange 2010 mailbox server and checks whether the Information Store, System Attendant, and Service Host services are running or not. Just put your relevant IP in the script.


REM Start-Sleep -s 60

# Start Exchange servers on Mailbox servers

Start-Service -InputObject $(Get-Service -Computer 192.168.0.1 -Name MSExchangeIS)
Start-Service -InputObject $(Get-Service -Computer 192.168.0.1 -Name MSExchangeSA)
Start-Service -InputObject $(Get-Service -Computer 192.168.0.1 -Name MSExchangeServiceHost)

REM Start-Sleep -s 60

# Check Exchange services Status

$ExchSvc = (get-service -ComputerName 192.168.0.1 -Name MSExchangeIS -ErrorAction SilentlyContinue)
IF($ExchSvc.status -ne "Running"){$host.SetShouldExit(1)}

$ExchSvc = (get-service -ComputerName 192.168.0.1 -Name MSExchangeSA -ErrorAction SilentlyContinue)
IF($ExchSvc.status -ne "Running"){$host.SetShouldExit(1)}

$ExchSvc = (get-service -ComputerName 192.168.0.1 -Name MSExchangeServiceHost -ErrorAction SilentlyContinue)
IF($ExchSvc.status -ne "Running"){$host.SetShouldExit(1)}

Exit



SureBackup can't run a Power Shell script directly, so you need to call the script from a batch file. In this situation the PS script is saved as exchangembx.ps1 in the root of C.


powershell.exe -noninteractive -noprofile -command "& {C:\exchangembx.ps1}"
EXIT /B %errorlevel%


Veeam SureBackup errors out after changing Veeam server IP address

After changing the IP address of your Veeam server you might receive an invalid datastore path error upon powering on the first VM in a SureBackup job. To fix this you need to recreate the NFS volume on the host running your SureBackup. In vSphere Client go to the configuration tab of your host and select storage. Have it rescan all your storage, and you should see the VeeamBackup datastore disappear. Once it's gone select add storage and then NFS. Enter the IP or FQDN of your Veeam server, for the folder enter "/VeeamBackup_<server name>" where <server name> is the name of your Veeam server, and enter "VeeamBackup_<server name>" for the datastore name. Do not check "Mount NFS read-only". Once that's done SureBackup should work correctly again.

Tuesday, April 2, 2013

9782 errors on Exchange 2010

When performing a Veeam backup of an Exchange 2010 mailbox server you might find the following errors in the application log:


Log Name:      Application
Source:        MSExchangeIS
Date:          4/2/2013 8:57:54 PM
Event ID:      9782
Task Category: Exchange VSS Writer
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <FDQN>
Description:
Exchange VSS Writer (instance 4763eabf-a53b-4686-a94e-afb456c3d5da:182) has completed the backup of database '<DB>' with errors. The backup did not complete successfully, and no log files were truncated for this database.

This happens when you perform an application aware backup using Veeam but have set it not to truncate the transaction logs. This appears to be normal behavior and can be safely ignored.

Event ID 8220 when performing VSS backup of Exchange 2010

When you perform a VSS compliant backup of Exchange 2010 you might get an 8220 information alert in the application similar to this:


Event ID:      8220
Task Category: None
Level:         Information
Keywords:      Classic
User:          N/A
Computer:      <FQDN>
Description:
Ran out of time while deleting files.

This is perfectly normal and presents no issues per Microsoft:
http://technet.microsoft.com/en-us/library/ee264207(v=ws.10).aspx


Thursday, March 28, 2013

Exchange 2010 System Attendant errors/warning after removing default database

When an Exchange 2010 mailbox server is installed it automatically recreates a default database. After you've created your own databases it makes sense to delete the default one. If you research this you'll see you need to move the hidden arbitration mailbox from the default database to another one, and then you can delete it. However, after deleting it you may start getting System Attendant errors or warnings in the event log. It is caused by the HomeMDB attribute of the System Attendant object no longer being valid, even though the System Attendant doesn't actually reside in any database.

Sometimes just restarting the System Attendant service with fix the issue as it will put a valid database path into the HomeMDB field. However, sometimes it doesn't. To fix it manually you need to open ADSIedit and drill down to Configuration > Services > Microsoft Exchange > [your site] > Administrative Groups > [your Exchange 2010 administrative group] > Databases. Select one your databases, view its properties, and copy the value for distinguishedName. Now expand Servers, expand your mailbox server(s), and select the properties for Microsoft System Attendant. Past the distinguishedName value into the HomeMDB attribute. Repeat for any other mailbox servers, and then restart the System Attendant service on the affected servers.

Tuesday, March 26, 2013

VMware Converter not starting

Sometimes when you try to start the VMware Converter in the vSphere Client it'll throw a fatal error and say  there was a network connection failure. In this case just log onto the vCenter server and restart the VMware vCenter Converter Integrated Worker and VMware vCenter Converter Integrated Server services.

Tuesday, March 12, 2013

Citrix not deleting user profiles after log off

A very annoying problem with Citrix is when it doesn't delete a user's roaming profile from it's local storage when the user logs off. If you're running Citrix on VMware then there's a good chance the problem is caused either by "Shared Folders" being installed with VMware Tools or the VMwareUser.exe file is set to run on startup. Either of those can lock a profile when the system is trying to delete it.

Thursday, January 31, 2013

Replacing failed drive in IBM DS3500 SAN

On an IBM DS3500 SAN you need to regularly check the event log because certain informational events won't trigger an alert but do indicate a likely impending drive failure. Seeing an isolated VDD repair is normal, but when you see one paired with a "Destination drive error" or a "Drive returned unrecoverable media error" then there's a good chance that drive will eventually fail, and IBM support will send a replacement drive without argument.

Once you have the drive in hand there are a couple steps to successfully replace the drive. First you need to unassign all your hot spares (of which you should have at least one). You do that by going to the "Physical" tab in IBM Storage Manager for the SAN, right clicking on the hot spare, selecting "Hot Spare Coverage", and then manually unassigning individual drives. This needs to be done otherwise as soon as you fail the bad drive it'll start rebuilding onto a spare.

Once your hot spare are unassigned click on the bad drive, and then go to "Advanced" and select "Fail Drive". Once the drive is failed you can physically remove it from the enclosure, wait several seconds, and then insert the new drive.

You don't need to do anything to start the rebuild process, as the system automatically does it. You can look at current operations to see the rebuild progress. It will rebuild each logical drive in turn. Once it starts rebuilding you can reassign the hot spares.