Exchange 2010: Removing a stuck move request

When you create a move request in Exchange 2010, an entry for that account is created in the "Move Request" under "Recipient Configuration". Once it's completed you cannot create another move request until you clear the existing move request. However, there are certain situations where the move request can get stuck and cannot be deleted, for example if the user is moved to a different OU after the move, but before deleting the move request.

To clear a stuck move request you need to use ADSIedit to clear the attributes msExchMailboxMoveSourceMDBLink, msExchMailboxMoveTargetMDBLink, msExchMailboxMoveFlags, and msExchMailboxMoveStatus from the user's account. As always, save the attributes to notepad before clearing them. Once they're cleared, synchronize your domain controllers, and the move request should disappear from the Exchange 2010 EMS. You may need to close and reopen EMS.

Hitting the 4GB database limit in VMWare vCenter

When you install VMWare vCenter to manage your virtual environment, unless you specify an existing database, it will automatically install MS SQL 2005 Express. This is great and works fine. However, one of hte important limitations of SQL 2005 Express is a 4GB database size limit. As you use VMWare, performance data, tasks history, and event history will all be recorded to the database and will slowly fill it up. Depending on size and complixty of your environment, some time later your DB will hit 4GB, the SQL server will stop accepting data, and vCenter will simply stop. As always, before making changes to DB make sure you have a good backup.

The solution is to quickly shrink your DB size, restart SQL, and start the vCenter service again. The easiest way to shrink the DB is to clear out all the old performance, task, and event data. If you haven't already, download and install the SQL Management Studio. Open up your vCenter database, and then run the following query to clear performance data:

truncate table VPX_HIST_STAT1;
truncate table VPX_SAMPLE_TIME1;

truncate table VPX_HIST_STAT2;
truncate table VPX_SAMPLE_TIME2;

truncate table VPX_HIST_STAT3;
truncate table VPX_SAMPLE_TIME3;

truncate table VPX_HIST_STAT4;
truncate table VPX_SAMPLE_TIME4;


Then open the table in the vCenter database called dbo.VPX_PARAMETER, modify "event.maxAge" to how many days of events you want to keep (I usually keep 90 days), modify "event.maxAgeEnabled" to "true", modify "task.maxAge" to how many days of tasks you want to keep (again I usually keep 90 days), and finally set "task.maxAgeEnabled" to "true". Once you're done open the "programmability" folder for the vCenter DB in Management Studio, open "stored procedures", and execute the one called "dbo.cleanup_events_tasks_proc".

The final step is right clicking on the database, going to tasks, and selecting shrink database (not file). Let it run, and your DB should suddenly be well under 4GB. Restart your vCenter service.

You can either monitor the growth of your DB and clean it up periodically, or you can upgrade to SQL standard. Because of Microsoft's SQL licensing, and because you're probably already running SQL in your virtual environment, you can probably upgrade to standard for no cost to you.

Fast way to check an ESEUTIL /G log

You can run ESEUTIL /G against an unmounted Exchange 2010 database to check it for errors. It's a good tool, especially if you can bring up your Exchange 2010 backup in a test environment to run checks. The problem is the log is very verbose, and there's no summary regarding if it found any errors. All you get is a line saying "finishes with error 0 (0x0)" if it didn't find an error. If it finds an error it'll say "finishes with error" and then an error code, so you need to parse the log file looking for any line which isn't "finishes with error 0 (0x0)", and even for a small database you'll have thousands of those entries to look at.

The trick is to open the log file in notepad, run Find and Replace, and replace all instances of "finishes with error 0 (0x0)" with something like "CHECK OK". Now you can search the log for "error" and quickly find the entries, if any, where it finished with an error.

Monitoring an Exchange 2010 index crawl

If you need to rebuild an Exchange 2010 DB search index, the index state for the DB changes to "crawling". Unfortunately it doesn't give any indication if it's actually doing anything, and considering it's possible to get the status stuck as crawling, you cannot activate the database in a DAG while it's crawling, and it can take a long time to complete, the ability to actually monitor it's progress is important. 

So here's a trick to help monitor the progress of the index crawl:

- Start Performance Monitor by going to Start>Run>Perfmon

- Under "Monitoring Tools" select "Performance Monitor"

- Click the green plus sign to add a monitor.

- Select the local computer, scroll down to "MSExchange Search Indices", select "Number of Indexed Recipients" and "Number of Mailboxes Left to Crawl", and hit OK.

- Right click in the graph, select properties, select the "graph" tab, and change the view to report. 

Now you can see as the number of mailboxes remaining drops, making it much easier to monitor its progress and ensure it's working. Be aware some mailboxes take much longer than other mailboxes.

Restoring vCenter

After our SAN reset itself and lost connection with our VMWare 4.1 ESX hosts there were several servers which were understandably not happy. However, vCenter did not appear to be one of them. In fact, we used it extensively while fixing the various issues caused by the SAN outage, and it never caused an issue. It was therefore a little surprising today when the vCenterServer Service stopped on it, and when I restarted the service the SQL Server 2005 Express instance shot to 99% CPU usage, and the tempdb system database started growing, and wouldn't stop until it had exhausted all available storage space, and then crash. We found restarting the SQL server would free up the space, but the cycle would then start again. We also found stopping the vCenterServer Service would drop the SQL service CPU utilization back to 0 and stop the tempdb file growth.

Naturally I engaged VMWare technical support on this, and we worked the problem for four hours without resolution. We tried truncating the temp tables and history tables in the vCenter DB, reset the credentials for the SQL services, and looked through extensive logs. No luck.

Finally, I tried an Instant Recovery of the vCenter VM from our Veeam backup system, and it worked beautifully; CPU utilization stayed down, and storage stayed stable. I immediately did a standard restore of it, and was back up and running.

All vCenter stores is historical information like performance statistics and your higher level configuration like HA. If you're a smaller shop and you keep your configuration basic, then vCenter isn't storing much of anything, and rolling it back a couple days won't harm anything. In fact, you could do a completely new installation of vCenter and reconfigure it manually pretty fast.