Thursday, October 23, 2008

Troubleshooting SharePoint: When Good Servers Go Bad



It summarizes the most-common issues with SharePoint, how to avoid them, and how
to fix them. Here is a list of topics:



  • Install Service Pack 1

  • Runaway Transaction Logs

  • Runaway SharePoint Logs

  • Protected Memory Errors

  • IISWAMREG Admin Service Errors

  • Timer Job Errors

  • Update Conflicts

  • Performance Issues

  • Customization Errors

  • Security Errors



Install Service Pack 1



The first recommendation is to make sure that you have SP1 installed. SP1 has
over 2500 fixes, and can go a long way to helping you avoid issues.



WSS SP1 Download



MOSS SP1 Download





Runaway Transaction Logs



One of the most common issues with SharePoint farms is runaway transaction logs
associated with the SQL Server databases. This is really not a SharePoint issue,
but a SQL Server issue. This issue is caused by the fact that most SharePoint
databases are created in Full-Recovery mode. This means that SQL Server retains
a log of every transaction that occurs on the database since the last database
backup. If you never backup your SharePoint databases, then the transaction log
will continue to grow until all the disk space is used up.



Symptoms



  • Little or no available disk space on SQL Server

  • Huge transaction log files



Recovery



Step 1: Backup the Transaction Log manually from the UI. This will truncate
the log, but not shrink it.



From
http://support.microsoft.com/kb/873235



When the transaction logs grow to an unacceptable limit, you must immediately
back up your transaction log file. While the backup of your transaction log
files is created, SQL Server automatically truncates the inactive part of the
transaction log. The inactive part of the transaction log file contains the
completed transactions, and therefore, the transaction log file is no longer
used by SQL Server during the recovery process. SQL Server reuses this
truncated, inactive space in the transaction log instead of permitting the
transaction log to continue to grow and to use more space.



Step 2: Manually shrink the log file from the UI.



From
http://support.microsoft.com/kb/873235



The backup operation does not reduce the log file size. To reduce the size of
the transaction log file, you must shrink the transaction log file. To shrink a
transaction log file to the requested size and to remove the unused pages, you
must use the DBCC SHRINKFILE operation. The DBCC SHRINKFILE Transact-SQL
statement can only shrink the inactive part inside the log file.



Prevention



Option 1: Switch to Simple recovery Mode for non-production SharePoint
databases.



This will prevent the transaction log from growing because simple recovery mode
does not keep transaction history. Use the following SQL Statement:



USE Master

SELECT Name, Recovery_Model_Desc FROM Sys.Databases



ALTER DATABASE [name] SET RECOVERY SIMPLE WITH NO_WAIT



Option 2: Create a database maintenance plan for all SharePoint databases




Creating the Database Maintenance Plan



1. Open SQL Server Management Studio



2. Right click the SQL Server Agent and select Start from the context menu.



3. Expand the tree and select Management►Maintenance Plans



4. Right click the Maintenance Plans folder and select Maintenance Plan Wizard
from the context menu.



5. On the Welcome screen, click the Next button.



6. On the Select Plan Properties screen, name the plan “MOSS Maintenance”



7. Click the Change button



8. Modify the schedule (daily recommended) and click the OK button.



9. Click the Next button



10. On the Select Maintenance Tasks screen, check the following tasks:



  • Check Database Integrity

  • Reorganize Index

  • Update Statistics

  • Clean Up History

  • Backup Database (Full)



11. Click the Next button.



12. On the Select Maintenance Task Order screen, click the Next button.



13. On the Define Database Check Integrity Task screen, drop the list and select
all of the SharePoint databases.



14. Check the Include Indexes box.



15. Click the Next button.



16. On the Define Reorganize Index Task screen, drop the list and select all of
the SharePoint databases.



17. Check the Compact Large Objects box.



18. Click the Next button.



19. On the Define Update Statistics Task screen, drop the list and select all of
the SharePoint databases.



20. Select the All Existing Statistics option.



21. Select the Full Scan option.



22. Click the Next button.



23. On the Define History Cleanup Task screen, click the Next button.



24. On the Define Back Up Database (Full) Task screen, drop the list and select
all of the SharePoint databases.



25. Check the box titled “Create a subdirectory for each database”.



26. Change the backup location, if desired.



27. Click the Next button.



28. On the Select Report Options screen, click Next.





Creating the Transaction Log Maintenance Plan



If this installation is not a production environment, consider changing the
recovery mode for all databases to SIMPLE using the following

ALTER DATABASE [name] SET RECOVERY SIMPLE WITH NO_WAIT



1. Open SQL Server Management Studio



2. Right click the SQL Server Agent and select Start from the context menu.



3. Expand the tree and select Management►Maintenance Plans



4. Right click the Maintenance Plans folder and select Maintenance Plan Wizard
from the context menu.



5. On the Welcome screen, click the Next button.



6. On the Select Plan Properties screen, name the plan “MOSS Log Maintenance”



7. Click the Change button



8. Under the Frequency section, Select Daily from the Occurs drop-down list



9. Under the Daily frequency section, select the Occurs Every option.



10. Set the Start Time to 9:00:00 AM.



11. Set the End Time to 5:30:00 PM



12. Click the OK button.



13. Click the Next button.



14. On the Select Maintenance Tasks screen, check Backup Database (Transaction
Log)



15. Click the Next button.



16. On the Select Maintenance Task Order screen, click the Next button.



17. On the Define Back Up Database (Transaction Log) Task screen, drop the list
and select all of the SharePoint content databases.



SSP, WSS Search and MOSS Search databases are set to simple recovery model, so
their transaction logs will not grow.



18. Check the box titled “Create a subdirectory for each database”.



19. Change the backup location, if desired.



20. Click the Next button.



21. On the Select Report Options screen, click Next.






Runaway SharePoint Logs



Another area where disk space can be used up quickly is through the SharePoint
Unified Logging Service (ULS). The ULS writes text-file logs to the LOGS folder
in the System Directory (\Program Files\Common Files\Microsoft Shared\web server
extensions\12).



Symptoms



Many large text files located in the LOGS directory



Recovery



Delete old files to regain space as necessary



Prevention





  • Throttle the logging service by opening Central Administration, clicking the
    "Operations" tab and going to "Diagnotic Logging". On the Diagnostic logging
    page, throttle the logged events by selecting a minimum level of severity for
    logging.




  • Make use of the
    LogViewer feature to examine logs. This adds a log
    viewing utility to Central Admin.






Protected Memory Errors



This error shows itself as several different errors in the Event Log all saying
that SharePoint attempted to read or write protected memory.



Symptoms



  • Event Log entries: Attempted to read or write to protected memory

  • Can’t open IIS Manager

  • Can’t make new web applications



Recovery & Prevention






IISWAMREG Admin Service Errors



Symptoms



In the Event Log, you see the following:



Type: Error

Source: DCOM

Category: None

Event ID: 10017

Description:

The application-specific permissions settings do not grant Local Activation
permission for the COM Server application with CLSID {CLSID} to the user
DomainName\UserName SID {SID}. This security permission can be modified using
the Component Services administration tool.



Recovery & Prevention




1. Click Start, click Run, type dcomcnfg in the Open box, and
then click OK.

2. Expand Component Services, expand Computers, expand My
Computer, and then click DCOM Config.

3. Right-click IIS WAMREG admin Service, and then click
Properties.

4. Click the Security tab.

5. Under Launch and Activation Permissions, click Edit.

6. In the Launch Permission dialog box, click Add.

7. In the Select Users, Computers, or Groups dialog box, type
the domain user account that you specified as the Windows SharePoint Services
3.0 service account, click Check Names, and then click OK.

8. In the Permissions for UserName list, click to select the
Allow check box that is next to Local Activation, and then click OK two times.




Here are some additional references:




http://support.microsoft.com/kb/920783




http://blogs.vertigo.com/personal/steventap/Blog/Lists/Posts/Post.aspx?ID=21






Timer Job Errors



There are two timer services in SharePoint that are responsible for running
various jobs that keep the farm healthy. If you have issues with these services,
then you may see these jobs fail.



Symptoms



  • Failed jobs in Central Admin

  • Installed solutions fail to deploy

  • Created web apps fail to create on every server



Recovery



  1. Recycle "Windows SharePoint Services Timer" and "Windows SharePoint Services
    Administration" on every server in the farm.

  2. Run stsadm -o execadmsvcjobs on every server in the
    farm



Prevention



To prevent these issues, create a batch file that periodically recycles the
services.




1. Open NotePad and add the following lines:



NET STOP "Windows SharePoint Services Timer"

NET START "Windows SharePoint Services Timer"

NET STOP "Windows SharePoint Services Administration"

NET START "Windows SharePoint Services Administration"

STSADM -o ExecAdmSvcJobs



2. Save the file as "C:\Program Files\Common Files\Microsoft Shared\web server
extensions\12\ISAPI\RecycleSharePointServiceTimer.bat"

3. Select Control Panel►Scheduled Tasks►Add Scheduled Task.

4. In the Schedule Task Wizard, click the Next button.

5. Click the Browse button and locate the file "C:\Program Files\Common
Files\Microsoft Shared\web server
extensions\12\ISAPI\RecycleSharePointServiceTimer.bat."

6. Select to perform the task daily and click the Next button.

7. Schedule the task to start at 4:00 AM Every Day and click the Next button.

8. Enter credentials for executing the task and click the Next button.

9. Check the box to open the Advanced Properties and click the Finish button.

10. Click the Schedule tab and then click the Advanced button.

11. Check the Repeat Task box.

12. Schedule the task to repeat every 4 hours.

13. Select the Time option and enter 3:00AM.

14. Click the OK button.

15. Click the OK button.






Update Conflicts



Whenever you update SharePoint system passwords, you can have issues with
SharePoint being able to access databases and run jobs.



Symptoms



In the Event Log, you see the following error:



An update conflict has occurred, and you must re-try this action. The object
SPApplicationPool Name=SharePoint Central Administration v3 Parent=SPWebService
Name=WSS_Administration is being updated by {...}, in the STSADM process, on
machine {...}. View the tracing log for more information about the
conflict.



Recovery & Prevention



Clear the File System cache on the SharePoint front end servers using this
procedure: (From
http://support.microsoft.com/kb/939308
).




1. Stop the Timer service. To do this, follow these steps:



a. Click Start, point to Administrative Tools, and then click
Services.


b. Right-click Windows SharePoint Services Timer, and then click
Stop.



2. Delete or move the contents of the following folder:

%ALLUSERSPROFILE% \Application Data\Microsoft\SharePoint\Config\GUID


3. Start the Timer service. To do this, follow these steps:



a. Click Start, point to Administrative Tools, and then click
Services.


b. Right-click Windows SharePoint Services Timer, and then click
Start.




Note The file system cache is re-created after you perform this procedure. Make
sure that you perform this procedure on all servers in the server farm on which
the Timer service is running.




Here's a great article on password changes for SharePoint:

http://blogs.msdn.com/joelo/archive/2006/08/22/712945.aspx






Performance Issues



Various performance issues like slow page loads.



Symptoms



  • Slow Page loads



Recovery



  • Reduce number/complexity of web parts

  • Evaluate lists

  • Evaluate web part architecture



Prevention



  • Control number of web parts

  • Delete unused web parts

  • List Maintenance (2000 Items, Indexed columns)

  • Avoid programming pitfalls (undisposed objects, excessive looping)

  • Enable caching



<BlobCache location="C:\blobCache"
path="\.(gifjpgpngcssjs)$" max enabled="true"
/>





Customization Errors



Symptoms



  • Generic SharePoint Error Page



Recovery



  • Add "?contents=1" to the web of the web page request to open web part management
    page. Delete offending web part

  • Restore default master page if your custom one is causing the problem.

  • Deactivate offending feature

  • Disable custom errors by editing web.config file by making the following
    changes:



<compilation batch="false" debug="true">



<customErrors mode="Off" />



<SafeMode MaxControls="200" CallStack="true" DirectFileDependencies="10"
TotalFileDependencies="50" AllowPageLevelTrace="false">



Prevention



  • Implement a Dev->QA->Production process






Security Errors



Symptoms



Request for the permission of type
'Microsoft.SharePoint.Security.SharePointPermission,
Microsoft.SharePoint.Security, Version=12.0.0.0, Culture=neutral,
PublicKeyToken=71e9bce111e9429c' failed



Recovery



Adjust security policy in web.config file as follows:



<trust level="WSS_Medium"
originUrl="" />



Prevention



Implement custom security policies on all web parts. Start with Dan Larson's
blog entry on the topic.


No comments: