Thursday, May 28, 2009

SharePoint Walktrough - Displaying List Items with their attachments Part 2/2

image

Recap

The SharePoint Out Of the Box List View Web Part is quite powerful, yet it isn’t able to display the attachments of a List Item. It can show if an item has any attachments (with a nice paper-clip icon) but you won’t be able to download these attachments directly.

I presented in Part 1 of this walkthrough how to create a DataView part consuming the Lists.asmx Web Service in SharePoint Designer to display the elements of a list. We are now going to display the attachments of these list elements.

To retrieve the attachments URLs we need to add an option to the Web Service Soap query, the “IncludeAttachmentUrls” option (details of the query options is available at http://msdn.microsoft.com/en-us/library/microsoft.sharepoint.spquery_properties.aspx). We could just open the datasource and change the QueryOptions parameter, that is if the SharePoint WSDL files were correctly defined… As it is not the case and SharePoint Designer itself isn’t perfect, you won’t be able to do that (see http://blogs.msdn.com/sharepointdesigner/archive/2008/06/20/data-source-issues-and-workarounds.aspx for reference).

But there is a workaround!

4 – Modify the DataSource QueryOptions parameters

imageFirst you need to have access to the page source, so let’s switch SharePoint Designer to Split or Code view :

In the code, search for the line starting with: “SharePoint:SoapDataSource”. On the same line you shall see something like “ </listName></GetListItems></soap:Body>”.

image

As you can see the parameters we defined in the Data Source Gui are found here and guess what, we can modify these parameters from here! So let’s add the parameter requesting the attachments url :

<queryOptions><QueryOptions><IncludeAttachmentUrls>TRUE</IncludeAttachmentUrls></QueryOptions></queryOptions>

So that the end of the soap request body looks like :

</listName><queryOptions><QueryOptions><IncludeAttachmentUrls>TRUE</IncludeAttachmentUrls></QueryOptions></queryOptions></GetListItems></soap:Body>

Save the page and check the result in the design view :

image

The ows_attachments field now contains the attachments URLs or 0 for list items with no attachments using the following format :

;#http://moss/Lists/MyList/Attachments/1/Piece.txt;#http://moss/Lists/MyList/Attachments/1/Piece2.txt;#

We still need to make some nice links out of this field. Time to put on your XSL designer gear.

5 – Modify the XSL Presentation

We first create a XSL template to transform the string given by the Web Service into a nice series of picture. You might use my sample below, paste this piece of code in the XSL part of the aspx page you are editing in SharePoint Designer (maybe just before the node “</xsl:stylesheet>”).

<xsl:template name="SplitAttachments">
<xsl:param name="str"/>
<xsl:choose>
<xsl:when test="contains($str,';#')">
<xsl:variable name="attachmentUrl" select="substring-before($str,';#')"/>
<xsl:if test="string-length($attachmentUrl) != 0">
<a href="{$attachmentUrl}"><img style="border:0px" src="/_layouts/images/attach.gif" alt='Open'/></a>
</xsl:if>
<xsl:call-template name="SplitAttachments">
<xsl:with-param name="str" select="substring-after($str,';#')" />
</xsl:call-template>
</xsl:when>
<xsl:otherwise>
</xsl:otherwise>
</xsl:choose>
</xsl:template>

This XSL template uses a recursive template to parse the attachments and generate corresponding links and pictures.

Now replace

<xsl:value-of select="@ows_Attachments"/>

By

<xsl:call-template name="SplitAttachments">
<xsl:with-param name="str" select="@ows_Attachments" />
</xsl:call-template>

And you should see this :

image

The paper clip icon you see are actually pointing to the attachments.

6 – Pimp up my Data View

At this point our Data View Web Part works ok but isn't as nice as you might want it to be.

As an example you might want to change these paper clips icons with icons representing the file type. The good news is that the XSL extension provided by WSS (default prefix used by SharePoint Designer is DDWRT) can handle this through a MapToIcon template. If you give a file extension to this template, it will return the icon filename corresponding to this filetype.

Using the substring-after method to get the filetype we can imagine replacing :

/_layouts/images/attach.gif

By

/_layouts/images/{ddwrt:MapToIcon('',substring-after($attachmentUrl,'.'))}

imageNow save the file and check the result :

It still isn't perfect but from this point on things will be much easier to improve if you know your HTML and a bit of XSL.

7 – Conclusion

That’s it, you have got a functional Web Part to display your List.

Using this method you can specify every parameters of the GetListItems method, as an example the ViewName parameter can prove usefull to filter items. If you need to test your parameter I would suggest you to try soapui.org.

You can of course export this web part to put it another page, even on another page of another SharePoint farm (in this case, you will have to use the basic authentication mode unless kerberos authentication can do the trick). If you want to display another list you can just export the WebPart and edit the file before reimporting it. This might prove very useful to put this web part on a publication page as these pages can’t be edited using SharePoint Designer.

If you need more details you can contact me by leaving a comment or using the "IM Me" box on the right of this page.

SharePoint Walktrough - Displaying List Items with their attachments Part 1/2

image The SharePoint Out Of the Box List View Web Part is quite powerful, yet it isn’t able to display the attachments of a List Item. It can show if an item has any attachments (with a nice paper-clip icon) but you won’t be able to download these attachments directly. As with most SharePoint limitation, there is a workaround. Drasko Popovic wrote about a JavaScript based workaround on CodeProject you might want to check out at http://www.codeproject.com/KB/sharepoint/DataViewAttachments.aspx.

In this post I want to presents you another workaround where all processing are done server-side with a single Web Service call (Lists.asmx). You will need SharePoint Designer 2007 (freely downloadable since the 2nd April 2009 at http://www.microsoft.com/downloads/details.aspx?FamilyID=baa3ad86-bfc1-4bd4-9812-d9e710d44f42&displaylang=en ) and of course a MOSS or WSS instance.

As this walkthrough is pretty long for a blog post, it is split into two parts. If you already how to create a Data View web part to display the elements of a list through calls to the Lists.asmx web service, I would suggest you to move to part 2 of this walkthrough.

0 – Principle/Rationale

We have a list in a SharePoint site and want to display the titles of this list’s elements along with direct links to their attachments.

We are going to create a Data View Web Part using SharePoint Designer, this Data View will consume the SharePoint Web Service to get the items of a list along with their attachments. The data view will display these items using a XSL we will define.

As a bonus this method will allows us to display the list items on a different SharePoint site or collection than the list itself.

1 – Create or Open a page

In SharePoint Designer after opening your site, either open an existing page or create a new one. I choose this last option:

image

Make some room for our content in the PlaceHolderMain zone of the page by creating a custom content.

image

Adding a Web Part zone isn’t mandatory but I strongly recommend it at this point. Click in the new Custom Content Zone you created in the design view, then click Insert|SharePoint Controls| Web Part Zone.

image

You should get something looking like that in the Design View:

image

SharePoint Designer is sometimes quite buggy, saving frequently from this point on might be a good idea. You can save the page wherever you want to. If you are creating multiple custom pages, creating a dedicated Document Library to hold these pages is a good idea; such a library already exists if you have activated the publication feature.

2 – Create a Web Service Data Source

imageThe page is now ready to host our Data View web part. Open the Data Source view by clicking Data View| Manage Data Sources.

imageChoose to connect to a web service:

In the Service Description Location enter: http://SP_SITE_URL/_vti_bin/lists.asmx, then click connect and choose the GetListItems operation:

image

imageDouble click on the listName parameter and enter the name or guid (between brackets {}) of your list.



imageValidate the Data Source Properties and check that it is working correctly by trying to see the data returned by this data source:

If a new panel opens with the list data, it means it’s working.

image

Troubleshooting:

If it isn’t working you will be presented with an enigmatic error message giving you absolutely no details:

image

This can be caused by numerous reasons you want to check:

  • The name or guid of the list is invalid
  • You tried to specify other parameters of the Web Service such as QueryOptions or ViewFields. These parameters can’t be specified at this point as the SharePoint Designer Team Blog attests it (http://blogs.msdn.com/sharepointdesigner/archive/2008/06/20/data-source-issues-and-workarounds.aspx). We will modify those latter on.
  • Problem of authentication, I need to investigate this problem but it seems like you sometimes can’t use the integrated authentication even when using a single WFE. In this case you have to change the Web Service Authentication method to basic and to specify a domain account that has the appropriate permission to read the list items and contact the web service.
Changing the Web Service authentication:

Open the Data Source properties

image

Open the login tab and choose basic authentication:

image

Save the details and try again to retrieve the data.

3 – Create the Data View Web Part

We now have a working Data Source, let’s create a web part to display it.

In the Data Source Details, select the ows_LinkTitle and ows_Attachments properties and choose insert this field as a Multiple Item View (Note: you might need to select the Web Part zone in the design view of the page beforehand).

image

You shall now see something like that in the design view.

image

We know have a very basic Data View Web Part but as you may have noticed the ows_attachments fields just display the number of attachments but not the actual links to these attachments.

We will see in part 2 of this walkthrough how to solve this.

Saturday, April 25, 2009

Quick, Easy and Free WAN emulation with WANem

Fibre Channel cable By Herrolm @ Flickr

A client recently asked us to deploy a MOSS based intranet over multiple geographical sites. The gist of it is to have a SharePoint farm in France and a second one in the US, with this dual-farms setup being as transparent as possible to the user. This in itself represents quite a challenge and as such will be the subject of a post once the project is finished.

We, of course, tried our different options in a mock up environment consisting of several virtual machines hosted on a single VMVare server instance.

As such, this set-up obviously lacked in representativeness of the actual environment. Being geographically dispersed the servers are not able to communicate in real life as efficiently as they can in the mock-up environment. They are indeed connected over WAN with round trip latencies over 100 milliseconds. We fiddled a bit with VMWare server options trying to find an option to limit the virtual network speeds, without any success. It would seems like VMWare ESX is better suited for this kind of WAN emulation but ESX wasn’t an option, especially considering that we had no certitude about its features regarding WAN emulation.

But wait, there is a free, open-source, easy –to-install, easy-to-use option available. Let me introduce you to WANem (for Wide Area Network Emulator). It’s a Linux live-cd (The numerous 2.09 % of linux visitors over the past month on this blog might be interested to know that it is based on Knoppix) bundled with the WANEm software itself that Tata Consulting provides us with at http://wanem.sourceforge.net.

As it is a live-cd, there is absolutely no installation to be done. Just boot up, select DHCP or enter the network information and you are good to go. You can then select from a direct-to-the-point interface what bandwith limitations and latencies you want to enforce. An advanced mode will even allow you to specify numerous others parameters such as the packet loss ratio. The only other piece of configuration you’ll need to do is to specify routes on your servers so that the packets actually go through the WANEm machine.

WANEm Web Console in Advanced Mode

So, we have a very easy-to-use, functional software for free! Seems great! Too good to be true, wait, it’s not all; the documentation included is a delight. I don’t know if it’s the same for you, but I have spent hours trying to make some open-source software work because of poor or inexistent documentation. Don’t get me wrong, open-source is great but sometimes the greatness of the piece of the software itself is at the cost of the documentation.

The WANEM Web console in basic mode

Anyway, they probably tought it wasn’t enough so they put a nice cherry at the top of the cake and created a VMWare server appliance with WANEm configured.

Finally because everything ought to have a default, WANEm is only available as a live-cd, even when provided in a virtual machine. Therefore the configuration isn’t persisted and you will have to reenter it every time you reboot the computer/vm, no big deal considering the few steps involved. It might even change soon as I read on their forum that a guy might have actually installed it on a drive (virtual or physical).

So if you need to emulate a WAN and don’t want/need to invest in an expensive Hardware solution, I strongly recommend you to give WANem a try at http://wanem.sourceforge.net.

Fiber Channel photo courtesy of Herrolm @ Flickr

Friday, April 17, 2009

Sorting / Filtering SharePoint RSS feeds

Really, REALLY BIG RSS feed button by photopia / HiMY SYeD on flickr.com Ever wondered if it was possible to sort or filter the automatically generated SharePoint RSS feeds? Well I did and, as a matter of fact, I found a solution. The idea is very simple, you create the view that will filter or sort your list elements according to your needs. You then take the guid of your view and append it to the feed URL.

Et Voila !

Here is an example, let’s say I have a Task List (screenshot from a French MOSS farm, I am sure you’ll get the idea anyway) with some items pending, other finished. This status is defined in a “State” (Etat) column.

image

I first create two views filtering on the Etat column. One returns the pending elements, the other returns the finished elements.

I then take note of the guid of my two views (Editing the views and extracting the &View=….. part of the URL does the trick).

After making sure that RSS feeding is activated for my task list, I can go check the RSS feed at :

http://MyServer/_layouts/listfeed.aspx?List=a4a88bd0%2Ddf29%2D4f54%2Da924%2D195be47346ed

The results as displayed by IE7 looks like :

image

All the items are displayed whatever their status but let’s say that I want to see only the elements pending, so that I know what I need to work on. I just append "&View=<VIEW_GUID>" to the URL and here is the result :

http://MyServer/_layouts/listfeed.aspx?List=a4a88bd0%2Ddf29%2D4f54%2Da924%2D195be47346ed&View=%7B9F3ACEAB%2D75DF%2D4C04%2D83C2%2D3A5733A6CA31%7D

image

By the way, I only tested this on a MOSS 2007 SP1 install. I would be interested to know if it works with WSS as well.

I wonder if this is documented anywhere as I stumbled upon this feature by pure chance ?

RSS picture credit Photopia on flickr

Sunday, April 05, 2009

Thinking of copying a locked SharePoint site collection ? Think twice !

clip_image002[7]I recently had to copy a locked SharePoint site collection from one farm to another farm. I soon realized this wasn’t such a great idea especially if the farms aren’t in the same AD domain. Here is a quick recap of what I noticed after a few more investigatory steps.

Eager readers will eagerly jump to the final conclusion.

Copying though stsadm backup/restore

Let’s try copying this site collection with the stsadm backup and restore commands. Both farms are MOSS RTM farms.

The backup worked smoothly. The restore is a total disaster (“Additions to this Web site have been blocked. Please contact the administrator to resolve this problem.”) :

screenshot.1

The weirder part is that the collection is visible in the central administration but it seems totally messed up. No way to delete it from the Central Adminsitration. The stsadm deletesite command returns the “Additions to this Web site have been blocked. Please contact the administrator to resolve this problem.” error.

Conclusion : Not a good idea ! Might actually works using the trick I finally used, see below.

Copying using a SQL Server backup

Seemed like today would be another day battling with SharePoint. But I wasn't short on idea. Indeed My locked site collection was alone in its content DB, therefore I was able to use a backup to restore the collection on my target farm.

The restore wentsmoothly. I was starting to believe it was actually going to be easy!

And then my frail hopes were cruelly shattered by the terrible SharePoint truth when I opened up the Central Administration to change the collection administrator.

The first obvious step is to select the site collection but when selecting the site collection in the “Select Site Collection” popup, none of the collection details would appear on the right.

clip_image004

Huh ? I tried validating the popup anyway and was presented with this screen:

clip_image006

It seemed just as if I had selected nothing, yet the adress bar contained the id of the content database :

clip_image008

Trying to unlock this site collection would produce kind of the same result.

I then realized that the farm administrator had no right on the collection. He indeed had no permissions on the collection when it was backuped. Lucky me, the farms were part of the same domain, therefore I tried a quick and dirty fix, I added the original farm administrator to the target farm administrator group and logged in the Central Adminsitration with this administrator.

It was a relief to be able to see the details of the collection. Thou shan’t prevail over me SharePoint !

But once again, I underestimated my adversary which quickly retaliated with a now too common “Additions to this Web site have been blocked” when I tried changing the collection administrator.

clip_image010

But victory was mine at long last when I unlocked the collection while logged in with the original farm administrator account on the target farm. After this step I was able to change the collection administrator to the target farm administrator as I intended from the beginning.

Conclusion : It works, but both farms have to be in the same AD domain

Final Conclusion

If possible I would advise you to unlock the site collection before proceeding with the copy. If it isn’t possible, I hope both of your farms are part of the same AD domain, otherwise good luck! Please keep me posted if the powers that be are you on your side.

By the way, for those of you thinking that I should have tried the stsadm export command, one simply can’t export a locked site collection this way.

Monday, March 23, 2009

IIS 5 and 6 time-taken field

IIS_time_takenWhile checking the performance problem a client was having with a legacy ASP.NET 1.1 application (yep, I am still young enough to call a 6 years old system, legacy J ), I asked myself what seems like a basic question.

Does the time-taken column in IIS 5 logs takes into account the network time ?

That is, Is the stop watch stopped when the response is ready or when the last TCP acknowledge is received from the client. After googling for a while I was quite disappointed by the lack of information regarding this specific column. The only useful information I found concerned IIS 6 and were …hum, let’s just say that Microsoft doesn’t seems to be able to make up its mind about this field:

According to :http://support.microsoft.com/kb/944884

Beginning in IIS 6.0, the time-taken field typically includes network time. Before HTTP.sys logs the value in the time-taken field, HTTP.sys usually waits for the client to acknowledge the last response packet send operation or HTTP.sys waits for the client to reset the underlying TCP connection. Therefore, when a large response or large responses are sent to a client over a slow network connection, the value of the time-taken field may be more than expected.

According to :http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/676400bc-8969-4aa7-851a-9319490a9bbb.mspx?mfr=true

For the time-taken field, the client-request timestamp is initialized when HTTP.sys receives the first byte, but before HTTP.sys begins parsing the request. The client-request timestamp is stopped when the last IIS send completion occurs. Time taken does not reflect time across the network. The first request to the site shows a slightly longer time taken than other similar requests because HTTP.sys opens the log file with the first request.

Perplexed, I figured out I could know the answer for sure by firing up Fiddler and comparing Fiddler time with the value of the time-taken columns in IIS log. I had limited access to the client environment but managed to get some measures nonetheless. Once again, it was a dead-end, the Fiddler times were sometimes higher than the IIS log time, sometimes lower.

That’s when I decided to contact the Microsoft Partner Support.

The definite answer they gave me for the time-taken field in IIS 5 logs is:

Since the calculation includes the time for the acknowledgement from the client, it may be skewed by any network latency issues.

The not so definite answer the support gave me for the IIS 6 time-taken field is that after checking the source code, the network time is not included in the time-taken field. However they also say that that if the request is aborted during its execution, no value are logged for the request; Therefore it would seems like IIS 6 waits for the last ACK to log BUT it doesn’t log the network. And that would be why the public available information is as clear as mud.

I hope it helps.


Edit : 29/06/2009

A reader (thank you Joe) pointed out to me that the http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/676400bc-8969-4aa7-851a-9319490a9bbb.mspx?mfr=true article as been updated with the following :

This is no longer true for II6, and you can check the KB94483. IIS TIME-TAKEN field takes into account the last ACK from the answer to the response, so in fact does reflect time across the network. There is only 2 exceptions for this:

  • The response size is less than or equal to 2 KB, and the response size is from memory.
  • TCP buffering is used.
So problem solved, but if anyone find the trace of the KB94483 mentionned, let me know because I can't !

Sunday, January 18, 2009

SharePoint : Search using « This Site » or “This List” scope returning no results

I met a problem with the Search scope on a MOSS 2007 farm the other day that I thought was worth blogging about.

On all the content sites of our farm, the “All Sites” scope worked as expected, but the “This Site” or “This List” search scopes wouldn’t return a single result.

There are several reasons that can explains this problem according to this technet forum thread (http://social.technet.microsoft.com/Forums/en-US/sharepointsearch/thread/906045d8-8782-4c39-be1b-7632f7239019/#page:1), in our case the problem was due to a difference between the Content Source defined for the crawl and the default zone defined in the Alternate Access mapping.

Our Alternate Access Mapping for this zone was defined as follow:

Private URL Zone Public URL
http://hostname Default http://hostname
http://intra.dnsname.com Intranet http://intra.dnsname.com

And we used http://intra.dnsname.com as the start address for the crawl.

We tried changing the content source start address from http://intra.dnsname.com to http://hostname, resetted the crawl content and launched a full crawl and that was it. The “This Site” and list scopes worked perfectly from then on.

That could have been it, except that this change caused another problem. Before this modification we had no trouble with links URL, I mean if I used http://intra.dnsname.com to reach the site, all the links in the pages would use this dns name, not the hostname. After the modification of the content source however, some search results would sometimes be presented with the hostname in the url instead of the dns name.

To solve that problem we changed the Alternate Acces Mapping configuration to:

Private URL Zone Public URL
http://hostname Default http://intra.dnsname.com
http://intra.dnsname.com Default http://intra.dnsname.com

Voila! The Site and List scopes are working and the URL are always based on the DNS name when someone uses the DNS name to reach to site.

By the way, you might find this article about Alternate Access Mapping very useful: http://blogs.msdn.com/sharepoint/archive/2007/03/06/what-every-sharepoint-administrator-needs-to-know-about-alternate-access-mappings-part-1.aspx

Tuesday, January 06, 2009

Infinite looping the MOSS OOTB approval workflow

A few days ago, I noticed something weird with the Microsoft Office SharePoint Server out of the box workflow, most specifically with the workflow option to start when an item is modified.

I was trying to set up the Approval Workflow on a SharePoint List with content approval activated. I first set up the workflow to ask for approval when a new item is submitted to the list. It worked fine but we also needed to ask for approval when an item was updated (e.g. the writer made a mistake and notice it after the approver did his job, thus he modify the item that as a result needs to be approved a second time).

I fired up the workflow configuration page and checked the infamous “Start Workflow when an item is modified” in the start options.

From a french MOSS install

And from then everything went wrong. Whenever an approvers did his job by approving an item, the workflow would be launched again and a new approval task would be added to the workflow task list.

I tried tweaking the workflow, modifying the tasks list used … but nothing would do the trick, the workflow was stuck in an infinite loop whenever the “Start on item edit” was activated.

And that’s when it hit me, changing the approval status of an item seemed to be considered as a modification of the item itself.
I am still surprised by what I can’t help but see as a major flaw of SharePoint. I definitely don’t think that the approval status of an item is part of the item itself, it is only a SharePoint information about OUR piece of information (the item).

I also was surprised by how few people seems to react to this fact based on a few google search I made. Yet it seems other people faced the same problem (http://social.msdn.microsoft.com/Forums/en-US/sharepointworkflow/thread/62f5921d-9d02-4924-a61f-97ad169d1043/ and http://unitedsupplyco.sharepointsite.net/publicfacing/bensblog/Lists/Posts/Post.aspx?ID=34 ).

If you have any information about this subject, I would enjoy to know it.

Tuesday, November 25, 2008

Sharepoint - Sorting people search results by name

Hello Dear Reader, for this new post. First some personal news, after working for quite sometime on load testing a java CRM application, I wanted to move on and finally work with the newest Microsoft technologies. And that’s what I did as I managed to land a job at a local Microsoft Gold Partner in Nantes, France. The team I ‘m part of is mainly working integrating the Sharepoint solution for local partner, and I got to say it’s a nice opportunity so far.

Background

For my first Sharepoint post I’ll speak about the limitation of the integrated People Search functionality. With MOSS 2007 comes the Enterprise Search Center that allows you to gives an easy acces to the search engine to your users. This Sharepoint site (because that’s exactly what it is) comewith two search mode OOB:

  • a standard search for searching through documents, lists, pages, ... basically whatever you let the Search Engine crawl and
  • a people search for, you guessed it, searching people based on any properties you want among the profile properties defined in the SSP (Shared Service Provider)

We recently had a problem where a client wanted to:

  • Display the list of employee for each office, sorted by name/firstname
  • Be able to search a client based on its name/firstname

The problem

We first imagined to fulfill these two needs with a single item, a people search page, where results per office would be obtained by adding the office as a searchable property. And that’s when we realized a huge shortcoming of the Search Center:

People search results can’t be sorted on arbitrary fields you will define, such as the name/firstname. Two standard Sorting modes are available and that’s it. Results can be sorted based on the ranking computed by the Search server. Results can also be sorted by social distance, the principle is fine enough, if you are looking for a John, it will first display the John who is your direct colleague instead of the John who is working in another department of another office.

The almost working workaround

Starting from this realization, we figured we might solve this problem by doing a quick sorting in the XSL displaying the results. This way, the sorting could be done on any fields necessary. While this solution is perfect if you have less than 50 results, it is not a full working solution.

The thing is, the search engine first returns the results. These results are then divided (by the Search Core Results web part) in x collections of y elements (people) where y is a webpart parameter capped at 50. Therefore if we have more than 50 matchs, the sorting will be done separately on each page in a discontinued manner.

That is , a hypothetical “Albert Alden” might be on page 2 while “Brittany Bern” is on page 1 if his rating is lower than Brittany’s. But we know for sure that “Cindy Costner” which is on page 2 will be presented after Albert whatever her rating is because the sorting on each page is done lexicographically by our XSL.

The working workaround

Keeping these limitations in mind we had to devise a quick solution. We chose to keep the standard people search for standard people search, considering the size of the company and the fact way it is going to be used. Yet we still needed a way to present the users with a list of all the employees of any given office (at this point, it’s probably needless to remind you that we REALLY want this list to be sorted by name/firstname J). That’s when we stumbled upon this superb article by Dietmar Kurok (http://www.codeproject.com/KB/sharepoint/DepartmentPeopleViewer1.aspx ).

The solution is to use the Web Service exposed by the search engine to get the information we need sorted as we need and then present it with a XSL of our own. In our case we used the same principle as the one used in Dietmar Kuork’s article, however we chose to generate our QueryEx input parameter (the Querypacket) with a very nice tool available on CodePlex. The tool is named SharePoint Search Service Tool and will be found at http://www.codeplex.com/SharePointSearchServ.

The general idea of the process is to:

image

I hope you enjoyed this first Sharepoint post and I hope to be able to post more often on this nice product in the future.

Wednesday, April 23, 2008

Diagnosing Java Memory Heap Leak with SAP Memory Analyzer

Hi dear reader, you might have been wondering why I haven’t been posting any new posts in quite a long time now. Well, you probably guessed the reason and it’s very common, my work has been really time-consuming. Indeed I joined a performance testing team and had much to learn. We are currently testing the performance and tuning a CRM application. As this is a J2EE application I learned a lot about the Java HotSpot Virtual Machine (especially the HP Ux flavor) memory performance and problems.

And here come this post, I thought I really need to drop a message about a very nice tool I found this week, SAP Memory Analyzer. This tool will analyze Java Heap dump (snapshots of the memory of a JVM) to help you fixing memory usage problems. This is a common feature of many programs (Sun Jhat, HP JMeter among others), yet I found Mem Analyzer very easy to use, full of functionality and something one really appreaciate after trying Sun JHat, it’s memory usage is quite low (and automatically managed) and it is very quick to open Heap Dump as opposed to some other tools. This TheServerSide article presents some striking features of this tool

If I was to stop here, it would be just be an Ad for this tool and nothing more, therefore I will give you a few hints that might help you diagnose Java application memory leak with this tool.

Making sure it is a Java Heap Leak

If your application crashs with an OutOfMemory error when it has been running with the same load for a few hours, then you probably have a Java Heap leak.

Whereas if the memory consumption of your java processes keeps increasing over time (once again with a constant load), it doesn’t necessary means that the Java Heap is leaking, indeed it might just be some JNI objects that you are using that are leaking (To put in a nutshell, the native heap is the heap used by your JVMs processes, it contains the Java Heap where the Java objects resides whereas non java object such as JNI objects resides in the remaining part of the native heap, thus the leak might be in either part of the native heap).

Because you might want to make sure that it is the Java heap that is leaking and because it is never a bad idea, you will probably need to log the Garbage Collection activity. I might post about how to do that latter, for now let’s just assume you added these commands to the command line of your JVMs (tested on a HP UV 1.4.2.x JVM) :

-XX:+PrintGCDetails -XX:+PrintGCTimeStamps -Xloggc:output.log

After running your application, you will get a output.log file that might be analyzed in the GCViewer application(http://www.tagtraum.com/gcviewer.html). If you want more detail, you will replace the Xloggc switch with the XVerbosegc:file:output.log switch and you will use the HP JMeter tool to analyze it (http://www.hp.com/products1/unix/java/hpjmeter/)

If the memory usage graph you get with either of these tools looks like a nice Saw Tooth shape (check http://en.wikipedia.org/wiki/Sawtooth_wave if I am not clear enough), then there is no Java Heap leak, you better check the other part the native heap. Aren’t your JNI Objects leaking, or is it the way your java code calls these objects?

If you see something else that tends to increase over time (probably something kind of like a SawTooth shape but that never goes down as much as it did the previous iteration) and more and more full garbage collection occurring until you eventually reach an OutOfMemory condition, then you probably have a leak (or memory retention as I think if more appropriate for these phenomenon when the memory is automatically managed).

Taking snapshots of the Java Heap

Now that we know we have got a memory retention in the Java Heap, we want to find what is responsible for it and hope we can solve it (I know, it’s so easy to blame external libraries J).

Here is a process I used recently, it’s only one among many other ways to analyze memory retentions, it might not be adapted to your case nor the easiest way, yet you might use it as a (good?) start. The general principle is always the same, taking a snapshot of your Java heap at two or more different times and diffing those.

First add the following attribute to your JVM : XX:HeapDumpOnCtrlBreak (it became available on the lattest 1.4.2.x JVMs).

With this attribute, you will be able to generate dump (snapshot of the memory) anytime you want. Anytime you will send a SIGQUIT signal (try kill -3 PID on Ux systems ) to the jvm process, instead of stopping it will generate a binary Heap Dump in a file entitled java_<pid>.hprof.<millitime>. By the way, mind the size of these dumps. If your heap is 1Go large, then the file will be 1Go as well.

A good way to find where the memory is retained is to take a snapshot when your application is in a nominal state (up, running with some users connected) and then wait a few hours (might be minutes if the heap is growing quickly or hours if it is very slowly gaining in size). Just be careful not to take the first snapshot while the system is warming up.

Finding what is retained in the Java Heap

You will then need to install SAP Memory Analyzer (available Here). Once it is done you can open the hprof generated beforehand. SAP Mem Analyzer will then parses and analyzes it.

This step might take quite some time (from a few minutes to half an hour for a typical heap on a typical computer). Yet don’t be afraid, you will be doing this step only once as the tool creates some files containing the result of the parsing along the HPROF file. If you open this hprof again later on, it will just read these parsing files.

TIP : You might want to do your test and snapshots on a specific JVM configuration with a smaller heap in order to quicken this step.

Once it has been parsed, SAP Memory Analyzer presents you with an histogram of the heap. While this can be useful you might want to jump to the Dominator view. This view will help you to analyze what is retained in the memory. If there are a handful of objects causing your memory retention, you will find those at the top when using this view. Linking these objects to your application code, and thus finding the cause of the problem might prove much easier that you think. I guess this is a good starting point for your analyze. It helped me quite a lot in a few cases even though it once proves quite useless (when we had tons of memory causing the retention). Another major feature that will help you is the Delta-Diffing feature available from the histogram view, it will compare two different Heap Dump. You might also want to try the "Calculate retained size" from the contextual menu.

You will see that this tool offers tons of functionality beyond the histogram and dominator view, you should really try as much of those as you can, it’s really useful and at least educating regarding your application memory usage.

One last thing, you might want to take a different approach for you analyze, that is instead of taking two snapshots before and after hours of nominal load, taking a snapshot before and after a very simple action on a platform you are the only to use might be your key to fixing your problem. It depends on the memory retention you are observing.

References: