+ Business Profile extraction from webpages or business directory report

Few days ago, we have tried to develop a business profile extraction parser. But if you want to write down a business profile parser then you have to do a proper R&D over that.

Right now I'm giving you a my first R&D report of a business profile parser.

Current R&D Target:

Developing an efficient intelligent profile parsing system that is able to parse profiles from at least 80% - 90% Business Directory WebPages.

Progress:

Our target Business-Directory-WebPages are classified into two groups:

1.      Multiple-Profile Pages

2.      Single-Profile Pages

Multiple Profile Pages:

-          These WebPages contains Multiple Business Profiles on a single page and there are common patterns in the html structure of those profiles.

-          Our intelligent parser would identify this repeated pattern and parse the profiles accordingly.

-          There is a big possibility that, some invalid data would be parse as business profiles, but our profile validator will discard them.

Single Profile Pages:

-          This case is a little harder than the previous case.

-          But, the approach is almost similar to the previous one.

-          In maximum business directories, different Single-Profile-Pages maintain the same pattern.

Currently we are focusing on the first case (Multiple-Profile Page). We are using selected pages from our crawled data.

Challenges:

The main challenge is to define and identify the pattern. Most of the html pages are semi-structured. So, there is no universal pattern that works for each one.

Here are some sample patterns:

Figure 1:

This is a part of the html, from a multiple-profile page. The highlighted (yellow) region contains muiltiple business profiles.

Figure 2:

In this html, each profile is contained in the <div class=”listing”> tag.

Figure 3:

This is a more complex pattern. Here, each of the 3 consecutive (highlighted) <tr> tags contains a single profile.

Figure 4:

The previous examples imply that, identifying repeated patterns allows us to parse multiple profiles. But, this figure shows a complex counter-example. Here, the pattern in the yellow region is repeated, but they don’t contain any business profiles. All the business profiles are contained in the <tr> tag (marked with a red rectangle).  The next image is the expansion of this <tr> tag.

Figure 5:

This is the expansion of the <tr> tag (marked with red rectangle) from the previous example (Figure 4). Each of the table in the yellow region contains a business profile.

Figure 6:

This image shows the expansion of the <table> tags from the previous example (Figure 5).

Figure 7:  This is the expansion of the 3 consecutive <tr> tags from the figure 3.

Approach:

After analyzing many business directory websites, we have found some possible techniques to identify the profile patterns.

Each business profile contains multiple entries (Company Name, Email, Address etc.). Each of these entries has a unique html tag-chain. Depending on these tag-chains we can identify the possible data segments in the following way:

-          At first, we have to identify the chain that signifies the start of a profile segment. We call it, the main-chain.

-          Initially, we considered the most frequent chain as the main-chain. But, this heuristic fails for some cases.

-          So, we consider all the chains in the non-increasing order of frequency and check for the validity of the result. But, the validity checking has many hard challenges that are part of Natural Language Processing.

Example:

Here is an example of a main-chain for a webpage:

Most Frequent Chain:   <tr>||<td width="64" valign="top">||</td>||<td width="102"                                                                                                valign="top">||</td>||<td width="368" valign="top">||</td>||</tr>||

Frequency:                         167

Using this chain we could parse profiles successfully. But, this is not always possible.

Counter Example:

Here are 2 of the tag-chains from the Webpage of figure 4:

Chain 1:                <tr>||<td width="100" align="right" class="SideMenuLink1">||</td>||<td class="SideMenuLink1">||</td>||</tr>||

Frequency:         80

Chain 2:                <tr>||<td colspan="2" class="SideMenuTitle">||</td>||</tr>||

Frequency:         31

Here, chain 2 is the main-chain, which is not the most-frequent chain.

Figure 8:

This is an example of a simple html-parse-tree. Every vertex represents a single html-tag. The red-vertices contain the profile-data. The path from the root to a leaf is the tag-chain that we previously described. But, most of the html-parse-trees are more complex.

But we are not going to describe those complex cases which has been skiped by us.