+ Anchor Text Identification from web pages or (from our web crawler data)

Anchor Text Identification

Anchor Text

The anchor text is the visible, clickable text in a hyperlink. The words contained in the anchor text can determine the ranking that the page will receive by search engines.  Some web browsers have added the ability to show a tooltip for a hyperlink before it is selected. Not all links have anchor texts because it may be obvious where the link will lead due to the context in which it is used. Anchor texts normally remain below 60 characters. Different browsers will display anchor texts differently.

Anchor text usually gives the user relevant descriptive or contextual information about the content of the link's destination. The anchor text may or may not be related to the actual text of the URL of the link. For example, a hyperlink with anchor text might take this form:

<a href="http://www.adventureescapades.co.za/activity-hanggliding"> Hang Gliding</a>

The anchor text in this example is “Hang Gliding"; the unwieldy URL http://www.adventureescapades.co.za/activity-hanggliding displays on the web page as Hang Gliding, contributing to clean, easy-to-read text

Why We Need Anchor Text?

For Single profile parser in Business Search, The Company name is the anchor text. So we need anchor text for Company name. Anchor Text is same as the company name for single profile as shown in figure 1.

Figure 1: Sample Web Page

When we click any url in the above figure, it goes a single 

profile. Figure 2, shows the profile of first url and it’s Anchor

 Text formation is: <ahref="compdet_company.asp?screen=Show&amp;iC_ID=27337" 

target="_parent">Altech Netstar Fleet Management Services (Pty) Ltd</a>

Figure 2: Sample Profile page

Architecture

System architecture of Anchor Text identifier is shown in Figure 3.

Figure 3: System Architecture of Anchor Text

Approach

Input: URLs

Output: Anchor Text of corresponding URL

Anchor Text exists in previous depth of the page, containing profile.

If the URL contains n-depth then

            Anchor Text will be found in the (n-1) depth’s pages of current HOST.

Directory Sorted by PHONE

For DataFoundURL info we need DirectorySortedByPhone. Sample information of Directory Sorted By Phone is shown in figure 4.

Figure 4: Directory Sorted By PHONE

Information of DataFoundURL is shown in figure 5.

Figure 5: DataFoundURL

In figure 5, marked areas are how many phone found and corresponding data location of that URL. If the Phone number is 1, we can say that, it is a single profile.

Depth Information

Depth information is found in Crawled Data. That is stored in HostData\WebSite\LinkDB. It’s format is shown in figure 6.

Figure 6: Download Link Path

F:\HostData/Website/content/www.100hills.kzn.org.za means location

2.html means page id

CRAWLER_DEPTH=1 means current depth

1 means index number of content

2261859 start byte of url

2261904 end byte of url

2261905 start byte of content

2347596 end byte of content

1 means igonore or not

Code for Relative URL Maker

For generating absolute url from relative url and host, we need some rules. The code part for generate absolute url is shown in figure 7 and 8.

Figure 7: Making Relative URL (Cont…)

Figure 8: Making Relative URL

Result of Anchor Text

Screen shot of sample output of Anchor Text is shown in Figure 9. Here selected part of Figure 9 is showing necessary information of one anchor text. <OUT_DEGREE> means the hyperlink found in a web page, <URL> means the address of current page, <ANCHOR_TEXT> is the anchor Text of <OUT_DEGREE>.

Figure 9: Result of Anchor Text

Index of Anchor Container

Figure 10 shows index of Anchor Container.

Figure 10a: Index of Anchor Container

Figure 10b: index of Anchor Container

Comparison

Matching found-URL in anchor-container with DataFoundURL.xml, and the data found url comes from directory detector. 

Challenges

Identifying Anchor test is a challenging task. Here Anchor tag start with <a and Hyperlink attributes start with href.  For example href=”http://www.adventureescapades.co.za/activity-airevents”. But there are lot of variations in right hand side of href, likes:

href=”http://www.adventureescapades.co.za/activity-airevents”

href=’http://www.adventureescapades.co.za/activity-airevents’

href=http://www.adventureescapades.co.za/activity-airevents

href= http://www.adventureescapades.co.za/activity-airevent’s

Different pages have different style of syntax for href. Some of the time one’s definition conflict with others, like:

href=’http://www.adventureescapades.co.za/activity-airevents’

here url start and end with ’. But there are some definition of href:

 href= http://www.adventureescapades.co.za/activity-airevent’s

where ‘ is not end marker.

There are also scripts for generating url in web pages and it is produced run-time. For example:

BannerTag.href  = Btargets[curBanner]

That is solved by omitting scripts and stylesheet.

In many cases, the hyperlink does not contain full url, it may contain relative url. For example:

href=”/ search/Bearings”

Finding absolute url from relative url is one of the most challenging task for anchor text identification.

There are also lots of variations in relative urls. But we have a general rules for finding absolute url as like Web Crawler. Sample Code is shown in figure 7 and 8.

Done So Far:

There are lots of unknown rules for hyperlink. We are following a “trail and error” basis to making absolute url and identifying anchor text.

+ Business Profile extraction from webpages or business directory report second (Intelligent Profile Parser (Multiple-Profiles-Page)(Front End))

Intelligent Profile Parser (Multiple-Profiles-Page)(Front End)

There are two parts in the design of the Intelligent Multiple-Profile Parser:

1)     Back End:

Input:

HTML files with the possibility of containing semi-structured multiple business profiles.

Output:

1)     An Analysis Statistics: The Back End analyzes the input html and identifies the html-tag-chains and their frequencies. (html-tag-chain is the tag-path from the root (<html> tag) to the container (<td>, <div>  etc) of the text)

2)     A more structured intermediate file: This file contains all the, possible business data as separate entries. They are grouped with their corresponding html-tag-chain.

 The tasks of the back end can be summarized as follows:

-          Get rid of unwanted text.

-          Impose a structure upon the semi-structured or unstructured text.

-          Detect relation among the texts.

-          Identify the importance of the texts.

-          Parse the html and classify the texts in groups according to different html-tag-chain.  

-          Generate the intermediate file that will be parsed by the Front End.

2)     Front End:

Input:

1)     The analysis statistics.

2)     The intermediate file.

Output:

Structured xml file with formatted business profiles.

            The steps of the Front End can be summarized as follows:

a)      From the analysis report, process each chain one by one and for each chain:

1)     From the intermediate file, parse the texts bounded by that particular chain.

2)     Detect the entity types (Company, Phone, Cell, email, host, Unknown_data etc.) for each entry.

3)     Run a linguistic analysis process on the unknown_data entries to detect the address entry, if the address was not detected in step 2.

4)     Run a linguistic analysis process on the unknown_data entries to detect the description entry, if it was not detected in step 2.

5)     Output each profile after converting to xml.

b)     Detect the correct chain that contains the business profiles.

c)      Validate the profiles.

d)     Convert the profiles in xml format.

Figure 1:  A sample webpage segment, containing multiple business profiles.

Figure 2: This is the html code of the sample html page from figure 1. Each single <tr> tag in the highlighted (in yellow) region contains a single business profile.

Figure 3: This is the html expansion of a single <tr> tag (mentioned in Figure 2).

Figure 4: part of the analysis statistics after analyzing the sample html from Figure 1. This is generated by the back end and used by the front end. The highlight (yellow) string is a html-tag-chain that has the highest frequency of 28 (highlighted in red).

Figure 5: This is a part of the intermediate file, generated by the back end. The highlighted (in red rectangle) strings are the instance of a single html-tag-chain that defines the profile segments. The highlighted (in yellow) texts are our target business data. One profile is assumed to be contained between two instances of an html-tag-chain. Detecting the correct html-tag-chain is one of the challenges we have.

Figure 6: This is an example of the output, produced by the back end after processing the intermediate file (Figure 5). The profiles are highlighted in yellow rectangles. These profiles are parsed with the highlighted chain (in red rectangles). Detecting the company name, address and description are some of the challenging parts in our R&D. We are using some linguistic analysis to detect them, which need to be improved significantly.

Figure 7: This is a more complex example of a business directory webpage. The profiles are highlighted in red rectangle.

Figure 8: This is the html of the page from Figure 7. Each <table> (highlighted in red) contains a business profile. The yellow region is the expansion of the first <table>.

Figure 9: This is the analysis statistics produced by the back end after analyzing the webpage from Figure 7. Here, all the red tags produce the same profiles. Sometimes, the same profile is parsed in two different forms for two different chains. So, it becomes challenging to identify the correct profiles. The yellow tag is the tag with the highest frequency but it doesn’t contain any profiles.

Figure 10: This is the intermediate file generated by the back end, after analyzing the html of Figure 7.

Figure 11: Business Profile output for the html of figure 7.

Figure 12: This is an example of an webpage that contains human-profiles. It is challenging to distinguish them from business-profiles.

Figure 10: This is the profile output for the page from Figure 9. Note that, the name of the human are treated as company-name. Hopefully, we have considered this problem in our Research and made some important progress which would be implemented in future.

Figure 11: This is an example of a page with a structure that challenges our Intelligent Profile Parser. Note that, on each row, the first entry is an address and the second entry is a company name. Sometimes in these cases, it becomes hard to distinguish between the address and the company name. In our parser, we are assuming (so far) that, there would be no repeated company name in a single page. This assumption leads us to ignore the unknown repeated strings. In that case, some addresses are ignored and some addresses gets considered as company name. This problem can be solved by some complex analysis of the linguistic features and html structure.

Figure 12: This is the html of the page from figure 11. Each of the yellow <tr> tag contains a single profile.

Figure 13: This is the intermediate file for the page of figure 11. The yellow texts are all addresses and the texts in red rectangles are company name.

Figure 14: These are the business profiles extracted from the page of Figure 11. The company names in yellow rectangles are correctly identified because we ignored the string “Alberton” before them because of its multiple occurrences. But, the company names in red rectangles are actually addresses. They were not repeated. This problem will be solved in future.

Figure 15: This page is a sample multiple-profile-page. The texts highlighted in red are the company name entries. The texts highlighted in yellow are description entries.

Figure 16: These are the output profiles parsed by the parser from the page in Figure 15. Here all the entries are correctly identified. The description-detection part is still in the research phase, so they are skipped. These profiles are parsed with the tag-chain (enclosed in red rectangle).

 

Figure 17: These are some incorrectly parsed profiles from the page in Figure 15. Here, the description entries are detected as company name (highlighted in blue rectangles). Detecting and discarding these profiles is a major challenge in our R&D. This problem would be solved by the Style-Sheet-Analysis that we shall start in future.

Figure 18: In this profile, the company name contains both the name of the company (highlighted in yellow) and the description/category of the company (highlighted in green). We shall try to separate them in two separate entries in our future research.