Visual Web Ripper: Using External Input Data Sources

Sometimes it is necessary to use external data sources to provide parameters for the scraping process. For example, you have a database with a bunch of ASINs and you need to scrape all product information for each one of them. As far as Visual Web Ripper is concerned, an input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values.

An input data source is normally used in one of these scenarios:

    To provide a list of input values for a web form
    To provide a list of start URLs
    To provide input values for Fixed Value elements
    To provide input values for scripts

Visual Web Ripper supports the following input data sources:

    SQL Server Database
    MySQL Database
    OleDB Database
    CSV File
    Script (A script can be used to provide data from almost any data source)

To see it in action you can download a sample project that uses an input CSV file with Amazon ASIN codes to generate Amazon start URLs and extract some product data. Place both the project file and the input CSV file in the default Visual Web Ripper project folder (My Documents\Visual Web Ripper\Projects).

For further information please look at the manual topic, explaining how to use an input data source to generate start URLs.


Source: http://extract-web-data.com/visual-web-ripper-using-external-input-data-sources/

Using External Input Data in Off-the-shelf Web Scrapers

There is a question I’ve wanted to shed some light upon for a long time already: “What if I need to scrape several URL’s based on data in some external database?“.

For example, recently one of our visitors asked a very good question (thanks, Ed):

    “I have a large list of amazon.com asin. I would like to scrape 10 or so fields for each asin. Is there any web scraping software available that can read each asin from a database and form the destination url to be scraped like http://www.amazon.com/gp/product/{asin} and scrape the data?”

This question impelled me to investigate this matter. I contacted several web scraper developers, and they kindly provided me with detailed answers that allowed me to bring the following summary to your attention:
Visual Web Ripper

An input data source can be used to provide a list of input values to a data extraction project. A data extraction project will be run once for each row of input values. You can find the additional information here.
Web Content Extractor

You can use the -at”filename” command line option to add new URLs from TXT or CSV file:

    WCExtractor.exe projectfile -at”filename” -s

projectfile: the file name of the project (*.wcepr) to open.
filename – the file name of the CSV or TXT file that contains URLs separated by newlines.
-s – starts the extraction process

You can find some options and examples here.
Mozenda

Since Mozenda is cloud-based, the external data needs to be loaded up into the user’s Mozenda account. That data can then be easily used as part of the data extracting process. You can construct URLs, search for strings that match your inputs, or carry through several data fields from an input collection and add data to it as part of your output. The easiest way to get input data from an external source is to use the API to populate data into a Mozenda collection (in the user’s account). You can also input data in the Mozenda web console by importing a .csv file or importing one through our agent building tool.

Once the data is loaded into the cloud, you simply initiate building a Mozenda web agent and refer to that Data list. By using the Load page action and the variable from the inputs, you can construct a URL like http://www.amazon.com/gp/product/%asin%.
Helium Scraper

Here is a video showing how to do this with Helium Scraper:


The video shows how to use the input data as URLs and as search terms. There are many other ways you could use this data, way too many to fit in a video. Also, if you know SQL, you could run a query to get the data directly from an external MS Access database like
SELECT * FROM [MyTable] IN "C:\MyDatabase.mdb"

Note that the database needs to be a “.mdb” file.
WebSundew Data Extractor
Basically this allows using input data from external data sources. This may be CSV, Excel file or a Database (MySQL, MSSQL, etc). Here you can see how to do this in the case of an external file, but you can do it with a database in a similar way (you just need to write an SQL script that returns the necessary data).
In addition to passing URLs from the external sources you can pass other input parameters as well (input fields, for example).
Screen Scraper

Screen Scraper is really designed to be interoperable with all sorts of databases. We have composed a separate article where you can find a tutorial and a sample project about scraping Amazon products based on a list of their ASINs.


Source: http://extract-web-data.com/using-external-input-data-in-off-the-shelf-web-scrapers/

Microsys A1 Website Scraper Review

The A1 scraper by Microsys is a program that is mainly used to scrape websites to extract data in large quantities for later use in webservices. The scraper works to extract text, URLs etc., using multiple Regexes and saving the output into a CSV file. This tool is can be compared with other web harvesting and web scraping services.
How it works
This scraper program works as follows:
Scan mode

    Go to the ScanWebsite tab and enter the site’s URL into the Path subtab.
    Press the ‘Start scan‘ button to cause the crawler to find text, links and other data on this website and cache them.



Important: URLs that you scrape data from have to pass filters defined in both analysis filters and output filters. The defining of those filters can be set at the Analysis filters and Output filters subtabs respectively. They must be set at the website analysis stage (mode).
Extract mode

    Go to the Scraper Options tab
    Enter the Regex(es) into the Regex input area.
    Define the name and path of the output CSV file.
    The scraper automatically finds and extracts the data according to Regex patterns.

The result will be stored in one CSV file for all the given URLs.

There is a need to mention that the set of regular expressions will be run against all the pages scraped.
Some more scraper features

Using the scraper as a website crawler also affords:

    URL filtering.
    Adjustment of the speed of crawling according to service needs rather than server load.

If  you need to extract data from a complex website, just disable Easy mode: out press the  button. A1 Scraper’s full tutorial is available here.
Conclusion

The A1 Scraper is good for mass gathering of URLs, text, etc., with multiple conditions set. However this scraping tool is designed for using only Regex expressions, which can increase the parsing process time greatly.



Source: http://extract-web-data.com/microsys-a1-website-scraper-review/

Distil Review: Anti-Scrape-Bot Service

Are you thinking of protecting your website content from theft and nonlegal scraping? Are you suspecting that some ‘innocent bots’ are continually visiting your web pages for data retrieval? Now we come to the anti scraping bot software and services. In this post we want to briefly review the new anti scrape bot service called Distil.

Distil started out helping digital publishers to protect their content from scrape, and now it claims an increase in scope resulting in it becoming a global service in this niche. Yes, illegal scraping creates the problem of data leakage and content farming. The latter is very much wrapped up with web publishing, but other industries might also need to be protected from data theft. These are E-commerce, Airline and Housing industry, Classifieds and others.
Distil is devoted to real-time scrape bot protecting, thus guarding multiple users with multiple domains. Once you plug in your domains, the service goes on alert to monitor HTTP traffic and detect both positive and negative visitors. Once some malicious bot is detected Distil may block it or put out a captcha or drop the bot request. The service allows you to manage as few as 5 domains up to an unlimited number with the Enterprise plan. You can read about other plan features here.

In the next post we review the results of the live-testing that we put Distil to.
Set up

After registering, you’ll get an email with your username/email and password. You should also read the instructions at the bottom of the letter for placing the domain under protection from scrapers… but initially I could not find the stated side panel item. So, as you login, at the left side panel go to Configuration -> Edit Subdomains and Server IP and follow the instructions. (Ensure that the IP address in the “Origin Server IP” field is the correct IP address for your server. After that you just do the changes in A and CNAME records of your domain with your DNS provider as stated in the email.)  At this config option you might also add subdomains for bot protection.
Protection

Distil uses a very professional approach to scrape bot protection. It works to profile the guarded site with date info, bandwidth load and geo access. This makes it possible to develop custom profiles for the business site under guard and thus sift off all suspected malicious access.

If your classified web portal gets loaded (not crushed :-) ) on a Friday night all the way until Monday morning with millions of queries (even from various IP addresses), why not suspect some scrapers’ assault? Distil’s strategy involves a layer of data mining and machine learning to effectively protect the websites.

The service tracks the users’ pages-per-minute speed, session length and pages per session for bot analysis. The detected scrapers are stored in a database, and Distil bans these from each of its clients’ sites.
Some Bot Protection features applied

    Uses statistics for bot behavior detection
    Good Bot, Bad Bot and browser’s identity checks (tries to execute JavaScript test to check if an actual browser is hitting your site). As we know, scrapers can leverage Selenium to automate innocent browsers for web scraping.
    Sets time thresholds for pages and maximum session length
    Tracks usage patterns by individual users to better highlight bots
    Blocks access to the protected site from some world segments.
    SSL Encryption

Performance change while being protected

Distil joins your domains to the cloud network, Content Delivery Network; thus they claim you’ll maintain and even get better site traffic performance while being in the scrape protection mode. With this system your site’s content or data get spread (cashed) geographically over multiple data centers for users’ convenience. They claim even to provide backup datacenter support (about 15 units worldwide) when your primary center goes offline (looks like a global backup server).

The service allows dynamic content caching to store frequently accessed content for increased performance. Automatically compressing content for faster delivery is another feature for maintaining performance. No doubt such options are because the startup is partnered with one of the worldwide internet infrastructure leader (Dyn).
Market opportunities

The anti-bot market related to web data scraping problems seems not to have been very developed until now. So the advance of the startup largely depends on if it can kindle companies’ interest in this anti scrape bot service. Some advice was recently given to Distil CEO Rami Essaid : “…These markets are very new and require quite a bit of education for your potential clients” by Elana Fine, Dingman Center for Entrepreneurship (source). Essaid’s reply: “We’ve been educating potential clients on the danger of bots through such activities as sending informative e-mails and holding educational webinars.” So market growth will provide more opportunities for this company and other similar services.




Source: http://extract-web-data.com/distil-review-anti-scrape-bot-service/

Outsourcing Data Entry Services

Data or raw information is the backbone of any industry or business organization. However, raw data is seldom useful in its pure form. For it to be of any use, data has to be recorded properly and organized in a particular manner. Only then can data be processed. That is why it is important to ensure accurate data entry. But because of the unwieldy nature of data, feeding data is a repetitive and cumbersome job and it requires heavy investment, both in terms of time and energy from staff. At the same time, it does not require a high level of technical expertise. Due to these factors, data entry can safely be outsourced, enabling companies to devote their time and energy on tasks that enhance their core competence.

Many companies, big and small, are therefore enhancing their productivity by outsourcing the endless monotonous tasks that tend to cut down the organization's productivity. In times to come, outsourcing these services will become the norm and the volume of work that is outsourced will multiply. The main reason for these kinds of development is the Internet. Web based customer service and instant client support has made it possible for service providers to act as one stop business process outsourcing partners to parent companies that require support.

Data entry services are not all alike. Different clients have different demands. While some clients may require recording information coupled with document management and research, others may require additional services like form processing or litigation support. Data entry itself could be from various sources. For instances, sometimes information may need to be typed out from existing documents while at other times, data needs to be extracted from images or scanned documents. To rise up to these challenges, service providers who offer these services must have the expertise and the software to ensure rapid and accurate data entry. That is why it is important to choose your service provider with a lot of care.

Before hiring your outsourcing partner, you need to ask yourself the following questions.

* What kind of reputation does the company enjoy? Do they have sufficient years of experience? What kind of history and background does the company enjoy?

* Do they have a local management arm that you can liaise with on a regular basis?

* Do the service personnel understand your requirements and can they handle them effectively?

* What are the steps taken by the company to ensure that there is absolutely no compromise in confidentiality and security while dealing with vital confidential data?

* Is there a guarantee in place?

* What about client references?

The answers to these questions will help you identify the right partner for outsourcing your data entry service requirements.




Source: http://ezinearticles.com/?Outsourcing-Data-Entry-Services&id=3568373

Time Saving and Money Saving Data Entry Services

If you have an organization than data-entry is definitely the section with which you have to deal. The main concern for any organization which hires data entry services is flexibility and value for money. People need services which provide fast accurate entry of any form of hand-written data.

Data entry is very straight forward work but requires enough man force. As a result, many companies prefer to outsource data entry services to offshore countries. Company just have to find reliable data-entry partner from offshore countries which provides accurate data-entry services at most affordable prices.

As competition grows, many data-entry firms from offshore countries gives the most competitive prices for data-entry services. Outsourcing is not a new concept and having vast market doing outsourcing work. If you are looking for outsourcing data-entry work than India is the best outsourcing destination.

Many firms in India has enough experience with data entry projects which gives the best possible data-entry solutions from advanced data-entry tools. Daily, number of companies wants to move their paper documents into electronic format. All these firms in offshore countries give data entry services from qualified and well trained data-entry professionals. Their experienced and professional team of data-entry is highly trained in handling and obtaining large quantities of data in the minimal time possible. Outsourcing data entry and document processing work will save your valuable time and money. Utilizing this time and money you will be able to concentrate on your more important parts of your business leads you to high profit in best time.

Effective policies leads business to continue progress and survive them in today's highly competitive market. As in many cases, non-core activities are creating headaches in the path of progress, it is also an essential to finish them accurately as they provide assistance to core business.

So with choosing outsourcing less important data-entry work as a business strategy, allow you to create more attention on your core business activities.

Author is related with offshore data entry services providing firm ServicesDataEntry.co.uk. Firm outsource data entry services like online data entry services and many more.




Source: http://ezinearticles.com/?Time-Saving-and-Money-Saving-Data-Entry-Services&id=2908114

Preference to Offshore Document Data Entry Services

A number or business organizations if different industries are seeking competent and precise document data entry services to maintain their business records safe for future references. Document data entry has advanced as a quickly developing and active industry structure almost accept in all major companies of the world. The companies doing businesses these days are undergoing rapid changes and therefore the need for services is becoming all the more crucial.

To get success you need to accomplish more understanding about the market, your business, clients as well as the prevailing factors that influence your business. A considerable amount of document is in one or the other way included in this entire process. These services is helpful in taking crucial decisions for the organization. It also provides you a standard in understanding the current and future business status of your company.

In this information age data-entry from documents and data conversion have become important elements for most business houses. The requirement for document services has reached zenith since companies work on processes like business merger and acquisitions, as well as new technology developments. In such scenarios having access to the right kind of data at the right time is very crucial and that is why companies opt for reliable services.

These services covers a range of professional business oriented activities such as document plus image processing to image editing as well as catalog processing. A few noteworthy examples of from documents include: PDF document indexing, insurance claim entry, online data capture as well as creating new databases. These services are important in industries like insurance companies, banks, government departments and airlines.

Companies such as Offshore and outsource and others offer an entire gamut of first rate data services. Actually, getting services from documents offshore to developing yet competent countries like India has made the process highly economical plus quality driven too.

Business giants around the world have realized multiple advantages associated in Offshore-Data-Entry. Companies not only prosper because of quality services but are also benefited because of better turn around time, maintaining confidentiality of data as well as economic rates.

Though the company works in all form of documents, there are few below mentioned areas where it specializes:

• Document data entry
• Document data entry conversion
• Document data processing
• Document data capture services
• Web data extraction
• Document scanning indexing

Since reputable companies like Offshore Data-Entry hire only well qualified and trained candidates work satisfaction is guaranteed. There are several steps involved in the quality check (QC) process and therefore accuracy level is maintained to 99.995% ensuring that the end result is delivered to the client far beyond his expectation.

With the amount of talent that India has outsourcing your data from documents is certainly an intelligent step. Visit our site: http://www.offshoredataentry.com, and drop us an email through a contact us feature and we will get back to you for your assistance.





Source: http://ezinearticles.com/?Preference-to-Offshore-Document-Data-Entry-Services&id=5570327