Scrapebox Dupe Remove Addon - Mass remove duplicate urls up to 180 million lines
The scrapebox dupe remove addon lets you remove duplicate urls and domains for large files. http://www.scrapestuff.com/dupe-remove-addon-for-scrapebox-remove-duplicate-urls-in-large-quantities You can work with files up to 180 million lines. The addon allows you to merge files together, split them and remove duplicate urls and domains. This is helpful since you can harvest millions of urls with scrapebox. In which case they are stored in the harvester sessions folder. Then you simply have to filter the files with this addon. Once your done you can split it down to file sizes you can work with in scrapebox.