Crawler-Based System Makes Sure all URLs are Found! Query String Values Still work!

• Finds hidden pages!  
• Ability to control Friendly URLs and SEO after finding each blog entry, articles, ecommerce products, events, etc. 
• Finds static pages and files. Provides SEO and friendly URL control of PDF Files, Excel files, including static images, Word files, static HTML files, etc. -- anything that is linked from your site. 
• Find and identify duplicate pages. Remove the SEO concern of duplicate pages by identifying URLs as a duplicate of another URL. 
• Designed as an "add-on" system. DNN core URLs still work including query string values for dynamic content. Friendly URLs in the system still take query string values into consideration with the system by creating a sophisticated redirect system. 

Global Controls for Setting Defaults

• Ability to include any url, group of urls, or the entire site in an XML sitemap with default change freq and page priority, then ability to modify at the page level 
• Ability to include any url, group of urls, or the entire site in a text sitemap 
• Ability to include any url, group of urls, or the entire site in a robot.txt file to have the search engines ignore what you want them to ignore 
• Ability to "commit" any or all of the site in advance so that when the crawler finds new urls, it approves them without an administrator's approval. This can be changed so that any new urls have to be approved/committed.  
• System provides a "friendly" url based on the title of the page crawled. This can be over-ridden for any url or group of urls by choosing to use the default DNN url or choosing a url manually and having it be the "default"  
• Changes at a global or parent level can be cascaded down to urls beneath them or retain the saved state at the individual level 
• Ability to set the timing for when the crawler will run. Alternately, the administrator can dig into the schedule tasks and create more sophisticated scheduled tasks. 

Individual URL and SEO Management

• Filter what group of urls to work on and sort by site structure, not reviewed yet, and most recently added urls by the crawler  
• Ability to commit or approve urls at a group or individual level 
• Ability to add as many friendly urls and 301 re-directs as wanted. Any of these can be made the "default". 
• The DNN url is always maintained in the background so that any query string parameters still work within the page.  
• SEO Title, Keywords, and Description are stored from the crawler. The system provides the ability to make modifications that are appropriate for the page after the crawler runs. 
• Hidden pages (not found by the crawler, but still part of the available urls on the site) will be highlighted. These pages will have all the same privileges as any other page.