Lynx browser. Creating sitemap.xml
There are more than few online services for sitemap.xml generation. But it is also possible to do it yourself, by means of lynx web browser and several Linux command line utilities. An example bash script employing them, named sitemap.sh is described below.
Bash script creating a sitemap.xml file:
#!/bin/bash
cd /home/me/sitemap/www/
lynx -crawl -traversal -accept_all_cookies -connect_timeout=30 http://www.compmiscellanea.com/ > /dev/null
cd /home/me/sitemap/www2/
lynx -crawl -traversal -accept_all_cookies -connect_timeout=30 http://compmiscellanea.com/ > /dev/null
cat /home/me/sitemap/www2/traverse.dat >> /home/me/sitemap/www/traverse.dat
cat /home/me/sitemap/www/traverse.dat | sed -e 's/\<www\>\.//g' | sort | uniq > /home/me/sitemap/sitemap/sitemap.xml
sed -i 's/\&/\&\;/g' /home/me/sitemap/sitemap/sitemap.xml
sed -i "s/'/\&apos\;/g" /home/me/sitemap/sitemap/sitemap.xml
sed -i 's/"/\"\;/g' /home/me/sitemap/sitemap/sitemap.xml
sed -i 's/>/\>\;/g' /home/me/sitemap/sitemap/sitemap.xml
sed -i 's/</\<\;/g' /home/me/sitemap/sitemap/sitemap.xml
sed -i 's/http:\/\//http:\/\/www\./g' /home/me/sitemap/sitemap/sitemap.xml
sed -i -e 's/^/<url><loc>/' /home/me/sitemap/sitemap/sitemap.xml
sed -i -e 's/$/<\/loc><\/url>/' /home/me/sitemap/sitemap/sitemap.xml
sed -i -e '1 i <?xml version="1\.0" encoding="UTF-8"?>\r\r<urlset xmlns="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9" xmlns:xsi="http:\/\/www\.w3\.org\/2001\/XMLSchema-instance" xsi:schemaLocation="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9 http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9\/sitemap\.xsd">\r\r<!-- created by sitemap.sh from http:\/\/www.compmiscellanea.com\/en\/lynx-browser-creating-sitemap.xml\.htm -->\r\r' /home/me/sitemap/sitemap/sitemap.xml
sed -i -e '$ a \\r</urlset>' /home/me/sitemap/sitemap/sitemap.xml
sed -i '/static/d' /home/me/sitemap/sitemap/sitemap.xml
echo "...Done"
After the bash script file is prepared: chmod +x sitemap.sh to make it executable.
Download sitemap.sh in sitemap.sh.tar.gz archive. After downloading and unpacking it, put a web site name with "www" instead of http://www.compmiscellanea.com/ and a web site name without "www" instead of http://compmiscellanea.com/ in the file. Replace "static" in the last line of the file by a string unnecessary links should possess to be removed. Then chmod +x sitemap.sh. Then run sitemap.sh.
Commentary
Download sitemap2.sh with line by line commentary in sitemap2.sh.tar.gz archive.
Before running the bash script, three folders should be created. Since lynx browser may miss some links if a web site domain name to be crawled is put with or without "www", bash script runs lynx twice, crawling the web site by its name with "www" and crawling the web site by its name without "www".
The two result files are put into two of these separate folders, here they are /home/me/sitemap/www/ and /home/me/sitemap/www2/. Folder /home/me/sitemap/sitemap/ is for sitemap.xml created in the end.
1. Path to bash:
#!/bin/bash
2. Going to a folder. Lynx is going to put there the files obtained from crawling a web site with "www" in its name:
cd /home/me/sitemap/www/
3. Running lynx browser to crawl a web site. Since some links may be missed by lynx if the domain name of the web site to be crawled is put with or without "www", bash script runs lynx browser twice, crawling the web site by its name with "www" and crawling the web site by its name without "www". Here it is with "www".
lynx will automatically go through all the pages and the links on them. All cookies are to be accepted. An amount of time lynx is to try to connect following each link may be set in seconds by the -connect_timeout option:
lynx -crawl -traversal -accept_all_cookies -connect_timeout=30 http://www.compmiscellanea.com/ > /dev/null
4. Going to another folder - lynx is going to put there the files obtained from crawling the web site without "www" in its name:
cd /home/me/sitemap/www2/
5. Running lynx browser to crawl a web site. Since some links may be missed by lynx if the domain name of the web site to be crawled is put with or without "www", bash script runs lynx browser twice, crawling the web site by its name with "www" and crawling the web site by its name without "www". Here it is without "www".
Lynx will automatically go through all the pages and the links on them. All cookies are to be accepted. An amount of time lynx is to try to connect following each link may be set in seconds by the -connect_timeout option:
lynx -crawl -traversal -accept_all_cookies -connect_timeout=30 http://compmiscellanea.com/ > /dev/null
6. Running lynx browser twice, crawling the web site by its name with "www" and crawling the web site by its name without "www", creates two files with the links collected. So here the content of the second file is added to the end of the first one:
cat /home/me/sitemap/www2/traverse.dat >> /home/me/sitemap/www/traverse.dat
7. Links gathered by lynx crawling the web site by its name without "www" have no "www." in the URLs, so to make links collection uniform, the rest of links are stripped from "www.". Then sorted alphabetically by sort. Then uniq removes duplicate entries. Then the result is written into a file named "sitemap.xml" created in the process:
cat /home/me/sitemap/www/traverse.dat | sed -e 's/\<www\>\.//g' | sort | uniq > /home/me/sitemap/sitemap/sitemap.xml
8. If there are &, ', ", >, < in URLs, they are to be replaced by &, ', ", >, <. Other special and non-ASCII characters are supposed to be made compliant with the current sitemap.xml file standards { 1 } and common practice { 2 } by the web site's developers or its CMS.
Otherwise lynx is going to attempt to understand the URLs according to its rules and abilities, to try and read them, then write them to traverse.dat. Depending on the environment lynx is run in, sometimes it will be more or less successful, sometimes more or less not.
So, & is replaced by &
sed -i 's/\&/\&\;/g' /home/me/sitemap/sitemap/sitemap.xml
9. ' is replaced by '
sed -i "s/'/\&apos\;/g" /home/me/sitemap/sitemap/sitemap.xml
10. " is replaced by "
sed -i 's/"/\"\;/g' /home/me/sitemap/sitemap/sitemap.xml
11. > is replaced by >
sed -i 's/>/\>\;/g' /home/me/sitemap/sitemap/sitemap.xml
12. < is replaced by <
sed -i 's/</\<\;/g' /home/me/sitemap/sitemap/sitemap.xml
13. www. is added to all the links:
sed -i 's/http:\/\//http:\/\/www\./g' /home/me/sitemap/sitemap/sitemap.xml
14. <url><loc> is added before every line:
sed -i -e 's/^/<url><loc>/' /home/me/sitemap/sitemap/sitemap.xml
15. </url></loc> is added after every line:
sed -i -e 's/$/<\/loc><\/url>/' /home/me/sitemap/sitemap/sitemap.xml
16. Opening tags of XML document and a comment are added before the content of the file:
sed -i -e '1 i <?xml version="1\.0" encoding="UTF-8"?>\r\r<urlset xmlns="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9" xmlns:xsi="http:\/\/www\.w3\.org\/2001\/XMLSchema-instance" xsi:schemaLocation="http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9 http:\/\/www\.sitemaps\.org\/schemas\/sitemap\/0\.9\/sitemap\.xsd">\r\r<!-- created by sitemap.sh from http:\/\/www.compmiscellanea.com\/en\/lynx-browser-creating-sitemap.xml\.htm -->\r\r' /home/me/sitemap/sitemap/sitemap.xml
17. Closing tag of XML document is added after the content of the file:
sed -i -e '$ a \\r</urlset>' /home/me/sitemap/sitemap/sitemap.xml
18. Unnecessary links with a given string in them are removed:
sed -i '/static/d' /home/me/sitemap/sitemap/sitemap.xml
19. Reporting the process is completed:
echo "...Done"
Documentation
- Lynx browser docs on -traversal and -crawl switches: CRAWL.announce
Operating systems
- Linux