Iterate endpoints from robots.txt (#471)

* Workflow to extract endpoints from robots.txt - requires projectdiscovery/nuclei/issues/304 fixed to work

Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>

* Correcting path after conducted testing

Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>

* Linting fixes

Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>

* Update of regex after changes to nuclei regex params in extractor

Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>

* Add detection of SEEEMS CMS

Signed-off-by: Casper Guldbech Nielsen <whopsec@protonmail.com>

* Updated template to support "iterate-all: true"

Co-authored-by: sandeep <sandeep@projectdiscovery.io>

miscellaneous/robots-txt-endpoint.yaml

@@ -0,0 +1,26 @@
+id: robots-txt-endpoint
+
+info:
+  name: robots.txt endpoint prober
+  author: CasperGN,pdteam
+  severity: info
+
+requests:
+  - method: GET
+    path:
+      - "{{RootURL}}/robots.txt"
+      - "{{RootURL}}{{endpoint}}"
+
+    extractors:
+      - type: regex
+        name: endpoint
+        group: 1
+        regex:
+          - '(?m:\s(/[[:alpha:]]+[[:graph:]]+))'
+        internal: true
+
+    iterate-all: true
+    matchers:
+      - type: status
+        status:
+          - 200