Files
nuclei/pkg/protocols/javascript/js.go

878 lines
33 KiB
Go
Raw Normal View History

javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
package javascript
import (
"bytes"
"context"
"fmt"
"maps"
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
"net"
"strings"
"sync/atomic"
"time"
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
"github.com/Mzack9999/goja"
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
"github.com/alecthomas/chroma/quick"
"github.com/ditashi/jsbeautifier-go/jsbeautifier"
"github.com/pkg/errors"
"github.com/projectdiscovery/gologger"
"github.com/projectdiscovery/nuclei/v3/pkg/js/compiler"
"github.com/projectdiscovery/nuclei/v3/pkg/js/gojs"
"github.com/projectdiscovery/nuclei/v3/pkg/model"
"github.com/projectdiscovery/nuclei/v3/pkg/operators"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/extractors"
"github.com/projectdiscovery/nuclei/v3/pkg/operators/matchers"
"github.com/projectdiscovery/nuclei/v3/pkg/output"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/contextargs"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/expressions"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/generators"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/helpers/eventcreator"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/interactsh"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/protocolstate"
"github.com/projectdiscovery/nuclei/v3/pkg/protocols/common/utils/vardump"
protocolutils "github.com/projectdiscovery/nuclei/v3/pkg/protocols/utils"
templateTypes "github.com/projectdiscovery/nuclei/v3/pkg/templates/types"
"github.com/projectdiscovery/nuclei/v3/pkg/types"
"github.com/projectdiscovery/utils/errkit"
iputil "github.com/projectdiscovery/utils/ip"
mapsutil "github.com/projectdiscovery/utils/maps"
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
sliceutil "github.com/projectdiscovery/utils/slice"
2024-04-03 17:50:57 +02:00
syncutil "github.com/projectdiscovery/utils/sync"
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
urlutil "github.com/projectdiscovery/utils/url"
)
// Request is a request for the javascript protocol
type Request struct {
// Operators for the current request go here.
operators.Operators `yaml:",inline,omitempty" json:",inline,omitempty"`
CompiledOperators *operators.Operators `yaml:"-" json:"-"`
// description: |
// ID is request id in that protocol
ID string `yaml:"id,omitempty" json:"id,omitempty" jsonschema:"title=id of the request,description=ID is the optional ID of the Request"`
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
// description: |
// Init is javascript code to execute after compiling template and before executing it on any target
// This is helpful for preparing payloads or other setup that maybe required for exploits
Init string `yaml:"init,omitempty" json:"init,omitempty" jsonschema:"title=init javascript code,description=Init is the javascript code to execute after compiling template"`
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
// description: |
// PreCondition is a condition which is evaluated before sending the request.
PreCondition string `yaml:"pre-condition,omitempty" json:"pre-condition,omitempty" jsonschema:"title=pre-condition for the request,description=PreCondition is a condition which is evaluated before sending the request"`
// description: |
// Args contains the arguments to pass to the javascript code.
Args map[string]interface{} `yaml:"args,omitempty" json:"args,omitempty"`
// description: |
// Code contains code to execute for the javascript request.
Code string `yaml:"code,omitempty" json:"code,omitempty" jsonschema:"title=code to execute in javascript,description=Executes inline javascript code for the request"`
// description: |
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
// StopAtFirstMatch stops processing the request at first match.
StopAtFirstMatch bool `yaml:"stop-at-first-match,omitempty" json:"stop-at-first-match,omitempty" jsonschema:"title=stop at first match,description=Stop the execution after a match is found"`
// description: |
// Attack is the type of payload combinations to perform.
//
// Sniper is each payload once, pitchfork combines multiple payload sets and clusterbomb generates
// permutations and combinations for all payloads.
AttackType generators.AttackTypeHolder `yaml:"attack,omitempty" json:"attack,omitempty" jsonschema:"title=attack is the payload combination,description=Attack is the type of payload combinations to perform,enum=sniper,enum=pitchfork,enum=clusterbomb"`
// description: |
// Payload concurreny i.e threads for sending requests.
// examples:
// - name: Send requests using 10 concurrent threads
// value: 10
Threads int `yaml:"threads,omitempty" json:"threads,omitempty" jsonschema:"title=threads for sending requests,description=Threads specifies number of threads to use sending requests. This enables Connection Pooling"`
// description: |
// Payloads contains any payloads for the current request.
//
// Payloads support both key-values combinations where a list
// of payloads is provided, or optionally a single file can also
// be provided as payload which will be read on run-time.
Payloads map[string]interface{} `yaml:"payloads,omitempty" json:"payloads,omitempty" jsonschema:"title=payloads for the webosocket request,description=Payloads contains any payloads for the current request"`
generator *generators.PayloadGenerator
// cache any variables that may be needed for operation.
options *protocols.ExecutorOptions `yaml:"-" json:"-"`
preConditionCompiled *goja.Program `yaml:"-" json:"-"`
scriptCompiled *goja.Program `yaml:"-" json:"-"`
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
// Compile compiles the request generators preparing any requests possible.
func (request *Request) Compile(options *protocols.ExecutorOptions) error {
request.options = options
var err error
if len(request.Payloads) > 0 {
request.generator, err = generators.New(request.Payloads, request.AttackType.Value, request.options.TemplatePath, options.Catalog, options.Options.AttackType, options.Options)
if err != nil {
return errors.Wrap(err, "could not parse payloads")
}
// default to 20 threads for payload requests
request.Threads = options.GetThreadsForNPayloadRequests(request.Requests(), request.Threads)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
if len(request.Matchers) > 0 || len(request.Extractors) > 0 {
compiled := &request.Operators
compiled.ExcludeMatchers = options.ExcludeMatchers
compiled.TemplateID = options.TemplateID
for _, matcher := range compiled.Matchers {
if matcher.Part == "" && matcher.Type.MatcherType != matchers.DSLMatcher {
matcher.Part = "response"
}
}
for _, extractor := range compiled.Extractors {
if extractor.Part == "" {
extractor.Part = "response"
}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if err := compiled.Compile(); err != nil {
return errkit.Newf("could not compile operators got %v", err)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
request.CompiledOperators = compiled
}
// "Port" is a special variable and it should not contains any dsl expressions
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
ports := request.getPorts()
for _, port := range ports {
if strings.Contains(port, "{{") {
return errkit.New("'Port' variable cannot contain any dsl expressions")
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
if request.Init != "" {
// execute init code if any
if request.options.Options.Debug || request.options.Options.DebugRequests {
gologger.Debug().Msgf("[%s] Executing Template Init\n", request.TemplateID)
var highlightFormatter = "terminal256"
if request.options.Options.NoColor {
highlightFormatter = "text"
}
var buff bytes.Buffer
_ = quick.Highlight(&buff, beautifyJavascript(request.Init), "javascript", highlightFormatter, "monokai")
prettyPrint(request.TemplateID, buff.String())
}
opts := &compiler.ExecuteOptions{
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
ExecutionId: request.options.Options.ExecutionId,
TimeoutVariants: request.options.Options.GetTimeouts(),
Source: &request.Init,
Context: context.Background(),
}
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
// register 'export' function to export variables from init code
// these are saved in args and are available in pre-condition and request code
opts.Callback = func(runtime *goja.Runtime) error {
err := gojs.RegisterFuncWithSignature(runtime, gojs.FuncOpts{
Name: "set",
Signatures: []string{
"set(string, interface{})",
},
Description: "set variable from init code. this function is available in init code block only",
FuncDecl: func(varname string, value any) error {
if varname == "" {
return fmt.Errorf("variable name cannot be empty")
}
if value == nil {
return fmt.Errorf("variable value cannot be empty")
}
if request.Args == nil {
request.Args = make(map[string]interface{})
}
request.Args[varname] = value
return nil
},
})
if err != nil {
return err
}
return gojs.RegisterFuncWithSignature(runtime, gojs.FuncOpts{
Name: "updatePayload",
Signatures: []string{
"updatePayload(string, interface{})",
},
Description: "update/override any payload from init code. this function is available in init code block only",
FuncDecl: func(varname string, Value any) error {
if request.Payloads == nil {
request.Payloads = make(map[string]interface{})
}
if request.generator != nil {
request.Payloads[varname] = Value
request.generator, err = generators.New(request.Payloads, request.AttackType.Value, request.options.TemplatePath, options.Catalog, options.Options.AttackType, options.Options)
if err != nil {
return err
}
} else {
return fmt.Errorf("payloads not defined and cannot be updated")
}
return nil
},
})
}
opts.Cleanup = func(runtime *goja.Runtime) {
_ = runtime.GlobalObject().Delete("set")
_ = runtime.GlobalObject().Delete("updatePayload")
}
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
args := compiler.NewExecuteArgs()
allVars := generators.MergeMaps(options.Variables.GetAll(), options.Options.Vars.AsMap(), request.options.Constants)
// proceed with whatever args we have
args.Args, _ = request.evaluateArgs(allVars, options, true)
initCompiled, err := compiler.SourceAutoMode(request.Init, false)
if err != nil {
return errkit.Newf("could not compile init code: %s", err)
}
result, err := request.options.JsCompiler.ExecuteWithOptions(initCompiled, args, opts)
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
if err != nil {
return errkit.Newf("could not execute pre-condition: %s", err)
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
}
if types.ToString(result["error"]) != "" {
gologger.Warning().Msgf("[%s] Init failed with error %v\n", request.TemplateID, result["error"])
return nil
} else {
if request.options.Options.Debug || request.options.Options.DebugResponse {
gologger.Debug().Msgf("[%s] Init executed successfully\n", request.TemplateID)
gologger.Debug().Msgf("[%s] Init result: %v\n", request.TemplateID, result["response"])
}
}
}
// compile pre-condition if any
if request.PreCondition != "" {
preConditionCompiled, err := compiler.SourceAutoMode(request.PreCondition, false)
if err != nil {
return errkit.Newf("could not compile pre-condition: %s", err)
}
request.preConditionCompiled = preConditionCompiled
}
// compile actual source code
if request.Code != "" {
scriptCompiled, err := compiler.SourceAutoMode(request.Code, false)
if err != nil {
return errkit.Newf("could not compile javascript code: %s", err)
}
request.scriptCompiled = scriptCompiled
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
return nil
}
// Options returns executer options for http request
func (r *Request) Options() *protocols.ExecutorOptions {
return r.options
}
// Requests returns the total number of requests the rule will perform
func (request *Request) Requests() int {
pre_conditions := 0
if request.PreCondition != "" {
pre_conditions = 1
}
if request.generator != nil {
payloadRequests := request.generator.NewIterator().Total()
return payloadRequests + pre_conditions
}
return 1 + pre_conditions
}
// GetID returns the ID for the request if any.
func (request *Request) GetID() string {
return request.ID
}
// ExecuteWithResults executes the protocol requests and returns results instead of writing them.
func (request *Request) ExecuteWithResults(target *contextargs.Context, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
// Get default port(s) if specified in template
ports := request.getPorts()
var errs []error
for _, port := range ports {
err := request.executeWithResults(port, target, dynamicValues, previous, callback)
if err != nil {
errs = append(errs, err)
}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
return errkit.Join(errs...)
}
// executeWithResults executes the request
func (request *Request) executeWithResults(port string, target *contextargs.Context, dynamicValues, previous output.InternalEvent, callback protocols.OutputEventCallback) error {
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
input := target.Clone()
// use network port updates input with new port requested in template file
// and it is ignored if input port is not standard http(s) ports like 80,8080,8081 etc
// idea is to reduce redundant dials to http ports
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
if err := input.UseNetworkPort(port, request.getExcludePorts()); err != nil {
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
gologger.Debug().Msgf("Could not network port from constants: %s\n", err)
}
hostPort, err := getAddress(input.MetaInput.Input)
if err != nil {
request.options.Progress.IncrementFailedRequestsBy(1)
return err
}
hostname, port, _ := net.SplitHostPort(hostPort)
if hostname == "" {
hostname = hostPort
}
requestOptions := request.options
templateCtx := request.options.GetTemplateCtx(input.MetaInput)
payloadValues := generators.BuildPayloadFromOptions(request.options.Options)
maps.Copy(payloadValues, dynamicValues)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
payloadValues["Hostname"] = hostPort
payloadValues["Host"] = hostname
payloadValues["Port"] = port
hostnameVariables := protocolutils.GenerateDNSVariables(hostname)
values := generators.MergeMaps(payloadValues, hostnameVariables, request.options.Constants, templateCtx.GetAll())
variablesMap := request.options.Variables.Evaluate(values)
payloadValues = generators.MergeMaps(variablesMap, payloadValues, request.options.Constants, hostnameVariables)
var interactshURLs []string
if request.options.Interactsh != nil {
for payloadName, payloadValue := range payloadValues {
var urls []string
payloadValue, urls = request.options.Interactsh.Replace(types.ToString(payloadValue), interactshURLs)
if len(urls) > 0 {
interactshURLs = append(interactshURLs, urls...)
payloadValues[payloadName] = payloadValue
}
}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
// export all variables to template context
templateCtx.Merge(payloadValues)
if vardump.EnableVarDump {
gologger.Debug().Msgf("JavaScript Protocol request variables: %s\n", vardump.DumpVariables(payloadValues))
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
if request.PreCondition != "" {
payloads := generators.MergeMaps(payloadValues, previous)
if request.options.Options.Debug || request.options.Options.DebugRequests {
gologger.Debug().Msgf("[%s] Executing Precondition for request\n", request.TemplateID)
var highlightFormatter = "terminal256"
if requestOptions.Options.NoColor {
highlightFormatter = "text"
}
var buff bytes.Buffer
_ = quick.Highlight(&buff, beautifyJavascript(request.PreCondition), "javascript", highlightFormatter, "monokai")
prettyPrint(request.TemplateID, buff.String())
}
argsCopy, err := request.getArgsCopy(input, payloads, requestOptions, true)
if err != nil {
return err
}
argsCopy.TemplateCtx = templateCtx.GetAll()
result, err := request.options.JsCompiler.ExecuteWithOptions(request.preConditionCompiled, argsCopy,
&compiler.ExecuteOptions{
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
ExecutionId: requestOptions.Options.ExecutionId,
TimeoutVariants: requestOptions.Options.GetTimeouts(),
Source: &request.PreCondition, Context: target.Context(),
})
// if precondition was successful
if err == nil && result.GetSuccess() {
if request.options.Options.Debug || request.options.Options.DebugRequests {
request.options.Progress.IncrementRequests()
gologger.Debug().Msgf("[%s] Precondition for request was satisfied\n", request.TemplateID)
}
} else {
var outError error
// if js code failed to execute
if err != nil {
outError = errkit.Append(errkit.New("pre-condition not satisfied skipping template execution"), err)
} else {
// execution successful but pre-condition returned false
outError = errkit.New("pre-condition not satisfied skipping template execution")
}
results := map[string]interface{}(result)
results["error"] = outError.Error()
// generate and return failed event
data := request.generateEventData(input, results, hostPort)
data = generators.MergeMaps(data, payloadValues)
event := eventcreator.CreateEventWithAdditionalOptions(request, data, request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
allVars := argsCopy.Map()
allVars = generators.MergeMaps(allVars, data)
wrappedEvent.OperatorsResult.PayloadValues = allVars
})
callback(event)
return err
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
}
if request.generator != nil && request.Threads > 1 {
request.executeRequestParallel(target.Context(), hostPort, hostname, input, payloadValues, callback)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
return nil
}
var gotMatches bool
if request.generator != nil {
iterator := request.generator.NewIterator()
for {
value, ok := iterator.Value()
if !ok {
return nil
}
select {
case <-input.Context().Done():
return input.Context().Err()
default:
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if err := request.executeRequestWithPayloads(hostPort, input, hostname, value, payloadValues, func(result *output.InternalWrappedEvent) {
if result.OperatorsResult != nil && result.OperatorsResult.Matched {
gotMatches = true
request.options.Progress.IncrementMatched()
}
callback(result)
}, requestOptions, interactshURLs); err != nil {
if errkit.IsNetworkPermanentErr(err) {
// gologger.Verbose().Msgf("Could not execute request: %s\n", err)
return err
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
// If this was a match, and we want to stop at first match, skip all further requests.
shouldStopAtFirstMatch := request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch
if shouldStopAtFirstMatch && gotMatches {
return nil
}
}
}
return request.executeRequestWithPayloads(hostPort, input, hostname, nil, payloadValues, callback, requestOptions, interactshURLs)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
func (request *Request) executeRequestParallel(ctxParent context.Context, hostPort, hostname string, input *contextargs.Context, payloadValues map[string]interface{}, callback protocols.OutputEventCallback) {
threads := request.Threads
if threads == 0 {
threads = 1
}
ctx, cancel := context.WithCancelCause(ctxParent)
defer cancel(nil)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
requestOptions := request.options
gotmatches := &atomic.Bool{}
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
// if request threads matches global payload concurrency we follow it
shouldFollowGlobal := threads == request.options.Options.PayloadConcurrency
2024-04-03 17:50:57 +02:00
sg, _ := syncutil.New(syncutil.WithSize(threads))
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if request.generator != nil {
iterator := request.generator.NewIterator()
for {
value, ok := iterator.Value()
if !ok {
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
break
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
select {
case <-input.Context().Done():
return
default:
}
// resize check point - nop if there are no changes
if shouldFollowGlobal && sg.Size != request.options.Options.PayloadConcurrency {
if err := sg.Resize(ctxParent, request.options.Options.PayloadConcurrency); err != nil {
gologger.Warning().Msgf("Could not resize workpool: %s\n", err)
}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
sg.Add()
go func() {
defer sg.Done()
if ctx.Err() != nil {
// work already done exit
return
}
shouldStopAtFirstMatch := request.options.Options.StopAtFirstMatch || request.StopAtFirstMatch
if err := request.executeRequestWithPayloads(hostPort, input, hostname, value, payloadValues, func(result *output.InternalWrappedEvent) {
if result.OperatorsResult != nil && result.OperatorsResult.Matched {
gotmatches.Store(true)
}
callback(result)
}, requestOptions, []string{}); err != nil {
if errkit.IsNetworkPermanentErr(err) {
cancel(err)
return
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
// If this was a match, and we want to stop at first match, skip all further requests.
if shouldStopAtFirstMatch && gotmatches.Load() {
cancel(nil)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
return
}
}()
}
}
sg.Wait()
if gotmatches.Load() {
request.options.Progress.IncrementMatched()
}
}
func (request *Request) executeRequestWithPayloads(hostPort string, input *contextargs.Context, _ string, payload map[string]interface{}, previous output.InternalEvent, callback protocols.OutputEventCallback, requestOptions *protocols.ExecutorOptions, interactshURLs []string) error {
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
payloadValues := generators.MergeMaps(payload, previous)
argsCopy, err := request.getArgsCopy(input, payloadValues, requestOptions, false)
if err != nil {
return err
}
if request.options.HasTemplateCtx(input.MetaInput) {
argsCopy.TemplateCtx = request.options.GetTemplateCtx(input.MetaInput).GetAll()
} else {
argsCopy.TemplateCtx = map[string]interface{}{}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if request.options.Interactsh != nil {
if argsCopy.Args != nil {
for k, v := range argsCopy.Args {
var urls []string
v, urls = request.options.Interactsh.Replace(fmt.Sprint(v), []string{})
if len(urls) > 0 {
interactshURLs = append(interactshURLs, urls...)
argsCopy.Args[k] = v
}
}
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
results, err := request.options.JsCompiler.ExecuteWithOptions(request.scriptCompiled, argsCopy,
&compiler.ExecuteOptions{
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
ExecutionId: requestOptions.Options.ExecutionId,
TimeoutVariants: requestOptions.Options.GetTimeouts(),
Source: &request.Code,
Context: input.Context(),
})
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if err != nil {
// shouldn't fail even if it returned error instead create a failure event
results = compiler.ExecuteResult{"success": false, "error": err.Error()}
}
request.options.Progress.IncrementRequests()
requestOptions.Output.Request(requestOptions.TemplateID, hostPort, request.Type().String(), err)
gologger.Verbose().Msgf("[%s] Sent Javascript request to %s", request.options.TemplateID, hostPort)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if requestOptions.Options.Debug || requestOptions.Options.DebugRequests || requestOptions.Options.StoreResponse {
msg := fmt.Sprintf("[%s] Dumped Javascript request for %s:\nVariables:\n %v", requestOptions.TemplateID, input.MetaInput.Input, vardump.DumpVariables(argsCopy.Args))
if requestOptions.Options.Debug || requestOptions.Options.DebugRequests {
gologger.Debug().Str("address", input.MetaInput.Input).Msg(msg)
var highlightFormatter = "terminal256"
if requestOptions.Options.NoColor {
highlightFormatter = "text"
}
var buff bytes.Buffer
_ = quick.Highlight(&buff, beautifyJavascript(request.Code), "javascript", highlightFormatter, "monokai")
prettyPrint(request.TemplateID, buff.String())
}
if requestOptions.Options.StoreResponse {
request.options.Output.WriteStoreDebugData(input.MetaInput.Input, request.options.TemplateID, request.Type().String(), msg)
}
}
values := mapsutil.Merge(payloadValues, results)
// generate event data
data := request.generateEventData(input, values, hostPort)
// add and get values from templatectx
request.options.AddTemplateVars(input.MetaInput, request.Type(), request.GetID(), data)
data = generators.MergeMaps(data, request.options.GetTemplateCtx(input.MetaInput).GetAll())
if requestOptions.Options.Debug || requestOptions.Options.DebugRequests || requestOptions.Options.StoreResponse {
msg := fmt.Sprintf("[%s] Dumped Javascript response for %s:\n%v", requestOptions.TemplateID, input.MetaInput.Input, vardump.DumpVariables(results))
if requestOptions.Options.Debug || requestOptions.Options.DebugRequests {
gologger.Debug().Str("address", input.MetaInput.Input).Msg(msg)
}
if requestOptions.Options.StoreResponse {
request.options.Output.WriteStoreDebugData(input.MetaInput.Input, request.options.TemplateID, request.Type().String(), msg)
}
}
if _, ok := data["error"]; ok {
event := eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
wrappedEvent.OperatorsResult.PayloadValues = payload
})
callback(event)
return err
}
if request.options.Interactsh != nil {
request.options.Interactsh.MakePlaceholders(interactshURLs, data)
}
var event *output.InternalWrappedEvent
if len(interactshURLs) == 0 {
event = eventcreator.CreateEventWithAdditionalOptions(request, generators.MergeMaps(data, payloadValues), request.options.Options.Debug || request.options.Options.DebugResponse, func(wrappedEvent *output.InternalWrappedEvent) {
wrappedEvent.OperatorsResult.PayloadValues = payload
})
callback(event)
} else if request.options.Interactsh != nil {
event = &output.InternalWrappedEvent{InternalEvent: data, UsesInteractsh: true}
request.options.Interactsh.RequestEvent(interactshURLs, &interactsh.RequestData{
MakeResultFunc: request.MakeResultEvent,
Event: event,
Operators: request.CompiledOperators,
MatchFunc: request.Match,
ExtractFunc: request.Extract,
})
}
return nil
}
// generateEventData generates event data for the request
func (request *Request) generateEventData(input *contextargs.Context, values map[string]interface{}, matched string) map[string]interface{} {
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
dialers := protocolstate.GetDialersWithId(request.options.Options.ExecutionId)
if dialers == nil {
panic(fmt.Sprintf("dialers not initialized for %s", request.options.Options.ExecutionId))
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
data := make(map[string]interface{})
maps.Copy(data, values)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
data["type"] = request.Type().String()
data["request-pre-condition"] = beautifyJavascript(request.PreCondition)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
data["request"] = beautifyJavascript(request.Code)
data["host"] = input.MetaInput.Input
data["matched"] = matched
data["template-path"] = request.options.TemplatePath
data["template-id"] = request.options.TemplateID
data["template-info"] = request.options.TemplateInfo
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
if request.StopAtFirstMatch || request.options.StopAtFirstMatch {
data["stop-at-first-match"] = true
}
// add ip address to data
if input.MetaInput.CustomIP != "" {
data["ip"] = input.MetaInput.CustomIP
} else {
// context: https://github.com/projectdiscovery/nuclei/issues/5021
hostname := input.MetaInput.Input
if strings.Contains(hostname, ":") {
host, _, err := net.SplitHostPort(hostname)
if err == nil {
hostname = host
} else {
// naive way
if !strings.Contains(hostname, "]") {
hostname = hostname[:strings.LastIndex(hostname, ":")]
}
}
}
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
data["ip"] = dialers.Fastdialer.GetDialedIP(hostname)
// if input itself was an ip, use it
if iputil.IsIP(hostname) {
data["ip"] = hostname
}
// if ip is not found,this is because ssh and other protocols do not use fastdialer
// although its not perfect due to its use case dial and get ip
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
dnsData, err := dialers.Fastdialer.GetDNSData(hostname)
if err == nil {
for _, v := range dnsData.A {
data["ip"] = v
break
}
if data["ip"] == "" {
for _, v := range dnsData.AAAA {
data["ip"] = v
break
}
}
}
}
return data
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
func (request *Request) getArgsCopy(input *contextargs.Context, payloadValues map[string]interface{}, requestOptions *protocols.ExecutorOptions, ignoreErrors bool) (*compiler.ExecuteArgs, error) {
// Template args from payloads
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
argsCopy, err := request.evaluateArgs(payloadValues, requestOptions, ignoreErrors)
if err != nil {
requestOptions.Output.Request(requestOptions.TemplateID, input.MetaInput.Input, request.Type().String(), err)
requestOptions.Progress.IncrementFailedRequestsBy(1)
}
// "Port" is a special variable that is considered as network port
// and is conditional based on input port and default port specified in input
argsCopy["Port"] = input.Port()
return &compiler.ExecuteArgs{Args: argsCopy}, nil
}
// evaluateArgs evaluates arguments using available payload values and returns a copy of args
func (request *Request) evaluateArgs(payloadValues map[string]interface{}, _ *protocols.ExecutorOptions, ignoreErrors bool) (map[string]interface{}, error) {
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
argsCopy := make(map[string]interface{})
mainLoop:
for k, v := range request.Args {
if vVal, ok := v.(string); ok && strings.Contains(vVal, "{") {
finalAddress, dataErr := expressions.Evaluate(vVal, payloadValues)
if dataErr != nil {
return nil, errors.Wrap(dataErr, "could not evaluate template expressions")
}
if finalAddress == vVal && ignoreErrors {
argsCopy[k] = ""
continue mainLoop
}
argsCopy[k] = finalAddress
} else {
argsCopy[k] = v
}
}
add 'fs' module to javascript protocol (#4156) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary * add fs library * add fs module * add init code block and 'updatePayload' * use go native func for isPortOpen * docgen improvements + 'fs' module docs * update func signature and more * prompt improvements * fix inconsitencies in jsdocs * remove debug statements --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-26 16:55:25 +05:30
return argsCopy, nil
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
// RequestPartDefinitions contains a mapping of request part definitions and their
// description. Multiple definitions are separated by commas.
// Definitions not having a name (generated on runtime) are prefixed & suffixed by <>.
var RequestPartDefinitions = map[string]string{
"type": "Type is the type of request made",
"response": "Javascript protocol result response",
"host": "Host is the input to the template",
"matched": "Matched is the input which was matched upon",
}
// getAddress returns the address of the host to make request to
func getAddress(toTest string) (string, error) {
urlx, err := urlutil.Parse(toTest)
if err != nil {
// use given input instead of url parsing failure
return toTest, nil
}
return urlx.Host, nil
}
// Match performs matching operation for a matcher on model and returns:
// true and a list of matched snippets if the matcher type is supports it
// otherwise false and an empty string slice
func (request *Request) Match(data map[string]interface{}, matcher *matchers.Matcher) (bool, []string) {
return protocols.MakeDefaultMatchFunc(data, matcher)
}
// Extract performs extracting operation for an extractor on model and returns true or false.
func (request *Request) Extract(data map[string]interface{}, matcher *extractors.Extractor) map[string]struct{} {
return protocols.MakeDefaultExtractFunc(data, matcher)
}
// MakeResultEvent creates a result event from internal wrapped event
func (request *Request) MakeResultEvent(wrapped *output.InternalWrappedEvent) []*output.ResultEvent {
return protocols.MakeDefaultResultEvent(request, wrapped)
}
// GetCompiledOperators returns a list of the compiled operators
func (request *Request) GetCompiledOperators() []*operators.Operators {
return []*operators.Operators{request.CompiledOperators}
}
// Type returns the type of the protocol request
func (request *Request) Type() templateTypes.ProtocolType {
return templateTypes.JavascriptProtocol
}
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
func (request *Request) getPorts() []string {
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
for k, v := range request.Args {
if strings.EqualFold(k, "Port") {
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
portStr := types.ToString(v)
ports := []string{}
for _, p := range strings.Split(portStr, ",") {
trimmed := strings.TrimSpace(p)
if trimmed != "" {
ports = append(ports, trimmed)
}
}
return sliceutil.Dedupe(ports)
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
}
v3.6.0 (#6657) * Multi Port Support Added - JS * minor -changes * restoring basic sequential multiport support * better error handling * feat(openapi/swagger): direct fuzzing using target url * fix (openapi/swagger): improve error handling and tmpDir cleanup * fix(openapi/swagger): err shadowing on write failure * fix(openapi/swagger): remove discarded error in defer * fix(openapi/swagger): linter and url validation * fix(openapi/swagger): remove code duplication * reusing dialer * removing debug log * fix: restore parallel processing in workflow & file proto add missing `go` keyword to anonymous funcs that were intended to run as goroutines but were executing synchronously instead. Fixes #6492 Signed-off-by: Dwi Siswanto <git@dw1.io> * test: adds `Test(FileProtocol|Workflows)ConcurrentExecution` tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(file): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(integration-test): enhance debug mode detects * replace hardcoded `DEBUG` env var check with extensible helper func. * add support for GitHub Actions Runner env var. * accept multiple truthy value variants. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(core): race cond in workflow execution caused by shared context callbacks. it was exposed after adding concurrent exec to workflow processing and occurred when multiple goroutines attempted to write to the same `ctx.OnResult` callback field simultaneously, causing data races during workflow template exec. Signed-off-by: Dwi Siswanto <git@dw1.io> * introducing workflow sequential mode * Revert "introducing workflow sequential mode" This reverts commit 1093bbc62dd10837a177ab3f844b977f19cde6bb. * refactor(core): keep workflow exec seq Signed-off-by: Dwi Siswanto <git@dw1.io> * test(core): rm unused tests Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(sdk): configure tmpDir for SDK Closes #6595. * docs(sdk): update comment to more accurately reflect purpose * feat(sdk): add tmpDir configuration option for SDK users * fix(sdk): init default engine tmpDir when unconfigured * style(sdk): remove unnecessary else block * feat(sdk): create parent & tmp dir in WithTemporaryDirectory * test(cmd): enable `BenchmarkRunEnumeration/Default` bench Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): collect CPU & heap profiles Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(cmd): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * Merge pull request #6610 from projectdiscovery/feat-result-upload allow custom id for upload * feat: write resume file specified by flag * updating docs * chore(deps): bump the modules group with 6 updates Bumps the modules group with 6 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.59` | `1.1.60` | | [github.com/projectdiscovery/httpx](https://github.com/projectdiscovery/httpx) | `1.7.2-0.20250911192144-fc425deb041a` | `1.7.2` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.27` | `0.1.28` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1-0.20251030144701-ce5c4b44e1e6` | `0.6.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.54` | `0.2.55` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.9` | `1.2.10` | Updates `github.com/projectdiscovery/gologger` from 1.1.59 to 1.1.60 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.59...v1.1.60) Updates `github.com/projectdiscovery/httpx` from 1.7.2-0.20250911192144-fc425deb041a to 1.7.2 - [Release notes](https://github.com/projectdiscovery/httpx/releases) - [Changelog](https://github.com/projectdiscovery/httpx/blob/dev/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/httpx/commits/v1.7.2) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.27 to 0.1.28 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.27...v0.1.28) Updates `github.com/projectdiscovery/utils` from 0.6.1-0.20251030144701-ce5c4b44e1e6 to 0.6.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/commits/v0.6.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.54 to 0.2.55 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.54...v0.2.55) Updates `github.com/projectdiscovery/cdncheck` from 1.2.9 to 1.2.10 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.9...v1.2.10) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.60 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/httpx dependency-version: 1.7.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.28 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.6.1 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.55 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.10 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * refactor(sdk): don't create parentDir when configuring tmpDir * adding test case * lint * removing unused check * adding multiport template * refactor test * chore(deps): bump golang.org/x/crypto Bumps the go_modules group with 1 update in the / directory: [golang.org/x/crypto](https://github.com/golang/crypto). Updates `golang.org/x/crypto` from 0.43.0 to 0.45.0 - [Commits](https://github.com/golang/crypto/compare/v0.43.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect dependency-group: go_modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(variables): check for undefined params for lazy eval (#6618) * feat(variables): check for undefined params for lazy eval Signed-off-by: Dwi Siswanto <git@dw1.io> * test(variables): add TestCheckForLazyEval Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(variables): fail safe on err compile expr Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/fastdialer@v0.4.16 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(interactsh): skip DNS lookups on interactsh domains (#6614) * fix(interactsh): skip DNS lookups on interactsh domains to prevent false positives. Prevents nuclei from resolving interactsh domains injected in Host headers, which would cause self-interactions to be incorrectly reported as matches. Changes: * Add `GetHostname()` method to `interactsh.Client` to expose active server domain. * Skip CNAME DNS lookups in `(*http.Request).addCNameIfAvailable` when hostname matches the `(*interactsh.Client).GetHostname`. Fixes #6613 Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(http): prevent false `interactshDomain` matches Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * feat: bump dsl with deserialization helpers * chore: omit unnecessary reassignment (#6622) Signed-off-by: ledigang <shuangcui@msn.com> * disable stale workflow for enhancements * ci: cache go-rod browser (#6640) Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump actions/checkout from 5 to 6 in the workflows group Bumps the workflows group with 1 update: [actions/checkout](https://github.com/actions/checkout). Updates `actions/checkout` from 5 to 6 - [Release notes](https://github.com/actions/checkout/releases) - [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md) - [Commits](https://github.com/actions/checkout/compare/v5...v6) --- updated-dependencies: - dependency-name: actions/checkout dependency-version: '6' dependency-type: direct:production update-type: version-update:semver-major dependency-group: workflows ... Signed-off-by: dependabot[bot] <support@github.com> * do not exempt abandoned issues and prs * ci: apply free-disk-space on tests Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump PD modules & update `httputil` calls (#6629) * chore(deps): bump the modules group across 1 directory with 11 updates Bumps the modules group with 11 updates in the / directory: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.16` | `0.4.17` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.95` | `0.0.96` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.108` | `1.0.109` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.131` | `1.0.132` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.60` | `1.1.61` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.28` | `0.1.29` | | [github.com/projectdiscovery/tlsx](https://github.com/projectdiscovery/tlsx) | `1.2.1` | `1.2.2` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.102` | `0.0.103` | | [github.com/projectdiscovery/utils](https://github.com/projectdiscovery/utils) | `0.6.1` | `0.7.1` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.55` | `0.2.56` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.10` | `1.2.11` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.16 to 0.4.17 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.16...v0.4.17) Updates `github.com/projectdiscovery/hmap` from 0.0.95 to 0.0.96 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.95...v0.0.96) Updates `github.com/projectdiscovery/retryabledns` from 1.0.108 to 1.0.109 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.108...v1.0.109) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.131 to 1.0.132 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.131...v1.0.132) Updates `github.com/projectdiscovery/gologger` from 1.1.60 to 1.1.61 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.60...v1.1.61) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.28 to 0.1.29 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.28...v0.1.29) Updates `github.com/projectdiscovery/tlsx` from 1.2.1 to 1.2.2 - [Release notes](https://github.com/projectdiscovery/tlsx/releases) - [Changelog](https://github.com/projectdiscovery/tlsx/blob/main/.goreleaser.yml) - [Commits](https://github.com/projectdiscovery/tlsx/compare/v1.2.1...v1.2.2) Updates `github.com/projectdiscovery/useragent` from 0.0.102 to 0.0.103 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.102...v0.0.103) Updates `github.com/projectdiscovery/utils` from 0.6.1 to 0.7.1 - [Release notes](https://github.com/projectdiscovery/utils/releases) - [Changelog](https://github.com/projectdiscovery/utils/blob/main/CHANGELOG.md) - [Commits](https://github.com/projectdiscovery/utils/compare/v0.6.1...v0.7.1) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.55 to 0.2.56 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.55...v0.2.56) Updates `github.com/projectdiscovery/cdncheck` from 1.2.10 to 1.2.11 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Changelog](https://github.com/projectdiscovery/cdncheck/blob/main/.goreleaser.yaml) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.10...v1.2.11) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.17 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.96 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.109 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.132 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.61 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.29 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/tlsx dependency-version: 1.2.2 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.103 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/utils dependency-version: 0.7.1 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.56 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.11 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * chore: update utils.httputil calls Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump github.com/projectdiscovery/utils => v0.7.3 Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Dwi Siswanto <git@dw1.io> * chore(deps): bump the modules group with 11 updates Bumps the modules group with 11 updates: | Package | From | To | | --- | --- | --- | | [github.com/projectdiscovery/fastdialer](https://github.com/projectdiscovery/fastdialer) | `0.4.17` | `0.4.18` | | [github.com/projectdiscovery/hmap](https://github.com/projectdiscovery/hmap) | `0.0.96` | `0.0.97` | | [github.com/projectdiscovery/retryabledns](https://github.com/projectdiscovery/retryabledns) | `1.0.109` | `1.0.110` | | [github.com/projectdiscovery/retryablehttp-go](https://github.com/projectdiscovery/retryablehttp-go) | `1.0.132` | `1.0.133` | | [github.com/projectdiscovery/dsl](https://github.com/projectdiscovery/dsl) | `0.8.5` | `0.8.6` | | [github.com/projectdiscovery/gologger](https://github.com/projectdiscovery/gologger) | `1.1.61` | `1.1.62` | | [github.com/projectdiscovery/networkpolicy](https://github.com/projectdiscovery/networkpolicy) | `0.1.29` | `0.1.30` | | [github.com/projectdiscovery/uncover](https://github.com/projectdiscovery/uncover) | `1.1.0` | `1.2.0` | | [github.com/projectdiscovery/useragent](https://github.com/projectdiscovery/useragent) | `0.0.103` | `0.0.104` | | [github.com/projectdiscovery/wappalyzergo](https://github.com/projectdiscovery/wappalyzergo) | `0.2.56` | `0.2.57` | | [github.com/projectdiscovery/cdncheck](https://github.com/projectdiscovery/cdncheck) | `1.2.11` | `1.2.12` | Updates `github.com/projectdiscovery/fastdialer` from 0.4.17 to 0.4.18 - [Release notes](https://github.com/projectdiscovery/fastdialer/releases) - [Commits](https://github.com/projectdiscovery/fastdialer/compare/v0.4.17...v0.4.18) Updates `github.com/projectdiscovery/hmap` from 0.0.96 to 0.0.97 - [Release notes](https://github.com/projectdiscovery/hmap/releases) - [Commits](https://github.com/projectdiscovery/hmap/compare/v0.0.96...v0.0.97) Updates `github.com/projectdiscovery/retryabledns` from 1.0.109 to 1.0.110 - [Release notes](https://github.com/projectdiscovery/retryabledns/releases) - [Commits](https://github.com/projectdiscovery/retryabledns/compare/v1.0.109...v1.0.110) Updates `github.com/projectdiscovery/retryablehttp-go` from 1.0.132 to 1.0.133 - [Release notes](https://github.com/projectdiscovery/retryablehttp-go/releases) - [Commits](https://github.com/projectdiscovery/retryablehttp-go/compare/v1.0.132...v1.0.133) Updates `github.com/projectdiscovery/dsl` from 0.8.5 to 0.8.6 - [Release notes](https://github.com/projectdiscovery/dsl/releases) - [Commits](https://github.com/projectdiscovery/dsl/compare/v0.8.5...v0.8.6) Updates `github.com/projectdiscovery/gologger` from 1.1.61 to 1.1.62 - [Release notes](https://github.com/projectdiscovery/gologger/releases) - [Commits](https://github.com/projectdiscovery/gologger/compare/v1.1.61...v1.1.62) Updates `github.com/projectdiscovery/networkpolicy` from 0.1.29 to 0.1.30 - [Release notes](https://github.com/projectdiscovery/networkpolicy/releases) - [Commits](https://github.com/projectdiscovery/networkpolicy/compare/v0.1.29...v0.1.30) Updates `github.com/projectdiscovery/uncover` from 1.1.0 to 1.2.0 - [Release notes](https://github.com/projectdiscovery/uncover/releases) - [Commits](https://github.com/projectdiscovery/uncover/compare/v1.1.0...v1.2.0) Updates `github.com/projectdiscovery/useragent` from 0.0.103 to 0.0.104 - [Release notes](https://github.com/projectdiscovery/useragent/releases) - [Commits](https://github.com/projectdiscovery/useragent/compare/v0.0.103...v0.0.104) Updates `github.com/projectdiscovery/wappalyzergo` from 0.2.56 to 0.2.57 - [Release notes](https://github.com/projectdiscovery/wappalyzergo/releases) - [Commits](https://github.com/projectdiscovery/wappalyzergo/compare/v0.2.56...v0.2.57) Updates `github.com/projectdiscovery/cdncheck` from 1.2.11 to 1.2.12 - [Release notes](https://github.com/projectdiscovery/cdncheck/releases) - [Commits](https://github.com/projectdiscovery/cdncheck/compare/v1.2.11...v1.2.12) --- updated-dependencies: - dependency-name: github.com/projectdiscovery/fastdialer dependency-version: 0.4.18 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/hmap dependency-version: 0.0.97 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryabledns dependency-version: 1.0.110 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/retryablehttp-go dependency-version: 1.0.133 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/dsl dependency-version: 0.8.6 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/gologger dependency-version: 1.1.62 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/networkpolicy dependency-version: 0.1.30 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/uncover dependency-version: 1.2.0 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: modules - dependency-name: github.com/projectdiscovery/useragent dependency-version: 0.0.104 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/wappalyzergo dependency-version: 0.2.57 dependency-type: direct:production update-type: version-update:semver-patch dependency-group: modules - dependency-name: github.com/projectdiscovery/cdncheck dependency-version: 1.2.12 dependency-type: indirect update-type: version-update:semver-patch dependency-group: modules ... Signed-off-by: dependabot[bot] <support@github.com> * feat(loader): implement persistent metadata cache (#6630) * feat(loader): implement persistent metadata cache for template filtering optimization. Introduce a new template metadata indexing system with persistent caching to dramatically improve template loading perf when filters are applied. The implementation adds a new index pkg that caches lightweight template metadata (ID, tags, authors, severity, .etc) and enables filtering templates before expensive YAML parsing occurs. The index uses an in-memory LRU cache backed by `otter` pkg for efficient memory management with adaptive sizing based on entry weight, defaulting to approx. 40MB for 50K templates. Metadata is persisted to disk using gob encoding at "~/.cache/nuclei/index.gob" with atomic writes to prevent corruption. The cache automatically invalidates stale entries using `ModTime` to detect file modifications, ensuring metadata freshness w/o manual intervention. Filtering has been refactored from the previous `TagFilter` and `PathFilter` approach into a unified `index.Filter` type that handles all basic filtering ops including severity, authors, tags, template IDs with wildcard support, protocol types, and path-based inclusion and exclusion. The filter implements OR logic within each field type and AND logic across different field types, with exclusion filters taking precedence over inclusion filters and forced inclusion via `IncludeTemplates` and `IncludeTags` overriding exclusions. The `loader` integration creates an index filter from store configuration via `buildIndexFilter` and manages the cache lifecycle through `loadTemplatesIndex` and `saveTemplatesIndex` methods. When `LoadTemplatesOnlyMetadata` or `LoadTemplatesWithTags` is called, the system first checks the metadata cache for each template path. If cached metadata exists and passes validation, the filter is applied directly against the metadata without parsing. Only templates matching the filter criteria proceed to full YAML parsing, resulting in significant performance gains. Advanced filtering via "-tc" flag (`IncludeConditions`) still requires template parsing as these are expression-based filters that cannot be evaluated from metadata alone. The `TagFilter` has been simplified to handle only `IncludeConditions` while all other filtering ops are delegated to the index-based filtering system. Cache management is fully automatic with no user configuration required. The cache gracefully handles errors by logging warnings & falling back to normal op w/o caching. Cache files use schema versioning to invalidate incompatible cache formats across nuclei updates (well, specifically `Index` and `Metadata` changes). This optimization particularly benefits repeated scans with the same filters, CI/CD pipelines running nuclei regularly, development and testing workflows with frequent template loading, and any scenario with large template collections where filtering would exclude most templates. * test(loader): adds `BenchmarkLoadTemplates{,OnlyMetadata}` benchs Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei-templates index Signed-off-by: Dwi Siswanto <git@dw1.io> * chore(index): satisfy lints Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): correct metadata filter logic for proper template matching. The `filter.matchesIncludes()` was using OR logic across different filter types, causing incorrect template matching. Additionally, ID matching was case-sensitive, failing to match patterns like 'CVE-2021-*'. The filter now correctly implements: (author1 OR author2) AND (tag1 OR tag2) AND (severity1 OR severity2) - using OR within each filter type and AND across different types. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): resolve test timing issue in CI environments. Some test was failing in CI due to filesystem timestamp resolution limitations. On filesystems with 1s ModTime granularity (common in CI), modifying a file immediately after capturing its timestamp resulted in identical ModTime values, causing IsValid() to incorrectly return true. Signed-off-by: Dwi Siswanto <git@dw1.io> * ci: cache nuclei with composite action Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(index): file locking issue on Windows during cache save/load. Explicitly close file handles before performing rename/remove ops in `Save` and `Load` methods. * In `Save`, close temp file before rename. * In `Load`, close file before remove during error handling/version mismatch. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(index): flaky index tests on Windows Fix path separator mismatch in `TestCacheSize` and `TestCachePersistenceWithLargeDataset` by using `filepath.Join` consistently instead of hardcoded forward slashes. Signed-off-by: Dwi Siswanto <git@dw1.io> * test(cmd): init logger to prevent nil pointer deref The integration tests were panicking with a nil pointer dereference in `pkg/catalog/loader` because the logger was not init'ed. When `store.saveMetadataIndexOnce` attempted to log the result of the metadata cache op, it dereferenced the nil logger, causing a crash. Signed-off-by: Dwi Siswanto <git@dw1.io> * fix(loader): resolve include/exclude paths for metadata cache filter. The `indexFilter` was previously init'ed using raw relative paths from the config for `IncludeTemplates` and `ExcludeTemplates`. But the persistent metadata cache stores templates using their absolute paths. This mismatch caused the `matchesPath` check to fail, leading to templates being incorrectly excluded even when explicitly included via flags (e.g., "-include-templates loader/excluded-template.yaml"). This commit updates `buildIndexFilter` to resolve these paths to their absolute versions using `store.config.Catalog.GetTemplatesPath` before creating the filter, ensuring consistent path matching against the metadata cache. Signed-off-by: Dwi Siswanto <git@dw1.io> * feat(index): adds `NewMetadataFromTemplate` func Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(index): return metadata when `(*Index).cache` is nil Signed-off-by: Dwi Siswanto <git@dw1.io> * refactor(loader): restore pre‑index behavior semantics Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> * chore: bump version Signed-off-by: Dwi Siswanto <git@dw1.io> --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Signed-off-by: dependabot[bot] <support@github.com> Signed-off-by: ledigang <shuangcui@msn.com> Co-authored-by: pussycat0x <65701233+pussycat0x@users.noreply.github.com> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: tvroi <roy.oswaldha@traveloka.com> Co-authored-by: Niek den Breeje <n.denbreeje@guardian360.nl> Co-authored-by: circleous <circleousdev@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ice3man <nizamulrana@gmail.com> Co-authored-by: Dogan Can Bakir <65292895+dogancanbakir@users.noreply.github.com> Co-authored-by: ledigang <shuangcui@msn.com> Co-authored-by: Doğan Can Bakır <dogancanbakir@protonmail.com>
2025-12-05 03:37:18 +07:00
return []string{}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
2025-09-27 21:00:13 +05:30
// getPorts returns a slice of ports from the Port argument
func (request *Request) getPorts() []string {
portStr := request.getPort()
if portStr == "" {
return []string{}
}
// Split by comma and clean up whitespace
ports := strings.Split(portStr, ",")
var cleanedPorts []string
for _, port := range ports {
cleaned := strings.TrimSpace(port)
if cleaned != "" {
cleanedPorts = append(cleanedPorts, cleaned)
}
}
return cleanedPorts
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
func (request *Request) getExcludePorts() string {
for k, v := range request.Args {
if strings.EqualFold(k, "exclude-ports") {
return types.ToString(v)
}
}
return ""
}
func (request *Request) MakeResultEventItem(wrapped *output.InternalWrappedEvent) *output.ResultEvent {
fields := protocolutils.GetJsonFieldsFromURL(types.ToString(wrapped.InternalEvent["host"]))
if types.ToString(wrapped.InternalEvent["ip"]) != "" {
fields.Ip = types.ToString(wrapped.InternalEvent["ip"])
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
data := &output.ResultEvent{
TemplateID: types.ToString(wrapped.InternalEvent["template-id"]),
TemplatePath: types.ToString(wrapped.InternalEvent["template-path"]),
Info: wrapped.InternalEvent["template-info"].(model.Info),
TemplateVerifier: request.options.TemplateVerifier,
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
Type: types.ToString(wrapped.InternalEvent["type"]),
Host: fields.Host,
Port: fields.Port,
URL: fields.URL,
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
Matched: types.ToString(wrapped.InternalEvent["matched"]),
Metadata: wrapped.OperatorsResult.PayloadValues,
ExtractedResults: wrapped.OperatorsResult.OutputExtracts,
Timestamp: time.Now(),
MatcherStatus: true,
Request: types.ToString(wrapped.InternalEvent["request"]),
Response: types.ToString(wrapped.InternalEvent["response"]),
IP: fields.Ip,
TemplateEncoded: request.options.EncodeTemplate(),
Error: types.ToString(wrapped.InternalEvent["error"]),
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
}
return data
}
func beautifyJavascript(code string) string {
opts := jsbeautifier.DefaultOptions()
beautified, err := jsbeautifier.Beautify(&code, opts)
if err != nil {
return code
}
return beautified
}
func prettyPrint(templateId string, buff string) {
cache, goroutine and unbounded workers management (#6420) * Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints. * Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency. * feat: improve concurrency to be bound * refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling * feat: add regex matching tests and benchmarks for performance evaluation * feat: add prefix check in regex extraction to optimize matching process * feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation * feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability * feat: enhance error handling in template execution and improve regex caching logic for better performance * Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency. * Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes - Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches. - Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes. * Refactor matchers and update load test generator to use io package - Removed maxRegexScanBytes constant from match.go. - Replaced ioutil with io package in load_test.go for NopCloser usage. - Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality. * Add cancellation support in template execution and enhance test coverage - Updated executeTemplateWithTargets to respect context cancellation. - Introduced fakeTargetProvider and slowExecuter for testing. - Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
2025-09-15 23:48:02 +05:30
if buff == "" {
return
}
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
lines := strings.Split(buff, "\n")
cache, goroutine and unbounded workers management (#6420) * Enhance matcher compilation with caching for regex and DSL expressions to improve performance. Update template parsing to conditionally retain raw templates based on size constraints. * Implement caching for regex and DSL expressions in extractors and matchers to enhance performance. Introduce a buffer pool in raw requests to reduce memory allocations. Update template cache management for improved efficiency. * feat: improve concurrency to be bound * refactor: replace fmt.Sprintf with fmt.Fprintf for improved performance in header handling * feat: add regex matching tests and benchmarks for performance evaluation * feat: add prefix check in regex extraction to optimize matching process * feat: implement regex caching mechanism to enhance performance in extractors and matchers, along with tests and benchmarks for validation * feat: add unit tests for template execution in the core engine, enhancing test coverage and reliability * feat: enhance error handling in template execution and improve regex caching logic for better performance * Implement caching for regex and DSL expressions in the cache package, replacing previous sync.Map usage. Add unit tests for cache functionality, including eviction by capacity and retrieval of cached items. Update extractors and matchers to utilize the new cache system for improved performance and memory efficiency. * Add tests for SetCapacities in cache package to ensure cache behavior on capacity changes - Implemented TestSetCapacities_NoRebuildOnZero to verify that setting capacities to zero does not clear existing caches. - Added TestSetCapacities_BeforeFirstUse to confirm that initial cache settings are respected and not overridden by subsequent capacity changes. * Refactor matchers and update load test generator to use io package - Removed maxRegexScanBytes constant from match.go. - Replaced ioutil with io package in load_test.go for NopCloser usage. - Restored TestValidate_AllowsInlineMultiline in load_test.go to ensure inline validation functionality. * Add cancellation support in template execution and enhance test coverage - Updated executeTemplateWithTargets to respect context cancellation. - Introduced fakeTargetProvider and slowExecuter for testing. - Added Test_executeTemplateWithTargets_RespectsCancellation to validate cancellation behavior during template execution.
2025-09-15 23:48:02 +05:30
final := make([]string, 0, len(lines))
javascript protocol for scripting (includes 15+ proto libs) (#4109) * rebase js-layer PR from @ice3man543 * package restructuring * working * fix duplicated event & matcher status * fix lint error * fix response field * add new functions * multiple minor improvements * fix incorrect stats in js protocol * sort output metadata in cli * remove temp files * remove dead code * add unit and integration test * fix lint error * add jsdoclint using llm * fix error in test * add js lint using llm * generate docs of libs * llm lint * remove duplicated docs * update generated docs * update prompt in doclint * update docs * temp disable version check test * fix unit test and add retry * fix panic in it * update and move jsdocs * updated jsdocs * update docs * update container platform in test * dir restructure and adding docs * add api_reference and remove markdown docs * fix imports * add javascript design and contribution docs * add js protocol documentation * update integration test and docs * update doc ext mdx->md * minor update to docs * new integration test and more * move go libs and add docs * gen new net docs and more * final docs update * add new devtool * use fastdialer * fix build fail * use fastdialer + network sandbox support * add reserved keyword 'Port' * update Port to new syntax * misc update * always enable templatectx in js protocol * move docs to 'js-proto-docs' repo * remove scrapefuncs binary --------- Co-authored-by: sandeep <8293321+ehsandeep@users.noreply.github.com>
2023-09-16 16:02:17 +05:30
for _, v := range lines {
if v != "" {
final = append(final, "\t"+v)
}
}
gologger.Debug().Msgf(" [%v] Javascript Code:\n\n%v\n\n", templateId, strings.Join(final, "\n"))
}
Remove singletons from Nuclei engine (continuation of #6210) (#6296) * introducing execution id * wip * . * adding separate execution context id * lint * vet * fixing pg dialers * test ignore * fixing loader FD limit * test * fd fix * wip: remove CloseProcesses() from dev merge * wip: fix merge issue * protocolstate: stop memguarding on last dialer delete * avoid data race in dialers.RawHTTPClient * use shared logger and avoid race conditions * use shared logger and avoid race conditions * go mod * patch executionId into compiled template cache * clean up comment in Parse * go mod update * bump echarts * address merge issues * fix use of gologger * switch cmd/nuclei to options.Logger * address merge issues with go.mod * go vet: address copy of lock with new Copy function * fixing tests * disable speed control * fix nil ExecuterOptions * removing deprecated code * fixing result print * default logger * cli default logger * filter warning from results * fix performance test * hardcoding path * disable upload * refactor(runner): uses `Warning` instead of `Print` for `pdcpUploadErrMsg` Signed-off-by: Dwi Siswanto <git@dw1.io> * Revert "disable upload" This reverts commit 114fbe6663361bf41cf8b2645fd2d57083d53682. * Revert "hardcoding path" This reverts commit cf12ca800e0a0e974bd9fd4826a24e51547f7c00. --------- Signed-off-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Mzack9999 <mzack9999@protonmail.com> Co-authored-by: Dwi Siswanto <git@dw1.io> Co-authored-by: Dwi Siswanto <25837540+dwisiswant0@users.noreply.github.com>
2025-07-09 14:47:26 -05:00
// UpdateOptions replaces this request's options with a new copy
func (r *Request) UpdateOptions(opts *protocols.ExecutorOptions) {
r.options.ApplyNewEngineOptions(opts)
}